Something went wrong!

Cycling Workflow Orchestration:
the Cylc Workflow Engine

cylc-logo-white

Hilary Oliver
NIWA

Why Cylc?
What's new? (Cylc 8)
Cylc 8 preview demo
Website
https://cylc.org
Developer chat and VC
https://riot.im/app/#/group/+cylc:matrix.org
User discussion forum
cylc.discourse.group
Code & development hub
https://github.com/cylc

light-bulb example-research-plot research

clock-icon real-time
cycling

seamless
(meta-scheduling)
transition

simulated
time

cycling

operations weather-symbols

Production ready
Cycling: date-time & arbitrary
Job retry on failure
Restart from checkpoint
Distributed architecture
Live monitoring
Manual intervention
Simulation mode
External triggering
Open Source GPLv3

Smaller workflows are very easy!

Efficient programmatic generation of large and complex workflows

Task config inheritance
Task parameter expansion
General Pythonic templating

System Architecture

metoffice-logo-black
  • Cylc: ~full HPC workload - research, and (since Jan 2014) production:
    • 460,672 cores
  • Demonstrated single-workflow scaling to:
    • ˜1000s of recurrences
    • ˜10,000s of tasks
    • ˜100,000s of dependencies

The road to
Cylc 8!

Python 3 & web UI

Cylc 7 Architecture

* "User A" can be across multiple hosts
  • local(ish) client/server
  • all clients equal (incl. jobs)
  • all runs as the user
  • auto owner-only authentication
  • owner/public authorization
  • https network layer REST API
  • details

cylc-7.8.4 - current stable production system

Cylc 8 Architecture

* "User A" can be across multiple hosts
Single point of access
Site identity management integration
Fine-grained authorization in WFS
In-browser UI, with no need for:
  • front-end Cylc installation
  • shared FS with WF platforms
Retire suite-specific passphrase

Server and CLI

python-logo
2
python-logo
3

Data & network layer

rest-logo graphql-logo zmq-logo
& Protocol Buffers

Web framework

cherrypy-logo tornado-logo

Front-end

gtk-logo css-html-logos vue-logo

Documentation

latex-logo sphinx-logo

Cylc Packaging

Cylc 8 Workflow Services

Server program & CLI. Mostly unchanged, but:

  • Python 3!
  • ZMQ network layer
  • Incremental update to UI Server
  • Reduced UI loading!

Cylc 8 UI Servers

Serves the UI to the browser, and:

  • Collates data from user WFS's
  • Takes the load off of WFS's
  • Presents stopped WFs etc. too
  • Tornado server
    • to UI: GraphQL by WebSocket
    • to WFS: ProtoBuf by ZMQ
  • details

Cylc 8 Hub

Uses JupyterHub as a generic application spawner.
* "User A" can be across multiple hosts
  • single point of access
  • site identity management
  • spawn and proxy UI Servers
  • local and remote spawners
  • stores session information
  • details

Cylc 8 UI

Javascript with Vue.js

  • unified (cylc gui, gscan)
  • dash board
  • responsive design
  • simplified set of task states
  • (n=1,2,3 window?)
  • shared data-store with incremental update

Cylc 8 Architecture

* "User A" can be across multiple hosts

Questions?

website cylc.org
github github.com/cylc
discourse cylc.discourse.group
mail hilary.oliver@niwa.co.nz