ecFlow Workflows

ecFlow is a workflow manager developed by ECMWF and popular in the meteorological community. It defines workflows as suites of tasks with dependencies, resource requirements, and scheduling logic. ecFlow uses a suite definition file (e.g. suite.def) to describe the workflow, and ecf scripts to carry out individual tasks.

The uw ecflow tool defines a UW YAML language that can be easily manipulated like any other key-value configuration file and translated into the artifacts required by ecFlow: a suite definition file and, optionally, a set of ecf scripts.

Top-Level Structure

UW YAML for ecFlow nests under a top-level ecflow key, and the suite definition under a suitedef child:

ecflow:
  suitedef:
    ...

Suite-Definition Root Level Keys

The following keys may appear at the top of the suite definition, directly under ecflow.suitedef:

ecflow:
  suitedef:
    extern:
      - /other_suite/family/task
    scheduler: slurm
    suite_forecast:
      ...
    suites_example:
      ...
    vars:
      ECF_HOME: /path/to/ecf
      ACCOUNT: myproject

extern:

An optional list of external node paths to declare in the suite definition. Useful for triggering tasks across suites.

scheduler:

The batch scheduler to target when generating ecf scripts. Supported values are slurm, pbs, and lsf. When set, scheduler directives defined under batchargs: in a task_<name> block are automatically added to generated ecf scripts. Omit this key if scripts are not needed or if no batch directives are required.

suite_<name>:

A suite definition. The portion of the key following suite_ becomes the suite name in ecFlow. For example, suite_forecast creates a suite named forecast. See Suite and Node Structure for the contents of a suite.

suites_<name>:

An expand block for generating multiple suites from a parameterized template. See Expand Blocks.

vars:

A mapping of variable name/value pairs set as ecFlow edit variables at the workflow (Defs) level. These become globally accessible within the suite.

Variables beginning with ECF_ are reserved by ecFlow and a defined set are supported: suite definition variables, and generated variables. Values for the latter are automatically supplied by the ecFlow server for use in ecf scripts, but can be overridden by users in a suite-definition file. Apart from the supported set, variables starting with ECF_ are not permitted.

See the ecFlow documentation for more information on available variables and their meanings.

Suite and Node Structure

Suites, families, and tasks are defined as nested YAML blocks. Keys are prefixed with suite_, family_, or task_ to indicate their type; the remainder of the key is used as the node name.

ecflow:
  suitedef:
    suite_workflow:
      family_data_prep:
        task_fetch:
          script:
            body: echo Fetching data...
            manual: This task fetches data.
        task_process:
          script:
            body: echo Processing data...
            manual: This task processes data.
          trigger: /workflow/data_prep/fetch == complete
      task_run_model:
        script:
          body: echo Running model...
          manual: This task runs the model.
        trigger: /workflow/data_prep == complete
      vars:
        WORKFLOW_VAR: production

This example defines a suite workflow containing a family data_prep with tasks fetch and process, and a top-level task run_model.

Important

Task Naming Convention: Task keys must follow the pattern task_<name>. When ecf scripts are generated, the <name> portion becomes the script filename with a .ecf extension. See ecFlow Workflows for more information about the structured UW YAML for ecFlow.

Examples:

  • task_fetch → fetch.ecf

  • task_run_model → run_model.ecf

  • task_process_output_files → process_output_files.ecf

Names for suites and families follow a corresponding naming convention.

Node Attributes

The following attributes may be set on suite, family, or task nodes, at any level in the config hierarchy:

defstatus:

Sets the default status of a node (see ecflow.DState). Accepted values are complete, suspended, aborted, queued, submitted, active, and unknown. For example:

task_get_obs:
  defstatus: complete

events:

A list of named events attached to a node (see ecflow.Event). Each item may be a string (event name) or a two-element list [number, name]. For example:

task_get_obs:
  events:
    - obs_ready
    - [2, model_ready]

inlimits:

A list of limits that a node consumes (see ecflow.InLimit). Each item may be a one-, two-, or three-element list [limit_name], [limit_path, limit_name], or [limit_path, limit_name, tokens]. For example:

task_run_model:
  inlimits:
    - [/forecast, max_jobs]

labels:

A list of label name/value pairs on a node (see ecflow.Label). Each item is a two-element list [name, value]. For example:

task_run_model:
  labels:
    - [progress, "0%"]

late:

Defines late notification thresholds for a node (see ecflow.Late). Accepts submitted, active, and complete as keys, each with a time value. For example:

task_run_model:
  late:
    active: "02:00:00"
    submitted: "00:05:00"

limits:

A list of limits defined on a suite or family (see ecflow.Limit). Each item is a two-element list [name, max_count]. For example:

suite_forecast:
  limits:
    - [max_jobs, 4]

meters:

A list of meters on a node (see ecflow.Meter). Each item is a three- or four-element list [name, min, max] or [name, min, max, threshold]. For example:

task_run_model:
  meters:
    - [progress, 0, 100]

trigger:

A string expression defining the conditions under which a node may run (see ecflow.Trigger). Only one trigger: is allowed per node. For example:

task_run_model:
  trigger: /forecast/prep/get_obs == complete

vars:

A mapping of variable name/value pairs to set as ecFlow edit variables on the node (see ecflow.Variable). For example:

task_run_model:
  vars:
    MEMBER: "001"

See this section for more information on ECF_ and other variables that can be set on nodes to override those set at the suite level.

Repeat Attributes

Only one repeat_*: attribute is allowed per node. The available repeat types are:

repeat_date:

Repeats over a date range (see ecflow.RepeatDate). Requires variable, start, and end (as YYYYMMDD integers); step is optional (default 1).

suite_forecast:
  repeat_date:
    variable: YMD
    start: 20240101
    end: 20240131
    step: 1

repeat_datelist:

Repeats over an explicit list of dates (see ecflow.RepeatDateList) (as YYYYMMDD integers). Requires variable and list.

repeat_datetime:

Repeats over a datetime range (see ecflow.RepeatDateTime). Requires variable, start, and end (as YYYYMMDDTHHmmss strings); step (as HH:mm:ss) is optional.

repeat_day:

Repeats by day increment (see ecflow.RepeatDay). Requires step.

repeat_enumerated:

Repeats over an explicit list of strings (see ecflow.RepeatEnumerated). Requires variable and list.

repeat_int:

Repeats over an integer range (see ecflow.RepeatInteger). Requires variable, start, and end; step is optional.

repeat_string:

Repeats over an explicit list of strings (see ecflow.RepeatString). Requires variable and list.

Task Script Block

Tasks are required to have a script: block that defines the ecf script to generate. The value of script: may be

  • A string specifying a path to an ecf script to be passed verbatim to ecFlow. The script can reference ecFlow %<VAR>% variables, which will be rendered by the server when converting the ecf script to a final job script, but must otherwise be complete. Note that this option is discouraged in most cases, as it will inhibit workflow portability and bypasses correctness checks supplied by uwtools.

script: /path/to/script.sh

or

  • A YAML mapping with the following required and optional keys:

batchargs:

Optional. Batch-scheduler directives to add to the ecf script. See batchargs: for more information.

body:

Required. Executable statements for the task. This may be a single command, including the path to a script to execute, or a multiline YAML string specifying a series of statements to execute in a bash shell.

script:
  body: |
    statement1
    statement2
    # etc.

includes:

Optional: Subkeys entry: and exit: specify sequences of files (typically with .h extensions) to include before and after the body statements, respectively, translated to ecFlow %include statements. By default, the ecFlow server looks for include under the directory specified by ecflow.server.vars.ECF_HOME in the config; this can be overridden by defining ecflow.server.vars.ECF_INCLUDE (see here). The include file server.h, which exports several crucial environment variables, is created by uw ecflow server at startup time and is available for inclusion in application ecf files.

script:
  includes:
    entry:
    - server.h
    - prep.h
    exit:
    - post.h

In this example prep.h and post.h would need to be placed by the application in the ECF_HOME directory, or in a directory specified by ECF_INCLUDE.

manual:

Optional. A brief description of the task’s purpose, embedded in the ecf script’s %manual section. The manual can, for example, be viewed in the ecFlow GUI.

Expand Blocks

The expand: mechanism generates multiple nodes from a parameterized template. It is used with the plural forms of node prefixes: suites_, families_, and tasks_.

The expand: key under a node block defines one or more variables to expand on, and their lists of values. The other keys in the block are treated as a template, rendered once for each element in the value lists, with {{ ec.VARNAME }} placeholders substituted. This implies that, when using the uw ecflow tool, ec is a reserved top-level key and should not be explicitly defined in YAML.

ecflow:
  suite_ensemble:
    tasks_member_{{ ec.MEM }}:
      expand:
        MEM: ["01", "02", "03"]
      script:
        body: /path/to/run.sh
        manual: Run ensemble member {{ ec.MEM }}

This expands to three tasks named member_01, member_02, and member_03.

Multiple value lists may be specified, provided they all have the same length:

tasks_member_{{ ec.MEM }}_{{ ec.LABEL }}:
  expand:
    MEM: ["01", "02"]
    LABEL: ["ctrl", "pert"]

This expands to two tasks, member_01_ctrl and member_02_pert.

Server Configuration

The uw ecflow server command reads configuration from an ecflow.server block:

ecflow:
  server:
    ECF_HOME: /path/to/run

All keys in the server: block are passed as environment variables to ecflow_server. Only ECF_HOME is required. All supported variables, and values accepted by UW YAML for ecFlow, are described in ecFlow documentation. The lone exception is ECF_SSL, which accepts values as described below.

ECF_SSL

Optionally controls SSL-based server security. Accepts a boolean or a certificate-filename prefix string:

  • true: The default when --insecure is not specified. Enable SSL using the default certificate triplet (server.crt / server.key / dh2048.pem) in $HOME/.ecflowrc/ssl. If these do not exist, uw ecflow server will generate them.

  • false: Disable SSL. Equivalent to specifying --insecure.

  • A certificate-filename prefix string (e.g. myhost.8888): Enable SSL using the specified certificate files. Files with the given prefix and the extensions .crt, .key, and .pem must exist under $HOME/.ecflowrc/ssl/.

For example, to use a certificate-filename prefix when running on a static port:

ecflow:
  server:
    ECF_SSL: myhost.8888

The --port value passed to uw ecflow server must match the port in the prefix. The three files myhost.8888.crt, myhost.8888.key, and myhost.8888.pem must already exist in $HOME/.ecflowrc/ssl; they are not auto-generated.

When --insecure is given to uw ecflow server, SSL is disabled entirely regardless of the ECF_SSL setting.

Generated Artifacts

suite.def

The ecFlow suite definition file, written to <output-path>/suite.def if --output-path is given, otherwise to stdout.

ecf scripts

One ecf script per task, written under the directory specified by --scripts-path. Each script is nested under this directory in the same manner as it is in the suite definition. This suite definition will create <scripts-path>/workflow/data_prep/fetch.ecf, <scripts-path>/workflow/data_prep/process.ecf, and <scripts-path>/workflow/run_model.ecf. Scripts are only generated when --scripts-path is specified.

Demonstration Workflow

View or run a simple ecFlow demonstration workflow.