`ecflow`

The uw mode for realizing and validating ecFlow suite definitions.

uw ecflow --help

usage: uw ecflow [-h] [--version] ACTION ...

Handle ecflow suite definitions

Optional arguments:
  -h, --help
      Show help and exit
  --version
      Show version info and exit

Positional arguments:
  ACTION
    realize
      Realize an ecFlow suite definition
    validate
      Validate an ecFlow YAML config

`realize`

In uw terminology, to realize a configuration file is to transform it from its raw form into its final, usable state. In the case of uw ecflow, that means transforming a structured UW YAML file into an ecFlow suite definition file (suite.def) and, optionally, a set of .ecf scripts. The structured YAML language required by UW closely follows the concepts defined by ecFlow.

See ecFlow Workflows for more information about the structured UW YAML for ecFlow.

uw ecflow realize --help

usage: uw ecflow realize [-h] [--version] [--config-file PATH]
                         [--output-dir PATH] [--scripts-dir PATH] [--quiet]
                         [--verbose]

Realize an ecFlow suite definition

Optional arguments:
  -h, --help
      Show help and exit
  --version
      Show version info and exit
  --config-file PATH, -c PATH
      Path to UW YAML config file (default: read from stdin)
  --output-dir PATH
      Path to output directory for suite definition (default: write to stdout)
  --scripts-dir PATH
      Path to output directory for ecf scripts (default: no scripts are
      generated)
  --quiet, -q
      Print no logging messages
  --verbose, -v
      Print all logging messages

Examples

The examples in this section use a UW YAML file ecflow.yaml with contents:

ecflow:
  suitedef:
    suite_forecast:
      family_prep:
        task_get_obs:
          trigger: "1==1"
          script:
            execution:
              incantation: /path/to/get_obs.sh
            manual: Retrieve observation data
      task_run_model:
        trigger: /forecast/prep/get_obs == complete
        script:
          execution:
            incantation: /path/to/run_model.sh
          manual: Run the forecast model

To realize a UW YAML input file to stdout in ecFlow suite definition format:

uw ecflow realize --config-file ecflow.yaml

[2025-01-02T03:04:05]     INFO Validating config against internal schema: ecflow
[2025-01-02T03:04:05]     INFO Schema validation succeeded for ecFlow config
#5.15.2
suite forecast
  family prep
    task get_obs
      trigger 1==1
  endfamily
  task run_model
    trigger /forecast/prep/get_obs == complete
endsuite
# enddef

To realize a UW YAML input file to a directory (writes suite.def inside that directory):

rm -f suite.def
uw ecflow realize --config-file ecflow.yaml --output-dir .
echo
cat suite.def

[2025-01-02T03:04:05]     INFO Validating config against internal schema: ecflow
[2025-01-02T03:04:05]     INFO Schema validation succeeded for ecFlow config

#5.15.2
suite forecast
  family prep
    task get_obs
      trigger 1==1
  endfamily
  task run_model
    trigger /forecast/prep/get_obs == complete
endsuite
# enddef

To read the UW YAML from stdin and write the suite definition to stdout:

cat ecflow.yaml | uw ecflow realize

[2025-01-02T03:04:05]     INFO Validating config against internal schema: ecflow
[2025-01-02T03:04:05]     INFO Schema validation succeeded for ecFlow config
#5.15.2
suite forecast
  family prep
    task get_obs
      trigger 1==1
  endfamily
  task run_model
    trigger /forecast/prep/get_obs == complete
endsuite
# enddef

To also generate .ecf scripts, using a UW YAML file ecflow-workflow.yaml with contents:

ecflow:
  suitedef:
    suite_workflow:
      vars:
        WORKFLOW_VAR: production
      family_data_prep:
        task_fetch:
          trigger: "1==1"
          script:
            execution:
              incantation: echo Fetching data...
        task_process:
          trigger: /workflow/data_prep/fetch == complete
          script:
            execution:
              incantation: echo Processing data...
      task_run_model:
        trigger: /workflow/data_prep == complete
        script:
          execution:
            incantation: echo Running model...

uw ecflow realize --config-file ecflow-workflow.yaml --output-dir ./workflow_output --scripts-dir ./workflow_output

[2025-01-02T03:04:05]     INFO Validating config against internal schema: ecflow
[2025-01-02T03:04:05]     INFO Schema validation succeeded for ecFlow config

The --scripts-path option specifies the directory under which .ecf scripts are written. Each script is placed at the same nested subdirectory level under <scripts-path> as dictated by the nesting level of the task node in the suite definition. For the example above, the generated scripts are:

workflow_output/workflow/data_prep/fetch.ecf:

model=%MODEL%

echo Fetching data...
if [[ $? -ne 0 ]]; then
   ecflow_client --msg="***JOB $ECF_NAME ERROR RUNNING J-SCRIPT ***"
   ecflow_client --abort
   exit 1
fi

%manual
Script to run fetch
%end

workflow_output/workflow/data_prep/process.ecf:

model=%MODEL%

echo Processing data...
if [[ $? -ne 0 ]]; then
   ecflow_client --msg="***JOB $ECF_NAME ERROR RUNNING J-SCRIPT ***"
   ecflow_client --abort
   exit 1
fi

%manual
Script to run process
%end

workflow_output/workflow/model.ecf:

model=%MODEL%

echo Running model...
if [[ $? -ne 0 ]]; then
   ecflow_client --msg="***JOB $ECF_NAME ERROR RUNNING J-SCRIPT ***"
   ecflow_client --abort
   exit 1
fi

%manual
Script to run run_model
%end

Important

Task Naming Convention: Task keys must follow the pattern task_<name>. When .ecf scripts are generated, the <name> portion becomes the script filename with a .ecf extension. See ecFlow Workflows for more information about the structured UW YAML for ecFlow.

Examples:

task_fetch → fetch.ecf

task_run_model → model.ecf

task_process_output_files → process_output_files.ecf

`validate`

uw ecflow validate --help

usage: uw ecflow validate [-h] [--version] [--config-file PATH] [--quiet]
                          [--verbose]

Validate an ecFlow YAML config

Optional arguments:
  -h, --help
      Show help and exit
  --version
      Show version info and exit
  --config-file PATH, -c PATH
      Path to UW YAML config file (default: read from stdin)
  --quiet, -q
      Print no logging messages
  --verbose, -v
      Print all logging messages

Examples

The examples in this section use the UW YAML file ecflow.yaml shown above.

To validate a UW YAML config file:

uw ecflow validate --config-file ecflow.yaml

[2025-01-02T03:04:05]     INFO Validating config against internal schema: ecflow
[2025-01-02T03:04:05]     INFO Schema validation succeeded for ecFlow config

To validate a UW YAML config from stdin:

cat ecflow.yaml | uw ecflow validate

[2025-01-02T03:04:05]     INFO Validating config against internal schema: ecflow
[2025-01-02T03:04:05]     INFO Schema validation succeeded for ecFlow config

When the config is invalid:

In this example, the top-level ecflow: key is missing.

uw ecflow validate --config-file ecflow-bad.yaml

[2025-01-02T03:04:05]     INFO Validating config against internal schema: ecflow
[2025-01-02T03:04:05]    ERROR 1 schema-validation error found in ecFlow config
[2025-01-02T03:04:05]    ERROR Error at top level:
[2025-01-02T03:04:05]    ERROR   'ecflow' is a required property
[2025-01-02T03:04:05]    ERROR YAML validation errors

ecflow

realize

Examples

validate

Examples

`ecflow`

`realize`

`validate`