rocoto

The uw mode for realizing and validating Rocoto XML documents.

uw rocoto --help
usage: uw rocoto [-h] [--version] ACTION ...

Realize and validate Rocoto XML documents

Optional arguments:
  -h, --help
      Show help and exit
  --version
      Show version info and exit

Positional arguments:
  ACTION
    realize
      Realize a Rocoto XML workflow document
    validate
      Validate Rocoto XML

realize

In uw terminology, to realize a configuration file is to transform it from its raw form into its final, usable state. In the case of uw rocoto, that means transforming a structured UW YAML file into a Rocoto XML file. The structured YAML language required by UW closely follows the XML language defined by Rocoto.

More information about the structured UW YAML file for Rocoto can be found here.

uw rocoto realize --help
usage: uw rocoto realize [-h] [--version] [--config-file PATH]
                         [--output-file PATH] [--quiet] [--verbose]

Realize a Rocoto XML workflow document

Optional arguments:
  -h, --help
      Show help and exit
  --version
      Show version info and exit
  --config-file PATH, -c PATH
      Path to UW YAML config file (default: read from stdin)
  --output-file PATH, -o PATH
      Path to output file (defaults to stdout)
  --quiet, -q
      Print no logging messages
  --verbose, -v
      Print all logging messages

Examples

The examples in this section use a UW YAML file rocoto.yaml with contents:

workflow:
  attrs:
    realtime: false
    scheduler: slurm
  cycledef:
    - attrs:
        group: howdy
      spec: 202209290000 202209300000 06:00:00
  entities:
    ACCOUNT: myaccount
    FOO: test.log
  log: /some/path/to/&FOO;
  tasks:
    task_hello:
      attrs:
        cycledefs: howdy
      account: "&ACCOUNT;"
      command: "echo hello $person"
      jobname: hello
      native: --reservation my_reservation
      nodes: 1:ppn=1
      walltime: 00:01:00
      envars:
        person: siri
  • To realize a UW YAML input file to stdout in Rocoto XML format:

    uw rocoto realize --config-file rocoto.yaml
    
    [2024-08-26T23:11:42]     INFO 0 schema-validation errors found in Rocoto config
    [2024-08-26T23:11:42]     INFO 0 Rocoto XML validation errors found
    <?xml version='1.0' encoding='utf-8'?>
    <!DOCTYPE workflow [
      <!ENTITY ACCOUNT "myaccount">
      <!ENTITY FOO "test.log">
    ]>
    <workflow realtime="False" scheduler="slurm">
      <cycledef group="howdy">202209290000 202209300000 06:00:00</cycledef>
      <log>/some/path/to/&FOO;</log>
      <task name="hello" cycledefs="howdy">
        <account>&ACCOUNT;</account>
        <nodes>1:ppn=1</nodes>
        <walltime>00:01:00</walltime>
        <command>echo hello $person</command>
        <jobname>hello</jobname>
        <native>--reservation my_reservation</native>
        <envar>
          <name>person</name>
          <value>siri</value>
        </envar>
      </task>
    </workflow>
    
  • To realize a UW YAML input file to a file named rocoto.xml:

    rm -f rocoto.xml
    uw rocoto realize --config-file rocoto.yaml --output-file rocoto.xml
    echo
    cat rocoto.xml
    
    [2024-08-26T23:11:41]     INFO 0 schema-validation errors found in Rocoto config
    [2024-08-26T23:11:41]     INFO 0 Rocoto XML validation errors found
    
    <?xml version='1.0' encoding='utf-8'?>
    <!DOCTYPE workflow [
      <!ENTITY ACCOUNT "myaccount">
      <!ENTITY FOO "test.log">
    ]>
    <workflow realtime="False" scheduler="slurm">
      <cycledef group="howdy">202209290000 202209300000 06:00:00</cycledef>
      <log>/some/path/to/&FOO;</log>
      <task name="hello" cycledefs="howdy">
        <account>&ACCOUNT;</account>
        <nodes>1:ppn=1</nodes>
        <walltime>00:01:00</walltime>
        <command>echo hello $person</command>
        <jobname>hello</jobname>
        <native>--reservation my_reservation</native>
        <envar>
          <name>person</name>
          <value>siri</value>
        </envar>
      </task>
    </workflow>
    
  • To read the UW YAML from stdin and write the XML to stdout:

    cat rocoto.yaml | uw rocoto realize
    
    [2024-08-26T23:11:42]     INFO 0 schema-validation errors found in Rocoto config
    [2024-08-26T23:11:42]     INFO 0 Rocoto XML validation errors found
    <?xml version='1.0' encoding='utf-8'?>
    <!DOCTYPE workflow [
      <!ENTITY ACCOUNT "myaccount">
      <!ENTITY FOO "test.log">
    ]>
    <workflow realtime="False" scheduler="slurm">
      <cycledef group="howdy">202209290000 202209300000 06:00:00</cycledef>
      <log>/some/path/to/&FOO;</log>
      <task name="hello" cycledefs="howdy">
        <account>&ACCOUNT;</account>
        <nodes>1:ppn=1</nodes>
        <walltime>00:01:00</walltime>
        <command>echo hello $person</command>
        <jobname>hello</jobname>
        <native>--reservation my_reservation</native>
        <envar>
          <name>person</name>
          <value>siri</value>
        </envar>
      </task>
    </workflow>
    
  • To see verbose log output (Rocoto XML and some output elided for brevity):

    rm -f rocoto.log
    uw rocoto realize --config-file rocoto.yaml --verbose >/dev/null 2>rocoto.log
    head -n10 rocoto.log
    echo ...
    tail -n10 rocoto.log
    
    [2025-01-05T21:15:07]    DEBUG Command: uw rocoto realize --config-file rocoto.yaml --verbose
    [2025-01-05T21:15:07]    DEBUG [dereference] Dereferencing, current value:
    [2025-01-05T21:15:07]    DEBUG [dereference]   workflow:
    [2025-01-05T21:15:07]    DEBUG [dereference]     attrs:
    [2025-01-05T21:15:07]    DEBUG [dereference]       realtime: false
    [2025-01-05T21:15:07]    DEBUG [dereference]       scheduler: slurm
    [2025-01-05T21:15:07]    DEBUG [dereference]     cycledef:
    [2025-01-05T21:15:07]    DEBUG [dereference]     - attrs:
    [2025-01-05T21:15:07]    DEBUG [dereference]         group: howdy
    [2025-01-05T21:15:07]    DEBUG [dereference]       spec: 202209290000 202209300000 06:00:00
    ...
    [2025-01-05T21:15:07]    DEBUG [dereference]     cycledefs: howdy
    [2025-01-05T21:15:07]    DEBUG [dereference]   account: '&ACCOUNT;'
    [2025-01-05T21:15:07]    DEBUG [dereference]   command: echo hello $person
    [2025-01-05T21:15:07]    DEBUG [dereference]   jobname: hello
    [2025-01-05T21:15:07]    DEBUG [dereference]   native: --reservation my_reservation
    [2025-01-05T21:15:07]    DEBUG [dereference]   nodes: 1:ppn=1
    [2025-01-05T21:15:07]    DEBUG [dereference]   walltime: 00:01:00
    [2025-01-05T21:15:07]    DEBUG [dereference]   envars:
    [2025-01-05T21:15:07]    DEBUG [dereference]     person: siri
    [2025-01-05T21:15:07]     INFO 0 Rocoto XML validation errors found
    

validate

uw rocoto validate --help
usage: uw rocoto validate [-h] [--version] [--input-file PATH] [--quiet]
                          [--verbose]

Validate Rocoto XML

Optional arguments:
  -h, --help
      Show help and exit
  --version
      Show version info and exit
  --input-file PATH, -i PATH
      Path to input file (defaults to stdin)
  --quiet, -q
      Print no logging messages
  --verbose, -v
      Print all logging messages

Examples

The examples in this section use a Rocoto XML file rocoto-good.xml with contents:

<?xml version='1.0' encoding='utf-8'?>
<!DOCTYPE workflow [
  <!ENTITY ACCOUNT "myaccount">
  <!ENTITY FOO "test.log">
]>
<workflow realtime="False" scheduler="slurm">
  <cycledef group="howdy">202209290000 202209300000 06:00:00</cycledef>
  <log>/some/path/to/&FOO;</log>
  <task name="hello" cycledefs="howdy">
    <account>&ACCOUNT;</account>
    <nodes>1:ppn=1</nodes>
    <walltime>00:01:00</walltime>
    <command>echo hello $person</command>
    <jobname>hello</jobname>
    <envar>
      <name>person</name>
      <value>siri</value>
    </envar>
  </task>
</workflow>
  • To validate XML from stdin:

    cat rocoto-good.xml | uw rocoto validate
    
    [2024-08-26T23:11:42]     INFO 0 Rocoto XML validation errors found
    
  • To validate XML from file rocoto-good.xml:

    uw rocoto validate --input-file rocoto-good.xml
    
    [2024-08-26T23:11:42]     INFO 0 Rocoto XML validation errors found
    
  • When the XML is invalid:

    In this example, the <command> line was removed from rocoto-good.xml to create rocoto-bad.xml.

    uw rocoto validate --input-file rocoto-bad.xml
    
    [2024-08-26T23:11:41]    ERROR 3 Rocoto XML validation errors found
    [2024-08-26T23:11:41]    ERROR <string>:9:0:ERROR:RELAXNGV:RELAXNG_ERR_NOELEM: Expecting an element command, got nothing
    [2024-08-26T23:11:41]    ERROR <string>:9:0:ERROR:RELAXNGV:RELAXNG_ERR_INTERSEQ: Invalid sequence in interleave
    [2024-08-26T23:11:41]    ERROR <string>:9:0:ERROR:RELAXNGV:RELAXNG_ERR_CONTENTVALID: Element task failed to validate content
    [2024-08-26T23:11:41]    ERROR Invalid Rocoto XML:
    [2024-08-26T23:11:41]    ERROR  1 <?xml version='1.0' encoding='utf-8'?>
    [2024-08-26T23:11:41]    ERROR  2 <!DOCTYPE workflow [
    [2024-08-26T23:11:41]    ERROR  3   <!ENTITY ACCOUNT "myaccount">
    [2024-08-26T23:11:41]    ERROR  4   <!ENTITY FOO "test.log">
    [2024-08-26T23:11:41]    ERROR  5 ]>
    [2024-08-26T23:11:41]    ERROR  6 <workflow realtime="False" scheduler="slurm">
    [2024-08-26T23:11:41]    ERROR  7   <cycledef group="howdy">202209290000 202209300000 06:00:00</cycledef>
    [2024-08-26T23:11:41]    ERROR  8   <log>/some/path/to/&FOO;</log>
    [2024-08-26T23:11:41]    ERROR  9   <task name="hello" cycledefs="howdy">
    [2024-08-26T23:11:41]    ERROR 10     <account>&ACCOUNT;</account>
    [2024-08-26T23:11:41]    ERROR 11     <nodes>1:ppn=1</nodes>
    [2024-08-26T23:11:41]    ERROR 12     <walltime>00:01:00</walltime>
    [2024-08-26T23:11:41]    ERROR 13     <jobname>hello</jobname>
    [2024-08-26T23:11:41]    ERROR 14     <envar>
    [2024-08-26T23:11:41]    ERROR 15       <name>person</name>
    [2024-08-26T23:11:41]    ERROR 16       <value>siri</value>
    [2024-08-26T23:11:41]    ERROR 17     </envar>
    [2024-08-26T23:11:41]    ERROR 18   </task>
    [2024-08-26T23:11:41]    ERROR 19 </workflow>
    

    To decode the three ERROR:RELAXNGV messages, it is easiest to read from the bottom up. They say:

    • At line 9 column 0 (i.e. <string>:9:0), the element (i.e. <task>) failed to validate.

    • The sequence of interleaved elements under <task> was invalid.

    • A <command> element was expected, but it wasn’t found.