`config`

The uw mode for handling configuration files (configs).

uw config --help

usage: uw config [-h] [--version] ACTION ...

Handle configs

Optional arguments:
  -h, --help
      Show help and exit
  --version
      Show version info and exit

Positional arguments:
  ACTION
    compare
      Compare configs
    compose
      Compose configs
    realize
      Realize config
    validate
      Validate config

`compare`

The compare action compares two config files.

uw config compare --help

usage: uw config compare --path1 PATH --path2 PATH [-h] [--version]
                         [--format1 {ini,nml,sh,yaml}]
                         [--format2 {ini,nml,sh,yaml}] [--quiet] [--verbose]

Compare configs

Required arguments:
  --path1 PATH
      Path to file 1
  --path2 PATH
      Path to file 2

Optional arguments:
  -h, --help
      Show help and exit
  --version
      Show version info and exit
  --format1 {ini,nml,sh,yaml}
      Format of file 1
  --format2 {ini,nml,sh,yaml}
      Format of file 2
  --quiet, -q
      Print no logging messages
  --verbose, -v
      Print all logging messages

Examples

The examples that follow use identical namelist files a.nml and b.nml with contents:

&values
  greeting = "Hello"
  recipient = "World"
/

To compare two config files with the same contents:

uw config compare --path1 a.nml --path2 b.nml

[2025-01-02T03:04:05]     INFO - a.nml
[2025-01-02T03:04:05]     INFO + b.nml

If there are differences between the config files, they will be shown below the dashed line. For example, c.nml is missing the line recipient: World:

&values
  greeting = "Hello"
/

uw config compare --path1 a.nml --path2 c.nml

[2025-01-02T03:04:05]     INFO - a.nml
[2025-01-02T03:04:05]     INFO + c.nml
[2025-01-02T03:04:05]     INFO ---------------------------------------------------------------------
[2025-01-02T03:04:05]     INFO ↓ ? = info | -/+ = line unique to - or + file | blank = matching line
[2025-01-02T03:04:05]     INFO ---------------------------------------------------------------------
[2025-01-02T03:04:05]     INFO   values:
[2025-01-02T03:04:05]     INFO     greeting: Hello
[2025-01-02T03:04:05]     INFO -   recipient: World

If a config file has an unrecognized (or no) extension, uw will not know how to parse its contents:

uw config compare --path1 a.txt --path2 c.nml

[2025-01-02T03:04:05]    ERROR Formats do not match: yaml vs nml

The format must be explicitly specified (a.txt is a copy of a.nml):

uw config compare --path1 a.txt --format1 nml --path2 c.nml

[2025-01-02T03:04:05]     INFO - a.txt
[2025-01-02T03:04:05]     INFO + c.nml
[2025-01-02T03:04:05]     INFO ---------------------------------------------------------------------
[2025-01-02T03:04:05]     INFO ↓ ? = info | -/+ = line unique to - or + file | blank = matching line
[2025-01-02T03:04:05]     INFO ---------------------------------------------------------------------
[2025-01-02T03:04:05]     INFO   values:
[2025-01-02T03:04:05]     INFO     greeting: Hello
[2025-01-02T03:04:05]     INFO -   recipient: World

To request verbose log output:

uw config compare --path1 a.nml --path2 c.nml --verbose

[2025-01-02T03:04:05]    DEBUG Command: uw config compare --path1 a.nml --path2 c.nml --verbose
[2025-01-02T03:04:05]     INFO - a.nml
[2025-01-02T03:04:05]     INFO + c.nml
[2025-01-02T03:04:05]     INFO ---------------------------------------------------------------------
[2025-01-02T03:04:05]     INFO ↓ ? = info | -/+ = line unique to - or + file | blank = matching line
[2025-01-02T03:04:05]     INFO ---------------------------------------------------------------------
[2025-01-02T03:04:05]     INFO   values:
[2025-01-02T03:04:05]     INFO     greeting: Hello
[2025-01-02T03:04:05]     INFO -   recipient: World

Note that uw logs to stderr. Use shell redirection as needed.

Note

Comparisons are supported only for configs of the same format, e.g. YAML vs YAML, Fortran namelist vs Fortran namelist, etc. uw will flag invalid comparisons:

uw config compare --path1 a.nml --path2 b.yaml

[2025-01-02T03:04:05]    ERROR Formats do not match: nml vs yaml

`compose`

The compose action builds up a final config by repeatedly updating a base config with the contents of other configs of the same format.

uw config compose --help

usage: uw config compose [-h] [--version] [--realize] [--output-file PATH]
                         [--input-format {ini,nml,sh,yaml}]
                         [--output-format {ini,nml,sh,yaml}] [--quiet]
                         [--verbose]
                         CONFIG [CONFIG ...]

Compose configs

positional arguments:
  CONFIG

Optional arguments:
  -h, --help
      Show help and exit
  --version
      Show version info and exit
  --realize
      Render template expressions where possible
  --output-file PATH, -o PATH
      Path to output file (default: write to stdout)
  --input-format {ini,nml,sh,yaml}
      Input format (default: yaml)
  --output-format {ini,nml,sh,yaml}
      Output format (default: yaml)
  --quiet, -q
      Print no logging messages
  --verbose, -v
      Print all logging messages

Examples

Consider three YAML configs:
Listing 1 compose-base.yaml
```
constants:
  pi: 3.142
color: blue
flower: violet
```
Listing 2 compose-update-1.yaml
```
constants:
  e: 2.718
```
Listing 3 compose-update-2.yaml
```
flower: rose
```
Compose the three together, writing to stdout:
```
uw config compose compose-base.yaml compose-update-1.yaml compose-update-2.yaml
```
```
constants:
  pi: 3.142
  e: 2.718
color: blue
flower: rose
```
Values provided by update configs override or augment values provided in the base config, while unaffected values survive to the final config. Priority of values increases from left to right.

Additionally:
- Sets of configs in the ini, nml, and sh formats can be similarly composed.
- The --input-format and --output-format options can be used to specify the format of the input and output configs, respectively, for cases when uwtools cannot deduce the format of configs from their filename extensions. When the formats are neither explicitly specified or deduced, yaml is assumed.
- The --output-file / -o option can be added to write the final config to a file instead of to stdout.

The optional --realize switch can be used to render as many Jinja2 template expressions as possible in the final config, using the config itself as a source of values. For example:

Listing 4 compose-template.yaml

radius: !float '{{ 2.0 * pi * r }}'

Listing 5 compose-values.yaml

pi: 3.142
r: 1.0

Without the --realize switch:

uw config compose compose-template.yaml compose-values.yaml

radius: !float '{{ 2.0 * pi * r }}'
pi: 3.142
r: 1.0

And with --realize:

uw config compose compose-template.yaml compose-values.yaml --realize

radius: 6.284
pi: 3.142
r: 1.0

The compose action supports defining YAML anchors and aliases in separate files, something not supported natively by the underlying PyYAML library. For example:
Listing 6 alias.yaml
```
values:
  <<: *CONST
  foo: bar
```
Listing 7 anchor.yaml
```
constants: &CONST
  e: 2.718
  pi: 3.142
```
```
uw config compose alias.yaml anchor.yaml
```
```
values:
  e: 2.718
  pi: 3.142
  foo: bar
constants:
  e: 2.718
  pi: 3.142
```
This example is trivial, but a YAML-anchored block could provide many more configuration values, and an alias to that block could be repeated many times in a config, or across many configs, avoiding hard-to-maintain and error-prone repeated values.

Note

Anchor names must be unique across all files passed to uw config compose, as YAML does not permit repetition of anchor names.

Note

Anchors must be defined either in the current file, or in a file that follows the current file in the list of YAML configs to compose. For example, given the invocation uw config compose 1.yaml 2.yaml 3.yaml, if *A appears in 1.yaml, then &A may be defined in any one of the three YAML files. If, *A appears instead in 2.yaml, then &A must be defined either in 2.yaml or 3.yaml. And if *A appears in 3.yaml, then &A must be defined in 3.yaml.

Configs in any format supported by uwtools can be composed, but all composed configs must be of the same format. For example, Fortran namelist configs may be composed:

Listing 8 compose-base.nml

&values
  pi = 3.142
  color = 'blue'
  flower = 'violet'
/

Listing 9 compose-update-1.nml

&values
  e = 2.718
/

Listing 10 compose-update-2.nml

&values
  flower = 'rose'
/

uw config compose compose-base.nml compose-update-1.nml compose-update-2.nml --output-format nml

&values
    pi = 3.142
    color = 'blue'
    flower = 'rose'
    e = 2.718
/

`realize`

In uw terminology, to realize a configuration file is to transform it from its raw form into its final, usable state. The realize action can build a complete config file from two or more separate files.

uw config realize --help

usage: uw config realize [-h] [--version] [--input-file PATH]
                         [--input-format {ini,nml,sh,yaml}]
                         [--update-file PATH]
                         [--update-format {ini,nml,sh,yaml}]
                         [--output-file PATH]
                         [--output-format {ini,nml,sh,yaml}]
                         [--key-path KEY[.KEY...]] [--values-needed] [--total]
                         [--dry-run] [--quiet] [--verbose]

Realize config

Optional arguments:
  -h, --help
      Show help and exit
  --version
      Show version info and exit
  --input-file PATH, -i PATH
      Path to input file (default: read from stdin)
  --input-format {ini,nml,sh,yaml}
      Input format (default: yaml)
  --update-file PATH, -u PATH
      Path to update file (default: read from stdin)
  --update-format {ini,nml,sh,yaml}
      Update format
  --output-file PATH, -o PATH
      Path to output file (default: write to stdout)
  --output-format {ini,nml,sh,yaml}
      Output format (default: yaml)
  --key-path KEY[.KEY...]
      Dot-separated path of keys to the block to be output
  --values-needed
      Print report of values needed to realize config
  --total
      Require rendering of all Jinja2 variables/expressions
  --dry-run
      Only log info, making no changes
  --quiet, -q
      Print no logging messages
  --verbose, -v
      Print all logging messages

Examples

The initial examples in this section use YAML file config.yaml with contents:

values:
  date: '{{ yyyymmdd }}'
  empty:
  greeting: Hello
  message: '{{ ((greeting + " " + recipient + " ") * repeat) | trim }}'
  recipient: World
  repeat: 1

and YAML file update.yaml with contents:

values:
  date: 20240105
  greeting: Good Night
  recipient: Moon
  repeat: 2

For a report of input-config values with unrendered Jinja2 variables/expressions or empty/null keys:

uw config realize --input-file config.yaml --output-format yaml --values-needed

[2025-01-02T03:04:05]     INFO Keys that are complete:
[2025-01-02T03:04:05]     INFO   values
[2025-01-02T03:04:05]     INFO   values.empty
[2025-01-02T03:04:05]     INFO   values.greeting
[2025-01-02T03:04:05]     INFO   values.message
[2025-01-02T03:04:05]     INFO   values.recipient
[2025-01-02T03:04:05]     INFO   values.repeat
[2025-01-02T03:04:05]     INFO 
[2025-01-02T03:04:05]     INFO Keys with unrendered Jinja2 variables/expressions:
[2025-01-02T03:04:05]     INFO   values.date: {{ yyyymmdd }}

To realize the config to stdout, the output format must be explicitly specified:

uw config realize --input-file config.yaml --output-format yaml

values:
  date: '{{ yyyymmdd }}'
  empty: null
  greeting: Hello
  message: Hello World
  recipient: World
  repeat: 1

Shell redirection may also be used to stream output to a file, another process, etc.

Values in the input file can be updated via an optional update file:

uw config realize --input-file config.yaml --update-file update.yaml --output-format yaml

values:
  date: 20240105
  empty: null
  greeting: Good Night
  message: Good Night Moon Good Night Moon
  recipient: Moon
  repeat: 2

To realize the config to a file via command-line argument:

rm -f realized.yaml
uw config realize --input-file config.yaml --update-file update.yaml --output-file realized.yaml
cat realized.yaml

values:
  date: 20240105
  empty: null
  greeting: Good Night
  message: Good Night Moon Good Night Moon
  recipient: Moon
  repeat: 2

With the --dry-run flag specified, nothing is written to stdout (or to a file if --output-file is specified), but a report of what would have been written is logged to stderr:

uw config realize --input-file config.yaml --update-file update.yaml --output-file realized.yaml --dry-run

[2025-01-02T03:04:05]     INFO values:
[2025-01-02T03:04:05]     INFO   date: 20240105
[2025-01-02T03:04:05]     INFO   empty: null
[2025-01-02T03:04:05]     INFO   greeting: Good Night
[2025-01-02T03:04:05]     INFO   message: Good Night Moon Good Night Moon
[2025-01-02T03:04:05]     INFO   recipient: Moon
[2025-01-02T03:04:05]     INFO   repeat: 2

If the config file has an unrecognized (or no) extension, uw will not automatically know how to parse its contents:

uw config realize --input-file config.txt --update-file update.yaml --output-format yaml

values:
  date: 20240105
  empty: null
  greeting: Good Night
  message: Good Night Moon Good Night Moon
  recipient: Moon
  repeat: 2

The format must be explicitly specified (here, config.txt is a copy of config.yaml):

uw config realize --input-file config.txt --update-file update.yaml --output-format yaml --input-format yaml

values:
  date: 20240105
  empty: null
  greeting: Good Night
  message: Good Night Moon Good Night Moon
  recipient: Moon
  repeat: 2

Similarly, if an input file is read from stdin, uw will not automatically know how to parse its contents:

cat config.yaml | uw config realize --update-file update.yaml --output-format yaml

values:
  date: 20240105
  empty: null
  greeting: Good Night
  message: Good Night Moon Good Night Moon
  recipient: Moon
  repeat: 2

The format must be explicitly specified:

cat config.yaml | uw config realize --update-file update.yaml --output-format yaml --input-format yaml

values:
  date: 20240105
  empty: null
  greeting: Good Night
  message: Good Night Moon Good Night Moon
  recipient: Moon
  repeat: 2

This example demonstrates: 1. Reading a config from stdin, 2. Extracting a specific subsection with the --key-path option, and 3. Writing the output in a different format:

cat config.yaml | uw config realize --input-format yaml --update-file update.yaml --key-path values --output-format sh

date=20240105
empty=None
greeting='Good Night'
message='Good Night Moon Good Night Moon'
recipient=Moon
repeat=2

Note

Combining configs with incompatible depths is not supported. ini configs are depth-2, as they organize their key-value pairs (one level) under top-level sections (a second level). sh configs are depth-1, and yaml configs have arbitrary depth.

For example, when attempting to generate a sh config from the original depth-2 config.yaml:

uw config realize --input-file config.yaml --output-format sh

[2025-01-02T03:04:05]    ERROR Cannot treat depth-2 config as 'sh'

nml configs are technically depth-2, but in order to support specification of Fortran derived type (aka user-defined type) members, a mapping between arbitrary-depth YAML and Fortran namelist is supported. For example, derived-type.yaml with contents

config:
  resolution:
    nx: 1440
    ny: 721

would be rendered as a Fortran namelist like this:

uw config realize --input-file derived-type.yaml --output-format nml

&config
    resolution%nx = 1440
    resolution%ny = 721
/

Fortran array-item/slice syntax (e.g. a(1) = 11, a(2,3) = 22, 33, etc.) is not currently supported.

It is possible to provide the update config, rather than the input config, on stdin. Usage rules are as follows:
- Only if either --update-file or --update-config are specified will uw attempt to read and apply update values to the input config.
- If --update-file is provided with an unrecognized (or no) extension, or if the update values are provided on stdin, --update-format must be used to specify the correct format.
- When updating, the input config, the update config, or both must be provided via file; they cannot be streamed from stdin simultaneously.
For example, here the update config is provided on stdin and the input config is read from a file:
```
echo "yyyymmdd: 20240520" | uw config realize --input-file config.yaml --update-format yaml --output-format yaml
```
```
values:
  date: '20240520'
  empty: null
  greeting: Hello
  message: Hello World
  recipient: World
  repeat: 1
yyyymmdd: 20240520
```

By default, variables/expressions that cannot be rendered are passed through unchanged in the output. For example, given config file flowers.yaml with contents

roses: "{{ color1 }}"
violets: "{{ color2 }}"
color1: red

uw config realize --input-file flowers.yaml --output-format yaml
echo -e "\nExit status: $?"

roses: red
violets: '{{ color2 }}'
color1: red

Exit status: 0

Adding the --total flag, however, requires uw to totally realize the config, and to exit with error status if it cannot:

uw config realize --input-file flowers.yaml --output-format yaml --total
echo -e "\nExit status: $?"

[2025-01-02T03:04:05]    ERROR Config could not be realized. Try with --values-needed for details.

Exit status: 1

Realization of individual values is all-or-nothing. If a single value contains a mix of renderable and unrenderable variables/expressions, then the entire value remains unrealized. For example, given roses.yaml with contents
```
roses: "{{ color1 }} or {{ color2 }}"
color1: red
```
```
uw config realize --input-file roses.yaml --output-format yaml
```
```
roses: '{{ color1 }} or {{ color2 }}'
color1: red
```

To request verbose log output:

echo "{hello: '{{ recipient }}', recipient: world}" | uw config realize --input-format yaml --output-format yaml --verbose

[2025-01-02T03:04:05]    DEBUG Command: uw config realize --input-format yaml --output-format yaml --verbose
[2025-01-02T03:04:05]    DEBUG Reading input from stdin
[2025-01-02T03:04:05]    DEBUG [dereference] Dereferencing, current value:
[2025-01-02T03:04:05]    DEBUG [dereference]   hello: '{{ recipient }}'
[2025-01-02T03:04:05]    DEBUG [dereference]   recipient: world
[2025-01-02T03:04:05]    DEBUG [dereference] Rendering: hello
[2025-01-02T03:04:05]    DEBUG [dereference] Rendered: hello
[2025-01-02T03:04:05]    DEBUG [dereference] Rendering: {{ recipient }}
[2025-01-02T03:04:05]    DEBUG [dereference] Rendered: world
[2025-01-02T03:04:05]    DEBUG [dereference] Rendering: recipient
[2025-01-02T03:04:05]    DEBUG [dereference] Rendered: recipient
[2025-01-02T03:04:05]    DEBUG [dereference] Rendering: world
[2025-01-02T03:04:05]    DEBUG [dereference] Rendered: world
[2025-01-02T03:04:05]    DEBUG [dereference] Dereferencing, current value:
[2025-01-02T03:04:05]    DEBUG [dereference]   hello: world
[2025-01-02T03:04:05]    DEBUG [dereference]   recipient: world
[2025-01-02T03:04:05]    DEBUG [dereference] Rendering: hello
[2025-01-02T03:04:05]    DEBUG [dereference] Rendered: hello
[2025-01-02T03:04:05]    DEBUG [dereference] Rendering: world
[2025-01-02T03:04:05]    DEBUG [dereference] Rendered: world
[2025-01-02T03:04:05]    DEBUG [dereference] Rendering: recipient
[2025-01-02T03:04:05]    DEBUG [dereference] Rendered: recipient
[2025-01-02T03:04:05]    DEBUG [dereference] Rendering: world
[2025-01-02T03:04:05]    DEBUG [dereference] Rendered: world
[2025-01-02T03:04:05]    DEBUG [dereference] Dereferencing, final value:
[2025-01-02T03:04:05]    DEBUG [dereference]   hello: world
[2025-01-02T03:04:05]    DEBUG [dereference]   recipient: world
[2025-01-02T03:04:05]    DEBUG Writing output to stdout
hello: world
recipient: world

Note that uw logs to stderr and writes non-log output to stdout, so the streams can be redirected separately via shell redirection.

`validate`

The validate action ensures that a given config file is structured properly.

uw config validate --help

usage: uw config validate --schema-file PATH [-h] [--version]
                          [--input-file PATH] [--quiet] [--verbose]

Validate config

Required arguments:
  --schema-file PATH
      Path to schema file to use for validation

Optional arguments:
  -h, --help
      Show help and exit
  --version
      Show version info and exit
  --input-file PATH, -i PATH
      Path to input file (default: read from stdin)
  --quiet, -q
      Print no logging messages
  --verbose, -v
      Print all logging messages

Examples

The examples that follow use the JSON Schema file schema.jsonschema with contents:

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "type": "object",
  "properties": {
    "values": {
      "type": "object",
      "properties": {
        "greeting": {
          "type": "string"
        },
        "recipient": {
          "type": "string"
        }
      },
      "required": ["greeting", "recipient"],
      "additionalProperties": false
    }
  },
  "required": ["values"],
  "additionalProperties": false
}

and the YAML file values.yaml with contents:

values:
  greeting: Hello
  recipient: World

To validate a YAML config against a given JSON schema:

uw config validate --schema-file schema.jsonschema --input-file values.yaml

[2025-01-02T03:04:05]     INFO Schema validation succeeded for config

Shell redirection may also be used to stream output to a file, another process, etc.

To read the config from stdin and print validation results to stdout:

cat values.yaml | uw config validate --schema-file schema.jsonschema

[2025-01-02T03:04:05]     INFO Schema validation succeeded for config

If a config fails validation, differences from the schema will be displayed. For example, values-bad.yaml:

values:
  greeting: Hello

uw config validate --schema-file schema.jsonschema --input-file values-bad.yaml

[2025-01-02T03:04:05]    ERROR 1 schema-validation error found in config
[2025-01-02T03:04:05]    ERROR Error at values:
[2025-01-02T03:04:05]    ERROR   'recipient' is a required property

To request verbose log output:

uw config validate --schema-file schema.jsonschema --input-file values.yaml --verbose

[2025-01-02T03:04:05]    DEBUG Command: uw config validate --schema-file schema.jsonschema --input-file values.yaml --verbose
[2025-01-02T03:04:05]    DEBUG [dereference] Dereferencing, current value:
[2025-01-02T03:04:05]    DEBUG [dereference]   values:
[2025-01-02T03:04:05]    DEBUG [dereference]     greeting: Hello
[2025-01-02T03:04:05]    DEBUG [dereference]     recipient: World
[2025-01-02T03:04:05]    DEBUG [dereference] Rendering: values
[2025-01-02T03:04:05]    DEBUG [dereference] Rendered: values
[2025-01-02T03:04:05]    DEBUG [dereference] Rendering: greeting
[2025-01-02T03:04:05]    DEBUG [dereference] Rendered: greeting
[2025-01-02T03:04:05]    DEBUG [dereference] Rendering: Hello
[2025-01-02T03:04:05]    DEBUG [dereference] Rendered: Hello
[2025-01-02T03:04:05]    DEBUG [dereference] Rendering: recipient
[2025-01-02T03:04:05]    DEBUG [dereference] Rendered: recipient
[2025-01-02T03:04:05]    DEBUG [dereference] Rendering: World
[2025-01-02T03:04:05]    DEBUG [dereference] Rendered: World
[2025-01-02T03:04:05]    DEBUG [dereference] Dereferencing, final value:
[2025-01-02T03:04:05]    DEBUG [dereference]   values:
[2025-01-02T03:04:05]    DEBUG [dereference]     greeting: Hello
[2025-01-02T03:04:05]    DEBUG [dereference]     recipient: World
[2025-01-02T03:04:05]    DEBUG Validating config against external schema file: schema.jsonschema
[2025-01-02T03:04:05]     INFO Schema validation succeeded for config

Note that uw logs to stderr, so the stream can be redirected.

config

compare

Examples

compose

Examples

realize

Examples

validate

Examples

`config`

`compare`

`compose`

`realize`

`validate`