`config`

The uw mode for handling configuration files (configs).

uw config --help

usage: uw config [-h] [--version] ACTION ...

Handle configs

Optional arguments:
  -h, --help
      Show help and exit
  --version
      Show version info and exit

Positional arguments:
  ACTION
    compare
      Compare configs
    realize
      Realize config
    validate
      Validate config

`compare`

The compare action compares two config files.

uw config compare --help

usage: uw config compare --file-1-path PATH --file-2-path PATH [-h]
                         [--version] [--file-1-format {ini,nml,sh,yaml}]
                         [--file-2-format {ini,nml,sh,yaml}] [--quiet]
                         [--verbose]

Compare configs

Required arguments:
  --file-1-path PATH
      Path to file 1
  --file-2-path PATH
      Path to file 2

Optional arguments:
  -h, --help
      Show help and exit
  --version
      Show version info and exit
  --file-1-format {ini,nml,sh,yaml}
      Format of file 1
  --file-2-format {ini,nml,sh,yaml}
      Format of file 2
  --quiet, -q
      Print no logging messages
  --verbose, -v
      Print all logging messages

Examples

The examples that follow use identical namelist files a.nml and b.nml with contents:

&values
  greeting = "Hello"
  recipient = "World"
/

To compare two config files with the same contents:

uw config compare --file-1-path a.nml --file-2-path b.nml

[2024-05-23T19:39:15]     INFO - a.nml
[2024-05-23T19:39:15]     INFO + b.nml
[2024-05-23T19:39:15]     INFO ---------------------------------------------------------------------

If there are differences between the config files, they will be shown below the dashed line. For example, c.nml is missing the line recipient: World:

&values
  greeting = "Hello"
/

uw config compare --file-1-path a.nml --file-2-path c.nml

[2024-05-23T19:39:16]     INFO - a.nml
[2024-05-23T19:39:16]     INFO + c.nml
[2024-05-23T19:39:16]     INFO ---------------------------------------------------------------------
[2024-05-23T19:39:16]     INFO values:       recipient:  - World + None

If a config file has an unrecognized (or no) extension, uw will not know how to parse its contents:

uw config compare --file-1-path a.txt --file-2-path c.nml

Cannot deduce format of 'a.txt' from unknown extension 'txt'

The format must be explicitly specified (a.txt is a copy of a.nml):

uw config compare --file-1-path a.txt --file-1-format nml --file-2-path c.nml

[2024-05-23T19:39:15]     INFO - a.txt
[2024-05-23T19:39:15]     INFO + c.nml
[2024-05-23T19:39:15]     INFO ---------------------------------------------------------------------
[2024-05-23T19:39:15]     INFO values:       recipient:  - World + None

To request verbose log output:

uw config compare --file-1-path a.nml --file-2-path c.nml --verbose

[2024-05-23T19:39:15]    DEBUG Command: uw config compare --file-1-path a.nml --file-2-path c.nml --verbose
[2024-05-23T19:39:15]     INFO - a.nml
[2024-05-23T19:39:15]     INFO + c.nml
[2024-05-23T19:39:15]     INFO ---------------------------------------------------------------------
[2024-05-23T19:39:15]     INFO values:       recipient:  - World + None

Note that uw logs to stderr. Use shell redirection as needed.

Note

Comparisons are supported only for configs of the same format, e.g. YAML vs YAML, Fortran namelist vs Fortran namelist, etc. uw will flag invalid comparisons:

uw config compare --file-1-path a.nml --file-2-path b.yaml

[2024-05-23T19:39:15]    ERROR Formats do not match: nml vs yaml

`realize`

In uw terminology, to realize a configuration file is to transform it from its raw form into its final, usable state. The realize action can build a complete config file from two or more separate files.

uw config realize --help

usage: uw config realize [-h] [--version] [--input-file PATH]
                         [--input-format {ini,nml,sh,yaml}]
                         [--update-file PATH]
                         [--update-format {ini,nml,sh,yaml}]
                         [--output-file PATH]
                         [--output-format {ini,nml,sh,yaml}]
                         [--key-path KEY[.KEY...]] [--values-needed] [--total]
                         [--dry-run] [--quiet] [--verbose]

Realize config

Optional arguments:
  -h, --help
      Show help and exit
  --version
      Show version info and exit
  --input-file PATH, -i PATH
      Path to input file (defaults to stdin)
  --input-format {ini,nml,sh,yaml}
      Input format
  --update-file PATH, -u PATH
      Path to update file (defaults to stdin)
  --update-format {ini,nml,sh,yaml}
      Update format
  --output-file PATH, -o PATH
      Path to output file (defaults to stdout)
  --output-format {ini,nml,sh,yaml}
      Output format
  --key-path KEY[.KEY...]
      Dot-separated path of keys to the block to be output
  --values-needed
      Print report of values needed to realize config
  --total
      Require rendering of all Jinja2 variables/expressions
  --dry-run
      Only log info, making no changes
  --quiet, -q
      Print no logging messages
  --verbose, -v
      Print all logging messages

Examples

The initial examples in this section use YAML file config.yaml with contents:

values:
  date: '{{ yyyymmdd }}'
  empty:
  greeting: Hello
  message: '{{ ((greeting + " " + recipient + " ") * repeat) | trim }}'
  recipient: World
  repeat: 1

and YAML file update.yaml with contents:

values:
  date: 20240105
  greeting: Good Night
  recipient: Moon
  repeat: 2

For a report of input-config values with unrendered Jinja2 variables/expressions or empty/null keys:

uw config realize --input-file config.yaml --output-format yaml --values-needed

[2024-05-28T16:43:43]     INFO Keys that are complete:
[2024-05-28T16:43:43]     INFO   values
[2024-05-28T16:43:43]     INFO   values.empty
[2024-05-28T16:43:43]     INFO   values.greeting
[2024-05-28T16:43:43]     INFO   values.message
[2024-05-28T16:43:43]     INFO   values.recipient
[2024-05-28T16:43:43]     INFO   values.repeat
[2024-05-28T16:43:43]     INFO 
[2024-05-28T16:43:43]     INFO Keys with unrendered Jinja2 variables/expressions:
[2024-05-28T16:43:43]     INFO   values.date: {{ yyyymmdd }}

To realize the config to stdout, the output format must be explicitly specified:

uw config realize --input-file config.yaml --output-format yaml

values:
  date: '{{ yyyymmdd }}'
  empty: null
  greeting: Hello
  message: Hello World
  recipient: World
  repeat: 1

Shell redirection may also be used to stream output to a file, another process, etc.

Values in the input file can be updated via an optional update file:

uw config realize --input-file config.yaml --update-file update.yaml --output-format yaml

values:
  date: 20240105
  empty: null
  greeting: Good Night
  message: Good Night Moon Good Night Moon
  recipient: Moon
  repeat: 2

To realize the config to a file via command-line argument:

rm -f realized.yaml
uw config realize --input-file config.yaml --update-file update.yaml --output-file realized.yaml
cat realized.yaml

values:
  date: 20240105
  empty: null
  greeting: Good Night
  message: Good Night Moon Good Night Moon
  recipient: Moon
  repeat: 2

With the --dry-run flag specified, nothing is written to stdout (or to a file if --output-file is specified), but a report of what would have been written is logged to stderr:

uw config realize --input-file config.yaml --update-file update.yaml --output-file realized.yaml --dry-run

[2024-05-23T19:39:16]     INFO values:
[2024-05-23T19:39:16]     INFO   date: 20240105
[2024-05-23T19:39:16]     INFO   empty: null
[2024-05-23T19:39:16]     INFO   greeting: Good Night
[2024-05-23T19:39:16]     INFO   message: Good Night Moon Good Night Moon
[2024-05-23T19:39:16]     INFO   recipient: Moon
[2024-05-23T19:39:16]     INFO   repeat: 2

If the config file has an unrecognized (or no) extension, uw will not automatically know how to parse its contents:

uw config realize --input-file config.txt --update-file update.yaml --output-format yaml

Cannot deduce format of 'config.txt' from unknown extension 'txt'

The format must be explicitly specified (here, config.txt is a copy of config.yaml):

uw config realize --input-file config.txt --update-file update.yaml --output-format yaml --input-format yaml

values:
  date: 20240105
  empty: null
  greeting: Good Night
  message: Good Night Moon Good Night Moon
  recipient: Moon
  repeat: 2

Similarly, if an input file is read from stdin, uw will not automatically know how to parse its contents:

cat config.yaml | uw config realize --update-file update.yaml --output-format yaml

Specify --input-format when --input-file is not specified

The format must be explicitly specified:

cat config.yaml | uw config realize --update-file update.yaml --output-format yaml --input-format yaml

values:
  date: 20240105
  empty: null
  greeting: Good Night
  message: Good Night Moon Good Night Moon
  recipient: Moon
  repeat: 2

This example demonstrates: 1. Reading a config from stdin, 2. Extracting a specific subsection with the --key-path option, and 3. Writing the output in a different format:

cat config.yaml | uw config realize --input-format yaml --update-file update.yaml --key-path values --output-format sh

date=20240105
empty=None
greeting='Good Night'
message='Good Night Moon Good Night Moon'
recipient=Moon
repeat=2

Note

Combining configs with incompatible depths is not supported. ini configs are depth-2, as they organize their key-value pairs (one level) under top-level sections (a second level). sh configs are depth-1, and yaml configs have arbitrary depth.

For example, when attempting to generate a sh config from the original depth-2 config.yaml:

uw config realize --input-file config.yaml --output-format sh

[2024-05-23T19:39:15]    ERROR Cannot realize depth-2 config to type-'sh' config

nml configs are technically depth-2, but in order to support specification of Fortran derived type (aka user-defined type) members, a mapping between arbitrary-depth YAML and Fortran namelist is supported. For example, derived-type.yaml with contents

config:
  resolution:
    nx: 1440
    ny: 721

would be rendered as a Fortran namelist like this:

uw config realize --input-file derived-type.yaml --output-format nml

&config
    resolution%nx = 1440
    resolution%ny = 721
/

Fortran array-item/slice syntax (e.g. a(1) = 11, a(2,3) = 22, 33, etc.) is not currently supported.

It is possible to provide the update config, rather than the input config, on stdin. Usage rules are as follows:
- Only if either --update-file or --update-config are specified will uw attempt to read and apply update values to the input config.
- If --update-file is provided with an unrecognized (or no) extension, or if the update values are provided on stdin, --update-format must be used to specify the correct format.
- When updating, the input config, the update config, or both must be provided via file; they cannot be streamed from stdin simultaneously.
For example, here the update config is provided on stdin and the input config is read from a file:
```
echo "yyyymmdd: 20240520" | uw config realize --input-file config.yaml --update-format yaml --output-format yaml
```
```
values:
  date: '20240520'
  empty: null
  greeting: Hello
  message: Hello World
  recipient: World
  repeat: 1
yyyymmdd: 20240520
```

By default, variables/expressions that cannot be rendered are passed through unchanged in the output. For example, given config file flowers.yaml with contents

roses: "{{ color1 }}"
violets: "{{ color2 }}"
color1: red

uw config realize --input-file flowers.yaml --output-format yaml
echo -e "\nExit status: $?"

roses: red
violets: '{{ color2 }}'
color1: red

Exit status: 0

Adding the --total flag, however, requires uw to totally realize the config, and to exit with error status if it cannot:

uw config realize --input-file flowers.yaml --output-format yaml --total
echo -e "\nExit status: $?"

[2024-05-23T19:39:14]    ERROR Config could not be realized. Try with --values-needed for details.

Exit status: 1

Realization of individual values is all-or-nothing. If a single value contains a mix of renderable and unrenderable variables/expressions, then the entire value remains unrealized. For example, given roses.yaml with contents
```
roses: "{{ color1 }} or {{ color2 }}"
color1: red
```
```
uw config realize --input-file roses.yaml --output-format yaml
```
```
roses: '{{ color1 }} or {{ color2 }}'
color1: red
```

To request verbose log output:

echo "{hello: '{{ recipient }}', recipient: world}" | uw config realize --input-format yaml --output-format yaml --verbose

[2024-05-23T19:39:16]    DEBUG Command: uw config realize --input-format yaml --output-format yaml --verbose
[2024-05-23T19:39:16]    DEBUG Reading input from stdin
[2024-05-23T19:39:16]    DEBUG Dereferencing, current value:
[2024-05-23T19:39:16]    DEBUG   hello: '{{ recipient }}'
[2024-05-23T19:39:16]    DEBUG   recipient: world
[2024-05-23T19:39:16]    DEBUG [dereference] Rendering: {{ recipient }}
[2024-05-23T19:39:16]    DEBUG [dereference] Rendered: world
[2024-05-23T19:39:16]    DEBUG [dereference] Rendering: hello
[2024-05-23T19:39:16]    DEBUG [dereference] Rendered: hello
[2024-05-23T19:39:16]    DEBUG [dereference] Rendering: world
[2024-05-23T19:39:16]    DEBUG [dereference] Rendered: world
[2024-05-23T19:39:16]    DEBUG [dereference] Rendering: recipient
[2024-05-23T19:39:16]    DEBUG [dereference] Rendered: recipient
[2024-05-23T19:39:16]    DEBUG Dereferencing, current value:
[2024-05-23T19:39:16]    DEBUG   hello: world
[2024-05-23T19:39:16]    DEBUG   recipient: world
[2024-05-23T19:39:16]    DEBUG [dereference] Rendering: world
[2024-05-23T19:39:16]    DEBUG [dereference] Rendered: world
[2024-05-23T19:39:16]    DEBUG [dereference] Rendering: hello
[2024-05-23T19:39:16]    DEBUG [dereference] Rendered: hello
[2024-05-23T19:39:16]    DEBUG [dereference] Rendering: world
[2024-05-23T19:39:16]    DEBUG [dereference] Rendered: world
[2024-05-23T19:39:16]    DEBUG [dereference] Rendering: recipient
[2024-05-23T19:39:16]    DEBUG [dereference] Rendered: recipient
[2024-05-23T19:39:16]    DEBUG Dereferencing, final value:
[2024-05-23T19:39:16]    DEBUG   hello: world
[2024-05-23T19:39:16]    DEBUG   recipient: world
[2024-05-23T19:39:16]    DEBUG Writing output to stdout
hello: world
recipient: world

Note that uw logs to stderr and writes non-log output to stdout, so the streams can be redirected separately via shell redirection.

`validate`

The validate action ensures that a given config file is structured properly.

uw config validate --help

usage: uw config validate --schema-file PATH [-h] [--version]
                          [--input-file PATH] [--quiet] [--verbose]

Validate config

Required arguments:
  --schema-file PATH
      Path to schema file to use for validation

Optional arguments:
  -h, --help
      Show help and exit
  --version
      Show version info and exit
  --input-file PATH, -i PATH
      Path to input file (defaults to stdin)
  --quiet, -q
      Print no logging messages
  --verbose, -v
      Print all logging messages

Examples

The examples that follow use the JSON Schema file schema.jsonschema with contents:

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "type": "object",
  "properties": {
    "values": {
      "type": "object",
      "properties": {
        "greeting": {
          "type": "string"
        },
        "recipient": {
          "type": "string"
        }
      },
      "required": ["greeting", "recipient"],
      "additionalProperties": false
    }
  },
  "required": ["values"],
  "additionalProperties": false
}

and the YAML file values.yaml with contents:

values:
  greeting: Hello
  recipient: World

To validate a YAML config against a given JSON schema:

uw config validate --schema-file schema.jsonschema --input-file values.yaml

[2024-05-23T19:39:17]     INFO 0 UW schema-validation errors found

Shell redirection may also be used to stream output to a file, another process, etc.

To read the config from stdin and print validation results to stdout:

cat values.yaml | uw config validate --schema-file schema.jsonschema

[2024-05-23T19:39:15]     INFO 0 UW schema-validation errors found

If a config fails validation, differences from the schema will be displayed. For example, values-bad.yaml:

values:
  greeting: Hello

uw config validate --schema-file schema.jsonschema --input-file values-bad.yaml

[2024-05-23T19:39:17]    ERROR 1 UW schema-validation error found
[2024-05-23T19:39:17]    ERROR Error at values:
[2024-05-23T19:39:17]    ERROR   'recipient' is a required property

To request verbose log output:

uw config validate --schema-file schema.jsonschema --input-file values.yaml --verbose

[2024-05-23T19:39:16]    DEBUG Command: uw config validate --schema-file schema.jsonschema --input-file values.yaml --verbose
[2024-05-23T19:39:16]    DEBUG Dereferencing, current value:
[2024-05-23T19:39:16]    DEBUG   values:
[2024-05-23T19:39:16]    DEBUG     greeting: Hello
[2024-05-23T19:39:16]    DEBUG     recipient: World
[2024-05-23T19:39:16]    DEBUG [dereference] Rendering: Hello
[2024-05-23T19:39:17]    DEBUG [dereference] Rendered: Hello
[2024-05-23T19:39:17]    DEBUG [dereference] Rendering: greeting
[2024-05-23T19:39:17]    DEBUG [dereference] Rendered: greeting
[2024-05-23T19:39:17]    DEBUG [dereference] Rendering: World
[2024-05-23T19:39:17]    DEBUG [dereference] Rendered: World
[2024-05-23T19:39:17]    DEBUG [dereference] Rendering: recipient
[2024-05-23T19:39:17]    DEBUG [dereference] Rendered: recipient
[2024-05-23T19:39:17]    DEBUG [dereference] Rendering: values
[2024-05-23T19:39:17]    DEBUG [dereference] Rendered: values
[2024-05-23T19:39:17]    DEBUG Dereferencing, final value:
[2024-05-23T19:39:17]    DEBUG   values:
[2024-05-23T19:39:17]    DEBUG     greeting: Hello
[2024-05-23T19:39:17]    DEBUG     recipient: World
[2024-05-23T19:39:17]     INFO 0 UW schema-validation errors found

Note that uw logs to stderr, so the stream can be redirected.

config

compare

Examples

realize

Examples

validate

Examples

`config`

`compare`

`realize`

`validate`