fs

The uw mode for handling filesystem items (files and directories).

uw fs --help

usage: uw fs [-h] [--version] ACTION ...

Handle filesystem items (files and directories)

Optional arguments:
  -h, --help
      Show help and exit
  --version
      Show version info and exit

Positional arguments:
  ACTION
    copy
      Copy files
    link
      Link files
    makedirs
      Make directories

copy

The copy action stages files in a target directory by copying files. Any KEY positional arguments are used to navigate, in the order given, from the top of the config to the file block.

Source paths prefixed with http:// or https:// will be copied from their upstream network locations to the local filesystem.

uw fs copy --help

usage: uw fs copy [-h] [--version] [--config-file PATH] [--target-dir PATH]
                  [--cycle CYCLE] [--leadtime LEADTIME] [--dry-run]
                  [--key-path KEY[.KEY...]] [--report] [--quiet] [--verbose]

Copy files

Optional arguments:
  -h, --help
      Show help and exit
  --version
      Show version info and exit
  --config-file PATH, -c PATH
      Path to UW YAML config file (default: read from stdin)
  --target-dir PATH
      Root directory for relative destination paths
  --cycle CYCLE
      The cycle in ISO8601 format (e.g. 2024-05-29T12)
  --leadtime LEADTIME
      The leadtime as hours[:minutes[:seconds]]
  --dry-run
      Only log info, making no changes
  --key-path KEY[.KEY...]
      Dot-separated path of keys to config block to use
  --report
      Show JSON report on [non]ready assets
  --quiet, -q
      Print no logging messages
  --verbose, -v
      Print all logging messages

Examples

Given copy-config.yaml containing a mapping from local-filesystem destination paths to source paths

config:
  files:
    foo: src/foo
    licenses/gpl: https://www.gnu.org/licenses/gpl-3.0.txt
    subdir/bar: src/bar

rm -rf copy-dst
uw fs copy --target-dir copy-dst --config-file copy-config.yaml --key-path config.files
echo
tree copy-dst

[2025-01-18T07:29:30]     INFO Validating config against internal schema: files-to-stage
[2025-01-18T07:29:30]     INFO 0 schema-validation errors found in fs config
[2025-01-18T07:29:32]     INFO File src/foo: Ready
[2025-01-18T07:29:33]     INFO Remote object https://www.gnu.org/licenses/gpl-3.0.txt: Ready
[2025-01-18T07:29:33]     INFO File src/bar: Ready
[2025-01-18T07:29:33]     INFO Copy src/foo -> copy-dst/foo: Executing
[2025-01-18T07:29:33]     INFO Copy src/foo -> copy-dst/foo: Ready
[2025-01-18T07:29:33]     INFO Copy https://www.gnu.org/licenses/gpl-3.0.txt -> copy-dst/licenses/gpl: Executing
[2025-01-18T07:29:33]     INFO Copy https://www.gnu.org/licenses/gpl-3.0.txt -> copy-dst/licenses/gpl: Ready
[2025-01-18T07:29:33]     INFO Copy src/bar -> copy-dst/subdir/bar: Executing
[2025-01-18T07:29:33]     INFO Copy src/bar -> copy-dst/subdir/bar: Ready
[2025-01-18T07:29:33]     INFO File copies: Ready

copy-dst
├── foo
├── licenses
│   └── gpl
└── subdir
    └── bar

3 directories, 3 files

Here, foo and bar are copies of their respective local-filesystem source files, and gpl is a copy of the upstream network source.

The --cycle and --leadtime options can be used to make Python datetime and timedelta objects, respectively, available for use in Jinja2 expression in the config. For example:

config:
  files:
    baz-{{ validtime }}: src/{{ yyyymmdd }}/{{ hh }}/{{ nnn }}/baz
yyyymmdd: "{{ cycle.strftime('%Y%m%d') }}"
hh: "{{ cycle.strftime('%H') }}"
nnn: "{{ '%03d' % (leadtime.total_seconds() // 3600) }}"
validtime: "{{ (cycle + leadtime).strftime('%Y-%m-%dT%H') }}"

rm -rf copy-dst-timedep
uw fs copy --target-dir copy-dst-timedep --config-file copy-config-timedep.yaml --cycle 2024-05-29T12 --leadtime 6 --key-path config.files
echo
tree copy-dst-timedep

[2024-11-23T00:30:43]     INFO Validating config against internal schema: files-to-stage
[2024-11-23T00:30:43]     INFO 0 schema-validation errors found in fs config
[2024-11-23T00:30:44]     INFO File src/20240529/12/006/baz: Ready
[2024-11-23T00:30:44]     INFO Copy src/20240529/12/006/baz -> copy-dst-timedep/baz-2024-05-29T18: Executing
[2024-11-23T00:30:44]     INFO Copy src/20240529/12/006/baz -> copy-dst-timedep/baz-2024-05-29T18: Ready
[2024-11-23T00:30:44]     INFO File copies: Ready

copy-dst-timedep
└── baz-2024-05-29T18

1 directory, 1 file

The --target-dir option need not be specified when all destination paths are absolute, and will never be applied to absolute destination paths. If any destination paths are relative, however, it is an error not to provide a target directory:

config:
  files:
    foo: src/foo
    licenses/gpl: https://www.gnu.org/licenses/gpl-3.0.txt
    subdir/bar: src/bar

uw fs copy --config-file copy-config.yaml --key-path config.files

[2024-12-07T01:01:51]     INFO Validating config against internal schema: files-to-stage
[2024-12-07T01:01:53]     INFO 0 schema-validation errors found in fs config
[2024-12-07T01:01:53]    ERROR Relative path 'foo' requires target directory to be specified

When the --report option is specified, a report of files not copied (“not-ready”) and copied (“ready”) will be printed to stdout as machine-readable JSON. For example, using a config specifying both available and unavailable source files:

foo: src/foo
qux: src/qux

rm -rf copy-dst-report-jq
uw fs copy --report --target-dir copy-dst-report-jq --config-file copy-config-report.yaml

[2025-01-30T06:36:44]     INFO Validating config against internal schema: files-to-stage
[2025-01-30T06:36:44]     INFO 0 schema-validation errors found in fs config
[2025-01-30T06:36:48]     INFO File src/foo: Ready
[2025-01-30T06:36:48]  WARNING File src/qux: Not ready [external asset]
[2025-01-30T06:36:48]     INFO Copy src/foo -> copy-dst-report-jq/foo: Executing
[2025-01-30T06:36:48]     INFO Copy src/foo -> copy-dst-report-jq/foo: Ready
[2025-01-30T06:36:48]  WARNING Copy src/qux -> copy-dst-report-jq/qux: Not ready
[2025-01-30T06:36:48]  WARNING Copy src/qux -> copy-dst-report-jq/qux: Requires:
[2025-01-30T06:36:48]  WARNING Copy src/qux -> copy-dst-report-jq/qux: ✖ File src/qux
[2025-01-30T06:36:48]  WARNING File copies: Not ready
[2025-01-30T06:36:48]  WARNING File copies: Requires:
[2025-01-30T06:36:48]  WARNING File copies: ✔ Copy src/foo -> copy-dst-report-jq/foo
[2025-01-30T06:36:48]  WARNING File copies: ✖ Copy src/qux -> copy-dst-report-jq/qux
{
  "not-ready": [
    "copy-dst-report-jq/qux"
  ],
  "ready": [
    "copy-dst-report-jq/foo"
  ]
}

Since uwtools logs to stderr, log and report output can be separated and the latter processed with a tool like jq:

rm -rf copy-dst
uw fs copy --report --target-dir copy-dst --config-file copy-config-report.yaml 2>/dev/null | jq -r .ready[]

copy-dst/foo

makedirs

The makedirs action creates directories. Any KEY positional arguments are used to navigate, in the order given, from the top of the config to the makedirs block.

uw fs makedirs --help

usage: uw fs makedirs [-h] [--version] [--config-file PATH]
                      [--target-dir PATH] [--cycle CYCLE]
                      [--leadtime LEADTIME] [--dry-run]
                      [--key-path KEY[.KEY...]] [--report] [--quiet]
                      [--verbose]

Make directories

Optional arguments:
  -h, --help
      Show help and exit
  --version
      Show version info and exit
  --config-file PATH, -c PATH
      Path to UW YAML config file (default: read from stdin)
  --target-dir PATH
      Root directory for relative destination paths
  --cycle CYCLE
      The cycle in ISO8601 format (e.g. 2024-08-12T00)
  --leadtime LEADTIME
      The leadtime as hours[:minutes[:seconds]]
  --dry-run
      Only log info, making no changes
  --key-path KEY[.KEY...]
      Dot-separated path of keys to config block to use
  --report
      Show JSON report on [non]ready assets
  --quiet, -q
      Print no logging messages
  --verbose, -v
      Print all logging messages

Examples

Given makedirs-config.yaml containing

config:
  makedirs:
    - foo
    - bar

rm -rf makedirs-parent
uw fs makedirs --target-dir makedirs-parent --config-file makedirs-config.yaml --key-path config
echo
tree -F makedirs-parent

[2024-11-23T00:30:43]     INFO Validating config against internal schema: makedirs
[2024-11-23T00:30:43]     INFO 0 schema-validation errors found in fs config
[2024-11-23T00:30:44]     INFO Directory makedirs-parent/foo: Executing
[2024-11-23T00:30:44]     INFO Directory makedirs-parent/foo: Ready
[2024-11-23T00:30:44]     INFO Directory makedirs-parent/bar: Executing
[2024-11-23T00:30:44]     INFO Directory makedirs-parent/bar: Ready
[2024-11-23T00:30:44]     INFO Directories: Ready

makedirs-parent/
├── bar/
└── foo/

3 directories, 0 files

The --cycle and --leadtime options can be used to make Python datetime and timedelta objects, respectively, available for use in Jinja2 expression in the config. For example:

config:
  makedirs:
    - foo/{{ yyyymmdd }}/{{ hh }}/{{ nnn }}/bar
    - baz/{{ yyyymmdd }}/{{ hh }}/{{ nnn }}/qux
yyyymmdd: "{{ cycle.strftime('%Y%m%d') }}"
hh: "{{ cycle.strftime('%H') }}"
nnn: "{{ '%03d' % (leadtime.total_seconds() // 3600) }}"
validtime: "{{ (cycle + leadtime).strftime('%Y-%m-%dT%H') }}"

rm -rf makedirs-parent-timedep
uw fs makedirs --target-dir makedirs-parent-timedep --config-file makedirs-config-timedep.yaml --cycle 2024-05-29T12 --leadtime 6 --key-path config
echo
tree -F makedirs-parent-timedep

[2024-11-23T00:30:43]     INFO Validating config against internal schema: makedirs
[2024-11-23T00:30:43]     INFO 0 schema-validation errors found in fs config
[2024-11-23T00:30:44]     INFO Directory makedirs-parent-timedep/foo/20240529/12/006/bar: Executing
[2024-11-23T00:30:44]     INFO Directory makedirs-parent-timedep/foo/20240529/12/006/bar: Ready
[2024-11-23T00:30:44]     INFO Directory makedirs-parent-timedep/baz/20240529/12/006/qux: Executing
[2024-11-23T00:30:44]     INFO Directory makedirs-parent-timedep/baz/20240529/12/006/qux: Ready
[2024-11-23T00:30:44]     INFO Directories: Ready

makedirs-parent-timedep/
├── baz/
│   └── 20240529/
│       └── 12/
│           └── 006/
│               └── qux/
└── foo/
    └── 20240529/
        └── 12/
            └── 006/
                └── bar/

11 directories, 0 files

The --target-dir option need not be specified when all directory paths are absolute, and will never be applied to absolute paths. If any paths are relative, however, it is an error not to provide a target directory:

config:
  makedirs:
    - foo
    - bar

uw fs makedirs --config-file makedirs-config.yaml --key-path config

[2024-12-07T01:01:55]     INFO Validating config against internal schema: makedirs
[2024-12-07T01:01:55]     INFO 0 schema-validation errors found in fs config
[2024-12-07T01:01:55]    ERROR Relative path 'foo' requires target directory to be specified

When the --report option is specified, a report of directories not created (“not-ready”) and created (“ready”) will be printed to stdout as machine-readable JSON. For example, using a config specifying both available and unavailable source files:

makedirs:
  - foo
  - subdir/bar

d=makedirs-parent-report
rm -rf $d
mkdir -p $d/subdir
chmod 550 $d/subdir # read-only
uw fs makedirs --report --target-dir $d --config-file makedirs-config-report.yaml
chmod 750 $d/subdir # read-write

[2025-02-13T17:19:49]     INFO Validating config against internal schema: makedirs
[2025-02-13T17:19:49]     INFO 0 schema-validation errors found in fs config
[2025-02-13T17:19:50]     INFO Directory makedirs-parent-report/foo: Executing
[2025-02-13T17:19:50]     INFO Directory makedirs-parent-report/foo: Ready
[2025-02-13T17:19:50]     INFO Directory makedirs-parent-report/subdir/bar: Executing
[2025-02-13T17:19:50]    ERROR [Errno 13] Permission denied: 'makedirs-parent-report/subdir/bar'
[2025-02-13T17:19:50]  WARNING Directory makedirs-parent-report/subdir/bar: Not ready
[2025-02-13T17:19:50]  WARNING Directories: Not ready
[2025-02-13T17:19:50]  WARNING Directories: Requires:
[2025-02-13T17:19:50]  WARNING Directories: ✔ Directory makedirs-parent-report/foo
[2025-02-13T17:19:50]  WARNING Directories: ✖ Directory makedirs-parent-report/subdir/bar
{
  "not-ready": [
    "makedirs-parent-report/subdir/bar"
  ],
  "ready": [
    "makedirs-parent-report/foo"
  ]
}

Since uwtools logs to stderr, log and report output can be separated and the latter processed with a tool like jq:

d=makedirs-parent-report-jq
rm -rf $d
mkdir -p $d/subdir
chmod 550 $d/subdir # read-only
uw fs makedirs --report --target-dir $d --config-file makedirs-config-report.yaml 2>/dev/null | jq -r .ready[]
chmod 750 $d/subdir # read-write

makedirs-parent-report-jq/foo