fs

The uw mode for handling filesystem items (files and directories).

uw fs --help

usage: uw fs [-h] [--version] ACTION ...

Handle filesystem items (files and directories)

Optional arguments:
  -h, --help
      Show help and exit
  --version
      Show version info and exit

Positional arguments:
  ACTION
    copy
      Copy files
    link
      Link files
    makedirs
      Make directories

copy

The copy action stages files in a target directory by copying files. Any KEY positional arguments are used to navigate, in the order given, from the top of the config to the file block.

uw fs copy --help

usage: uw fs copy [-h] [--version] [--config-file PATH] [--target-dir PATH]
                  [--cycle CYCLE] [--leadtime LEADTIME] [--dry-run] [--quiet]
                  [--verbose]
                  [KEY ...]

Copy files

Optional arguments:
  -h, --help
      Show help and exit
  --version
      Show version info and exit
  --config-file PATH, -c PATH
      Path to UW YAML config file (default: read from stdin)
  --target-dir PATH
      Root directory for relative destination paths
  --cycle CYCLE
      The cycle in ISO8601 format (e.g. 2024-05-29T12)
  --leadtime LEADTIME
      The leadtime as hours[:minutes[:seconds]]
  --dry-run
      Only log info, making no changes
  --quiet, -q
      Print no logging messages
  --verbose, -v
      Print all logging messages
  KEY
      YAML key leading to file dst/src block

Examples

Given copy-config.yaml containing

config:
  files:
    foo: src/foo
    subdir/bar: src/bar

rm -rf copy-dst
uw fs copy --target-dir copy-dst --config-file copy-config.yaml config files
echo
tree copy-dst

[2024-08-26T23:03:41]     INFO Validating config against internal schema: files-to-stage
[2024-08-26T23:03:41]     INFO 0 UW schema-validation errors found in fs config
[2024-08-26T23:03:41]     INFO File copies: Initial state: Not Ready
[2024-08-26T23:03:41]     INFO File copies: Checking requirements
[2024-08-26T23:03:41]     INFO Copy src/foo -> copy-dst/foo: Initial state: Not Ready
[2024-08-26T23:03:41]     INFO Copy src/foo -> copy-dst/foo: Checking requirements
[2024-08-26T23:03:41]     INFO Copy src/foo -> copy-dst/foo: Requirement(s) ready
[2024-08-26T23:03:41]     INFO Copy src/foo -> copy-dst/foo: Executing
[2024-08-26T23:03:41]     INFO Copy src/foo -> copy-dst/foo: Final state: Ready
[2024-08-26T23:03:41]     INFO Copy src/bar -> copy-dst/subdir/bar: Initial state: Not Ready
[2024-08-26T23:03:41]     INFO Copy src/bar -> copy-dst/subdir/bar: Checking requirements
[2024-08-26T23:03:41]     INFO Copy src/bar -> copy-dst/subdir/bar: Requirement(s) ready
[2024-08-26T23:03:41]     INFO Copy src/bar -> copy-dst/subdir/bar: Executing
[2024-08-26T23:03:41]     INFO Copy src/bar -> copy-dst/subdir/bar: Final state: Ready
[2024-08-26T23:03:41]     INFO File copies: Final state: Ready

copy-dst
├── foo
└── subdir
    └── bar

2 directories, 2 files

Here, foo and bar are copies of their respective source files.

The --cycle and --leadtime options can be used to make Python datetime and timedelta objects, respectively, available for use in Jinja2 expression in the config. For example:

config:
  files:
    baz-{{ validtime }}: src/{{ yyyymmdd }}/{{ hh }}/{{ nnn }}/baz
yyyymmdd: "{{ cycle.strftime('%Y%m%d') }}"
hh: "{{ cycle.strftime('%H') }}"
nnn: "{{ '%03d' % (leadtime.total_seconds() // 3600) }}"
validtime: "{{ (cycle + leadtime).strftime('%Y-%m-%dT%H') }}"

rm -rf copy-dst-timedep
uw fs copy --target-dir copy-dst-timedep --config-file copy-config-timedep.yaml --cycle 2024-05-29T12 --leadtime 6 config files
echo
tree copy-dst-timedep

[2024-08-26T23:03:42]     INFO Validating config against internal schema: files-to-stage
[2024-08-26T23:03:43]     INFO 0 UW schema-validation errors found in fs config
[2024-08-26T23:03:43]     INFO File copies: Initial state: Not Ready
[2024-08-26T23:03:43]     INFO File copies: Checking requirements
[2024-08-26T23:03:43]     INFO Copy src/20240529/12/006/baz -> copy-dst-timedep/baz-2024-05-29T18: Initial state: Not Ready
[2024-08-26T23:03:43]     INFO Copy src/20240529/12/006/baz -> copy-dst-timedep/baz-2024-05-29T18: Checking requirements
[2024-08-26T23:03:43]     INFO Copy src/20240529/12/006/baz -> copy-dst-timedep/baz-2024-05-29T18: Requirement(s) ready
[2024-08-26T23:03:43]     INFO Copy src/20240529/12/006/baz -> copy-dst-timedep/baz-2024-05-29T18: Executing
[2024-08-26T23:03:43]     INFO Copy src/20240529/12/006/baz -> copy-dst-timedep/baz-2024-05-29T18: Final state: Ready
[2024-08-26T23:03:43]     INFO File copies: Final state: Ready

copy-dst-timedep
└── baz-2024-05-29T18

1 directory, 1 file

The --target-dir option need not be specified when all destination paths are absolute, and will never be applied to absolute destination paths. If any destination paths are relative, however, it is an error not to provide a target directory:

config:
  files:
    foo: src/foo
    subdir/bar: src/bar

uw fs copy --config-file copy-config.yaml config files

[2024-08-26T23:03:40]     INFO Validating config against internal schema: files-to-stage
[2024-08-26T23:03:40]     INFO 0 UW schema-validation errors found in fs config
[2024-08-26T23:03:40]    ERROR Relative path 'foo' requires the target directory to be specified

makedirs

The makedirs action creates directories. Any KEY positional arguments are used to navigate, in the order given, from the top of the config to the makedirs block.

uw fs makedirs --help

usage: uw fs makedirs [-h] [--version] [--config-file PATH]
                      [--target-dir PATH] [--cycle CYCLE]
                      [--leadtime LEADTIME] [--dry-run] [--quiet] [--verbose]
                      [KEY ...]

Make directories

Optional arguments:
  -h, --help
      Show help and exit
  --version
      Show version info and exit
  --config-file PATH, -c PATH
      Path to UW YAML config file (default: read from stdin)
  --target-dir PATH
      Root directory for relative destination paths
  --cycle CYCLE
      The cycle in ISO8601 format (e.g. 2024-08-12T00)
  --leadtime LEADTIME
      The leadtime as hours[:minutes[:seconds]]
  --dry-run
      Only log info, making no changes
  --quiet, -q
      Print no logging messages
  --verbose, -v
      Print all logging messages
  KEY
      YAML key leading to file dst/src block

Examples

Given makedirs-config.yaml containing

config:
  makedirs:
    - foo
    - bar

rm -rf makedirs-parent
uw fs makedirs --target-dir makedirs-parent --config-file makedirs-config.yaml config
echo
tree -F makedirs-parent

[2024-08-26T23:03:46]     INFO Validating config against internal schema: makedirs
[2024-08-26T23:03:46]     INFO 0 UW schema-validation errors found in fs config
[2024-08-26T23:03:46]     INFO Directories: Initial state: Not Ready
[2024-08-26T23:03:46]     INFO Directories: Checking requirements
[2024-08-26T23:03:46]     INFO Directory makedirs-parent/foo: Initial state: Not Ready
[2024-08-26T23:03:46]     INFO Directory makedirs-parent/foo: Checking requirements
[2024-08-26T23:03:46]     INFO Directory makedirs-parent/foo: Requirement(s) ready
[2024-08-26T23:03:46]     INFO Directory makedirs-parent/foo: Executing
[2024-08-26T23:03:46]     INFO Directory makedirs-parent/foo: Final state: Ready
[2024-08-26T23:03:46]     INFO Directory makedirs-parent/bar: Initial state: Not Ready
[2024-08-26T23:03:46]     INFO Directory makedirs-parent/bar: Checking requirements
[2024-08-26T23:03:46]     INFO Directory makedirs-parent/bar: Requirement(s) ready
[2024-08-26T23:03:46]     INFO Directory makedirs-parent/bar: Executing
[2024-08-26T23:03:46]     INFO Directory makedirs-parent/bar: Final state: Ready
[2024-08-26T23:03:46]     INFO Directories: Final state: Ready

makedirs-parent/
├── bar/
└── foo/

3 directories, 0 files

The --cycle and --leadtime options can be used to make Python datetime and timedelta objects, respectively, available for use in Jinja2 expression in the config. For example:

config:
  makedirs:
    - foo/{{ yyyymmdd }}/{{ hh }}/{{ nnn }}/bar
    - baz/{{ yyyymmdd }}/{{ hh }}/{{ nnn }}/qux
yyyymmdd: "{{ cycle.strftime('%Y%m%d') }}"
hh: "{{ cycle.strftime('%H') }}"
nnn: "{{ '%03d' % (leadtime.total_seconds() // 3600) }}"
validtime: "{{ (cycle + leadtime).strftime('%Y-%m-%dT%H') }}"

rm -rf makedirs-parent-timedep
uw fs makedirs --target-dir makedirs-parent-timedep --config-file makedirs-config-timedep.yaml --cycle 2024-05-29T12 --leadtime 6 config
echo
tree -F makedirs-parent-timedep

[2024-08-26T23:03:46]     INFO Validating config against internal schema: makedirs
[2024-08-26T23:03:46]     INFO 0 UW schema-validation errors found in fs config
[2024-08-26T23:03:46]     INFO Directories: Initial state: Not Ready
[2024-08-26T23:03:46]     INFO Directories: Checking requirements
[2024-08-26T23:03:46]     INFO Directory makedirs-parent-timedep/foo/20240529/12/006/bar: Initial state: Not Ready
[2024-08-26T23:03:46]     INFO Directory makedirs-parent-timedep/foo/20240529/12/006/bar: Checking requirements
[2024-08-26T23:03:46]     INFO Directory makedirs-parent-timedep/foo/20240529/12/006/bar: Requirement(s) ready
[2024-08-26T23:03:46]     INFO Directory makedirs-parent-timedep/foo/20240529/12/006/bar: Executing
[2024-08-26T23:03:46]     INFO Directory makedirs-parent-timedep/foo/20240529/12/006/bar: Final state: Ready
[2024-08-26T23:03:46]     INFO Directory makedirs-parent-timedep/baz/20240529/12/006/qux: Initial state: Not Ready
[2024-08-26T23:03:46]     INFO Directory makedirs-parent-timedep/baz/20240529/12/006/qux: Checking requirements
[2024-08-26T23:03:46]     INFO Directory makedirs-parent-timedep/baz/20240529/12/006/qux: Requirement(s) ready
[2024-08-26T23:03:46]     INFO Directory makedirs-parent-timedep/baz/20240529/12/006/qux: Executing
[2024-08-26T23:03:46]     INFO Directory makedirs-parent-timedep/baz/20240529/12/006/qux: Final state: Ready
[2024-08-26T23:03:46]     INFO Directories: Final state: Ready

makedirs-parent-timedep/
├── baz/
│   └── 20240529/
│       └── 12/
│           └── 006/
│               └── qux/
└── foo/
    └── 20240529/
        └── 12/
            └── 006/
                └── bar/

11 directories, 0 files

The --target-dir option need not be specified when all directory paths are absolute, and will never be applied to absolute paths. If any paths are relative, however, it is an error not to provide a target directory:

config:
  makedirs:
    - foo
    - bar

uw fs makedirs --config-file makedirs-config.yaml config

[2024-08-26T23:03:44]     INFO Validating config against internal schema: makedirs
[2024-08-26T23:03:45]     INFO 0 UW schema-validation errors found in fs config
[2024-08-26T23:03:45]    ERROR Relative path 'foo' requires the target directory to be specified