`fs`

The uw mode for handling filesystem items (files and directories).

uw fs --help

usage: uw fs [-h] [--version] ACTION ...

Handle filesystem items (files and directories)

Optional arguments:
  -h, --help
      Show help and exit
  --version
      Show version info and exit

Positional arguments:
  ACTION
    copy
      Copy files
    hardlink
      Create hardlinks
    link
      Create symlinks
    makedirs
      Make directories

`copy`

The copy action stages files in a target directory by copying files. Any KEY positional arguments are used to navigate, in the order given, from the top of the config to the file block.

Source paths prefixed with http://, https://, hsi://, or htar:// will be copied from their upstream network locations to the local filesystem.

Source paths prefixed with file:// will be treated as local-filesystem paths.

uw fs copy --help

usage: uw fs copy [-h] [--version] [--config-file PATH] [--target-dir PATH]
                  [--cycle CYCLE] [--leadtime LEADTIME] [--dry-run]
                  [--key-path KEY[.KEY...]] [--report] [--quiet] [--verbose]

Copy files

Optional arguments:
  -h, --help
      Show help and exit
  --version
      Show version info and exit
  --config-file PATH, -c PATH
      Path to UW YAML config file (default: read from stdin)
  --target-dir PATH
      Root directory for relative destination paths
  --cycle CYCLE
      The cycle in ISO8601 format (e.g. yyyy-mm-ddThh)
  --leadtime LEADTIME
      The leadtime as hours[:minutes[:seconds]]
  --dry-run
      Only log info, making no changes
  --key-path KEY[.KEY...]
      Dot-separated path of keys to config block to use
  --report
      Show JSON report on [non]ready assets
  --quiet, -q
      Print no logging messages
  --verbose, -v
      Print all logging messages

Local-filesystem and HTTP Sources

Consider this source directory

tree src

src
├── 20240529
│   └── 12
│       └── 006
│           └── baz
├── bar
└── foo

4 directories, 3 files

and a config mapping local-filesystem destination paths to source paths:

config:
  files:
    foo: src/foo
    licenses/uwtools: https://raw.githubusercontent.com/ufs-community/uwtools/refs/heads/main/LICENSE
    subdir/bar: src/bar

rm -rf dst/copy
uw fs copy --target-dir dst/copy --config-file copy.yaml --key-path config.files
echo
tree dst/copy

[2025-01-02T03:04:05]     INFO Validating config against internal schema: files-to-stage
[2025-01-02T03:04:05]     INFO Schema validation succeeded for fs config
[2025-01-02T03:04:05]     INFO Local src/foo -> dst/copy/foo: Executing
[2025-01-02T03:04:05]     INFO Local src/foo -> dst/copy/foo: Ready
[2025-01-02T03:04:05]     INFO HTTP https://raw.githubusercontent.com/ufs-community/uwtools/refs/heads/main/LICENSE -> dst/copy/licenses/uwtools: Executing
[2025-01-02T03:04:05]     INFO HTTP https://raw.githubusercontent.com/ufs-community/uwtools/refs/heads/main/LICENSE -> dst/copy/licenses/uwtools: Ready
[2025-01-02T03:04:05]     INFO Local src/bar -> dst/copy/subdir/bar: Executing
[2025-01-02T03:04:05]     INFO Local src/bar -> dst/copy/subdir/bar: Ready
[2025-01-02T03:04:05]     INFO File copies: Ready

dst/copy
├── foo
├── licenses
│   └── uwtools
└── subdir
    └── bar

3 directories, 3 files

Here, foo and bar are copies of their respective local-filesystem source files, and the uwtools license file is a copy of the upstream network source.

The --cycle and --leadtime options can be used to make Python datetime and timedelta objects, respectively, available for use in Jinja2 expression in the config. For example:

config:
  files:
    baz-{{ validtime }}: src/{{ yyyymmdd }}/{{ hh }}/{{ nnn }}/baz
yyyymmdd: "{{ cycle.strftime('%Y%m%d') }}"
hh: "{{ cycle.strftime('%H') }}"
nnn: "{{ '%03d' % (leadtime.total_seconds() // 3600) }}"
validtime: "{{ (cycle + leadtime).strftime('%Y-%m-%dT%H') }}"

rm -rf dst/copy-timedep
uw fs copy --target-dir dst/copy-timedep --config-file copy-timedep.yaml --cycle 2024-05-29T12 --leadtime 6 --key-path config.files
echo
tree dst/copy-timedep

[2025-01-02T03:04:05]     INFO Validating config against internal schema: files-to-stage
[2025-01-02T03:04:05]     INFO Schema validation succeeded for fs config
[2025-01-02T03:04:05]     INFO Local src/20240529/12/006/baz -> dst/copy-timedep/baz-2024-05-29T18: Executing
[2025-01-02T03:04:05]     INFO Local src/20240529/12/006/baz -> dst/copy-timedep/baz-2024-05-29T18: Ready
[2025-01-02T03:04:05]     INFO File copies: Ready

dst/copy-timedep
└── baz-2024-05-29T18

1 directory, 1 file

The --target-dir option need not be specified when all destination paths are absolute, and will never be applied to absolute destination paths. If any destination paths are relative, however, it is an error not to provide a target directory:

config:
  files:
    foo: src/foo
    licenses/uwtools: https://raw.githubusercontent.com/ufs-community/uwtools/refs/heads/main/LICENSE
    subdir/bar: src/bar

uw fs copy --config-file copy.yaml --key-path config.files

[2025-01-02T03:04:05]     INFO Validating config against internal schema: files-to-stage
[2025-01-02T03:04:05]     INFO Schema validation succeeded for fs config
[2025-01-02T03:04:05]    ERROR Relative path 'foo' requires target directory to be specified

When the --report option is specified, a report of files not copied (“not-ready”) and copied (“ready”) will be printed to stdout as machine-readable JSON. For example, using a config specifying both available and unavailable source files:

foo: src/foo
qux: src/qux

rm -rf dst/copy-report
uw fs copy --report --target-dir dst/copy-report --config-file copy-report.yaml

[2025-01-02T03:04:05]     INFO Validating config against internal schema: files-to-stage
[2025-01-02T03:04:05]     INFO Schema validation succeeded for fs config
[2025-01-02T03:04:05]     INFO Local src/foo -> dst/copy-report/foo: Executing
[2025-01-02T03:04:05]     INFO Local src/foo -> dst/copy-report/foo: Ready
[2025-01-02T03:04:05]  WARNING File src/qux: Not ready [external asset]
[2025-01-02T03:04:05]  WARNING Local src/qux -> dst/copy-report/qux: Not ready
[2025-01-02T03:04:05]  WARNING Local src/qux -> dst/copy-report/qux: Requires:
[2025-01-02T03:04:05]  WARNING Local src/qux -> dst/copy-report/qux: ✖ File src/qux
[2025-01-02T03:04:05]  WARNING File copies: Not ready
[2025-01-02T03:04:05]  WARNING File copies: Requires:
[2025-01-02T03:04:05]  WARNING File copies: ✔ Local src/foo -> dst/copy-report/foo
[2025-01-02T03:04:05]  WARNING File copies: ✖ Local src/qux -> dst/copy-report/qux
{
  "not-ready": [
    "dst/copy-report/qux"
  ],
  "ready": [
    "dst/copy-report/foo"
  ]
}

Since uwtools logs to stderr, log and report output can be separated and the latter processed with a tool like jq, for example for extracting values for use elsewhere:

rm -rf dst/copy-report-jq
uw fs copy --report --target-dir dst/copy-report-jq --config-file copy-report.yaml 2>/dev/null | jq -r .ready[]

dst/copy-report-jq/foo

Use the !glob tag to specify that a local-filesystem source-path value should be treated as a glob pattern:

dst/<file>: !glob src/*

rm -rf dst/copy-glob
uw fs copy --report --target-dir dst/copy-glob --config-file copy-glob.yaml
echo
tree dst/copy-glob

[2025-01-02T03:04:05]     INFO Validating config against internal schema: files-to-stage
[2025-01-02T03:04:05]     INFO Schema validation succeeded for fs config
[2025-01-02T03:04:05]  WARNING Ignoring directory src/20240529
[2025-01-02T03:04:05]     INFO Local src/foo -> dst/copy-glob/dst/foo: Executing
[2025-01-02T03:04:05]     INFO Local src/foo -> dst/copy-glob/dst/foo: Ready
[2025-01-02T03:04:05]     INFO Local src/bar -> dst/copy-glob/dst/bar: Executing
[2025-01-02T03:04:05]     INFO Local src/bar -> dst/copy-glob/dst/bar: Ready
[2025-01-02T03:04:05]     INFO File copies: Ready
{
  "not-ready": [],
  "ready": [
    "dst/copy-glob/dst/foo",
    "dst/copy-glob/dst/bar"
  ]
}

dst/copy-glob
└── dst
    ├── bar
    └── foo

2 directories, 2 files

The --report output can be especially useful in combination with glob patterns to allow downstream logic to process a set of copied files whose identity is not known in advance.

See Glob Support for more examples and information on the semantics of the !glob tag.

HPSS Sources

The examples in this section use the following files

hsi -q ls -1 /BMC/rtrr/5year/uwtools/*.tar

[connecting to hpsscore1.fairmont.rdhpcs.noaa.gov/1217]
/BMC/rtrr/5year/uwtools/code.tar
/BMC/rtrr/5year/uwtools/data.tar

with this content:

htar -qtf /BMC/rtrr/5year/uwtools/code.tar
htar -qtf /BMC/rtrr/5year/uwtools/data.tar

[connecting to hpsscore1.fairmont.rdhpcs.noaa.gov/1217]
HTAR: -rw-r--r--  Paul.Madden/rtruc         64 2025-04-04 04:37  a1.c
HTAR: -rw-r--r--  Paul.Madden/rtruc         22 2025-04-04 04:38  a1.py
HTAR: -rw-------  Paul.Madden/rtruc        256 2025-04-04 12:58  /tmp/HTAR_CF_CHK_2834605_1743771535
HTAR: HTAR SUCCESSFUL
[connecting to hpsscore1.fairmont.rdhpcs.noaa.gov/1217]
HTAR: -rw-r--r--  Paul.Madden/rtruc          3 2025-03-29 16:47  a1.txt
HTAR: -rw-r--r--  Paul.Madden/rtruc          3 2025-03-29 16:48  a2.dat
HTAR: -rw-r--r--  Paul.Madden/rtruc          3 2025-03-29 16:48  b1.txt
HTAR: -rw-r--r--  Paul.Madden/rtruc          3 2025-03-29 16:48  b2.dat
HTAR: -rw-------  Paul.Madden/rtruc        256 2025-03-29 17:14  /tmp/HTAR_CF_CHK_4120691_1743268474
HTAR: HTAR SUCCESSFUL

They also assume that the hsi and htar executable programs are available on PATH.

HSI Support

An hsi:// URL can be used as a source path to copy full files from HPSS to the local filesystem:

a.tar: hsi:///BMC/rtrr/5year/uwtools/data.tar

rm -rf dst/copy-hpss-hsi-single
uw fs copy --target-dir dst/copy-hpss-hsi-single --config-file copy-hpss-hsi-single.yaml
echo
tree dst/copy-hpss-hsi-single

[2025-01-02T03:04:05]     INFO Validating config against internal schema: files-to-stage
[2025-01-02T03:04:05]     INFO 0 schema-validation errors found in fs config
[2025-01-02T03:04:05]     INFO HPSS file /BMC/rtrr/5year/uwtools/data.tar: Executing
[2025-01-02T03:04:05]     INFO HPSS file /BMC/rtrr/5year/uwtools/data.tar: Running: hsi -q ls -1 '/BMC/rtrr/5year/uwtools/data.tar'
[2025-01-02T03:04:05]     INFO HPSS file /BMC/rtrr/5year/uwtools/data.tar: Ready
[2025-01-02T03:04:05]     INFO HSI /BMC/rtrr/5year/uwtools/data.tar -> dst/copy-hpss-hsi-single/a.tar: Executing
[2025-01-02T03:04:05]     INFO HSI /BMC/rtrr/5year/uwtools/data.tar -> dst/copy-hpss-hsi-single/a.tar: Running: hsi -q get 'dst/copy-hpss-hsi-single/a.tar' : '/BMC/rtrr/5year/uwtools/data.tar'
[2025-01-02T03:04:05]     INFO HSI /BMC/rtrr/5year/uwtools/data.tar -> dst/copy-hpss-hsi-single/a.tar: => [connecting to hpsscore1.fairmont.rdhpcs.noaa.gov/1217]
[2025-01-02T03:04:05]     INFO HSI /BMC/rtrr/5year/uwtools/data.tar -> dst/copy-hpss-hsi-single/a.tar: => get  '/home/Paul.Madden/git/uwtools/docs/sections/user_guide/cli/tools/fs/dst/copy-hpss-hsi-single/a.tar' : '/BMC/rtrr/5year/uwtools/data.tar' (2025/03/29 17:12:40 5632 bytes, 3982.9 KBS )
[2025-01-02T03:04:05]     INFO HSI /BMC/rtrr/5year/uwtools/data.tar -> dst/copy-hpss-hsi-single/a.tar: Ready
[2025-01-02T03:04:05]     INFO File copies: Ready

dst/copy-hpss-hsi-single
└── a.tar

1 directory, 1 file

See Full-File hsi Copies for more information on UW YAML support for hsi:// sources.

Use the !glob tag to copy multiple full HPSS files. Here, <f> is a placeholder for the actual name of each glob-matched file.

<f>: !glob hsi:///BMC/rtrr/5year/uwtools/data.*

rm -rf dst/copy-hpss-hsi-glob
uw fs copy --target-dir dst/copy-hpss-hsi-glob --config-file copy-hpss-hsi-glob.yaml
echo
tree dst/copy-hpss-hsi-glob

[2025-01-02T03:04:05]     INFO Validating config against internal schema: files-to-stage
[2025-01-02T03:04:05]     INFO 0 schema-validation errors found in fs config
[2025-01-02T03:04:05]     INFO Running: hsi -q ls -1 '/BMC/rtrr/5year/uwtools/data.*'
[2025-01-02T03:04:05]     INFO HSI /BMC/rtrr/5year/uwtools/data.tar -> dst/copy-hpss-hsi-glob/data.tar: Executing
[2025-01-02T03:04:05]     INFO HSI /BMC/rtrr/5year/uwtools/data.tar -> dst/copy-hpss-hsi-glob/data.tar: Running: hsi -q get 'dst/copy-hpss-hsi-glob/data.tar' : '/BMC/rtrr/5year/uwtools/data.tar'
[2025-01-02T03:04:05]     INFO HSI /BMC/rtrr/5year/uwtools/data.tar -> dst/copy-hpss-hsi-glob/data.tar: => [connecting to hpsscore1.fairmont.rdhpcs.noaa.gov/1217]
[2025-01-02T03:04:05]     INFO HSI /BMC/rtrr/5year/uwtools/data.tar -> dst/copy-hpss-hsi-glob/data.tar: => get  '/home/Paul.Madden/git/uwtools/docs/sections/user_guide/cli/tools/fs/dst/copy-hpss-hsi-glob/data.tar' : '/BMC/rtrr/5year/uwtools/data.tar' (2025/03/29 17:12:40 5632 bytes, 3305.2 KBS )
[2025-01-02T03:04:05]     INFO HSI /BMC/rtrr/5year/uwtools/data.tar -> dst/copy-hpss-hsi-glob/data.tar: Ready
[2025-01-02T03:04:05]     INFO HSI /BMC/rtrr/5year/uwtools/data.tar.idx -> dst/copy-hpss-hsi-glob/data.tar.idx: Executing
[2025-01-02T03:04:05]     INFO HSI /BMC/rtrr/5year/uwtools/data.tar.idx -> dst/copy-hpss-hsi-glob/data.tar.idx: Running: hsi -q get 'dst/copy-hpss-hsi-glob/data.tar.idx' : '/BMC/rtrr/5year/uwtools/data.tar.idx'
[2025-01-02T03:04:05]     INFO HSI /BMC/rtrr/5year/uwtools/data.tar.idx -> dst/copy-hpss-hsi-glob/data.tar.idx: => [connecting to hpsscore1.fairmont.rdhpcs.noaa.gov/1217]
[2025-01-02T03:04:05]     INFO HSI /BMC/rtrr/5year/uwtools/data.tar.idx -> dst/copy-hpss-hsi-glob/data.tar.idx: => get  '/home/Paul.Madden/git/uwtools/docs/sections/user_guide/cli/tools/fs/dst/copy-hpss-hsi-glob/data.tar.idx' : '/BMC/rtrr/5year/uwtools/data.tar.idx' (2025/03/29 17:12:40 2848 bytes, 1950.6 KBS )
[2025-01-02T03:04:05]     INFO HSI /BMC/rtrr/5year/uwtools/data.tar.idx -> dst/copy-hpss-hsi-glob/data.tar.idx: Ready
[2025-01-02T03:04:05]     INFO File copies: Ready

dst/copy-hpss-hsi-glob
├── data.tar
└── data.tar.idx

1 directory, 2 files

See Glob Support for Full-File hsi Copies for more information on the use of the !glob tag in combination with hsi:// sources.

HTAR Support

An htar:// URL can be used as a source path to extract a member from an HPSS archive file and copy it to the local filesystem. The URL should include the path to the archive file and, as the URL query string, the path to archive member to extract:

afile: htar:///BMC/rtrr/5year/uwtools/data.tar?a1.txt

rm -rf dst/copy-hpss-htar-single
uw fs copy --target-dir dst/copy-hpss-htar-single --config-file copy-hpss-htar-single.yaml
echo
tree dst/copy-hpss-htar-single

[2025-01-02T03:04:05]     INFO Validating config against internal schema: files-to-stage
[2025-01-02T03:04:05]     INFO 0 schema-validation errors found in fs config
[2025-01-02T03:04:05]     INFO HPSS file /BMC/rtrr/5year/uwtools/data.tar: Executing
[2025-01-02T03:04:05]     INFO HPSS file /BMC/rtrr/5year/uwtools/data.tar: Running: hsi -q ls -1 '/BMC/rtrr/5year/uwtools/data.tar'
[2025-01-02T03:04:05]     INFO HPSS file /BMC/rtrr/5year/uwtools/data.tar: Ready
[2025-01-02T03:04:05]     INFO HTAR /BMC/rtrr/5year/uwtools/data.tar:a1.txt -> dst/copy-hpss-htar-single/afile: Executing
[2025-01-02T03:04:05]     INFO HTAR /BMC/rtrr/5year/uwtools/data.tar:a1.txt -> dst/copy-hpss-htar-single/afile: Running: htar -qxf '/BMC/rtrr/5year/uwtools/data.tar' 'a1.txt' in dst/copy-hpss-htar-single/.tmpdir3fxd_w55
[2025-01-02T03:04:05]     INFO HTAR /BMC/rtrr/5year/uwtools/data.tar:a1.txt -> dst/copy-hpss-htar-single/afile: => Using fe5.boulder.rdhpcs.noaa.gov for data hostname
[2025-01-02T03:04:05]     INFO HTAR /BMC/rtrr/5year/uwtools/data.tar:a1.txt -> dst/copy-hpss-htar-single/afile: => [connecting to hpsscore1.fairmont.rdhpcs.noaa.gov/1217]
[2025-01-02T03:04:05]     INFO HTAR /BMC/rtrr/5year/uwtools/data.tar:a1.txt -> dst/copy-hpss-htar-single/afile: => HTAR: HTAR SUCCESSFUL
[2025-01-02T03:04:05]     INFO HTAR /BMC/rtrr/5year/uwtools/data.tar:a1.txt -> dst/copy-hpss-htar-single/afile: Moving dst/copy-hpss-htar-single/.tmpdir3fxd_w55/a1.txt -> dst/copy-hpss-htar-single/afile
[2025-01-02T03:04:05]     INFO HTAR /BMC/rtrr/5year/uwtools/data.tar:a1.txt -> dst/copy-hpss-htar-single/afile: Ready
[2025-01-02T03:04:05]     INFO File copies: Ready

dst/copy-hpss-htar-single
└── afile

1 directory, 1 file

See Archive-Member htar Copies for more information on UW YAML support for htar:// sources.

Use the !glob tag to extract multiple members from one or more HPSS archive files. Here, <f> is a placeholder for the actual name of each glob-matched file.

<f>: !glob htar:///BMC/rtrr/5year/uwtools/*.tar?a1.*

rm -rf dst/copy-hpss-htar-glob
uw fs copy --target-dir dst/copy-hpss-htar-glob --config-file copy-hpss-htar-glob.yaml
echo
tree dst/copy-hpss-htar-glob

[2025-01-02T03:04:05]     INFO Validating config against internal schema: files-to-stage
[2025-01-02T03:04:05]     INFO 0 schema-validation errors found in fs config
[2025-01-02T03:04:05]     INFO Running: hsi -q ls -1 '/BMC/rtrr/5year/uwtools/*.tar'
[2025-01-02T03:04:05]     INFO Running: htar -qtf '/BMC/rtrr/5year/uwtools/code.tar'
[2025-01-02T03:04:05]     INFO Running: htar -qtf '/BMC/rtrr/5year/uwtools/data.tar'
[2025-01-02T03:04:05]     INFO HTAR /BMC/rtrr/5year/uwtools/code.tar:a1.c -> dst/copy-hpss-htar-glob/a1.c: Executing
[2025-01-02T03:04:05]     INFO HTAR /BMC/rtrr/5year/uwtools/code.tar:a1.c -> dst/copy-hpss-htar-glob/a1.c: Running: htar -qxf '/BMC/rtrr/5year/uwtools/code.tar' 'a1.c' in dst/copy-hpss-htar-glob/.tmpdirfgmy4hxn
[2025-01-02T03:04:05]     INFO HTAR /BMC/rtrr/5year/uwtools/code.tar:a1.c -> dst/copy-hpss-htar-glob/a1.c: => [connecting to hpsscore1.fairmont.rdhpcs.noaa.gov/1217]
[2025-01-02T03:04:05]     INFO HTAR /BMC/rtrr/5year/uwtools/code.tar:a1.c -> dst/copy-hpss-htar-glob/a1.c: => HTAR: HTAR SUCCESSFUL
[2025-01-02T03:04:05]     INFO HTAR /BMC/rtrr/5year/uwtools/code.tar:a1.c -> dst/copy-hpss-htar-glob/a1.c: Moving dst/copy-hpss-htar-glob/.tmpdirfgmy4hxn/a1.c -> dst/copy-hpss-htar-glob/a1.c
[2025-01-02T03:04:05]     INFO HTAR /BMC/rtrr/5year/uwtools/code.tar:a1.c -> dst/copy-hpss-htar-glob/a1.c: Ready
[2025-01-02T03:04:05]     INFO HTAR /BMC/rtrr/5year/uwtools/code.tar:a1.py -> dst/copy-hpss-htar-glob/a1.py: Executing
[2025-01-02T03:04:05]     INFO HTAR /BMC/rtrr/5year/uwtools/code.tar:a1.py -> dst/copy-hpss-htar-glob/a1.py: Running: htar -qxf '/BMC/rtrr/5year/uwtools/code.tar' 'a1.py' in dst/copy-hpss-htar-glob/.tmpdir83p7619l
[2025-01-02T03:04:05]     INFO HTAR /BMC/rtrr/5year/uwtools/code.tar:a1.py -> dst/copy-hpss-htar-glob/a1.py: => [connecting to hpsscore1.fairmont.rdhpcs.noaa.gov/1217]
[2025-01-02T03:04:05]     INFO HTAR /BMC/rtrr/5year/uwtools/code.tar:a1.py -> dst/copy-hpss-htar-glob/a1.py: => HTAR: HTAR SUCCESSFUL
[2025-01-02T03:04:05]     INFO HTAR /BMC/rtrr/5year/uwtools/code.tar:a1.py -> dst/copy-hpss-htar-glob/a1.py: Moving dst/copy-hpss-htar-glob/.tmpdir83p7619l/a1.py -> dst/copy-hpss-htar-glob/a1.py
[2025-01-02T03:04:05]     INFO HTAR /BMC/rtrr/5year/uwtools/code.tar:a1.py -> dst/copy-hpss-htar-glob/a1.py: Ready
[2025-01-02T03:04:05]     INFO HTAR /BMC/rtrr/5year/uwtools/data.tar:a1.txt -> dst/copy-hpss-htar-glob/a1.txt: Executing
[2025-01-02T03:04:05]     INFO HTAR /BMC/rtrr/5year/uwtools/data.tar:a1.txt -> dst/copy-hpss-htar-glob/a1.txt: Running: htar -qxf '/BMC/rtrr/5year/uwtools/data.tar' 'a1.txt' in dst/copy-hpss-htar-glob/.tmpdir95rslkj_
[2025-01-02T03:04:05]     INFO HTAR /BMC/rtrr/5year/uwtools/data.tar:a1.txt -> dst/copy-hpss-htar-glob/a1.txt: => [connecting to hpsscore1.fairmont.rdhpcs.noaa.gov/1217]
[2025-01-02T03:04:05]     INFO HTAR /BMC/rtrr/5year/uwtools/data.tar:a1.txt -> dst/copy-hpss-htar-glob/a1.txt: => HTAR: HTAR SUCCESSFUL
[2025-01-02T03:04:05]     INFO HTAR /BMC/rtrr/5year/uwtools/data.tar:a1.txt -> dst/copy-hpss-htar-glob/a1.txt: Moving dst/copy-hpss-htar-glob/.tmpdir95rslkj_/a1.txt -> dst/copy-hpss-htar-glob/a1.txt
[2025-01-02T03:04:05]     INFO HTAR /BMC/rtrr/5year/uwtools/data.tar:a1.txt -> dst/copy-hpss-htar-glob/a1.txt: Ready
[2025-01-02T03:04:05]     INFO File copies: Ready

dst/copy-hpss-htar-glob
├── a1.c
├── a1.py
└── a1.txt

1 directory, 3 files

See Glob Support for Archive-Member htar Copies for more information on the use of the !glob tag in combination with htar:// sources.

`hardlink`

The hardlink action stages items in a target directory by creating hardlinks to files or other links. It behaves similarly to the link action (see below), but note the following:

Hardlinks cannot target:
- Items on other filesystems
- Items the user does not have write permission to
- Directories
If a hardlink cannot be created but the --fallback option is specified, a copy (given --fallback copy) or a symlink (given --fallback symlink) will be created instead, if possible.

uw fs hardlink --help

usage: uw fs hardlink [-h] [--version] [--config-file PATH]
                      [--target-dir PATH] [--cycle CYCLE]
                      [--leadtime LEADTIME] [--dry-run]
                      [--key-path KEY[.KEY...]] [--report] [--quiet]
                      [--verbose] [--fallback {copy,symlink}]

Create hardlinks

Optional arguments:
  -h, --help
      Show help and exit
  --version
      Show version info and exit
  --config-file PATH, -c PATH
      Path to UW YAML config file (default: read from stdin)
  --target-dir PATH
      Root directory for relative destination paths
  --cycle CYCLE
      The cycle in ISO8601 format (e.g. yyyy-mm-ddThh)
  --leadtime LEADTIME
      The leadtime as hours[:minutes[:seconds]]
  --dry-run
      Only log info, making no changes
  --key-path KEY[.KEY...]
      Dot-separated path of keys to config block to use
  --report
      Show JSON report on [non]ready assets
  --quiet, -q
      Print no logging messages
  --verbose, -v
      Print all logging messages
  --fallback {copy,symlink}
      Alternative if hardlink fails

`link`

The link action stages items in a target directory by creating symbolic links (symlinks) to files, directories, or other links. It otherwise behaves similarly to the copy action (see above), but note the following:

The link action links directories indentified by glob patterns, whereas the copy action ignores directories.
HTTP and HPSS sources are not supported.

uw fs link --help

usage: uw fs link [-h] [--version] [--config-file PATH] [--target-dir PATH]
                  [--cycle CYCLE] [--leadtime LEADTIME] [--dry-run]
                  [--key-path KEY[.KEY...]] [--report] [--quiet] [--verbose]

Create symlinks

Optional arguments:
  -h, --help
      Show help and exit
  --version
      Show version info and exit
  --config-file PATH, -c PATH
      Path to UW YAML config file (default: read from stdin)
  --target-dir PATH
      Root directory for relative destination paths
  --cycle CYCLE
      The cycle in ISO8601 format (e.g. yyyy-mm-ddThh)
  --leadtime LEADTIME
      The leadtime as hours[:minutes[:seconds]]
  --dry-run
      Only log info, making no changes
  --key-path KEY[.KEY...]
      Dot-separated path of keys to config block to use
  --report
      Show JSON report on [non]ready assets
  --quiet, -q
      Print no logging messages
  --verbose, -v
      Print all logging messages

`makedirs`

The makedirs action creates directories. Any KEY positional arguments are used to navigate, in the order given, from the top of the config to the makedirs block, which must nest under a makedirs: key.

uw fs makedirs --help

usage: uw fs makedirs [-h] [--version] [--config-file PATH]
                      [--target-dir PATH] [--cycle CYCLE]
                      [--leadtime LEADTIME] [--dry-run]
                      [--key-path KEY[.KEY...]] [--report] [--quiet]
                      [--verbose]

Make directories

Optional arguments:
  -h, --help
      Show help and exit
  --version
      Show version info and exit
  --config-file PATH, -c PATH
      Path to UW YAML config file (default: read from stdin)
  --target-dir PATH
      Root directory for relative destination paths
  --cycle CYCLE
      The cycle in ISO8601 format (e.g. yyyy-mm-ddThh)
  --leadtime LEADTIME
      The leadtime as hours[:minutes[:seconds]]
  --dry-run
      Only log info, making no changes
  --key-path KEY[.KEY...]
      Dot-separated path of keys to config block to use
  --report
      Show JSON report on [non]ready assets
  --quiet, -q
      Print no logging messages
  --verbose, -v
      Print all logging messages

Examples

Given a config containing

config:
  makedirs:
    - foo
    - bar

rm -rf makedirs-parent
uw fs makedirs --target-dir makedirs-parent --config-file makedirs.yaml --key-path config
echo
tree -F makedirs-parent

[2025-01-02T03:04:05]     INFO Validating config against internal schema: makedirs
[2025-01-02T03:04:05]     INFO Schema validation succeeded for fs config
[2025-01-02T03:04:05]     INFO Directory makedirs-parent/foo: Executing
[2025-01-02T03:04:05]     INFO Directory makedirs-parent/foo: Ready
[2025-01-02T03:04:05]     INFO Directory makedirs-parent/bar: Executing
[2025-01-02T03:04:05]     INFO Directory makedirs-parent/bar: Ready
[2025-01-02T03:04:05]     INFO Directories: Ready

makedirs-parent/
├── bar/
└── foo/

3 directories, 0 files

The --cycle and --leadtime options can be used to make Python datetime and timedelta objects, respectively, available for use in Jinja2 expression in the config. For example:

config:
  makedirs:
    - foo/{{ yyyymmdd }}/{{ hh }}/{{ nnn }}/bar
    - baz/{{ yyyymmdd }}/{{ hh }}/{{ nnn }}/qux
yyyymmdd: "{{ cycle.strftime('%Y%m%d') }}"
hh: "{{ cycle.strftime('%H') }}"
nnn: "{{ '%03d' % (leadtime.total_seconds() // 3600) }}"
validtime: "{{ (cycle + leadtime).strftime('%Y-%m-%dT%H') }}"

rm -rf makedirs-parent-timedep
uw fs makedirs --target-dir makedirs-parent-timedep --config-file makedirs-timedep.yaml --cycle 2024-05-29T12 --leadtime 6 --key-path config
echo
tree -F makedirs-parent-timedep

[2025-01-02T03:04:05]     INFO Validating config against internal schema: makedirs
[2025-01-02T03:04:05]     INFO Schema validation succeeded for fs config
[2025-01-02T03:04:05]     INFO Directory makedirs-parent-timedep/foo/20240529/12/006/bar: Executing
[2025-01-02T03:04:05]     INFO Directory makedirs-parent-timedep/foo/20240529/12/006/bar: Ready
[2025-01-02T03:04:05]     INFO Directory makedirs-parent-timedep/baz/20240529/12/006/qux: Executing
[2025-01-02T03:04:05]     INFO Directory makedirs-parent-timedep/baz/20240529/12/006/qux: Ready
[2025-01-02T03:04:05]     INFO Directories: Ready

makedirs-parent-timedep/
├── baz/
│   └── 20240529/
│       └── 12/
│           └── 006/
│               └── qux/
└── foo/
    └── 20240529/
        └── 12/
            └── 006/
                └── bar/

11 directories, 0 files

The --target-dir option need not be specified when all directory paths are absolute, and will never be applied to absolute paths. If any paths are relative, however, it is an error not to provide a target directory:

config:
  makedirs:
    - foo
    - bar

uw fs makedirs --config-file makedirs.yaml --key-path config

[2025-01-02T03:04:05]     INFO Validating config against internal schema: makedirs
[2025-01-02T03:04:05]     INFO Schema validation succeeded for fs config
[2025-01-02T03:04:05]    ERROR Relative path 'foo' requires target directory to be specified

The --report option behaves the same as for link (see above).

fs

copy