Unified Workflow Tools
uwtools is a modern, open-source Python package that helps automate common tasks needed for many standard numerical weather prediction (NWP) workflows. It also provides drivers to automate the configuration and execution of Unified Forecast System (UFS) components, providing flexibility, interoperability, and usability to various UFS Applications.
If you’re interested in contributing, check out the Contributor Guide.
For users who want to start using the toolbox and framework, take a peek at our User Guide.
The Tools
The tools are accessible from both a command-line interface (CLI) and a Python API. The CLI automates many operations commonly needed in NWP workflows. The API supports all CLI operations and additionally provides access to in-memory objects to facilitate more novel use cases. We hope these options will let you integrate the package into your pre-existing bash and Python scripts, and give you some handy tools to use in your day-to-day work with running NWP systems.
Configuration Management
uw config -himport uwtools.api.configThe config tool suite helps you compare, transform, modify, and even validate your configuration. The package supports YAML, shell, Fortran namelist, and INI file formats. Configuration in any of these formats may use Jinja2 syntax to express values. These values can reference others, or compute new values by evaluating mathematical expressions, building paths, manipulating strings, etc.
Compare Action
When the Linux diff tool just doesn’t work for comparing unordered namelists with mixed-case keys, this is your go-to! The Fortran namelists are the real catalyst behind this gem, but it also works on the other configuration formats.
Compose Action
To compose configs is to start with a base config and to update its values, structurally, from the contents of one or more other configs of the same type (YAML, Fortran namelist, etc.). Composition supports building up complex experiment configurations by, for example, combining default values appropriate to the overall application with values needed on a particular machine, and then with specific user requirements, each stored in their own config files. Such a hierarchical approach to configuration management provides flexibility and avoids repetition of common values across files.
Realize Action
To realize a config is to render values encoded in Jinja2 expressions, potentially overriding values in one file or object with those from others, not necessarily with the same configuration format. With uwtools, you can even reference the contents of other files to build up a configuration from its pieces.
Validate Action
Provide a JSON Schema file alongside your configuration to validate that it meets the requirements set by the schema. We’ve enabled robust logging to make it easier to repair your configs when problems arise.
Templating
uw template -himport uwtools.api.templateRender Action
This gives you the full power of rendering a Jinja2 template in the same easy-to-use interface as your other workflow tools.
Translate Action
This tool helps transform legacy configuration files templated with the atparse tool (common at NOAA) into Jinja2 templates for use with the uw config realize and uw template render tools, or their API equivalents.
File/Link/Directory Provisioning
uw fs -himport uwtools.api.fsThis tool helps users define the source and destination of files to be copied or linked, or directories to be created, in the same UW YAML language used by UW drivers.
There is a video demonstration of the use of the uw fs tool (formerly uw file) available via YouTube.
Rocoto Support
uw rocoto -himport uwtools.api.rocotoThis tool is all about creating a configurable interface to the Rocoto workflow manager tool that produces the Rocoto XML for a totally arbitrary set of tasks. The uwtools package defines a structured YAML interface that relies on tasks you define to run. Paired with the uw config tool suite, this interface becomes highly configurable and requires no XML syntax!
Iterate Action
Given a Rocoto XML workflow document, invoke Rocoto in a loop, monitoring its progress, until a specified task is complete.
Realize Action
This is where you put in your structured YAML that defines your workflow of choice, and it pops out a verified Rocoto XML.
Validate Action
Do you already have a Rocoto XML but don’t want to run Rocoto to make sure it works? Use rocoto validate-xml to check to see if Rocoto will be happy.
The Drivers
Drivers for NWP components are available as top-level CLI modes and API modules.
Provided with a valid UW YAML configuration file, and CLI arguments when required, uw can prepare a fully provisioned run directory and execute a component either directly on the current system, or via a batch job submitted to an HPC scheduler.
Each driver produces a list of available TASK arguments from its CLI --help, -h flag. The provisioned_rundir will do everything except run the executable, but any of the tasks may be requested and only the steps required to produce that task will be performed.
Over time, we’ll add many other drivers to support a variety of UFS components from pre-processing to post-processing, along with many data assimilation components.
Drivers for UFS
To prepare a complete forecast, drivers would typically be run in the order shown here (along with additional drivers still in development).
cdeps
uw cdeps -himport uwtools.api.cdepsesg_grid
uw esg_grid -himport uwtools.api.esg_gridfilter_topo
uw filter_topo -himport uwtools.api.filter_topoglobal_equiv_resol
uw global_equiv_resol -himport uwtools.api.global_equiv_resolmake_hgrid
uw make_hgrid -himport uwtools.api.make_hgridmake_solo_mosaic
uw make_solo_mosaic -himport uwtools.api.make_solo_mosaicorog
uw orog -himport uwtools.api.orogorog_gsl
uw orog_gsl -himport uwtools.api.orog_gslsfc_climo_gen
uw sfc_climo_gen -himport uwtools.api.sfc_climo_genshave
uw shave -himport uwtools.api.shavechgres_cube
uw chgres_cube -himport uwtools.api.chgres_cubeFV3
uw fv3 -himport uwtools.api.fv3UPP
uw upp -himport uwtools.api.uppDrivers for Data Assimilation
GSI
uw gsi -himport uwtools.api.gsiEnKF
uw enkf -himport uwtools.api.enkfIODA
uw ioda -himport uwtools.api.iodaJEDI
uw jedi -himport uwtools.api.jediDrivers for MPAS
Drivers for working with standalone Model for Prediction Across Scales (MPAS), typically run in the order shown here:
ungrib
uw ungrib -himport uwtools.api.ungribmpas_init
uw mpas_init -himport uwtools.api.mpas_initmpas
uw mpas -himport uwtools.api.mpasmpassit
uw mpassit -himport uwtools.api.mpassitSupport Policy
The latest three minor versions (in the semantic versioning sense) of uwtools are considered actively supported, meaning that bugfix releases will be made as needed. Older versions are considered deprecated, and users may be advised to update to newer versions where bugs either have been or will be fixed. Users are encouraged to raise an Issue to report a bug or to request help, even with deprecated versions.
Disclaimer
The United States Department of Commerce (DOC) GitHub project code is provided on an “as is” basis and the user assumes responsibility for its use. DOC has relinquished control of the information and no longer has responsibility to protect the integrity, confidentiality, or availability of the information. Any claims against the Department of Commerce stemming from the use of its GitHub project will be governed by all applicable Federal law. Any reference to specific commercial products, processes, or services by service mark, trademark, manufacturer, or otherwise, does not constitute or imply their endorsement, recommendation or favoring by the Department of Commerce. The Department of Commerce seal and logo, or the seal and logo of a DOC bureau, shall not be used in any manner to imply endorsement of any commercial product or activity by DOC or the United States Government.