realistic_global

The realistic_global task group contains tasks that use realistic global ocean meshes, bathymetry and forcing. It currently contains seven kinds of tasks:

• woa23, a mesh-independent preprocessing task that builds a reusable World Ocean Atlas 2023 (WOA23) hydrography product on the native 0.25-degree latitude-longitude grid.

• jra55, a mesh-independent preprocessing task that builds a reusable wind-stress product on the native JRA55-do TL319 grid.

• init, which creates mesh-specific ocean initial conditions from that hydrography and forcing together with the culled mesh from e3sm/init.

• forward and cached_forward, short forward runs on a mesh and initial condition the init workflow above builds.

• cached_forward, the same short runs from an E3SM initial condition cached in the Polaris input database, a temporary task described in the same section.

• analysis_output, one-year Omega runs from either source of initial condition, writing what the analysis tasks read.

• restart, a short run continued from a restart, checking that it matches the uninterrupted run, history output included.

The arrows show where each kind of task gets its inputs:

        flowchart LR
    woa23 --> init
    jra55 --> init
    e3sm_init["e3sm/init"] --> init
    init --> forward
    init --> analysis_output
    database[("Polaris input database")] --> cached_forward
    database --> analysis_output
    database --> restart
    

supported models

The woa23 and jra55 tasks are model-independent and do not require either MPAS-Ocean or Omega to be built.

The init and forward tasks support both MPAS-Ocean and Omega. The analysis_output tasks support only Omega, since the analysis reads only Omega output.

The restart task supports Omega only. It is a regression test for the way Omega writes its history time axis across a restart, and MPAS-Ocean stamps its output with xtime rather than an elapsed time measured from the start of the clock, so the failure it looks for cannot arise there.

woa23

This task is the Polaris port of the legacy Compass utility/extrap_woa workflow. It combines January and annual WOA23 climatologies, uses a cached e3sm/init combined-topography product on the WOA grid to define the ocean mask used during preprocessing, and then fills missing temperature and salinity values through staged horizontal and vertical extrapolation.

The task can be set up with:

polaris setup -t ocean/spherical/realistic_global/hydrography/woa23 ...

description

The task is organized into inspectable steps:

1. combine_topo from the e3sm/init component is used to combine topography GEBCO and Bedmap3 datasets on the WOA23 0.25-degree latitude-longitude grid.

2. combine creates woa_combined.nc by combining January and annual WOA23 in-situ temperature and practical-salinity fields, then deriving conservative temperature and absolute salinity.

3. extrapolate creates the final woa23_decav_0.25_jan_extrap.nc product.

4. viz produces horizontal maps and vertical sections of the extrapolated product. This step is not run by default.

This layout is intended to match Polaris shared-step conventions so the WOA23 preprocessing pipeline can later be reused by mesh-dependent realistic_global initialization tasks.

mesh

N/A. This task operates on the native WOA23 latitude-longitude grid rather than an MPAS mesh.

vertical grid

N/A. The task preserves the standard WOA23 depth levels.

initial conditions

The source fields come from the WOA23 January and annual climatologies in the Polaris input database.

forcing

N/A.

time step and run duration

N/A.

config options

# Options related to generating a reusable WOA23 hydrography product
[woa23]

# the minimum weight sum needed to mark a new cell valid in horizontal
# extrapolation
extrap_threshold = 0.01

# target depths for horizontal plots of the extrapolated product (m)
horizontal_plot_depths = 0.0, 200.0, 400.0, 600.0, 800.0

# maximum depth to include in section plots (m)
section_max_depth = 2000.0

# endpoints of a transect through Filchner Trough and into the Filchner
# ice-shelf cavity
filchner_start_lon = -46.0
filchner_start_lat = -71.5
filchner_end_lon = -38.5
filchner_end_lat = -81.2

# endpoints of a transect through the Ross Ice Shelf cavity
ross_start_lon = 176.0
ross_start_lat = -72.0
ross_end_lon = -171.0
ross_end_lat = -84.0

The viz step is further controlled by the [woa23_viz_temperature], [woa23_viz_salinity], [woa23_viz_section_temperature] and [woa23_viz_section_salinity] sections, each of which supports the standard Polaris colormap options described in global lat/lon plots.

cores

The local combine and extrapolate steps run serially. The combine_topo step is intended to use the cached e3sm/init output because regenerating the combined topography product is substantially more expensive.

jra55

This task builds a time-invariant global wind-stress product from JRA55-do 10-m winds. Its purpose is narrow: standalone runs need a realistic, constant-in-time momentum input so that dynamic adjustment can spin down fast waves against a physically sensible circulation. It is not intended as a climate forcing product, and it does not include surface restoring, thermal or freshwater fluxes, or any time variation.

polaris setup -t ocean/spherical/realistic_global/forcing/jra55 ...

The source is JRA55-do v1.5.0 (MRI-JRA55-do-1-5-0), variables uas and vas: 10-m winds on the TL319 Gaussian grid, 3-hourly, distributed as input4MIPs on ESGF. E3SM forces ocean-only (G-case) runs with JRA55, so the adjusted state is adjusted against something close to what a coupled run will apply. The default time window is January 1958, the first month of the record and of an interannually forced G-case.

Warning

Running the stress step downloads about 3.5 GiB of raw reanalysis into the initial_condition_database. The derived product is in the Polaris cache database, so this happens only when you run this task to regenerate it, and users of the init task get the few-MB product from the Polaris server instead.

The stress is computed at every 3-hourly step and then time-averaged, rather than computing the stress of the time-mean wind. Averaging the wind first discards the gust contribution and underestimates the stress in the storm tracks, which is why the 3-hourly data is needed at all. The bulk formula is the Large and Yeager (2004, 2009) neutral 10-m drag law; the stability correction, the ocean-current-relative wind and any sea-ice drag distinction are deliberately omitted as second-order for this purpose.

description

The task contains two steps:

1. stress downloads the raw winds and writes jra55_stress.nc on the native TL319 grid. It is cached by default, but this task always runs it.

2. viz plots global maps of the stress components and magnitude, plus a zonal-mean taux curve. It runs by default in this task, but is left out when the wind-stress steps are pulled into other workflows as shared dependencies, so the plots are not regenerated for every mesh.

The zonal-mean curve is the diagnostic worth looking at. For the default January window it peaks near +0.12 N m^-2 around 50S in the Southern Ocean westerlies, with a trade-wind minimum near -0.06 N m^-2. Do not compare those against published annual-mean stress climatologies, which are considerably stronger in the Southern Ocean – the Compass NCEP 1958-2000 annual mean peaks at +0.19 N m^-2 – because the Southern Ocean westerlies are weaker in austral summer. Compare like with like, or expect a January product to look weak.

mesh

N/A. This task operates on the native JRA55-do TL319 grid rather than an MPAS mesh.

vertical grid

N/A.

initial conditions

N/A.

forcing

N/A. This task produces wind stress for the init task rather than applying any forcing itself.

time step and run duration

N/A.

config options

# Options related to generating a reusable JRA55-do wind-stress product
[jra55]

# the input4MIPs source and version of the JRA55-do dataset
source_id = MRI-JRA55-do-1-5-0
version = v20200916

# the ESGF data node and root path used to download the raw winds.  The
# retired esgf-node.llnl.gov and esgf-node.ornl.gov paths no longer work.
base_url = https://esgf.ceda.ac.uk/thredds/fileServer/esg_cmip6/input4MIPs/CMIP6/OMIP/MRI

# the year and month to average.  January 1958 is the first month of the
# JRA55-do record and of an interannually forced E3SM G-case.
year = 1958
month = 1

# air density used in the bulk formula (kg m^-3)
rho_air = 1.22

# wind speeds below this value (m s^-1) are clamped before evaluating the
# drag law, whose 2.70/U term would otherwise diverge
min_wind_speed = 0.5

# number of time slices to process at once when averaging the stress
time_chunk_size = 24

The viz step is further controlled by the [jra55_viz_taux], [jra55_viz_tauy] and [jra55_viz_tau_mag] sections, each of which supports the standard Polaris colormap options described in global lat/lon plots.

cores

Both steps run serially.

init

The init task creates a mesh-specific ocean initial condition (and, for Omega, a vertical-coordinate file) from the WOA23 hydrography and the culled mesh from e3sm/init, together with a surface forcing file from the JRA55-do wind stress. There is one task for each base and unified mesh:

polaris setup -t ocean/spherical/realistic_global/<mesh_name>/init ...

The target model, MPAS-Ocean or Omega, is set by the [ocean] model config option, which polaris setup takes from --model or detects from the build given with -p.

description

The task runs the e3sm/init steps that build and cull the mesh and remap its topography (see Topography Tasks), the steps of the woa23 and jra55 tasks, which read their products from the cache, and then these steps of its own:

1. cull_topo moves the remapped topography from the base mesh onto the culled ocean mesh.

2. woa23_map and remap_woa23 remap the WOA23 conservative temperature and absolute salinity to the culled mesh.

3. jra55_map and remap_jra55 remap the JRA55-do wind stress to the culled mesh. The step fails if the mapping leaves any cell uncovered.

4. pstar_init builds the p-star vertical coordinate together with the temperature and salinity on it, since each depends on the other through the equation of state. It also fits the bathymetry to the depths the vertical grid can represent and fills isolated bathymetry holes.

5. initial_state writes mesh.nc, init.nc and, for Omega, vert_coord.nc.

6. forcing writes forcing.nc with the wind stress on the mesh.

7. viz writes sanity-check plots and ParaView exports for the initial condition and vertical coordinate: an initial_state_summary.png figure of histograms, a vertical_coordinate.png structure figure, global maps of temperature and salinity at several depths (plus surface and seafloor) and of topography and column diagnostics, vertical transects across the major ocean basins, and xdmf/ subdirectories for ParaView. For Omega, native surface/bottom pressure maps and a TEOS-10 in-situ density (stratification) check are also produced. Global maps of the on-mesh wind stress components and magnitude are also written, which is where remapping artifacts would show up.

On large meshes, viz is the most expensive step: it took 33 minutes of a run of about 1.8 hours on u-oi6to18-lr12p5. It runs only in this task; other workflows that reuse the init steps leave it out.

mesh

Any of the base meshes (Base Mesh Tasks) or unified meshes (Unified base-mesh tasks), culled to the ocean and sea-ice domain by e3sm/init.

vertical grid

The default is the 80-layer E3SMv1 reference grid with the p-star coordinate (see p-star) and partial bottom cells. The 240 km meshes use a cheaper 16-level grid instead (see per-mesh config options).

# Options related to the vertical grid
[vertical_grid]

# the type of vertical grid
grid_type = 80layerE3SMv1

# the type of vertical coordinate
coord_type = p-star

# Whether to use "partial" or "full", or "None" to not alter the topography
partial_cell_type = partial

# the minimum fraction of a layer for partial cells
min_pc_fraction = 0.1

# Minimum number of vertical levels a valid column must contain (3 is the
# practical minimum for baroclinic dynamics).  Also sets the minimum depth
# clamp together with min_bottom_depth.
min_vert_levels = 3

# Minimum geometric water-column depth (m); cells shallower than this (or than
# the depth needed for min_vert_levels layers) are clamped to it.
min_bottom_depth = 10.0

Columns deeper than the reference grid can represent are made shallower to fit it.

initial conditions

Temperature and salinity come from the extrapolated WOA23 January climatology built by the woa23 task. They are conservative temperature and absolute salinity for Omega, and potential temperature and practical salinity for MPAS-Ocean. The ocean starts at rest, with the sea surface at zero height. Ice-shelf cavities are not supported.

forcing

forcing.nc holds constant-in-time wind stress from the jra55 task: SfcStressZonal and SfcStressMeridional for Omega, windStressZonal and windStressMeridional for MPAS-Ocean. Both models take zonal and meridional components at cell centers and project onto edges themselves. Surface restoring and the thermal and freshwater fluxes are future work. The settings that switch the forcing on in the model are described under forward.

time step and run duration

N/A. This task does not run the ocean model.

config options

[realistic_global_init]
# Approximate number of MPAS cells to target per MPI task when building
# the WOA23 and JRA55-do remapping weights with mbtempest / ESMF.  Building
# weights is a purely horizontal, geometric problem with no vertical
# structure, so these counts are much larger than the ~200 cells per core
# used to size forward runs.
remap_cells_per_task = 10000

# Rough maximum cells per task, used to compute the minimum task count
# (min_tasks).  This is a rule of thumb, not a hard limit: the task count is
# rounded to the nearest whole number, so a task may get somewhat more cells.
remap_min_cells_per_task = 50000


# Options related the ocean component
[ocean]

# Equation of state type, defaults to mpas-ocean default
eos_type = teos-10


# Options related to the Coriolis force
[coriolis]

# Coriolis type for a realistic global (Earth-rotation) mesh, giving
# f = 2 * omega * sin(latitude)
type = spherical


# Options for the realistic global init visualization step
[realistic_global_init_viz]

# Projection for the global maps, must be supported by polaris.viz
projection = Robinson

# Longitude of the center of the global maps
central_longitude = 200.

# Depths (m below the surface) at which to plot global temperature/salinity
# maps.  One vertical level is selected per depth; its actual depth may vary.
depths = 0, 100, 500, 1000, 2000, 4000

# the type of norm used in the colormaps (set per-variable at run time)
norm_type = linear

# additional arguments to provide to the colormap norm (set at run time)
norm_args = {}


# Vertical transects to plot, crossing relevant ocean basins.  Each named
# transect is a list of an even number (>= 4) of values giving alternating
# lon, lat waypoints in degrees (a straight line uses two waypoints; a zonal
# circumnavigation needs intermediate waypoints).
[realistic_global_init_viz_transects]

# comma-separated list of transects to plot (each defined as an option below)
transects = atlantic_meridional, pacific_meridional, indian_meridional,
    southern_ocean_zonal

# Atlantic meridional section near 30 W
atlantic_meridional = -30.0, -60.0, -30.0, 65.0

# Pacific meridional section near 150 W
pacific_meridional = -150.0, -60.0, -150.0, 60.0

# Indian meridional section near 80 E
indian_meridional = 80.0, -60.0, 80.0, 25.0

# Southern Ocean zonal section near 60 S (waypoints around the globe)
southern_ocean_zonal = -180.0, -60.0, -90.0, -60.0, 0.0, -60.0, 90.0, -60.0,
    180.0, -60.0

The mapping steps use the map_tool option in the [mapping] section, moab by default. The colormap and its limits for each viz plot are set at run time from the variable and the range of its data, so the [realistic_global_init_viz] colormap options are not meant to be edited.

per-mesh config options

Some config options need to differ from one mesh to the next. These live in one optional file per mesh, <mesh_name>.cfg, under polaris/tasks/ocean/realistic_global/mesh_configs. When a realistic_global task is set up, the options for its mesh are added after the task’s own config file, so they override the defaults. Most meshes have no file there at all and simply use the defaults; you can still override any option for a single mesh in your own user config file.

These are separate from the per-mesh files in polaris/mesh/spherical/unified. The two are joined by the mesh name but have different owners: the mesh component describes the mesh itself (resolutions, river networks, sizing fields, total cell count), while the ocean component describes what the ocean does on that mesh. An option specific to the ocean belongs in mesh_configs, not in the mesh component.

Currently:

• qu240km, icos240km and u-oi240-lr240 replace the default 80-layer vertical grid with a cheap 16-level tanh_dz grid with a 3000 m bottom depth. The shallow ocean is deliberate: these coarse meshes exist for fast smoke-testing of E3SM, Omega and MPAS-Ocean rather than for physically realistic simulations.

• All five unified meshes set culled_ocean_cell_count, the approximate size of the ocean + sea-ice culled mesh that the ocean model actually runs. It is used to size MPI task counts and is smaller than the full mesh, which includes land and river-channel refinement that is culled away first.

• All five unified meshes, plus qu240km and icos240km, set the resolution-dependent forward-run options described under forward: the horizontal mixing coefficients, how mixing is scaled across the mesh, which eddy parameterizations are active and, where the default scaling is not appropriate, the time step. The three 240 km meshes are qualitatively the same mesh and share the same values.

For example, u-oi240-lr240.cfg contains:

# Ocean-specific properties of this mesh
[realistic_global_mesh]

# Approximate cell count of the ocean + sea-ice culled mesh that the ocean
# model actually runs, used to size ocean MPI task counts (ntasks/min_tasks).
culled_ocean_cell_count = 7500


# Options related to the vertical grid
[vertical_grid]

# the type of vertical grid
grid_type = tanh_dz

# Number of vertical levels
vert_levels = 16

# Depth of the bottom of the ocean
bottom_depth = 3000.0

# The minimum layer thickness
min_layer_thickness = 3.0

# The maximum layer thickness
max_layer_thickness = 500.0

Options the per-mesh file does not set (coord_type, min_vert_levels, min_bottom_depth, and so on) are inherited from the task’s config file as usual.

cores

woa23_map and jra55_map run on MPI tasks sized from the mesh’s ocean-culled cell count: one task per remap_cells_per_task cells, and no fewer than one per remap_min_cells_per_task cells. Both are rounded to the nearest whole task, so they are rules of thumb rather than hard limits. The other steps of this task run serially.

restart

This task checks that a run continued from a restart reproduces the uninterrupted run, history output included. It is shaped like the ERS (exact restart) test that CIME runs on E3SM cases, so that Omega’s restarts are tested here the way they will be tested once Omega runs in E3SM.

It is also a regression test for Omega #482, where a run continued from a restart silently overwrote the history frames written before the restart instead of appending to them, leaving a file that was well formed and simply missing the earlier half of the run.

It can be set up with:

polaris setup -t ocean/spherical/realistic_global/e3sm-qu240/restart ...

description

The task contains three steps:

1. full_run covers the whole four-hour period in one go and writes a restart two hours in. Besides its history, output.nc, it writes the same history up to the restart to output_before_restart.nc: what a run that stopped at the restart would have left behind.

2. restart_run continues from that restart and covers the last two hours, appending its history frames to a copy of output_before_restart.nc.

3. validate compares the history of restart_run with the history of full_run.

As in CIME’s ERS test, the restart is written partway through the full run rather than by a separate, shorter run. Each run writes its restarts into a directory of its own under the task’s restarts directory, and restart_run reads the full run’s restart at two hours by name. Starting restart_run from a copy of the history up to the restart means it finds on disk what a continuing run would find: a history file that already holds the earlier frames and has to be appended to.

validate checks three things. The history up to the restart has to hold fewer frames than the full run’s, or the restart run would have had nothing to append to and the next check would pass without testing anything. The time axis of restart_run’s history then has to hold every frame of the whole period, in increasing order, which is what #482 broke. Finally, the state has to match the uninterrupted run, which is the usual exact-restart check.

mesh

E3SM’s quasi-uniform 240-km mesh, e3sm-qu240, downloaded from the Polaris input database, as for the cached_forward tasks.

vertical grid

The vertical grid is the one supplied with the cached initial condition.

initial conditions

A cached TEOS-10 Omega initial condition from the realistic_global section of the Polaris input database, which supplies the mesh, vertical coordinate and initial state.

forcing

As for the cached_forward tasks: zonal and meridional wind stress from the initial-condition file, applied through bulk wind stress.

time step and run duration

Both runs are split_explicit cached_forward runs on e3sm-qu240, configured by the same [realistic_global_forward] options, including the per-mesh ones: a 1-hour baroclinic and a 3-minute barotropic time step, and a biharmonic momentum viscosity of 1.0e15 m^4/s at 240 km. The full run covers four hours and the restart run the last two, with a history frame every hour, so each half of the period has two frames and the whole period has four. The period is short on purpose: the task is about how the model restarts and writes its history across a restart, not about the circulation, and it is meant to be cheap enough for the pull-request suite.

config options

The task reads the [realistic_global_forward] options described under forward and cached_forward, including the per-mesh ones for e3sm-qu240. The run durations and history interval are fixed by the task, so its own config file sets only the equation of state:

[ocean]

# Equation of state type
eos_type = teos-10

cores

The number of cores used by each forward step is determined algorithmically from the number of cells in the mesh. The validate step runs serially.

forward and cached_forward

A forward run of the ocean model selected by the [ocean] model config option (MPAS-Ocean or Omega), starting from a realistic_global initial condition. The short step is a brief run: it checks that the model runs stably on the mesh and initial condition, not that the simulation is physically interesting. Two further steps, global_stats and viz, plot the run’s statistics and its state; neither is run by default.

The two tasks are built from the same steps and run the same physics; they differ only in where the initial condition comes from.

Each comes in two variants, which differ only in the time integrator. The split_explicit variant uses the default split time stepping in both models. The rk4 variant adds rk4.cfg, which runs RK4 in both models, for occasional testing:

[realistic_global_forward]

mpaso_time_integrator = RK4
omega_time_integrator = RK4

forward runs on a mesh and initial condition that init builds, and can be set up for any mesh Polaris knows how to build:

polaris setup -t ocean/spherical/realistic_global/qu240km/forward/split_explicit ...

cached_forward downloads a ready-made initial condition instead, so it runs without the WOA23 remapping and topography culling the init chain does first. It is defined for the two meshes whose initial conditions have been staged:

mesh	cells	split time steps	RK4 time step	run duration
e3sm-qu240	7,153	1 h, 3 min	10 min	5 days
e3sm-ec30to60e2r2	236,853	30 min, 1 min	45 s	1 day

polaris setup -t ocean/spherical/realistic_global/e3sm-qu240/cached_forward/rk4 ...

Both e3sm-qu240 cached_forward variants are in the omega_pr suite, and the split_explicit one is in mpaso_pr.

physics, and comparing with Omega

forward and cached_forward run the same physics, as close to production as each model allows. For MPAS-Ocean, that is E3SM’s: Gent-McWilliams, Redi, KPP, the submesoscale parameterization and the Jacobian_from_TS pressure gradient, with the resolution-dependent mixing each mesh needs. Omega runs what it has of these, which today is KPP and the submesoscale parameterization, so the two models do not run the same physics.

To compare them, turn off the MPAS-Ocean physics Omega lacks with the options in compare_with_omega.cfg, in the forward package. Add them to the user config file passed to polaris setup -f (or pass the file itself), or set them in a step’s forward.cfg before running it, since the run reads its options again then. Polaris never applies this file on its own:

# A reference for comparing MPAS-Ocean with Omega.  Polaris never reads this
# file on its own.  Forward runs are as close to production as each model
# allows, so MPAS-Ocean runs physics that Omega does not have.  To compare the
# two models, add these options to the user config file passed to
# "polaris setup -f" (or pass this file itself), or set them in a step's
# forward.cfg before running it.
#
# The list describes what Omega lacks today and should shrink as Omega gains
# capabilities.
[realistic_global_forward]

# Omega has no Gent-McWilliams or Redi, and is not expected to gain them
use_GM = False
use_Redi = False

# the ordinary counterpart to Omega's Centered pressure gradient, rather than
# Jacobian_from_TS
pressure_gradient_type = pressure_and_zmid

Both cached meshes scale their coefficients with cell width like every other mesh, but use more momentum viscosity than E3SM to reduce grid-scale noise in Omega’s velocity: on e3sm-qu240, a biharmonic viscosity of 1.0e15 m^4/s at 240 km rather than E3SM’s 2.0e14, and on e3sm-ec30to60e2r2, E3SM’s biharmonic viscosity rescaled to the mesh’s median cell width with a harmonic one of 1.0e4 m^2/s rather than E3SM’s 1.0e3. With E3SM’s values, one-year Omega runs checkerboard the divergence along coasts and, on e3sm-ec30to60e2r2, the Gulf Stream. These values are provisional until the fix to Omega’s pressure-gradient errors over bathymetry.

Note

cached_forward is temporary. Its initial conditions are E3SM’s, for E3SM’s QU240 and EC30to60E2r2 meshes (e3sm-qu240 and e3sm-ec30to60e2r2 here), converted for Omega. They let Omega run realistic global configurations, and be compared with MPAS-Ocean, before Polaris could build its own initial conditions. Now that it can, these tasks will be removed.

surface forcing

The run is forced by time-invariant JRA55-do wind stress — written by the init workflow’s forcing step for forward, and carried inside the cached initial-condition file for cached_forward. Wind stress is the only forcing: there is no surface restoring and there are no thermal or freshwater fluxes, so the tracers spin down from the initial condition while the momentum input holds up a circulation.

forcing.yaml turns it on with config_use_bulk_wind_stress, which mpaso_to_omega.yaml translates to Omega’s SfcStressForcingTendencyEnable. Both models default it to off, and it is set for every realistic global forward run: there is no config option to run without wind forcing.

forcing.yaml also sets config_bulk_wind_stress_interp_isotropic for MPAS-Ocean, on every forward run. Both models take the stress at cell centres and project it onto edges themselves, but not the same way by default: MPAS-Ocean uses the two cells adjacent to an edge, and Omega uses four. This is set because four is what E3SM runs with today, so it is the E3SM-like choice for the unified meshes; it also happens to be Omega’s only option, so the two models end up applying the same momentum without anything further being said. The isotropic form is slightly less accurate and Omega may not keep it, so this is worth revisiting when that is decided.

Where the wind stress comes from is a separate question, answered by the forward step’s initial condition. When it stages a forcing file of its own — as the init workflow does — forcing_streams.yaml points each model’s input stream at the file named by the [ocean_staged_files] forcing_filename config option (forcing.nc by default). A cached initial condition carries the stress inside the initial-condition file, so the same streams are pointed there instead. For MPAS-Ocean that is a forcing stream holding just the two wind-stress variables, rather than the Registry’s forcing_data stream, which would also demand the sea-ice and atmospheric pressures and the tracer restoring fields.

output

The run writes output.nc at output_interval and a restart at restart_interval.

output.nc holds more than the prognostic state. For MPAS-Ocean it is the tracers, velocity, layer thickness, kinetic energy and relative vorticity; for Omega it is the State, Tracers, AuxiliaryState, SshCell and Eos groups, which add the vorticity, divergence and del2 terms, the free surface and the specific volume and buoyancy frequency. The extra fields cost file size and are worth it: without them a run that goes wrong can be seen to have gone wrong but not diagnosed, and these runs exist partly to be compared against each other.

It also writes a time series of the global minimum, maximum, mean and RMS of the state variables, plus the global CFL number for MPAS-Ocean. This is the cheapest way to see a run going wrong before it NaNs, which matters most for the longer spin-ups that build on this task. MPAS-Ocean produces it through the globalStats analysis member, in global_stats.nc, and Omega through its GlobalStats analysis group, in global_stats_1DayInstants at the default stats_interval — Omega treats the configured file name as a prefix and appends the period and the kind of output, with no .nc extension.

The two files hold much the same quantities under different variable names, which Polaris builds from the field and the statistic rather than mapping one by one. One difference is not just a name: MPAS-Ocean’s rms* is a root mean square while Omega’s SpatialStdDev is a standard deviation, so the global_stats step converts the former before plotting it. Which fields it plots is set by [realistic_global_forward_stats] fields; left empty, as it is by default, the step plots every field the run wrote statistics for.

Omega samples the statistics instantaneously rather than averaging them over the period, because those are the names the readers build: Omega names a time-averaged quantity <name>_TimeMean<period> and an instantaneous one plainly <name>. Averaging would also constrain restart_interval, since Omega aborts unless the restart interval is a whole multiple of an averaging period; instantaneous sampling carries no such constraint. MPAS-Ocean writes an instantaneous sample too, so the two are comparable.

MPAS-Ocean additionally writes the temperature-threshold mixed-layer depth at output_interval, through the mixedLayerDepths analysis member, in mixed_layer_depths.nc. It is the standard first look at whether the surface boundary layer is behaving. Omega has no equivalent analysis member, so this file is written only for MPAS-Ocean.

physics options

The horizontal mixing coefficients, the eddy parameterizations and the way mixing is scaled across a variable-resolution mesh all depend on the mesh, so they are config options rather than fixed values, and most meshes override them in their per-mesh config file. The defaults follow how these meshes are configured in E3SM; see physics, and comparing with Omega for turning off what Omega lacks to compare the two models.

A coefficient and its on/off switch are a single option: leaving mom_del2 blank turns harmonic momentum mixing off, and giving it a value turns it on with that coefficient. This matters more than it might seem — the model defaults leave horizontal mixing off entirely, and a global run without it grows grid-scale noise until it produces NaNs.

hmix_scaling chooses how the coefficients are scaled across the mesh. The default, ref_cell_width, makes them apply at hmix_ref_cell_width and scale from there — as cellWidth for the del2 coefficients and as cellWidth**3 for the del4 ones — in both models. none uses them as given.

A blank hmix_ref_cell_width, the default, is the mesh’s median cell width, computed from the cell areas as Omega computes it. That is Omega’s own reference, and it makes the coefficients mean much the same thing on two meshes of the same nominal resolution. The median comes from the mesh when the forward step runs, and the step’s log reports it with the coefficients.

A mesh whose coefficients come from E3SM states its finest resolution instead, because E3SM’s per-grid mom_del2 and mom_del4 values are referenced to that: the u-oi6to18 meshes pair RRSwISC6to18E3r5’s 3.2e09 with 6 km, u-oi-so12to30-lr10 pairs SOwISC12to30E3r3’s 1.18e10 with 12 km, and the three 240 km meshes take oQU240’s 2.0e14 with 240 km. Omega has no reference width of its own to set, so when one is stated, Polaris converts the coefficients it gives Omega — del2 by the ratio of the median to the reference, del4 by its cube — and both models run the same mixing at every cell width.

Note

MPAS-Ocean spells this as two nested flags, and ref_cell_width sets both: config_hmix_use_ref_cell_width is read only inside if (config_hmix_scaleWithMesh). Setting the first without the second reads as a request for width-based scaling and silently gets none at all.

MPAS-Ocean’s third combination — scaleWithMesh with use_ref_cell_width false — scales by the legacy meshDensity field instead. Polaris does not offer it, and it should not be used: meshDensity is meaningless on E3SM v4 meshes, which write it as uniformly 1.0, so that branch applies no scaling either.

mom_del4_div_factor scales the divergence part of the biharmonic momentum operator alone, leaving its rotational part at mom_del4. Blank leaves the model default of 1.0, the true biharmonic. Raising it damps the divergent grid-scale mode — the one carrying C-grid checkerboard and gravity-wave noise — without blurring the resolved eddy field, which is why E3SM uses 10 on its eddying RRSwISC6to18E3r5 mesh and nowhere else. It reaches Omega as DivFactor.

Some settings are not config options because they should not vary: they are pinned in forward.yaml purely so that MPAS-Ocean and Omega agree: horizontal tracer advection order 3, implicit constant bottom drag with a coefficient of 1e-3, the CVMix convection, shear-mixing and KPP parameters, and the submesoscale parameters. Where the two models’ own defaults differ, the MPAS-Ocean default wins, so KPP’s Langmuir turbulence, on by default in Omega, is off. The submesoscale parameters are the exception: they are E3SM’s, which Omega’s defaults match, so the efficiency Ce is 0.08 rather than the MPAS-Ocean Registry default of 0.06. The rest already match both models, and stating them is what keeps them from drifting apart.

Gent-McWilliams (use_GM), Redi (use_Redi), the Leith closure (use_Leith_del2), frazil ice (use_frazil_ice_formation) and the pressure-gradient formulation (pressure_gradient_type) are applied only for MPAS-Ocean. Polaris has no Omega counterpart for any of them, and Omega is not expected to gain GM or Redi. The KPP boundary layer (use_KPP) and the submesoscale eddy parameterization (use_submesoscale) apply to both models.

They are config options rather than fixed values so that a run comparing MPAS-Ocean with Omega can turn off what Omega lacks; see physics, and comparing with Omega. The defaults are E3SM’s. Several of them are off by default in MPAS-Ocean itself, so leaving them alone would not have given E3SM’s configuration either.

Blank pressure_gradient_type leaves the MPAS-Ocean Registry default of pressure_and_zmid, the ordinary counterpart to Omega’s Centered; Jacobian_from_TS is how E3SM runs and has no Omega equivalent.

config_Redi_min_layers_diag_terms is pinned to 0 in forward.yaml rather than being an option — the Registry default of 6 skips the Redi diagnostic terms in the top six layers, and Compass computes them everywhere. It has an effect only where Redi is on.

time integrator

The time integrator is the one setting chosen per model rather than shared: mpaso_time_integrator defaults to split_explicit_ab2 and omega_time_integrator to split_explicit (translated to SplitExplicitRK2). Split time stepping is what makes the month-long spin-ups that build on this workflow affordable. Omega also supports RK4 (translated to RungeKutta4), but not split_explicit_ab2. The rk4 variant of each task sets both options to RK4. Both options use neutral (MPAS-Ocean) naming, and an omega_time_integrator that Omega does not support is an error at run time rather than at setup, so the option can still be changed after setting a task up.

time step and run duration

The time step is derived from the mesh’s minimum resolution: dt_per_km gives the baroclinic step and btr_dt_per_km the barotropic subcycling step, each in seconds per kilometer. Setting dt or btr_dt overrides the derived value. Meshes whose stable time step does not follow the default scaling set dt_per_km and btr_dt_per_km in their per-mesh config file.

The time step follows from the integrator. A split integrator advances on the long baroclinic step and subcycles the barotropic mode on the short one; a non-split integrator such as RK4 has no subcycling and so must advance on the short barotropic step, btr_dt_per_km times the minimum resolution. Setting rk4_dt overrides that. It is separate from dt so that a mesh config can give a time step for each variant, as the cached meshes’ configs do.

config options

# Options related to the ocean component
[ocean]

# The equation of state.  The whole realistic_global family is TEOS-10, and it
# is stated here as well as in the init package's init.cfg so that a forward
# task is self-contained: a run from a cached initial condition never sets the
# init workflow's config up at all, and still has to name the equation of
# state to find the right file in the database.
eos_type = teos-10


# Options for realistic global ocean forward runs
[realistic_global_forward]

# The time integrator, chosen separately for each ocean model because the two
# models do not support the same integrators.  Both are given in neutral
# (MPAS-Ocean) naming; the Omega one is translated to the Omega name.
#
# Both models default to split time stepping, which is far cheaper than RK4
# for the long spin-ups this workflow feeds into: 'split_explicit_ab2' for
# MPAS-Ocean and 'split_explicit' (translated to 'SplitExplicitRK2') for Omega.
# Omega also supports 'RK4' (translated to 'RungeKutta4'), and anything else
# raises an error at run time.  The rk4 variant of each forward task sets both
# to 'RK4'.
#
# The time step follows from this choice (see dt_per_km, btr_dt_per_km and
# rk4_dt).
mpaso_time_integrator = split_explicit_ab2
omega_time_integrator = split_explicit

# Run duration as an MPAS-style duration string (DDDD_HH:MM:SS)
run_duration = 0001_00:00:00

# Interval between writes to the output stream (DDDD_HH:MM:SS).  MPAS-Ocean
# also writes on startup; Omega cannot, so an Omega run shorter than this
# interval writes its final state instead.
output_interval = 0001_00:00:00

# Interval between writes to the restart stream (DDDD_HH:MM:SS); leave blank to
# default to run_duration (a single restart at the end of the run)
restart_interval =

# Interval between writes of the global statistics (DDDD_HH:MM:SS).  These are
# a handful of scalars, so they are written far more often than the 3-D output
# and are what makes an excursion within a run visible.  Drives MPAS-Ocean's
# globalStatsOutput stream and Omega's GlobalStats snapshot period alike; for
# Omega it must be a whole number of seconds.
stats_interval = 0001_00:00:00

# Baroclinic time step per km of the mesh minimum resolution (s/km).  Only used
# for split time stepping (split_explicit_ab2, or split_explicit for Omega).
dt_per_km = 30.0

# Barotropic time step per km of the mesh minimum resolution (s/km).  Used as
# config_btr_dt for split time stepping, and as the single time step for
# non-split integrators such as RK4 and Omega's RungeKutta4.
btr_dt_per_km = 1.5

# Explicit baroclinic/barotropic time steps for split time stepping
# (DDDD_HH:MM:SS) that override the *_per_km values when set; leave blank to
# derive from *_per_km and the mesh minimum resolution
dt =
btr_dt =

# Explicit time step for non-split integrators such as RK4 (DDDD_HH:MM:SS)
# that overrides btr_dt_per_km when set.  It is separate from dt so that a mesh
# config can give a time step for each kind of integrator.
rk4_dt =

# Rayleigh damping coefficient (1/s); leave blank for no Rayleigh damping.
# Applied in every layer by both models: through MPAS-Ocean's
# constant_and_rayleigh bottom drag and Omega's RayleighDamping.  For Omega, a
# non-blank value needs a build with RayleighDamping.
damping =

# Horizontal mixing coefficients.  A blank value turns the corresponding term
# off; a number turns it on with that coefficient.  These are usually set per
# mesh, since the right values depend on the resolution.
#   mom_del2, mom_del4: harmonic and biharmonic momentum viscosity (m^2/s,
#     m^4/s)
#   tracer_del2, tracer_del4: harmonic and biharmonic tracer diffusivity
#     (m^2/s, m^4/s)
mom_del2 = 1.0e3
mom_del4 = 1.2e11
tracer_del2 =
tracer_del4 =

# A factor applied to the divergence part of the biharmonic momentum operator
# alone, leaving its rotational part at mom_del4.  Blank leaves the model
# default of 1.0, which is the true biharmonic.  Raising it damps the divergent
# grid-scale mode -- the one that carries C-grid checkerboard and gravity-wave
# noise -- without blurring the resolved eddy field, which is why E3SM uses 10
# on its eddying RRSwISC6to18E3r5 mesh and nowhere else.
mom_del4_div_factor =

# Whether to use the Leith closure for harmonic momentum mixing
use_Leith_del2 = False

# How horizontal mixing coefficients are scaled across the mesh.  One of:
#   none            - the coefficients above are used as given
#   ref_cell_width  - the coefficients above apply at hmix_ref_cell_width, and
#                     scale as cellWidth for del2 and cellWidth**3 for del4
# ref_cell_width is the default, as it is in E3SM; on a mesh whose cells are
# all near the reference width the two are equivalent.  Omega scales from the
# mesh's median cell width rather than from a reference width, so for Omega the
# coefficients are converted at run time to give the same mixing as MPAS-Ocean.
#
# MPAS-Ocean spells this as two nested flags, and ref_cell_width sets *both*:
# config_hmix_use_ref_cell_width is read only when config_hmix_scaleWithMesh is
# true.  Setting only the former reads as a request for width-based scaling and
# silently gets none.
#
# MPAS-Ocean's third combination -- scaleWithMesh with use_ref_cell_width false
# -- scales by the legacy meshDensity field instead, and is deliberately not
# offered here.  E3SM v4 meshes, including every mesh Polaris builds, write
# meshDensity as uniformly 1.0, so that path silently applies no scaling at
# all.  Do not use it, and do not add a Polaris option for it.
hmix_scaling = ref_cell_width

# The reference cell width (m), used only when hmix_scaling = ref_cell_width:
# the width at which the coefficients above apply.  Blank uses the mesh's
# median cell width, computed from the cell areas as Omega does, so that the
# coefficients mean the same thing on any two meshes of the same nominal
# resolution.  A mesh whose coefficients come from E3SM sets it to its finest
# resolution, since that is what E3SM's per-grid values are referenced to.
hmix_ref_cell_width =

# Whether to use the Gent-McWilliams eddy transport parameterization, and the
# closure and constant kappa (m^2/s) it uses.  Leave the closure and kappa blank
# to use the model defaults.  MPAS-Ocean only; Omega has no GM.
use_GM = True
GM_closure =
GM_constant_kappa =

# Whether to use Redi isopycnal mixing.  MPAS-Ocean only; Omega has no Redi.
use_Redi = True

# Whether to use the CVMix KPP boundary-layer scheme, in both models.  Omega
# turns it on by default, so it is stated for Omega too.
use_KPP = True

# Whether to use the submesoscale eddy parameterization, in both models.  On
# as in E3SM and in the Compass global_ocean tests; both models' defaults are
# off.  Its parameters are E3SM's, fixed in forward.yaml.
use_submesoscale = True

# The MPAS-Ocean horizontal pressure gradient formulation.  Leave blank for the
# Registry default of pressure_and_zmid, which is the ordinary counterpart to
# Omega's Centered.  Jacobian_from_TS is how E3SM runs and has no Omega
# equivalent.
pressure_gradient_type = Jacobian_from_TS

# Whether to form frazil ice.  MPAS-Ocean only: Omega's frazil is off by
# default, and Polaris does not turn it on.  Off, unlike in E3SM, because these
# standalone runs have no sea ice or ice-shelf cavities for the frazil to go
# to.
use_frazil_ice_formation = False

# Whether to write density to the 3-D output stream.  MPAS-Ocean only; Omega
# writes specific volume through its Eos group instead.  On by default, since
# it is part of what makes the two models comparable on the same fields and a
# short run can afford it.  It is the only field in that stream with a switch
# of its own, because it is 3-D and is roughly 13% of the output volume, which
# starts to matter once a workflow writes the stream many times over.
output_density = True

# Simulation start time (config_start_time)
start_time = 0001-01-01_00:00:00


# options for the plots of the global statistics a forward run wrote
[realistic_global_forward_stats]

# The fields to plot global statistics for, using MPAS-Ocean (Polaris
# standard) names.  Leave this empty, as it is by default, to plot every
# field the run wrote statistics for, which is what the models' own default
# lists give.  A field the run did not write is skipped with a message, not
# an error.
fields =

mesh

forward runs on a mesh built by init, so the mesh is whichever one the task is registered for. cached_forward runs on the standard E3SM mesh that comes with its cached initial condition: the quasi-uniform 240-km e3sm-qu240, or the eddy-closure e3sm-ec30to60e2r2, which varies from 30 km to 60 km.

vertical grid

For forward, the vertical grid is the one the init workflow built, set by the [vertical_grid] section of the mesh’s per-mesh config file. For cached_forward it arrives with the cached initial condition, so those meshes have no [vertical_grid] section at all.

initial conditions

For forward, the mesh, vertical coordinate, initial state and wind-stress forcing are the outputs of the init workflow’s steps, which the task runs first.

For cached_forward, a single model-specific file from the realistic_global section of the Polaris input database supplies all of them. MPAS-Ocean reads a zerovel (zero initial velocity) file along with a prebuilt graph partition file; Omega reads a TEOS-10 file converted from it, and partitions internally.

cores

The number of cores used by the short step is determined algorithmically from the number of cells in the mesh — read from the mesh file once it exists, and estimated from the mesh definition before that. The global_stats and viz steps run serially.

analysis_output

The 1_year task runs Omega for a year with its MonthlyAverages and MOC analysis groups on, writing what the analysis tasks read. It is meant to become a cached step for regression testing the omega_analysis suite. It is defined for every mesh that has a forward or cached_forward task, and runs from the same initial condition: on the two cached meshes, the one in the Polaris input database, and on the others, the one the init steps build, which are part of the task. For example:

polaris setup -t ocean/spherical/realistic_global/e3sm-qu240/analysis_output/1_year ...
polaris setup -t ocean/spherical/realistic_global/u-oi30-lr10/analysis_output/1_year ...

The tasks are in no suite. The e3sm-qu240 year takes about 8 minutes on a Chrysalis node, and the e3sm-ec30to60e2r2 year about 50 minutes on 6 nodes, writing about 40 GB.

description

The task contains two steps:

1. forward is a split_explicit forward or cached_forward run (see forward and cached_forward) a year long, with monthly output added.

2. global_stats plots time series of the daily global statistics, as in those tasks. This step is not run by default.

On meshes without a cached initial condition, the shared init steps come first, as in the forward tasks.

The run writes, in the forward step’s work directory:

• monthly_means_1MonthTimeStats.<year>-<month>.nc, the monthly means from the MonthlyAverages group of conservative temperature, absolute salinity, pseudo-thickness, specific volume, the layer-midpoint and interface elevations, sea surface height, normal velocity and kinetic energy;

• moc_1MonthTimeStats.<year>-<month>.nc, the monthly-mean global overturning streamfunction from the MOC group, in moc_num_bins latitude bins: Omega’s default of 180, or 60 on the 240-km meshes;

• global_stats_1DayInstants, daily global statistics of the state, kinetic energy and sea surface height;

• output/ocn.hist.<year>-<month>.nc, a monthly snapshot of the same fields as the monthly means, taken at the instant each mean is finalized, as a check on the means.

Omega names a time-mean file for the instant it was finalized, the start of the following month, so the twelve files covering year 1 are named 0001-02 through 0002-01. The analysis accounts for this; see analysis. Every monthly file is declared as an output of the forward step, so a month that was not written fails the step.

To analyze the run, point the omega_analysis suite at the omega.yml in the forward step’s work directory:

[ocean_analysis]

omega_config_filename = <work_dir>/ocean/spherical/realistic_global/<mesh>/analysis_output/1_year/forward/omega.yml

[ocean_analysis_climatology]

start_year = 1
end_year = 1

[ocean_analysis_time_series]

start_year = 1
end_year = 1

mesh, vertical grid, initial conditions and forcing

As for the forward or cached_forward task on the same mesh; see forward and cached_forward.

time step and run duration

The run uses the mesh’s split time steps and momentum viscosities, given in forward and cached_forward. It lasts 365 days from 0001-01-01, one year on the No Leap calendar. The history stream and restarts are written monthly, because Omega requires the restart interval to be a multiple of every averaging period. These are runs for testing the analysis, not scientifically valid simulations.

config options

The task reads the [realistic_global_forward] options described under forward and cached_forward, including the per-mesh ones. Its own defaults come before the per-mesh options:

# Default options for the one-year analysis_output runs, added before the
# per-mesh options so that a mesh can override them
[analysis_output]

# The number of latitude bins of the overturning streamfunction: Omega's
# default of one degree.  Meshes coarser than about a degree set fewer.
moc_num_bins = 180

and it overrides the run duration after them:

# Options for the one-year analysis_output runs, added after the per-mesh
# options so that they take precedence
[realistic_global_forward]

# One year on the No Leap calendar.  Omega's TimeInterval parser understands
# only DDDD_HH:MM:SS, so the duration is given in days.  The history and
# restart streams are monthly, which is set in monthly_output.yaml rather than
# here, since months are not all the same length.
run_duration = 0365_00:00:00

The 240-km meshes’ config files set fewer MOC latitude bins, here for e3sm-qu240:

# Options for the one-year analysis_output run on this mesh
[analysis_output]

# The number of latitude bins of the overturning streamfunction: three
# degrees, since Omega's default of one degree is finer than the mesh
moc_num_bins = 60

cores

The number of cores used by the forward step is determined algorithmically from the number of cells in the mesh. The global_stats step runs serially.

dynamic_adjustment

The dynamic_adjustment task runs a sequence of short forward stages that dissipate the fast waves introduced by the initial condition, producing a relaxed restart suitable for a longer production run. One realistic_global_dynamic_adjustment task is registered per MPAS mesh; the target model (MPAS-Ocean or Omega) is set by the [ocean] model config option.

The task can be set up with:

polaris setup -t ocean/spherical/realistic_global/<mesh>/dynamic_adjustment ...

description

Each stage is a forward run that restarts from the previous stage, with its own run duration, time step, output/restart cadence, and (typically decreasing) Rayleigh damping. The stages are read from a schedule YAML: a mesh-specific <mesh_name>.yaml when the mesh has one, and otherwise default.yaml. Polaris ships schedules for all five unified meshes — u-oi240-lr240, u-oi30-lr10, u-oi6to18-lr12p5, u-oi6to18-lr25 and u-oi-so12to30-lr10 (the last four ported from Compass’ ec30to60, rrs6to18 (twice) and so12to30). default.yaml is what the base meshes such as icos240km and qu240km fall back to; it is only two stages, which is a quick check that a mesh runs rather than an adjustment worth keeping. The final stage, simulation, writes the relaxed restart, and its output.nc is compared against a baseline when one is provided.

The ported schedules, and default.yaml (Compass’ qu240), keep Compass’ stage durations, time steps and damping unchanged; they have not been retuned for Polaris. They were found by trial and error: where a stage crashed, its time step was shortened or its damping increased. Omega runs the same schedules, which have worked for it as they are, without a separate tuning. u-oi240-lr240 has no Compass counterpart; it uses the time steps of Compass’ qu240 schedule, which are also what dt_per_km and btr_dt_per_km give at 240 km, with the three-stage damping ramp of ec30to60.

The number and settings of the stages come from the schedule, so a user can retune them by editing the checked-in YAML or by pointing the schedule config option at an alternate file before setup. Because the Polaris step graph is fixed at setup, changing the schedule requires re-running polaris setup.

schedule format

A stage is a forward run, so it starts from the [realistic_global_forward] config options for the mesh — including the per-mesh overrides — and the schedule supplies only what varies from stage to stage. The horizontal mixing coefficients, the eddy parameterizations and the time integrators are therefore the same as they would be for a forward run on that mesh, and are not repeated in the schedule.

A schedule has a shared block of per-stage defaults and an ordered stages block:

dynamic_adjustment:
  shared:
    output_interval: 10_00:00:00
    restart_interval: 10_00:00:00
    start_time: 0001-01-01_00:00:00
  stages:
    damped_adjustment_1:
      run_duration: 10_00:00:00
      dt: 00:15:00
      btr_dt: 00:00:30
      damping: 1.0e-4
    simulation:
      run_duration: 10_00:00:00
      dt: 00:30:00
      btr_dt: 00:01:00
      # damping omitted => the config default, which is no Rayleigh damping

Every key in a schedule must be a setting that ForwardStage recognizes, which is to say a [realistic_global_forward] config option, and it overrides that option for that stage. An unknown or misspelled key is an error at setup rather than being ignored in favor of the config value, so a schedule cannot silently stop taking effect when an option is renamed. Each stage must specify run_duration, because it determines when each stage starts and stops. Other settings may be omitted and then come from the shared block or, failing that, from config. The exception is restart_interval, which defaults to the stage’s own run_duration, because each stage has to write the restart the next one starts from. Time steps may instead be given per km of mesh minimum resolution with dt_per_km / btr_dt_per_km.

Two intervals are checked at setup, because a bad value would otherwise only show up after the model has run. A restart_interval must put the stage’s stop time on MPAS-Ocean’s restart alarm, which is measured from a fixed reference of 0001-01-01_00:00:00 and not from where the stage starts — otherwise the restart the next stage reads is never written. And a stats_interval must be no longer than the stage, or the statistics file holds only the record written when the stage starts. output_interval is not checked: an interval longer than the stage is a legitimate way to say “no 3-D output during the damped stages”, which is what the ported Compass schedules do. Such a stage still writes one record: its initial state for MPAS-Ocean, and its final state for Omega, which cannot write on startup.

start_time belongs in the shared block: it is where the chain begins, and each stage’s own start time follows from the durations before it. The remaining restart settings (do_restart, restart_in, restart_out) are set by Polaris to link the stages together and cannot be set by a schedule.

Rayleigh damping runs the other way round from the other options: it is off in config (damping is blank) and turned on per stage by the schedule, so the simulation stage gets an undamped run simply by omitting damping.

The boundary layer, the submesoscale parameterization and the pressure gradient are stage-settable on the same terms as everything else, which is worth saying because the shipped schedules never touch them: use_KPP, use_submesoscale and pressure_gradient_type are [realistic_global_forward] options, so a schedule may vary them from stage to stage. All three default to what E3SM runs, and a stage that says nothing keeps that. Varying them is a real thing to want — use_KPP: false for the earliest, most heavily damped stages, say, where the boundary-layer scheme is mixing a surface layer that the damping is still holding down — but no shipped schedule does, so treat it as untried rather than merely unused.

Note

use_KPP and use_submesoscale apply to both models. pressure_gradient_type has no Omega counterpart, so a schedule that sets it is describing an MPAS-Ocean run. It does not raise for Omega; it is simply not translated, and an Omega run proceeds without it.

Note

damping applies to both models: MPAS-Ocean’s constant_and_rayleigh bottom drag and Omega’s RayleighDamping both damp the velocity in every layer.

restart chaining

The stages share a restarts directory beside them in the task work directory. Each stage writes restarts/rst.<timestamp>.nc at the end of its run and declares it as an output; the next stage declares the same file as an input and starts from it, so a break in the chain is caught before the model launches.

For MPAS-Ocean the restart stream is both an input and an output stream, so one filename_template serves both directions: a restarting stage sets config_do_restart and config_start_time to its predecessor’s timestamp and the template resolves to that file. There is no restart-pointer file.

Omega reads a restart through a separate RestartRead stream, which restart_streams.yaml supplies, and finds it through a pointer file. Each stage writes its restart and pointer into a directory under restarts named for its restart time, and a restarting stage reads its predecessor’s. The restart files are declared as step inputs and outputs for both models.

diagnostics and validation

A final summary step summarizes the sequence, and a viz step plots the underlying time series.

The summary is dynamic_adjustment_stats.csv, one row per stage, and the same table is written to the step’s log. It is the quickest way to see whether the damping ramp and the stage durations were chosen well: kinetic energy should be falling and flattening from stage to stage, the volume-weighted mean tracers should not be moving at all, and nothing should be approaching a blow-up. Columns ending _in_stage are the extreme reached at any point during the stage; the rest are end-of-stage values.

The two mean_*_change_per_day columns are a conservation check rather than a sign of settling. These runs are forced by wind alone, with no surface fluxes, so the volume-weighted mean temperature and salinity are conserved and both columns should be zero to within round-off — u-oi30-lr10 reaches -4e-17 °C/day and u-oi6to18-lr12p5 -5e-14 °C/day. A value larger than that means tracer was lost or gained, which says nothing about whether the fast waves have been damped out.

The numbers come from each stage’s global-statistics file wherever the model reports them. The model computes these over the whole domain every stats_interval, so an extreme in the middle of a stage shows up, which it would not in a field written only at the end of the stage. stats_interval is separate from output_interval, which controls the 3-D output: the statistics are a handful of numbers, so they can be written far more often than the 3-D fields without making the output much larger. Quantities the model does not report in the statistics file are computed from output.nc instead, but only for maxima and minima, which are the same however they are computed. A volume-weighted mean is not, so kinetic_energy_mean and kinetic_energy_total are left blank rather than replaced with an unweighted mean.

Omega’s GlobalStats covers temperature, salinity, layer thickness, normal velocity and kinetic energy but has no CFL number or volume-weighted sums, so kinetic_energy_total, ocean_volume and cfl_max_in_stage are blank for an Omega run. A blank means “this model does not report it”, not that something went wrong; the log says which source each metric came from.

In place of the CFL number, an Omega run fills cfl_estimate_max_in_stage with a conservative estimate: the larger magnitude of the minimum and maximum normal velocity, times the stage’s baroclinic time step, divided by the smallest cell spacing on the mesh. The fastest flow and the smallest cell need not be in the same place, so this is an upper bound rather than Omega’s CFL number, and it is reported for inspection, not checked against cfl_max.

There are four checks, all per stage, run in a <stage>_check step immediately after that stage. The next stage does not start until the check has passed, so a stage that is already out of bounds stops the sequence instead of using up the rest of the job:

• the maximum temperature in the stage must stay below temperature_max;

• the minimum temperature must stay above temperature_min, which catches water colder than the initial condition that only the numerics could have produced, such as a tracer overshoot in a thin bottom cell that does not blow up;

• the maximum salinity must stay below salinity_max;

• the maximum CFL number must stay below cfl_max, which is what catches a stage whose time step is too long for the flow it produced.

All four report when the extreme occurred as well as how large it was, which is usually most of the diagnosis. A NaN or infinite value fails any of them.

They deliberately ignore the sample written before a stage’s first time step. For the first stage that sample is the initial condition, which dynamic adjustment did not produce; for every later stage it is the previous stage’s final state, which that stage’s own check already covered. So nothing goes unexamined, and each stage is judged on what it did rather than on what it was handed. The viz figure still plots the full series, so an inherited extreme remains visible there.

The two tracer checks ignore a further startup_exclusion_duration at the beginning of the sequence, over which the initial condition’s artifacts are still mixing away. That window is measured from the start of the sequence, not of each stage, so it covers only the stages it actually reaches — a window measured from each stage would instead stop looking just after every damping change, which is where a schedule that steps the damping down too fast would show itself. For the same reason the CFL check does not get the window at all: the opening hours are exactly where a time step that is too long shows up.

This matters in practice. NOAA’s January WOA23 analysis contains one bad grid cell off Sumatra, reaching 36.9 °C at 70 m. It is a known source-data problem that cannot be filtered without discarding real hydrography, and how much of it survives into the initial condition depends on the mesh: a single cell at 33.46 °C on u-oi30-lr10, but 35.94 °C on u-oi6to18-lr12p5 and 36.11 °C on u-oi6to18-lr25, which average the bad source cell far less. It also erodes far more slowly there — about 34.3 °C after six hours, and by extrapolation more than a day to fall below 33 — so it is still present when the second and third stages begin. A second source artifact in the Red Sea does the same for salinity, and erodes more slowly still: it is a ~25-cell blob at 1150 m, where there is almost no flow, and it settles near 42.1 PSU for the rest of the adjustment.

The two tracer thresholds are therefore blow-up detectors rather than plausibility bounds, and they are set above the artifacts rather than above the physics. A real blow-up produces hundreds of degrees or NaN, not a fraction of a degree over, so there is ample room between the two: the warmest genuine water these runs reach is about 32 °C, and nothing in the u-oi6to18 initial conditions is above 42 PSU outside the Red Sea artifact.

The summary step applies no checks. It collects the per-stage diagnostics, writes dynamic_adjustment_stats.csv and logs the table.

Note

There used to be a fourth check there, on whether the mean kinetic energy was settling stage over stage. It was rewritten three times — comparing kinetic energy levels, then the fractional change from stage to stage, then that change per unit time — and every revision was forced by a healthy run failing it.

The reason is that the quantity is not comparable across stages. A schedule’s stages differ in length (0.25 to 18 days on u-oi6to18-lr12p5), so a longer stage changes more for no interesting reason; the final stage switches the Rayleigh damping off, so its kinetic energy rises by design; and a stage where the energy happens to dip makes the next ordinary rise look like an acceleration. Three samples cannot separate those from a run that is genuinely running away.

Whether an adjustment has settled is now read from the diagnostics table and the viz figure, which show the whole series. The checks that remain are bounds on quantities with real failure modes behind them. Please do not re-add an automated settling criterion without evidence that it survives schedules whose stages differ in length and damping.

Each check is skipped, with a log line, when the configured model does not report the quantity: Omega’s GlobalStats has no CFL number.

One caveat the thresholds cannot express: Omega’s temperature is conservative temperature where MPAS-Ocean’s is potential temperature, and its salinity is absolute salinity where MPAS-Ocean’s is practical salinity, so the bounds are not literally on the same quantities in the two models. Against these thresholds the difference is immaterial.

The viz step writes dynamic_adjustment_stats.png from the same statistics, plotting kinetic energy (maximum, mean and domain-integrated), maximum normal velocity, CFL number, tracer extremes, mean tracer change and minimum layer thickness on one continuous axis, with the stage boundaries marked and each stage labelled with its damping. The labels read upwards, abbreviated to d1, d2 and so on with sim for the final undamped stage, and a label climbs to a second or third row where its stage is too narrow to hold one — u-oi6to18-lr12p5 opens with two 6-hour stages on a 30-day axis. A summary row cannot distinguish a quantity that was flat through a stage from one that spiked and recovered, and those call for different changes to the schedule; the figure can. Reading it against the damping labels is usually the fastest way to tell a response to the ramp from a trend that would have happened anyway. For Omega, the CFL panel shows the conservative estimate described above, labelled as an upper bound.

The viz step also writes dynamic_adjustment_tracer_change.png, which shows how far the adjustment moved the ocean from its initial condition: histograms of the change in temperature and in salinity, cell by cell and layer by layer, from the initial state to the last record of the final stage. Each cell is weighted by its initial volume, so the histograms are probability densities over the ocean’s volume, drawn on a logarithmic axis so the rare large changes show. Each model is compared in its own convention (potential temperature and practical salinity for MPAS-Ocean, conservative temperature and absolute salinity for Omega). The volume-weighted mean and RMS change, the change that 99% of the volume stays within and the largest change are printed on the figure and in the step’s log.

The viz step belongs to the standalone task. If the stages are later reused as shared steps by a longer spin-up workflow, a figure describing a completed adjustment is not what that workflow is asking about.

config options

# Options for realistic global ocean dynamic-adjustment runs
[realistic_global_dynamic_adjustment]

# Path to an alternate schedule YAML that overrides the built-in per-mesh
# schedule; leave blank to use the checked-in schedule for the mesh
schedule =

# Maximum allowed temperature (deg C) reached during any stage; exceeding it is
# treated as numerical blow-up.  Neither the sample written before a stage's
# first time step nor the startup window below is checked, so this bounds what
# the stage did rather than the state it started from.
# Set above the WOA23 warm artifact off Sumatra rather than at a physical
# bound.  On the u-oi6to18 meshes that cell starts at 35.9 to 36.1 deg C and
# erodes far more slowly than it does at 30 km -- 34.3 after six hours, and by
# extrapolation more than a day to fall below 33 -- so a bound at 33 would fail
# the first three stages of a healthy run.
temperature_max = 35.0

# Minimum allowed temperature (deg C) reached during any stage, on the same
# terms as temperature_max.  A detector for numerical problems rather than a
# physical bound: with no surface heat loss, water colder than the initial
# condition can only come from the numerics, as in a tracer update that is not
# monotone in a thin partial bottom cell.  The coldest water in the WOA23
# initial conditions is about -2.12 deg C on every unified mesh, and healthy
# MPAS-Ocean adjustments stay above it.  None of these meshes has ice-shelf
# cavities, where water below this bound is real; revisit it if one gains them.
temperature_min = -2.5

# Maximum allowed salinity reached during any stage, on the same terms: practical
# salinity (PSU) for MPAS-Ocean and absolute salinity (g/kg) for Omega, about
# 0.5% higher.  A blow-up detector rather than a plausibility bound: the WOA23
# source data carries a Red Sea artifact that reaches 44.1 to 44.5 PSU (44.7
# g/kg) in the u-oi6to18 initial conditions.  In MPAS-Ocean it settles near 42.1
# within a few hours, but Omega erodes it far more slowly, since it is a
# ~25-cell blob at 1150 m where there is almost no flow to erode it.  Nothing
# else in those initial conditions is above 42, and a real blow-up produces far
# more than 45.
salinity_max = 45.0

# A window at the start of the adjustment, in addition to each stage's first
# sample, over which the tracer extremes are not checked, as an MPAS duration
# string.  It is measured from the start of the sequence rather than of each
# stage, because what it skips belongs to the initial condition: the WOA23
# artifacts need a couple of hours of mixing at 6-18 km before they fall back
# to the values they then hold for the rest of the run.  A per-stage window
# would instead stop looking just after every damping change, which is where a
# schedule that steps the damping down too fast would show itself.
# The CFL number is deliberately still checked inside this window, for the same
# reason: the opening hours are where a time step that is too long shows up.
startup_exclusion_duration = 00_02:00:00

# Maximum allowed CFL number at any point in any stage.  MPAS-Ocean is not
# usually pushed much past 0.1; the AB2 split-explicit integrator these
# schedules use tolerates more, so this is set at 0.2 as a loose upper bound
# rather than a tuned value.  Worth revisiting against experience.
# MPAS-Ocean only: Omega's GlobalStats reports no CFL number.
cfl_max = 0.2


# Forward-run options overridden for the dynamic-adjustment stages.  A stage is
# a forward run, so it starts from [realistic_global_forward]; this section is
# applied on top of it, and a per-mesh config file can still override this in
# turn.
[realistic_global_forward]

# Don't write density.  It is in the forward output stream so that MPAS-Ocean
# and Omega can be compared on the same fields, which is not what a dynamic
# adjustment is for, and it is 3-D: about 13% of the output volume, or 2.4 GB
# per stage on u-oi6to18-lr12p5, written eight times over.  ssh stays, being
# 2-D and nearly free.  Nothing here reads density -- the checks and the
# summary come from the global statistics, and the stratification questions
# density would answer are asked of the initial condition, not of a stage.
output_density = False