Tutorial

These examples are for use on LCRC machines (e.g., Chrysalis). Some parameters must be changed for use on other machines. These include the paths for input, output, www, mapping_file, reference_data_path, obs_ts, streamflow_obs_ts, and dc_obs_climo. Different machines also have different partition names, so partition may need to be changed as well.

Example 1

Let’s say we want to post-process 100 years of an existing simulation.

Copy and paste the following into post.mysimulation.cfg

  1[default]
  2input = <input>
  3input_subdir = archive/atm/hist
  4output = <output>
  5case = 20210122.v2_test01.piControl.ne30pg2_EC30to60E2r2-1900_ICG.chrysalis
  6www = <www>
  7partition = compute
  8campaign = "water_cycle"
  9
 10# Regridded atmosphere climatologies every 20 and 50 years
 11[climo]
 12active = True
 13years = "1:100:20", "1:100:50",
 14# Use default vars
 15
 16  # Monthly climatology
 17  [[ atm_monthly_180x360_aave ]]
 18  mapping_file = /home/ac.zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc
 19  frequency = "monthly"
 20
 21  # Diurnal climatology (specifically, 3-hour increments of a day averaged over every day of a month)
 22  [[ atm_monthly_diurnal_8xdaily_180x360_aave ]]
 23  input_subdir = "archive/atm/hist"
 24  input_files = "eam.h4"
 25  mapping_file = /home/ac.zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc
 26  vars = "PRECT"
 27  frequency = "diurnal_8xdaily"
 28
 29# Regridded atmosphere monthly time series files in 10 year chunks
 30[ts]
 31active = True
 32years = "1:100:10",
 33
 34  # Monthly time series
 35  [[ atm_monthly_180x360_aave ]]
 36  input_subdir = "archive/atm/hist"
 37  input_files = "eam.h0"
 38  frequency = "monthly"
 39  mapping_file = /home/ac.zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc
 40  # Use default vars
 41
 42  # Daily time series
 43  [[ atm_daily_180x360_aave ]]
 44  input_subdir = "archive/atm/hist"
 45  input_files = "eam.h1"
 46  frequency = "daily"
 47  mapping_file = /home/ac.zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc
 48  vars = "PRECT"
 49
 50  # Monthly global time series
 51  [[ atm_monthly_glb ]]
 52  input_subdir = "archive/atm/hist"
 53  input_files = "eam.h0"
 54  frequency = "monthly"
 55  mapping_file = "glb"
 56  # Use default vars
 57
 58  # Monthly land time series
 59  [[ land_monthly ]]
 60  input_subdir = "archive/lnd/hist"
 61  input_files = "elm.h0"
 62  frequency = "monthly"
 63  mapping_file = /home/ac.zender/data/maps/map_ne30pg2_to_cmip6_180x360_aave.20200201.nc
 64  vars = "FSH,RH2M"
 65
 66  [[ rof_monthly ]]
 67  input_subdir = "archive/rof/hist"
 68  input_files = "mosart.h0"
 69  frequency = "monthly"
 70  mapping_file = ""
 71  vars = "RIVER_DISCHARGE_OVER_LAND_LIQ"
 72  extra_vars = 'areatotal2'
 73
 74[tc_analysis]
 75active = boolean(default=True)
 76years = "1:100:20", "1:100:50",
 77
 78# E3SM Diags every 20 and 50 years
 79[e3sm_diags]
 80active = True
 81years = "1:100:20", "1:100:50",
 82ts_num_years = 10
 83
 84  # Monthly diagnostics
 85  [[ atm_monthly_180x360_aave ]]
 86  short_name = '20210122.v2_test01.piControl.ne30pg2_EC30to60E2r2-1900_ICG.chrysalis'
 87  grid = '180x360_aave'
 88  reference_data_path = '/lcrc/soft/climate/e3sm_diags_data/obs_for_e3sm_diags/climatology'
 89  # These two lines need to be included if sets requiring time series files are being run.
 90  obs_ts = '/lcrc/soft/climate/e3sm_diags_data/obs_for_e3sm_diags/time-series'
 91  ref_start_yr = 1979
 92  # This needs to be set for QBO Diags
 93  ref_final_yr = 2016
 94  # This needs to be set for streamflow diags
 95  streamflow_obs_ts = '/lcrc/group/e3sm/public_html/diagnostics/observations/Atm/time-series/'
 96  # These three lines need to be included if diurnal cycle diagnostics are being run.
 97  dc_obs_climo = '/lcrc/group/e3sm/public_html/diagnostics/observations/Atm/climatology'
 98  climo_diurnal_subsection = "atm_monthly_diurnal_8xdaily_180x360_aave"
 99  climo_diurnal_frequency = "diurnal_8xdaily"
100  # This is for tropical cyclone diags
101  tc_obs = '/lcrc/group/e3sm/public_html/diagnostics/observations/Atm/tc-analysis/'
102
103# MPAS-Analysis
104[mpas_analysis]
105active = True
106walltime = "24:00:00"
107parallelTaskCount = 6
108ts_years = "1-50", "1-100",
109enso_years = "11-50", "11-100",
110climo_years = "21-50", "51-100",
111mesh = "EC30to60E2r2"
112
113# Global time series plots
114[global_time_series]
115active = True
116years = "1-100",
117ts_num_years = 10
118figstr=coupled_v2_test01
119moc_file=mocTimeSeries_0001-0100.nc
120experiment_name=20210122.v2_test01.piControl.chrysalis
121ts_years = "1-50", "1-100",
122climo_years = "21-50", "51-100",

Now we can run zppy -c post.mysimulation.cfg.

The [climo] subsections ([[ atm_monthly_180x360_aave ]] and [[ atm_monthly_diurnal_8xdaily_180x360_aave ]]), the [ts] subsections ([[ atm_monthly_180x360_aave ]], [[ atm_daily_180x360_aave ]], [[ atm_monthly_glb ]] and [[ land_monthly ]]), and the [mpas_analysis] section will run first because they have no dependencies.

Once the [climo] subsections and the [ts] subsection [[ atm_monthly_180x360_aave ]] finish, [e3sm_diags] will run.

Once the [ts] subsection [[ atm_monthly_glb ]] and the [mpas_analysis] section finish, [global_time_series] will run.

Post-processing results will be located in output and www. Some machines have a web server. www should be pointed to that so that E3SM Diags, MPAS-Analysis, and the global time series plots will be visible online.

Because we have specified campaign = "water_cycle", some parameters will be automatically set. zppy/templates/water_cycle.cfg specifies what [e3sm_diags] > sets, and [mpas_analysis] > generate should be for the water cycle campaign. Users may specify their own values for any of these parameters, allowing for easy configuration changes. For example, a user could set campaign = "water_cycle" but specify their own value for [e3sm_diags] > sets.

Example 2

This is another example of a configuration file, this time using a RRM simulation.

 1[default]
 2input = <input>
 3input_subdir = archive/atm/hist
 4output = <output>
 5case = 20210603.v2rc3c.piControl.northamericax4v1pg2_WC14to60E2r3.chrysalis
 6www = <www>
 7partition = debug
 8campaign = "water_cycle"
 9
10[climo]
11active = True
12years = "1:60:20",
13
14  [[atm_monthly_180x360_aave]]
15  mapping_file = /home/ac.zender/data/maps/map_northamericax4v1pg2_to_cmip6_180x360_aave.20200401.nc
16
17  [[conus_0.23x0.31_aave]]
18  grid = 'conus_0.23x0.31_aave'
19  mapping_file = /home/ac.zender/data/maps/map_northamericax4v1pg2_to_conus_0.23x0.31_nco.20210301.nc
20
21[ts]
22active = True
23years = "1:60:10",
24mapping_file = /home/ac.zender/data/maps/map_northamericax4v1pg2_to_cmip6_180x360_aave.20200401.nc
25
26  [[ atm_monthly_180x360_aave ]]
27  input_subdir = "archive/atm/hist"
28  input_files = "eam.h0"
29  vars = "FSNTOA,FLUT,FSNT,FLNT,FSNS,FLNS,SHFLX,QFLX,TAUX,TAUY,PRECC,PRECL,PRECSC,PRECSL,TS,TREFHT,CLDTOT,CLDHGH,CLDMED,CLDLOW,U"
30
31  [[ atm_monthly_glb ]]
32  input_subdir = "archive/atm/hist"
33  input_files = "eam.h0"
34  mapping_file = "glb"
35  vars = "FSNTOA,FLUT,FSNT,FLNT,FSNS,FLNS,SHFLX,QFLX,TAUX,TAUY,PRECC,PRECL,PRECSC,PRECSL,TS,TREFHT,CLDTOT,CLDHGH,CLDMED,CLDLOW"
36
37  [[ land_monthly ]]
38  input_subdir = "archive/lnd/hist"
39  input_files = "elm.h0"
40  frequency = "monthly"
41  vars = "FSH,RH2M"
42
43[e3sm_diags]
44active = True
45reference_data_path = '/lcrc/soft/climate/e3sm_diags_data/obs_for_e3sm_diags/climatology'
46output_format_subplot = "pdf",
47short_name = '20210603.v2rc3c.piControl.naRRM'
48years = "1:60:20",
49
50  [[atm_monthly_180x360_aave]]
51  grid = '180x360_aave'
52  sets = "lat_lon","zonal_mean_xy","zonal_mean_2d","polar","cosp_histogram","meridional_mean_2d","enso_diags","qbo","area_mean_time_series",
53  ts_num_years = 10
54  obs_ts = '/lcrc/soft/climate/e3sm_diags_data/obs_for_e3sm_diags/time-series'
55  ref_start_yr = 1990
56  ref_end_yr = 1999
57  ref_final_yr = 2016
58
59  [[conus_0.23x0.31_aave]]
60  grid = 'conus_0.23x0.31_aave'
61  sets = 'lat_lon',
62  cfg = "inclusions/e3sm_diags/lat_lon_conus.cfg" # Built-in cfg for users to use
63
64[mpas_analysis]
65active = False
66walltime = "24:00:00"
67parallelTaskCount = 6
68ts_years = "1-50",
69enso_years = "1-50",
70climo_years ="1-50",
71mesh = "WC14to60E2r3"
72partition = compute
73
74[global_time_series]
75active = False

Example 3

MPAS-Analysis model vs. model

MPAS-Analysis supports “main vs. control” (model-vs-model) comparisons. In zppy, this is configured in the [mpas_analysis] section using reference_data_path (and optionally test_data_path), consistent with the terminology used for e3sm_diags model-vs-model runs.

Unlike e3sm_diags (where run_type = "model_vs_model" and reference_data_path points directly at reference climatology output), MPAS-Analysis comparisons are driven by MPAS-Analysis config files. For model-vs-model mode, zppy locates the matching config file(s) from prior MPAS-Analysis output and passes them to MPAS-Analysis.

 1[default]
 2input = <test simulation input>
 3# Where the `post/` directory will be written
 4output = <test simulation output>
 5case = <test case name>
 6www = <www>
 7partition = <partition>
 8
 9# MPAS-Analysis: model vs. model ("test" vs. "reference")
10[mpas_analysis]
11active = True
12walltime = "4:00:00"
13parallelTaskCount = 6
14mesh = "EC30to60E2r2"
15shortTermArchive = True
16
17# Test run year ranges (these determine the identifier used for the generated
18# MPAS-Analysis cfg: mpas_analysis_ts_<ts>_climo_<climo>.cfg)
19ts_years = "1850-2014",
20climo_years = "1985-2014",
21enso_years = "1850-2014",
22
23# Point at a *previous zppy run output directory* for the reference simulation.
24# zppy will locate the matching MPAS-Analysis cfg file under:
25#   <reference_data_path>/post/analysis/mpas_analysis/cfg/
26reference_data_path = <reference zppy output directory>
27
28# Required when reference_data_path is not a [[subsection]].
29# Used to build the MVM output directory name.
30reference_case = <reference case name>
31
32# Optional: point at a previous zppy output directory for the test simulation,
33# if you want MPAS-Analysis to reuse a completed test run as well.
34# If set to [[subsection]], zppy will use that subsection's year ranges
35# when ts_years/climo_years/enso_years are not provided.
36# test_data_path = <test zppy output directory>
37
38# Optional: override the reference year ranges (defaults to the test ranges).
39# If reference_data_path points to a prior [mpas_analysis] subsection using
40# the form [[subsection]], these default to that subsection's year ranges.
41# If you provide a single range, it will be reused for each test range.
42# ref_ts_years = "451-500",
43# ref_climo_years = "451-500",
44# ref_enso_years = "451-500",

Debugging failures

$ cd <output directory from cfg>/post/scripts
$ grep -v "OK" *status # See what failed
# Review `.o` files corresponding to failed `.status` files.
# If an error is obvious, make a fix in the bash file and rerun:
$ sbatch <failed job>.bash
# If the error is not obvious, do the following:
$ emacs <failed job>.bash
# In this file, set `debug = True`. This will provide more information.
# Note: another option is to set `debug = True` in your `cfg` and rerun `zppy`.
$ sbatch <failed job>.bash