Developing and Testing Your Code

Initial Notes for the Included Examples

For the purposes of this guide, we will employ the working example of adding a new atmosphere processes to the MAM4xx aerosol library, specifically the not-real, example process "Ash-Injection." The source files for this process will be found at src/physics/mam/. Note that here, and for the remainder of this guide, we will denote all paths relative to the EAMxx root directory E3SM/components/eamxx/ (see Code Structure for additional details on how EAMxx code is organized).

Example: Adding a New Atmosphere Process

The minimum files required for a new process will be those defining the interface to EAMxx. For our case, and following the current informal naming standard, these will be the eamxx_inject_ash_process_interface.<x>pp files, located at src/physics/mam/, and trivial skeleton examples are provided below.

eamxx_inject_ash_process_interface.hppeamxx_inject_ash_process_interface.cpp

#ifndef EAMXX_MAM_INJECT_ASH_HPP
#define EAMXX_MAM_INJECT_ASH_HPP

#include "share/atm_process/atmosphere_process.hpp"

namespace scream {
class MAMInjectAsh : public AtmosphereProcess {
public:
  MAMInjectAsh(const ekat::Comm &comm, const ekat::ParameterList &params);
  AtmosphereProcessType type() const override {
    return AtmosphereProcessType::Physics;
  }
  std::string name() const override { return "MAMInjectAsh"; }
  void
  set_grids(const std::shared_ptr<const GridsManager> grids_manager) override;

protected:
  void initialize_impl(const RunType run_type) override;
  void run_impl(const double dt) override;
  void finalize_impl() override { /* Nothing to do */ }

  Field source_mask;
  Field lat;
  Field lon;
}
} // namespace scream

#endif // EAMXX_MAM_INJECT_ASH_HPP

#include "ash_source_calculation.hpp"
#include "eamxx_inject_ash_process_interface.hpp"

namespace scream {

MAMInjectAsh::MAMInjectAsh(const ekat::Comm &comm,
                           const ekat::ParameterList &params)
    : AtmosphereProcess(comm, params) {
  [...]
}
void MAMInjectAsh::set_grids(
    const std::shared_ptr<const GridsManager> grids_manager) {
  constexpr auto kg = ekat::units::kg;
  auto grid = grids_manager->get_grid("Physics");
  add_tracer<Updated>("ash", grid, kg / kg);
  lat = grid->get_geometry_data("lat");
  lon = grid->get_geometry_data("lon");
}
void MAM_AshInjection::initialize_impl(const RunType run_type) {
  source_mask = get_field_out("ash").clone("source_mask");
  [...] // set source_mask=1 only on area of interest, using lat/lon field
}
void MAM_AshInjection::run_impl(const double dt) {
  auto t = this->timestamp() + dt; // timestamp() is the START of step time
  auto ash = get_field_out("ash");
  auto rate = compute_ash_injection_rate(
      t); // defined in ash_source_calculation.hpp header
  ash.update(source_mask, rate * dt,
             decay); // ash = decay*ash + rate*dt*source_mask
}
} // namespace scream

Add the New Process to CMake Build

In order to ensure the code you've added is configured and built by CMake, we need to add to the relevant CMakeLists.txt file. At minimum, we need to add eamxx_inject_ash_process_interface.cpp to the EAMxx library named mam, as shown below. Of course, there may be additional configuration commands that are required for a proper build, but this is beyond the scope of this guide.

src/physics/mam/CMakeLists.txt

[...]
# EAMxx mam4xx-based atmospheric processes
add_library(mam
            eamxx_mam_microphysics_process_interface.cpp
            eamxx_mam_optics_process_interface.cpp
            [...]
            eamxx_inject_ash_process_interface.cpp)
[...]

Adding Verification and Validation Tests

All new code that is added to EAMxx is required to be tested. At minimum, a process must be Validation Tested. That is, tested against known, trusted data, and this data can be samples taken by real-world instruments or comparable results generated by other code. Ideally, any new and sufficiently complex functions should also have Verification Tests that test for mathematical or scientific "correctness".¹

Single-process Validation Test

First, we add a validation test for (only) our new process to tests/single-process/mam/inject_ash/. Defining the test behavior and pass/fail criteria requires 3 files:

• CMakeLists.txt
  • Largely reusable boilerplate code but should be tailored to the proper test parameters.
  • Defines build and run behavior.
  • Enumerates the required NetCDF input files.
• input.yaml
  • Defines key parameters for the model and test run.
  • Enumerates the keys (std::string) used to access the input files from within the process source code.
  • Provides information about the test's initial condition.
    • Input file names/paths.
    • Other required input values.
• output.yaml
  • Defines the output fields used as pass/fail criteria.
  • Defines the tests runtime output behavior.

CMakeLists.txtinput.yamloutput.yaml

include (EAMxxUtils)

set (TEST_BASE_NAME mam4_inject_ash_standalone)
set (FIXTURES_BASE_NAME ${TEST_BASE_NAME}_generate_output_nc_files)
# Create the test
CreateADUnitTest(${TEST_BASE_NAME}
  LABELS mam4_inject_ash physics
  LIBS mam
  MPI_RANKS ${TEST_RANK_START} ${TEST_RANK_END}
  FIXTURES_SETUP_INDIVIDUAL ${FIXTURES_BASE_NAME}
)

# Set AD configurable options
set (ATM_TIME_STEP 1800)
SetVarDependingOnTestSize(NUM_STEPS 2 5 48)  # 1h 2.5h 24h
set (RUN_T0 2021-10-12-45000)

# Copy (and configure) yaml files needed by tests
configure_file(${CMAKE_CURRENT_SOURCE_DIR}/input.yaml
               ${CMAKE_CURRENT_BINARY_DIR}/input.yaml)
configure_file(${CMAKE_CURRENT_SOURCE_DIR}/output.yaml
               ${CMAKE_CURRENT_BINARY_DIR}/output.yaml)

# Ensure test input files are present in the data dir
set (TEST_INPUT_FILES
  eamxx/init/${inject_ash_IC_file}
  eamxx/mam4xx/.../inject_ash_input_data_file_ABC.nc
  eamxx/mam4xx/.../inject_ash_input_data_file_XYZ.nc
  [...]
  <path-to-required-input-file>
)
foreach (file IN ITEMS ${TEST_INPUT_FILES})
  GetInputFile(${file})
endforeach()

# Compare results between runs employing differing numbers of MPI ranks
# to ensure they are bfb
include (CompareNCFiles)
CompareNCFilesFamilyMpi (
  TEST_BASE_NAME ${TEST_BASE_NAME}
  FILE_META_NAME ${TEST_BASE_NAME}_output.INSTANT.nsteps_x1.npMPIRANKS.${RUN_T0}.nc
  MPI_RANKS ${TEST_RANK_START} ${TEST_RANK_END}
  LABELS mam4_inject_ash physics
  META_FIXTURES_REQUIRED ${FIXTURES_BASE_NAME}_npMPIRANKS_omp1
)
# Compare one of the output files with the baselines.
# Note: one is enough, since we already check that np1 is BFB with npX
if (EAMXX_ENABLE_BASELINE_TESTS)
  set (OUT_FILE ${TEST_BASE_NAME}_output.INSTANT.nsteps_x1.np${TEST_RANK_END}.${RUN_T0}.nc)
  CreateBaselineTest(${TEST_BASE_NAME} ${TEST_RANK_END} ${OUT_FILE} ${FIXTURES_BASE_NAME})
endif()

%YAML 1.1
---
driver_options:
  atmosphere_dag_verbosity_level: 5
time_stepping:
  time_step: ${ATM_TIME_STEP}
  run_t0: ${RUN_T0}  # YYYY-MM-DD-XXXXX
  number_of_steps: ${NUM_STEPS}
atmosphere_processes:
  atm_procs_list: [mam4_inject_ash]
  mam4_inject_ash:
    inject_ash_ABC_data: ${EAMXX_DATA_DIR}/mam4xx/.../inject_ash_input_data_file_ABC.nc
    inject_ash_XYZ_data: ${EAMXX_DATA_DIR}/mam4xx/.../inject_ash_input_data_file_XYZ.nc
    <identifying-string-used-by-inject_ash_xyz.xpp>: /path/to/.../data_file.nc
grids_manager:
  Type: Mesh Free
  geo_data_source: IC_FILE
  grids_names: [Physics GLL]
  Physics GLL:
    type: point_grid
    aliases: [Physics]
    number_of_global_columns:   218
    number_of_vertical_levels:  72
initial_conditions:
  # The name of the file containing the initial conditions for this test.
  Filename: ${EAMXX_DATA_DIR}/init/${inject_ash_IC_file}
  topography_filename: ${TOPO_DATA_DIR}/${EAMxx_tests_TOPO_FILE}

  # other variables to pass as input
  num_ABC : 42.0
  num_XYZ : 1.618
  <input_variable> : <value>
# The parameters for I/O control
Scorpio:
  output_yaml_files: ["output.yaml"]
...

%YAML 1.1
---
filename_prefix: mam4_inject_ash_standalone_output
Averaging Type: Instant
Fields:
  Physics:
    Field Names:
      - answer_field
output_control:
  Frequency: 1
  frequency_units: nsteps
...

Function-level Verification Tests

Next, because of the highly-technical nature of our new compute_ash_injection_rate() function, we add a verification test to the 'tests/' subdirectory, co-located with the source code (src/physics/mam/). This subdirectory contains a CMakeLists.txt that we modify to add our new test to the build system. Any required data files should be placed here, and the preferred format for data is yaml, though NetCDF can also be used.

src/physics/mam/tests/ash_injection.cpp

#include "catch2/catch.hpp"
#include "ash_source_calculation.hpp"

namespace {
using namespace scream;
TEST_CASE("verify compute_ash_injection_rate", "mam")
{
  // initialize the data and test variables
  tol = 1.0e-12;
  util::TimeStamp t1 = ...
  util::TimeStamp t2 = t1 + 86400 * 100; // 100 days later
  [...]
  // call the function we are testing
  auto rate1 = compute_ash_injection_rate(t1);
  auto rate2 = compute_ash_injection_rate(t2);
  // apply the REQUIRE() function from catch2/catch.hpp to test
  // whether the resulting test answer is within a set error tolerance
  // from the reference answer. Check all properties that are meaningful
  REQUIRE (rate1 >= 0);
  REQUIRE (rate2 >= 0);
  REQUIRE (rate2 < rate1);
  [...]
}

Multi-process Validation Tests

Finally, because the Ash-Injection process is not called in isolation, we must validate the results generated when Ash-Injection is called in combination with other processes. At time of writing, these tests are divided into the self-describing directories physics_only and dynamics_physics. The subdirectories and test directories are named according to the convention of listing the names of included processes and potentially relevant grid information. Generally speaking, these tests are configured in the same manner as the single-process tests, via the 3 files CMakeLists.txt, input.yaml, output.yaml, and for this reason we omit further details

Code, Build, Test... Repeat

Now that we've written all of the necessary code and there are no errors, we build EAMxx, run our tests, and watch them pass with flying colors. :white_check_mark:

However, for those of us that are mere mortals, we may need to iterate on this process a bit until everything works perfectly. In this case, the automated build and test workflow provided by scripts/test-all-eamxx is designed to make this cycle as painless as possible.

Suggested (Very Thorough and Conservative) Testing Progression

To fully cover all the potential interactions of the Ash-Injection process, a reasonable series of tests is provided in the example script below.

Note: There is no reason all of this could not be done at once, but we write the script in favor of being clear and explicit.

Thorough and Careful Testing Script

For the purposes of this example, we assume we are on a non-supported local workstation named "bingo-pajama" that has 64 CPU cores and an NVIDIA GPU. We also assume that we have working configuration files in ~/.cime/ for both CPU and GPU build configurations.

#!/bin/bash

# machine-related variables
machine_name="bingo-pajama"
total_ncores=64
id_cpu="${machine_name}-cpu"
id_gpu="${machine_name}-gpu"

# name of log file for conveniently capturing output
outfile_prefix="ash_injection_buildNtest"
ofile_cpu="${outfile_prefix}-${id_cpu}.log"
ofile_gpu="${outfile_prefix}-${id_gpu}.log"

# parallelism variables
make_ncores=60
test_ncores=60
pbuild="-p --make-parallel-level ${make_ncores}"
ptest="--ctest-parallel-level ${test_ncores}"

# flags to assist with testing/debugging
vflag="--extra-verbose"
# limit test configuration to double-precision debug
test_config="-t dbg"
bl_loc="${HOME}/eamxx-baselines"

# navigate to the root directory of eamxx
cd "${eamxx_root}"
# switch to the master branch
git checkout master
# pull so we are at the latest commit (origin/HEAD)
git pull origin master
# update the submodules (the below is typically unnecessary, but thorough)
git submodule sync --recursive && \
  git submodule update --init --recursive --progress

# flags common to all tests
base_flags="${pbuild} ${vflag} ${test_config}"
# flags common to this stage of tests
test_base_flags="${base_flags} --baseline-dir ${bl_loc} --generate"
# general test flags per architecture
cpu_flags="--work-dir ${PWD}/${id_cpu} --local ${id_cpu} ${ptest}"
gpu_flags="--work-dir ${PWD}/${id_gpu} --local ${id_gpu}"
# specific flags for this test per architecture
cpu_test_flags="${test_base_flags} ${cpu_flags}"
gpu_test_flags="${test_base_flags} ${gpu_flags}"

# generate baselines for cpu and gpu
echo "TEST FLAGS = ${cpu_test_flags}"
./scripts/test-all-eamxx ${cpu_test_flags}
echo "TEST FLAGS = ${gpu_test_flags}"
./scripts/test-all-eamxx ${gpu_test_flags}

# Verify the 'compute_ash_injection_rate()' function works as expected by
running only that test
test_choice="[^\s]*compute_ash_injection_rate_test[^\s]*"
test_base_flags="${base_flags} --limit-test-regex ${test_choice}"
cpu_test_flags="${test_base_flags} ${cpu_flags}"
gpu_test_flags="${test_base_flags} ${gpu_flags}"

# run verification tests
echo "TEST FLAGS = ${cpu_test_flags}"
# now that we're running tests that may fail, use 'tee' to
# duplicate stdout and stderr to file so debugging is slightly easier
./scripts/test-all-eamxx ${cpu_test_flags} |& tee "${ofile_cpu}"
echo "TEST FLAGS = ${gpu_test_flags}"
./scripts/test-all-eamxx ${gpu_test_flags} |& tee "${ofile_gpu}"

# Validate the Ash-Injection process generates the solutions we expect
test_choice="[^\s]*mam4_ash_injection_standalone[^\s]*"
test_base_flags="${base_flags} --limit-test-regex ${test_choice}"
cpu_test_flags="${test_base_flags} ${cpu_flags}"
gpu_test_flags="${test_base_flags} ${gpu_flags}"
# run single-process verification tests
echo "TEST FLAGS = ${cpu_test_flags}"
./scripts/test-all-eamxx ${cpu_test_flags} |& tee "${ofile_cpu}"
echo "TEST FLAGS = ${gpu_test_flags}"
./scripts/test-all-eamxx ${gpu_test_flags} |& tee "${ofile_gpu}"

# validate all the processes that interact with ash_injection generate
# expected solutions
ash_injection_multiproc_tests=(
                           # physics_only
                           shoc_mam4_ash_injection
                           p3_mam4_ash_injection
                           # dynamics_physics
                           homme_shoc_cld_p3_mam4_ash_injection_rrtmgp
                           homme_shoc_cld_spa_p3_rrtmgp_mam4_ash_injection
                           )
for tchoice in ${ash_injection_multiproc_tests[@]}; do
  test_choice="[^\s]*${tchoice}[^\s]*"
  test_base_flags="${base_flags} --limit-test-regex ${test_choice}"
  cpu_test_flags="${test_base_flags} ${cpu_flags}"
  gpu_test_flags="${test_base_flags} ${gpu_flags}"
  # run single-process verification tests
  echo "TEST FLAGS = ${cpu_test_flags}"
  ./scripts/test-all-eamxx ${cpu_test_flags} |& tee "${ofile_cpu}"
  echo "TEST FLAGS = ${gpu_test_flags}"
  ./scripts/test-all-eamxx ${gpu_test_flags} |& tee "${ofile_gpu}"
done
# NOTE: just setting 'test_choice="[^\s]*_mam4_ash_injection^\s]*"' would
#       also do the trick in this case

# run full eamxx test suite against pre-Ash-Injection baselines for all
# test configurations (not just 'dbg')
base_flags="${pbuild} ${vflag} --baseline-dir ${bl_loc}"
cpu_test_flags="${base_flags} ${cpu_flags}"
gpu_test_flags="${base_flags} ${gpu_flags}"
# run the full suite of eamxx baseline tests
echo "TEST FLAGS = ${cpu_test_flags}"
./scripts/test-all-eamxx ${cpu_test_flags} |& tee "${ofile_cpu}"
echo "TEST FLAGS = ${gpu_test_flags}"
./scripts/test-all-eamxx ${gpu_test_flags} |& tee "${ofile_gpu}"

Testing Modifications to Existing Code

Note that if you are merely modifying or augmenting existing code, you may not need to create additional tests, provided there is already a test that targets the piece of code that was edited. However, one case for which you should create a test is when adding a new function to existing code. In this case, a verification test should be added that checks the new functionality for correctness.

Submit Code to E3SM Repository for Review and Integration

Whew!

Now that we have added some exciting new functionality to EAMxx and confirmed that:

• The process sub-components (functions) work correctly.
  • Verification testing of the functions
• The process gives us expected answers both in isolation and when interacting with other processes.
  • Single- and multi-process Validation testing
• We have not introduced regressions from the baseline results.
  • Baseline testing

Now it's time to submit the code!

Getting your new code integrated into E3SM requires several steps involving git and completed using E3SM's GitHub repository. This process is collectively referred to as a Pull Request or PR.²

At a very high level, the steps involved are:

1. Fork the E3SM repository (referred to as a repo by the cool kids).
2. Push your development branch to your own E3SM fork.
3. Submit the Pull Request from your fork to the E3SM-Project/E3SM repository.
4. Respond to Reviews and comments provided by other developers, and make the necessary modifications to your code.
5. Once all reviewers are satisfied with the changes, your code will be Merged into the master branch of E3SM. :partying_face:

---

1. Admittedly, "correctness" is a slippery concept, but at the very least we can always test for properties like mass conservation, non-negativity, etc. ↩

2. Here is an E3SM-based reference about Submitting a Pull Request (PR) (along with a lot of other good info for developers), and here is another reference provided by GitHub. ↩