Testing directions for making a release
Have three shells open: one on Chrysalis, one on Compy, and one on Perlmutter. Do the following steps on each machine.
cd
to thezppy
directory.Check out a branch to test on.
test dev (run before making a new zppy RC)
git fetch upstream main git checkout -b test_pre_zppy_rc<#>_<machine_name> upstream/main git log # check the commits match https://github.com/E3SM-Project/zppy/commits/main
test new Unified RC
git fetch upstream main git checkout -b test_unified_rc<#>_<machine_name> upstream/main git log # check the commits match https://github.com/E3SM-Project/zppy/commits/main
test final Unified
git fetch upstream main git checkout -b test_unified_<#>_<machine_name> upstream/main git log # check the commits match https://github.com/E3SM-Project/zppy/commits/main
Set up a dev environment for E3SM Diags. This is used for the
diags_environment_commands
parameter. Normally, this is only used for checkingenvironment_commands
works properly. However, by modifying the test templates, it’s possible to run other E3SM Diags subtasks in a different environment. This can be useful if a bug fix in E3SM Diags hasn’t been incorporated into the latest Unified RC yet.# `cd` to e3sm_diags directory $ git checkout main $ git fetch upstream $ git reset --hard upstream/main $ git log # Should match https://github.com/E3SM-Project/e3sm_diags/commits/main $ mamba clean --all $ mamba env create -f conda-env/dev.yml -n e3sm_diags_<date> $ conda activate e3sm_diags_<date> $ pip install . $ `cd` back to zppy directory
Make sure you’re using the latest packages. In
tests/integration/utils.py
, do the following:test dev (run before making a new zppy RC):
If it’s the first zppy RC (i.e., a Unified RC hasn’t been made yet): Change the last line
generate_cfgs()
togenerate_cfgs(unified_testing=False)
. We want to use the latest officially released E3SM Unified environment.Otherwise: follow the (b: test new Unified RC) directions for this step.
test new Unified RC
Change the last line
generate_cfgs()
togenerate_cfgs(unified_testing=True)
. We don’t want to use the latest officially released E3SM Unified environment.Update
environment_commands_test
for the machine you’re currently working on. If this is not done, thenzppy
will use the existing value, which is likely an old E3SM Unified RC launch script that doesn’t even exist anymore. This parameter should be set to source the latest E3SM Unified RC launch script, since E3SM Unified is the only environment that allzppy
tasks can run from. For instance,ncclimo
isn’t in thezppy
dev environment.
test final Unified
Change the last line
generate_cfgs()
togenerate_cfgs(unified_testing=False)
. We want to use the latest officially released E3SM Unified environment.
For a, b, and c, always:
Update
diags_environment_commands
for the machine you’re currently working on to use the environment created in the above step. If this is not done, theatm_monthly_180x360_aave_environment_commands
subtask will use an older dev environment for E3SM Diags than we want.
Set up your environment.
test dev (run before making a new zppy RC): Set up a new development environment. This ensures that testing will use the latest conda changes. Note that you will need to run
conda remove -n zppy_dev_pre_zppy_rc<#> --all
first if you have previously done this step.mamba clean --all mamba env create -f conda/dev.yml -n zppy_dev_pre_zppy_rc<#> conda activate zppy_dev_pre_rc<#> pip install .
test new Unified RC: Launch the E3SM Unified environment for the machine you’re on. Change out the version numbers below.
Chrysalis:
source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.9.0rc16_chrysalis.sh
Compy:
source /share/apps/E3SM/conda_envs/test_e3sm_unified_1.9.0rc16_compy.sh
Perlmutter:
source /global/common/software/e3sm/anaconda_envs/test_e3sm_unified_1.9.0rc16_pm-cpu.sh
test final Unified: Launch the latest E3SM Unified environment for the machine you’re on.
Chrysalis:
source /lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh
Compy:
source /share/apps/E3SM/conda_envs/load_latest_e3sm_unified_compy.sh
Perlmutter:
source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_pm-cpu.sh
Run the unit tests with
python -u -m unittest tests/test_*.py
.test dev (run before making a new zppy RC):
If there are any failures, fix the code (or tests). If you make any conda changes, go back to step 6a. If you otherwise change zppy source code, run
pip install .
and then redo step 7. If you only make changes to tests, you can immediately redo step 7.
test new Unified RC:
If there are any failures, fix the code and go back to step 1, following the (a: test dev (run before making a new zppy RC)) directions.
test final Unified:
There should be no failures. If there are, a patch release of E3SM Unified may be required.
For a, b, and c:
If there are no failures, proceed to the next step.
Run the “Commands to run before running integration tests” for the current machine. To ensure you don’t encounter issues from running
zppy
commands simultaneously, wait for all the automatically launched jobs from one run to finish before runningzppy
again.Run the integration tests with
python -u -m unittest tests/integration/test_*.py
. Note thattest_complete_run.py
takes approximately 75 minutes to run on Compy.test dev (run before making a new zppy RC):
If there are any unexpected failures, fix the code (or tests). If you make any conda changes, go back to step 6a. If you otherwise change zppy source code, run
pip install .
and then go back to step 7. If you only make changes to tests, you can immediately redo step 9.
test new Unified RC:
If there are any unexpected failures, fix the code and go back to step 1, following the (a: test dev (run before making a new zppy RC)) directions.
test final Unified:
There should be no unexpected failures. If there are, a patch release of E3SM Unified may be required.
For a, b, and c:
If there are only expected failures, then update the expected files. Use the “Commands to run to replace outdated expected files” from the links on step 8. Then repeat step 9.
If there are no failures at all, proceed to the next step.
Make a pull request and merge any changes. This keeps the repo updated with the latest testing configurations. Mark yourself as the assignee, and mark “Testing” as the label. If you made bug fixes, add the “semver: bug” label.
Wrap up release testing:
test dev (run before making a new zppy RC): Create the next zppy RC by following the “release candidates” directions at https://e3sm-project.github.io/zppy/_build/html/main/dev_guide/release.html.
test new Unified RC: Create the next zppy release following the “production releases” directions at https://e3sm-project.github.io/zppy/_build/html/main/dev_guide/release.html.
test final Unified: Run the “Commands to generate official expected results for a release” from step 8. This will create a baseline of expected files for the release. You can now safely remove old branches and environments. At https://github.com/E3SM-Project/zppy/branches, delete any branches that are no longer needed. Also, run:
# Branches $ cd <zppy directory> $ git branch # Look at all branch names $ git branch -D <list branches you want to delete> # Environments $ conda env list # For each environment you want to delete, run: $ conda remove -n <environment_name> --all