Developer Tutorial: Running the mali_pr Suite on Perlmutter

This tutorial takes you from no Polaris at all to a finished run of the mali_pr suite on Perlmutter. Along the way, you clone Polaris, deploy its environment, let Polaris build MALI, and submit the suite as a batch job. You can use Perlmutter’s CPU nodes (pm-cpu) or its GPU nodes (pm-gpu); the steps are the same apart from a few names, which are given for both.

You need:

  • a NERSC account in the e3sm project (or another project you can charge jobs to; see Submitting the job)

  • an SSH key on GitHub that you can use from Perlmutter, since Polaris and its submodules are cloned with SSH

  • the bash shell (see Unix Shell)

Cloning Polaris

Log in to Perlmutter and clone Polaris:

ssh <username>@perlmutter.nersc.gov
git clone git@github.com:E3SM-Project/polaris.git
cd polaris
git submodule update --init jigsaw-python e3sm_submodules/MALI-Dev

Polaris keeps the model source code in submodules under e3sm_submodules. You only need two of them here: jigsaw-python, which the deployment builds, and MALI-Dev, the source code for MALI.

Deploying the Polaris environment

Polaris runs in a pixi environment that ./deploy.py creates in your clone. MALI also needs compiled libraries, including Albany and Trilinos for its first-order velocity solver. These are in shared Spack environments on Perlmutter, and --with-albany points Polaris to the ones that have Albany.

For pm-cpu:

./deploy.py --machine pm-cpu --compiler gnu --with-albany

For pm-gpu:

./deploy.py --machine pm-gpu --compiler gnugpu --with-albany

The first deployment takes a while, since it installs pixi and builds the environment. Do not add --deploy-spack: that rebuilds the Spack environments rather than using the shared ones, which takes hours.

When it is done, you have a load script in the root of your clone. Source it whenever you start a new shell to work with Polaris:

# pm-cpu
source load_polaris_pm-cpu_gnu_mpich.sh

# pm-gpu
source load_polaris_pm-gpu_gnugpu_mpich.sh

Polaris pixi and spack environments, compilers and system modules says more about deployment and its options.

Setting up the suite

Choose a work directory on Perlmutter’s scratch file system, where the suite will be built and run, and set up the suite there:

# pm-cpu
polaris suite -c landice -t mali_pr -m pm-cpu \
    -w $SCRATCH/polaris_work/mali_pr_pm-cpu

# pm-gpu
polaris suite -c landice -t mali_pr -m pm-gpu \
    -w $SCRATCH/polaris_work/mali_pr_pm-gpu

Polaris builds MALI from the MALI-Dev submodule before it sets up the tasks, so this step takes several minutes. The build goes in landice_build in the work directory, and its log in landice_build/build_mali.log. When setup finishes, it lists the four tasks in the suite:

Setting up tasks:
  landice/dome/2km/sia/decomp
  landice/dome/2km/sia/restart
  landice/dome/2km/fo/decomp
  landice/dome/2km/fo/restart

Running polaris suite again in the same work directory reuses the build. Add --build to rebuild MALI after you change its source code.

Submitting the job

Setup writes a job script, job_script.mali_pr.sh, to the work directory. Submit it:

cd $SCRATCH/polaris_work/mali_pr_pm-cpu   # or mali_pr_pm-gpu
sbatch job_script.mali_pr.sh

The job script charges the e3sm project. To charge another project, put its name in a config file, for example my_account.cfg:

[parallel]
account = <project>

then add -f my_account.cfg to the polaris suite command above and set up the suite again.

The suite runs in a few minutes once the job starts. You can check on the job with squeue --me.

Checking the results

The job’s output is in polaris_mali_pr.o<jobid> in the work directory. The suite’s steps run at the same time where they fit on the node, so the output ends with a summary of the steps (in the order they finished) and the result of the suite:

Step runtimes:
      23.9s SUCCESS landice/dome/2km/init
       3.8s SUCCESS landice/dome/2km/sia/decomp/1proc
       4.1s SUCCESS landice/dome/2km/sia/decomp/4proc
       4.1s SUCCESS landice/dome/2km/sia/restart/full_run
       3.7s SUCCESS landice/dome/2km/sia/restart/restart_run
       8.0s SUCCESS landice/dome/2km/fo/decomp/4proc
       8.2s SUCCESS landice/dome/2km/fo/restart/full_run
       4.7s SUCCESS landice/dome/2km/sia/decomp/validate
       2.2s SUCCESS landice/dome/2km/sia/restart/validate
      11.0s SUCCESS landice/dome/2km/fo/decomp/1proc
       4.5s SUCCESS landice/dome/2km/fo/restart/restart_run
       3.0s SUCCESS landice/dome/2km/fo/decomp/validate
       1.6s SUCCESS landice/dome/2km/fo/restart/validate
Total runtime: 0:00:38
Step time was 83s, so this run did 2.2x the work of running one step at a time for as long.
PASS: All passed successfully!

Each step’s log is in case_outputs, and each task’s own work directory, such as landice/dome/2km/fo/decomp, has its steps’ output. The job also writes mali_pr_output_for_pr.md, a summary of the run to paste into a pull request.

Next steps

  • To test a MALI branch of your own, check it out in e3sm_submodules/MALI-Dev, or point --branch at another E3SM checkout, and set up the suite with --build (see Building MALI).

  • To compare your results with an earlier run, set up the suite with -b <earlier work directory> (see Suites).

  • The Quick Start for Developers covers the deployment and build options this tutorial skipped.