Developer Tutorial: Running the mali_pr Suite on Perlmutter
This tutorial takes you from no Polaris at all to a finished run of the mali_pr suite on Perlmutter. Along the way, you clone Polaris, deploy its environment, let Polaris build MALI, and submit the suite as a batch job. You can use Perlmutter’s CPU nodes (pm-cpu) or its GPU nodes (pm-gpu); the steps are the same apart from a few names, which are given for both.
You need:
a NERSC account in the
e3smproject (or another project you can charge jobs to; see Submitting the job)an SSH key on GitHub that you can use from Perlmutter, since Polaris and its submodules are cloned with SSH
the
bashshell (see Unix Shell)
Cloning Polaris
Log in to Perlmutter and clone Polaris:
ssh <username>@perlmutter.nersc.gov
git clone git@github.com:E3SM-Project/polaris.git
cd polaris
git submodule update --init jigsaw-python e3sm_submodules/MALI-Dev
Polaris keeps the model source code in submodules under e3sm_submodules.
You only need two of them here: jigsaw-python, which the deployment builds,
and MALI-Dev, the source code for MALI.
Deploying the Polaris environment
Polaris runs in a pixi environment that ./deploy.py
creates in your clone. MALI also needs compiled libraries, including Albany
and Trilinos for its first-order velocity solver. These are in shared Spack
environments on Perlmutter, and --with-albany points Polaris to the ones
that have Albany.
For pm-cpu:
./deploy.py --machine pm-cpu --compiler gnu --with-albany
For pm-gpu:
./deploy.py --machine pm-gpu --compiler gnugpu --with-albany
The first deployment takes a while, since it installs pixi and builds the
environment. Do not add --deploy-spack: that rebuilds the Spack
environments rather than using the shared ones, which takes hours.
When it is done, you have a load script in the root of your clone. Source it whenever you start a new shell to work with Polaris:
# pm-cpu
source load_polaris_pm-cpu_gnu_mpich.sh
# pm-gpu
source load_polaris_pm-gpu_gnugpu_mpich.sh
Polaris pixi and spack environments, compilers and system modules says more about deployment and its options.
Setting up the suite
Choose a work directory on Perlmutter’s scratch file system, where the suite will be built and run, and set up the suite there:
# pm-cpu
polaris suite -c landice -t mali_pr -m pm-cpu \
-w $SCRATCH/polaris_work/mali_pr_pm-cpu
# pm-gpu
polaris suite -c landice -t mali_pr -m pm-gpu \
-w $SCRATCH/polaris_work/mali_pr_pm-gpu
Polaris builds MALI from the MALI-Dev submodule before it sets up the
tasks, so this step takes several minutes. The build goes in
landice_build in the work directory, and its log in
landice_build/build_mali.log. When setup finishes, it lists the four tasks
in the suite:
Setting up tasks:
landice/dome/2km/sia/decomp
landice/dome/2km/sia/restart
landice/dome/2km/fo/decomp
landice/dome/2km/fo/restart
Running polaris suite again in the same work directory reuses the build.
Add --build to rebuild MALI after you change its source code.
Submitting the job
Setup writes a job script, job_script.mali_pr.sh, to the work directory.
Submit it:
cd $SCRATCH/polaris_work/mali_pr_pm-cpu # or mali_pr_pm-gpu
sbatch job_script.mali_pr.sh
The job script charges the e3sm project. To charge another project,
put its name in a config file, for example my_account.cfg:
[parallel]
account = <project>
then add -f my_account.cfg to the polaris suite command above and
set up the suite again.
The suite runs in a few minutes once the job starts. You can check on the
job with squeue --me.
Checking the results
The job’s output is in polaris_mali_pr.o<jobid> in the work directory. The
suite’s steps run at the same time where they fit on the node, so the output
ends with a summary of the steps (in the order they finished) and the result
of the suite:
Step runtimes:
23.9s SUCCESS landice/dome/2km/init
3.8s SUCCESS landice/dome/2km/sia/decomp/1proc
4.1s SUCCESS landice/dome/2km/sia/decomp/4proc
4.1s SUCCESS landice/dome/2km/sia/restart/full_run
3.7s SUCCESS landice/dome/2km/sia/restart/restart_run
8.0s SUCCESS landice/dome/2km/fo/decomp/4proc
8.2s SUCCESS landice/dome/2km/fo/restart/full_run
4.7s SUCCESS landice/dome/2km/sia/decomp/validate
2.2s SUCCESS landice/dome/2km/sia/restart/validate
11.0s SUCCESS landice/dome/2km/fo/decomp/1proc
4.5s SUCCESS landice/dome/2km/fo/restart/restart_run
3.0s SUCCESS landice/dome/2km/fo/decomp/validate
1.6s SUCCESS landice/dome/2km/fo/restart/validate
Total runtime: 0:00:38
Step time was 83s, so this run did 2.2x the work of running one step at a time for as long.
PASS: All passed successfully!
Each step’s log is in case_outputs, and each task’s own work directory, such
as landice/dome/2km/fo/decomp, has its steps’ output. The job also writes
mali_pr_output_for_pr.md, a summary of the run to paste into a pull
request.
Next steps
To test a MALI branch of your own, check it out in
e3sm_submodules/MALI-Dev, or point--branchat another E3SM checkout, and set up the suite with--build(see Building MALI).To compare your results with an earlier run, set up the suite with
-b <earlier work directory>(see Suites).The Quick Start for Developers covers the deployment and build options this tutorial skipped.