# Polaris Version Update Testing

date: 2026/10/14

Contributors:

- Xylar Asay-Davis
- Claude

## Summary

A pull request that updates Polaris to a new version, such as #664 or #835,
usually changes the Spack environments and often the submodules. Its
checklist asks for the MPAS-Ocean `mpaso_pr` suite, and for the Omega
CTests and `omega_pr` suite, on every supported machine, compiler and MPI
library. The checklist is run twice: first with test Spack environments,
then with the shared environments deployed for the new version.

This design adds `utils/polaris_update_testing`, a utility for that
testing. It reuses the code in `utils/pr_testing` that the Omega PR testing
utility shares (see {doc}`omega_pr_testing`). The new parts are the rows,
the step that deploys Spack environments, and reports on the Polaris pull
request.

Success means that #835's deployment can be tested and reported with the
utility, and that a reviewer can tick each box of its checklist from the
`Testing` comments alone.

## Requirements

### Requirement: Every supported row is tested.

The rows are every machine, compiler and MPI library that the new version
supports, for MPAS-Ocean and for Omega.

### Requirement: Each report says which Spack environments it tested.

A report says whether it tested test environments or the shared ones, and
where they are.

### Requirement: The requester controls deploying Spack environments.

Deploying shared environments changes what every Polaris user on a machine
loads.

### Requirement: A baseline is optional.

When there is one, it is the previous version, from Polaris `main`, with
its shared environments.

### Requirement: Each machine's results are reported in one comment.

### Requirement: Progress across machines is visible.

The requester can see which rows have results with each kind of
environment, which were not run and why, and which results are for a
commit that is no longer the pull request's head.

### Requirement: Commits, provenance and permissions are handled as for Omega pull requests.

The requirements of {doc}`omega_pr_testing` hold, except those about
linting and the Omega pull request template: every machine tests the same
commits, each report carries its own provenance, results are reported as
Polaris and CTest write them, the requester controls every outward-facing
action, testers do not tick the checklist, and a matching baseline is
reused wherever it was run.

## Algorithm Design

### Algorithm Design: Every supported row is tested.

The rows come from `docs/developers_guide/supported_machines.yaml` at the
pull request's head. Each row is a machine, compiler, MPI library and
model. An MPAS-Ocean row runs `mpaso_pr`. An Omega row runs the CTests and
`omega_pr`. A machine and compiler that support both models give two rows.

Omega is built from the pull request's `e3sm_submodules/Omega` pin, in the
Omega worktrees the Omega utility makes. MPAS-Ocean is built from the
`e3sm_submodules/E3SM-Project` submodule of the Polaris checkout.
MPAS-Ocean builds inside its source tree, so the rows that share a tree
build one at a time (see [Decisions](#decisions)).

### Algorithm Design: Each report says which Spack environments it tested.

The environments are read from each row's load script, not set by the
requester. A load script whose Spack environment is under the machine's
`[deploy] spack` path uses the shared environments. Any other uses test
environments. The comment's heading names the kind, and the report gives
the path, the Polaris version and the mache version, and says if mache
came from a branch rather than a release.

> **Rationale:** The report states what was loaded, so it cannot claim the
> shared environments when a test deployment was sourced by mistake. #835
> starts with the shared environments, since its test environments were
> already tested by hand.

### Algorithm Design: The requester controls deploying Spack environments.

`deploy` prints the `./deploy.py --deploy-spack` command for a machine. It
deploys the compilers and MPI libraries of the machine's supported rows at
the pull request's head. With a test path, it adds
`--spack-path`. With a mache fork and branch, for a mache that is not
released yet, it adds `--mache-fork` and `--mache-branch`. On Perlmutter, pm-cpu and pm-gpu each get their own
command. Only with `--run` does `deploy` run the command, under `nohup`,
with its output in a log. Afterward, `deploy` checks that each of the
machine's rows has a load script in the checkout, and that the script
uses the environment just deployed.

### Algorithm Design: A baseline is optional.

`init --baseline main` pins the current `main` of E3SM-Project/polaris as
the baseline. The baseline runs from a second Polaris checkout that
contains that commit and not the pull request's head. It uses that
checkout's load scripts, and so the previous version's shared
environments. Omega and MPAS-Ocean come from that checkout's submodule
pins.

A row whose compiler the pull request renames uses the former name for the
baseline, as in the Omega utility. A row the previous version does not
support runs without a baseline, and its report says so.

Rows that the current `main` cannot run, such as Aurora's Omega rows after
mache 5.1.0 (#896), get their own manifest, with `init --baseline <older
commit> --rows ...`. A manifest whose baseline is not `main` adds the
baseline's short hash to its test branch and run directory, so it does not
collide with the manifest for the other rows.

Baseline differences can come from the environments, from the submodule
updates or from Polaris changes in the pull request. Each report names
the commits on both sides, so the tester can tell which changed.

### Algorithm Design: Each machine's results are reported in one comment.

Each machine gets one comment for each kind of environment, on the Polaris
pull request. Perlmutter's pm-cpu and pm-gpu rows share one, as in the
Omega utility. A table has a line for each row. Below it, each row has a
collapsed block with the files Polaris and CTest wrote, pasted unchanged.

### Algorithm Design: Progress across machines is visible.

Each comment's marker, prefixed `polaris-update-test`, lists its rows and
the kind of environment. `status` lists each row once for each kind of
environment, with its latest result.

### Algorithm Design: Commits, provenance and permissions are handled as for Omega pull requests.

`init` pins the pull request's head, its Omega and E3SM-Project pins and
the rows in a manifest. The manifest is committed on top of the head, in a
branch on the requester's Polaris fork. Testers fetch that branch. Their
Polaris checkout must contain the head, with this utility merged in
locally if the head does not have it.

The outward-facing steps are `init --push`, `deploy --run`,
`setup --submit` and `post`. Each needs the requester's permission in the
agent's session, as in the Omega utility. `report` only writes its comment,
and `post` posts it once someone has read it.

## Implementation

### Implementation: Every supported row is tested.

The shared `Row` gains an optional `model`, which the Omega utility leaves
out. `pr_test_setup` builds MPAS-Ocean as well as Omega:

- `polaris_suite()` already takes the suite and model. For MPAS-Ocean it
  passes no `--branch`, so the build uses the checkout's submodule.
- A lock on the E3SM-Project submodule, like the lock on an Omega
  worktree, makes rows build one at a time.
- A finished MPAS-Ocean build is recognized by its `ocean_model`.
- `BaselineCriteria.build_type` may be `None`, since MPAS-Ocean's
  provenance has no build type.

Only Omega rows set up CTests and compare build warnings.

```
polaris_update_test.py init --pr 835 [--baseline main] [--rows ...]
                            [--merge-omega-pr <number>]...
                            [--merge-e3sm-pr <number>]...
                            [--notes <text>] [--push]
```

`--merge-omega-pr` merges an Omega pull request, such as a build fix still
in review, into the Omega the pull request pins. `--merge-e3sm-pr` merges an
E3SM pull request after any Omega ones, such as an update to E3SM `master`
that Omega has yet to merge. The merge is pushed to the requester's E3SM
fork, recorded in the manifest's `omega_merge`, and built by every Omega row
in place of the pin. Checkouts are still checked against the pin. MPAS-Ocean
rows build the E3SM the pull request pins.

The manifest, `polaris_update_test.yaml`:

```yaml
schema_version: 1
requester: <GitHub user>
pull_request: 835
pr_head: <sha>
polaris_version: 1.2.0-alpha.1
omega_commit: <sha>
e3sm_commit: <sha>
omega_merge:            # or null
  commit: <sha>         # built in place of omega_commit
  pull_requests: [{pull_request: 600, commit: <sha>}]
  e3sm_pull_requests: [{pull_request: 9000, commit: <sha>}]
  fork: <the requester's E3SM fork>
  branch: polaris-update-test/835-<sha7>-omega
baseline:               # or null
  polaris_commit: <sha>
  polaris_version: 1.1.0-alpha.7
  omega_commit: <sha>
  e3sm_commit: <sha>
rows:
  - {machine: chrysalis, compiler: gnu, mpi: openmpi, model: mpas-ocean}
  - {machine: chrysalis, compiler: gnu, mpi: openmpi, model: omega}
notes: <free text for the requester>
```

### Implementation: Each report says which Spack environments it tested.

`setup` reads the Spack environment from the loaded load script and
compares it with the machine's `[deploy] spack` path. It records the kind
of environment, `shared` or `test`, with the path and the versions, in
the row's `setup.json`. mache came from a branch if pip installed the
loaded version from a directory, as `./deploy.py --mache-fork` does; a
conda package's metadata names conda as its installer. Row directories are kept apart by kind:

```
<work_base>/polaris-pr<N>-<head sha7>/<shared|test>/
    report_<machine>.md
    <machine>_<compiler>_<model>/
        build/  <suite>/  ctest/  setup.json
```

### Implementation: The requester controls deploying Spack environments.

```
polaris_update_test.py deploy --fork <fork> --branch <branch>
                              [--machine <machine>] [--spack-path <path>]
                              [--recreate] [--mache-fork <org>/mache]
                              [--mache-branch <branch>] [--run]
```

The compilers and MPI libraries come from the rows in
`supported_machines.yaml` at the pull request's head, all of them, even if
the manifest tests only some. `AGENTS.md` tells an agent to ask before
`--run` on each machine, and to say whether it deploys test or shared
environments. With permission, `deploy --run` deploys the shared
environments as well as test ones (see [Decisions](#decisions)).

### Implementation: A baseline is optional.

```
polaris_update_test.py setup --fork <fork> --branch <branch> [--submit]
                             [--baseline-load-script <script>]
                             [--no-baseline <reason>]
```

`setup` sets up every row the loaded load script tests. The baseline runs
from the checkout of `main` that the settings' `polaris_main` names, with
its load script for the same compiler, or for the compiler's name before a
rename, as in the Omega utility. `--baseline-load-script` names another. `setup` checks that the
checkout contains the manifest's baseline commit and pins its Omega and
E3SM-Project commits. The baseline is found or set up with the shared
`get_baseline()`, so an Omega baseline that the Omega utility ran from the
same `main` is reused.

### Implementation: Each machine's results are reported in one comment.

```
polaris_update_test.py report [--machine <machine>] [--spack <kind>]
                              [--notes <file>] [--not-run <reason>]
                              [--agent <name>]
```

```markdown
## Testing on chrysalis with the shared Spack environments
<!-- polaris-update-test {...} -->

Tested E3SM-Project/polaris#835 (`<sha>`), Polaris 1.2.0-alpha.1 with
mache 5.0.0, Omega `<sha>` and E3SM `<sha>`, with the shared Spack
environments in `<path>`. The baseline is `<sha>`, Polaris 1.1.0-alpha.7,
Omega `<sha>` and E3SM `<sha>`, with its shared Spack environments, so
differences can come from any of these.

| Row | Suite | CTests | New build warnings |
|---|---|---|---|
| chrysalis, gnu, openmpi (MPAS-Ocean) | `mpaso_pr`: 24 of 24 passed | | |
| chrysalis, intel, openmpi (Omega) | `omega_pr`: 29 of 29 passed | 59 of 59 passed | 0 |
| chrysalis, gnu, openmpi (Omega) | `omega_pr`: 29 of 29 passed | 59 of 59 passed | 0 |

<details>...</details>
```

The comment uses the shared `format_rows()`, `assemble()` and
`deliver()`, with `E3SM-Project/polaris` as the repository.

### Implementation: Progress across machines is visible.

```
polaris_update_test.py status --pr 835 [--fork <fork> --branch <branch>]
```

`status` uses the shared `get_row_entries()` and `format_status()`. A row
is named `<machine>/<compiler>/<model>/<shared|test>`.

### Implementation: Commits, provenance and permissions are handled as for Omega pull requests.

`utils/polaris_update_testing/` holds `AGENTS.md`, `README.md`,
`polaris_update_test.py` and its `polaris_update_*.py` modules. `AGENTS.md`
links to the Omega utility's `AGENTS.md` for the rules and the ssh steps
the two share, rather than repeating them. The settings come from
`~/.config/polaris_pr_test.cfg`, the Omega utility's file, so the two
share `work_base`, its Omega worktrees and its baselines.

## Testing

### Testing and Validation: Every supported row is tested.

Unit tests read the rows from a fixture `supported_machines.yaml`, and set
up an MPAS-Ocean row with a stand-in for `polaris suite`. They check that
two MPAS-Ocean rows from one tree wait on its lock.

### Testing and Validation: Each report says which Spack environments it tested.

Unit tests read fixture load scripts with a Spack environment under the
shared path and under a test path.

### Testing and Validation: The requester controls deploying Spack environments.

Unit tests build the `deploy` command from fixture machine configs,
including Perlmutter's two and one with a mache branch, and check that
nothing runs without `--run`. Others tell a mache from a branch from
conda packages and pip installs of a release.

### Testing and Validation: A baseline is optional.

Unit tests set up a row with and without a baseline, and with a baseline
load script that uses a compiler's former name.

### Testing and Validation: Each machine's results are reported in one comment.

Unit tests build a comment with MPAS-Ocean and Omega rows, and check that
the files appear byte for byte.

### Testing and Validation: Progress across machines is visible.

Unit tests parse markers for both kinds of environment.

### Testing and Validation: Commits, provenance and permissions are handled as for Omega pull requests.

The trial is #835's deployment, against a `main` baseline, on Chrysalis,
Perlmutter and Aurora over ssh and with an agent on Frontier.

## Decisions

- **A separate utility that shares code with the Omega one.** An update
  pull request is on another repository, has MPAS-Ocean rows, deploys
  Spack environments and needs no lint step. One command line for both
  would put two workflows in one `AGENTS.md`.
- **`deploy --run` does the deployment, shared environments included.**
  Handing the command back for the requester to run would defeat the
  utility. The requester's permission for each machine is what guards the
  shared environments.
- **The two utilities share one settings file**,
  `~/.config/polaris_pr_test.cfg`, which replaced the Omega utility's
  `~/.config/omega_pr_test.cfg`.
- **The pull request's head is tested, not a merge into `main`.**
  Deployment runs from the pull request's branch, and an update pull
  request is brought up to date with `main` before it merges.
- **The kind of environment is read from the load scripts**, so a report
  states what was loaded (see the algorithm design for each report saying
  which Spack environments it tested).
- **`deploy` covers only the compilers and MPI libraries of supported
  rows.** A machine's config can list combinations that are no longer
  supported, such as Frontier's `crayamd` in #835, or that do not build,
  such as pm-gpu's `intelgpu` (E3SM-Project/mache#526). The first trial
  deployed everything the config listed and stopped on both.
- **MPAS-Ocean builds in the checkout's submodule, one row at a time.**
  An E3SM worktree for each row would cost a large checkout for builds
  that take minutes.
- **New build warnings are compared for Omega rows only, at first.** The
  parser and build log names are Omega's. MPAS-Ocean can follow.
- **No lint step.** Polaris's own CI checks the pull request.