# Omega Pull Request Testing

date: 2026/09/29

Contributors:

- Xylar Asay-Davis
- Claude

## Summary

The Omega
[pull request template](https://github.com/E3SM-Project/Omega/blob/develop/.github/pull_request_template.md)
asks for the CTests and the Polaris `omega_pr` suite to pass on six machine
and compiler combinations, with the results recorded in a `Testing`
comment. For Omega#481, agents on four machines did this from a handoff
written for that pull request alone. This design turns that process into a
Polaris utility, `utils/omega/pr_testing`, that any developer can use.

The developer who wants the results, the requester, gives the work two
roles. An initiator pins the commits under test in a branch on the
requester's fork, then reports on linting and the documentation build. A
tester builds and runs both sides against that pin on each machine, then
reports its rows, including any new compiler warnings. By default, one agent
on the requester's own computer takes both roles and runs each machine's
commands over ssh, through connections the requester opens by logging in.
Frontier does not allow such connections, so an agent on Frontier tests its
rows. Scripts do the steps whose mistakes would pass unnoticed: choosing
the commits and producing the reports. Agent instructions cover the order of
setup and what to do when something fails, using the tools Polaris already
has.

Success means that a reviewer can check each item of the checklist from the
`Testing` comments alone: what was tested, against which baseline, and with
what result.

## Requirements

### Requirement: Any developer can request testing.

Nothing depends on one person's paths, fork, accounts or allocations.

### Requirement: Testing can start from the requester's computer or any supported machine.

### Requirement: GitHub credentials are needed on one computer only.

Pushing the test branch and posting comments need the requester's
credentials. Setting those up on every machine is a chore that should not
be needed.

### Requirement: Every machine tests the same commits.

### Requirement: The baseline is Polaris' Omega submodule unless the requester chooses otherwise.

Any other baseline, `develop` included, is stated in every report, with a
reason that rests on evidence, not a guess.

### Requirement: Each report carries its own provenance.

A report names the machine, compiler and build directories. It identifies
the Polaris commit and the baseline and pull request Omega commits, each
with a few commits of recent history.

### Requirement: Results are reported as Polaris and CTest write them.

Anything an agent adds is kept apart from them.

### Requirement: The requester controls every outward-facing action.

Pushing the test branch, submitting jobs and posting comments each need the
requester's permission.

### Requirement: Testers do not tick the checklist.

The requester ticks the boxes from the posted results.

### Requirement: Progress across machines is visible.

The requester can see which rows have results, which were not run and why,
and which results are for commits that are no longer current.

### Requirement: A matching baseline is reused wherever it was run.

Before building a baseline, the tester looks for an existing `omega_pr`
run that matches, whether or not this utility made it. A reused baseline is
reported with its own provenance, like any other.

### Requirement: Linting and the documentation build are checked once.

They do not depend on the machine, so they are checked once per pull
request head rather than once per row. A result is posted only if Omega's
CI does not already show it on the pull request.

### Requirement: Each row reports new compiler warnings.

A report lists the warnings from the pull request build that the baseline
build does not have. It says which of them are in files the pull request
changes.

## Algorithm Design

### Algorithm Design: Any developer can request testing.

The requester is usually the pull request's assignee, but may be its author
or another reviewer. Their agent runs in their own Polaris checkout, and
runs commands in their own checkouts on each machine. Settings come from a
config file the requester writes once on their computer and once per
machine, and the agent asks for any that are missing.

The agent instructions are committed in Polaris and reviewed like code. The
test branch carries data, not instructions.

### Algorithm Design: Testing can start from the requester's computer or any supported machine.

The initiator only does git work, so any computer with an Omega clone and a
Polaris environment will do. The initiator may also be the tester for any
machine it can run commands on.

### Algorithm Design: GitHub credentials are needed on one computer only.

The agent on the requester's computer runs the utility on each machine over
ssh, using a shared connection (`ControlMaster`) that the requester opens
by logging in, so the agent never handles a password or passcode. The
connection lasts as long as their login session, unless they choose to
keep it open longer (`ControlPersist`). The
utility runs on the machine as it would for an agent there; nothing in it
knows about ssh. The agent copies each report back unchanged and posts it
from the requester's computer, where `gh` is logged in.

An agent on a machine that the requester's computer cannot reach this way,
such as Frontier, tests that machine's rows itself. It posts with `gh` if
it is logged in there. Otherwise it gives the requester the report, which
the agent on their computer can post.

### Algorithm Design: Every machine tests the same commits.

The initiator makes the test commit by merging the pull request head into
the head of its base branch, as GitHub's test merge does. Any extra merges
the requester names, such as a build fix still in review, go on top. It
pushes two branches to the requester's fork:

```
omega-pr-test/<pr>-<head sha7>            test commit, then manifest commit
omega-pr-test/<pr>-<head sha7>-baseline   baseline commit
```

The tip of the first branch adds a single file, the manifest
`omega_pr_test.yaml`. Testers build its parent, so the manifest is not part
of the tested tree. Before building anything, a tester checks that the
parent contains the recorded pull request head and base branch head.

Only the initiator runs `init`, since testers may already be using the
branches. If the fork already has the test branch with a manifest that
pins the same pull request head, base, baseline, Polaris commits, merges
and rows, `init` leaves it alone rather than replace it with merges that
differ only in their hashes. Test and baseline commits are compared by
tree.

```yaml
schema_version: 1
requester: <GitHub user>
pull_request: 553
pr_head: <sha>
base_branch: develop
base_head: <sha>
test_commit: <sha>
baseline:
  commit: <sha>
  source: polaris-submodule   # or the ref the requester chose
  reason: null                # required unless polaris-submodule
  polaris_commit: <sha>       # polaris_commit by default
polaris_commit: <sha>
extra_merges:
  - {pull_request: 574, commit: <sha>, sides: [test]}
rows:
  - {machine: aurora, compiler: intel, mpi: mpich,
     template: "aurora, intel, mpich"}
notes: <free text for the requester>
```

Testers act only on the structured fields. `notes` is repeated in reports
and never followed.

An Omega pull request may need Polaris changes that the baseline Omega
cannot run, such as renamed config options. Then `polaris_commit` is a test
merge of the Polaris pull request, and the baseline runs from a second
Polaris checkout at `baseline.polaris_commit`, without those changes.

### Algorithm Design: The baseline is Polaris' Omega submodule unless the requester chooses otherwise.

By default, the baseline commit is the `e3sm_submodules/Omega` pin at
`baseline.polaris_commit`, which is normally Polaris `main` when the branch
is made. That pin states which Omega the Polaris code supports. So the
Polaris checkout the baseline runs from must pin the same Omega commit,
although it may be at a different Polaris commit, which is recorded.

The pin can lag far enough behind that differences would be hard to blame on
the pull request, as happened for Omega#481. In that case the requester can
choose another baseline, such as `base_head`. Doing so overrides the normal
practice, so the reason must rest on evidence: a merged PR's Testing comment
showing that it changed `omega_pr` results, or something checked that
suggests a merge may change them, stated as only that. A PR's title is not
evidence. `init` prints the reason in its summary, so the requester sees it
before the branch is pushed, and it goes in the manifest and in each report.

### Algorithm Design: Each report carries its own provenance.

The Omega commit is recorded when Omega is built, not when a suite is set
up. Otherwise a suite set up with `-p` on an existing build reports the
source tree's HEAD at setup time, which may have moved since the build (the
gap noted in #823, filed as #827 and fixed by #829).

`<suite>_output_for_pr.md` gains the full hash and the last five
first-parent commits of the Polaris checkout and of both Omega builds. The
baseline's come from the baseline work directory's `provenance` file. The
CTest utility writes a matching `ctest_output_for_pr.md` with the build
directory and Omega commit.

Human testers get this too, whether or not they use the utility.

### Algorithm Design: Results are reported as Polaris and CTest write them.

Each machine gets one comment, with Perlmutter's pm-cpu and pm-gpu rows in
one. A table at the top has a line for each row, generated from
`<suite>_results.json`, the CTest counts and the warnings comparison. Below
it, each row has a collapsed `<details>` block with its two `.md` files,
pasted unchanged, and its warnings section. After the results, the agent
may add a single `Notes` section for qualifications such as an extra merge
or a known failure. When an agent writes the report, it ends with the
signature that `AGENTS.md` prescribes.

### Algorithm Design: The requester controls every outward-facing action.

Each outward-facing step prints what it would do, and acts only when given
a flag: `init --push` and `setup --submit`. `lint` and `report` only write
their comments. The `post` command posts one, once someone has read it,
and is outward-facing in itself (see [Decisions](#decisions)). The agent passes one of these only when the requester has allowed
it in that session.
A manifest or any other document cannot grant permission.

Building on the login node is not outward-facing. `polaris suite --build`
initializes the submodules and builds. The two builds run one after the
other, because concurrent CIME configures race.

### Algorithm Design: Progress across machines is visible.

Each report starts with a hidden marker:

```
<!-- omega-pr-test {"schema": 2, "pr": 553, "pr_head": "<sha>",
     "test": "<sha>", "baseline": "<sha>",
     "rows": [{"row": "pm-cpu/gnu", "status": "complete", "result": "pass",
               "new_warnings": 0},
              {"row": "pm-gpu/gnugpu", "status": "not-run",
               "result": "none"}]} -->
```

The table uses the template's label for each row, so the requester can
match it to a box. A row that cannot be run, for example because its queue
is down, is reported with `"status": "not-run"`, and the table gives the
reason. A report for a machine that is down entirely is written on the
requester's computer.

### Algorithm Design: A matching baseline is reused wherever it was run.

The tester searches the roots the requester lists, and always the utility's
own `<work_base>/baselines`. A candidate is any directory with both a
`provenance` file and an `omega_pr_results.json`. A candidate matches only
if all of the following hold:

- It has the same machine, compiler and build type.
- Its Omega commit, as recorded at build time, is the baseline commit.
- Its Polaris commit is that of the checkout the baseline runs from, and
  neither tree was dirty.
- It was set up as the `omega_pr` suite, with no extra config file.
- Its results are complete, and no task failed or is pending.

A run whose Omega commit was recorded only at setup never matches. The
matcher lists the near misses and the test each one failed, so the agent
can tell the requester what almost matched.

If nothing matches, `setup` makes a new baseline in `<work_base>/baselines`.
A baseline there whose job is still queued or running becomes the pull
request suite's dependency. With the submodule as the default baseline,
pull requests tested against the same Polaris commit share one baseline per
row.

The warnings comparison needs only a build log. It can use the log from any
complete build of the baseline commit with the same machine, compiler and
build type, even when no suite run matches. A build is complete if it
started from an empty build directory, which the build-time record notes.

### Algorithm Design: Linting and the documentation build are checked once.

Omega's `omega-pr` workflow already runs pre-commit on the changed files and
`make html-strict` on each pull request, in its "lint and test docs" job.
When that job has passed on `pr_head`, the initiator tells the requester
and posts nothing, since the pull request's checks already show the
result. If the job has not run for `pr_head`, or has not passed, the
initiator runs the same two commands on the test commit, in an `omega_dev`
environment. It then posts a `Testing` comment that gives pre-commit and the
documentation build each its own result, passing or failing. The comment's
marker has `lint` as its only row.

Checking that the rendered documentation "looks as expected" still needs a
person.

### Algorithm Design: Each row reports new compiler warnings.

The baseline and pull request builds each leave a complete build log. A
warning is identified by its source path relative to the Omega tree, its
message and its warning flag, but not by its line number. A pull request
that adds lines above an existing warning therefore does not make that
warning look new. The report lists each identity that the baseline build
does not have, and marks the ones whose file is in
`git diff --name-only <base_head>...<pr_head>`. In those files, it also
lists an identity that occurs more often in the pull request build. In
other files, a higher count says nothing about the pull request's code: a
header's warning occurs once for each file that includes it, and a parallel
build can interleave the output of two compilers so that one occurrence
cannot be parsed.

With the submodule as the baseline, warnings added on `develop` since the
pin also appear as new. The mark separates those from the pull request's
own warnings.

Only complete build logs can be compared. When a build directory has to be
reused because an earlier build failed, `setup` rebuilds it with
`--clean_build` rather than building on top of the partial build.

## Implementation

### Implementation: Any developer can request testing.

`utils/omega/pr_testing/` holds:

- `AGENTS.md`: instructions for the initiator and tester roles, and the
  prompts the requester gives each one.
- `README.md`: the same workflow, written for people running the commands.
- `omega_pr_test.py`: the subcommands `init`, `lint`, `setup`, `report`,
  `post` and `status`, with the modules they use in `omega_pr_*.py`.
The modules that do not depend on Omega PRs are in `utils/pr_testing/`, as
`pr_test_*.py`, so that other PR testing utilities can share them: the
settings, git and GitHub helpers, the rows, setting up and submitting
suites and CTests, markers and the table of rows, status, and
`pr_test_warnings.py`, the build log parser.  `omega_pr_test.py` puts that
directory on `sys.path`.  The prefixes keep a module from shadowing a
standard one, such as `warnings`.

The settings for the requester's computer and each machine are in
`utils/pr_testing/example.cfg`, shared with other PR testing utilities.
Copy it to `~/.config/polaris_pr_test.cfg`, or pass another path with
`-f`:

```ini
[polaris_pr_test]
work_base = /path/to/scratch/polaris_pr_test
omega_repo = /path/to/a/clone/of/E3SM-Project/Omega
# needed by the initiator only
fork = git@github.com:<user>/E3SM.git
# needed by the initiator only if the PR needs Polaris changes
polaris_fork = git@github.com:<user>/polaris.git
# needed by the initiator only if Omega's CI has not passed
omega_dev_env = /path/to/conda/envs/omega_dev
# other places to look for a baseline to reuse, besides work_base
baseline_search_roots = /path/to/earlier/test/dirs
# a deployed Polaris checkout tracking main, which baselines of main run
# from (testers only)
polaris_main = /path/to/polaris/main

# on the requester's computer only: <ssh host>:<Polaris on the machine>
[machines]
chrysalis = chrys:/path/to/polaris
```

`init` ends by printing the prompt for an agent testing on a machine
itself, such as Frontier, which is the whole handoff and the same on every
machine. The utility's `AGENTS.md` shows its template, so a handoff never
needs anything added:

```
Test Omega PR 553 from branch omega-pr-test/553-54456ec on
git@github.com:xylar/E3SM.git, following utils/omega/pr_testing/AGENTS.md.
```

Each tester finds its rows with `status`, which names the manifest's rows
for the machine it runs on, or, on the requester's computer, the rows for
each machine. Perlmutter's pm-cpu and pm-gpu rows are tested from the same
login nodes.

When the PR needs Polaris changes, the prompt also names the Polaris
branches for the PR and the baseline, on `polaris_fork`.

### Implementation: Testing can start from the requester's computer or any supported machine.

```
omega_pr_test.py init --pr 553 [--baseline <ref> --reason <text>]
                      [--test-ref <ref>]
                      [--merge-pr <n>]... [--baseline-merge-pr <n>]...
                      [--polaris-ref <ref>] [--baseline-polaris-ref <ref>]
                      [--rows <machine>/<compiler>,...] [--push]
```

`init` asks GitHub for the pull request head and base branch, fetches them
from E3SM-Project/Omega, and makes the merges in a scratch worktree of
`omega_repo`. `polaris_commit` is the current `main` of
E3SM-Project/polaris, fetched rather than taken from the initiator's
checkout, unless `--polaris-ref` names another. `baseline.polaris_commit`
is `polaris_commit` unless `--baseline-polaris-ref` names another. If a
merge conflicts, `init`
stops, and the pull request's author must update the branch. Alternatively,
the requester can resolve the conflicts in a merge of their own, and
`--test-ref` names it, a branch on the fork or a commit. `init` checks that it
contains the pull request head and the base branch head and uses it as the
test commit. The default rows are the six in the template.

### Implementation: GitHub credentials are needed on one computer only.

`AGENTS.md` has the agent run each command on a machine with
`ssh -o BatchMode=yes <host> bash -l -s`, the commands on standard input.
Each command gets a fresh login shell, and `ssh` fails rather than prompting
when the requester's connection is gone. `setup` and `./deploy.py` run
under `nohup`, so that a dropped connection does not kill a build. The
`[machines]` section of the config file on the requester's computer names
each machine's ssh host and Polaris checkout. The utility checks that each
names a Polaris machine and has that form, and `status` shows them with
each machine's rows. Reports come back with `scp`,
and `post` posts them (see below, under results).

### Implementation: Every machine tests the same commits.

```
omega_pr_test.py setup --fork <fork> --branch <branch> [--submit]
                       [--baseline-dir <dir>]
                       [--baseline-load-script <script>]
                       [--no-baseline <reason>]
```

`setup` fetches both branches and reads the manifest. It stops with a
message unless all of these hold:

- it knows the manifest's schema version;
- the test commit is the tip's parent, and it contains `pr_head` and
  `base_head`;
- the Polaris checkout has no uncommitted changes to tracked files (not
  counting submodule checkouts, which the utility never builds from), and
  it contains `polaris_commit`;
- the Polaris checkout the baseline runs from pins `baseline.commit`, if
  the baseline is the submodule;
- if `baseline.polaris_commit` differs from `polaris_commit`, a second
  checkout, which is clean, contains `baseline.polaris_commit` and does not
  contain `polaris_commit`: by default the one in `polaris_main`, with its
  load script for the row, or the one whose load script the tester gave;
- the loaded Polaris environment belongs to this checkout
  (`POLARIS_BRANCH`), and its machine, compiler and MPI library give the
  row;
- the baseline checkout's load script is for the same row, or for the
  name the row's compiler had before Polaris renamed it.

Omega worktrees are made from `omega_repo` at `<work_base>/omega/<sha12>`.
Git keeps a worktree's submodule clones in the main clone's
`.git/worktrees/<name>/modules`, so `setup` refuses to make a worktree from
an `omega_repo` on another filesystem than `work_base`: a clone in home
would fill the home quota by several hundred megabytes per worktree.
They are keyed by commit, so baselines and pull requests share them.
`setup` makes each one and initializes the submodules its build needs
once, under a lock, and the build updates them only when they need it
(polaris#846), so that rows set up at the same time never update one
tree's submodules at once. Each
row gets its own directories:

```
<work_base>/pr<N>-<head sha7>/report_<machine>.md
<work_base>/pr<N>-<head sha7>/<machine>_<compiler>/
    build/       pull request build (-p)
    omega_pr/    pull request suite (-w, with -b <baseline>)
    ctest/       the CTest job script
    setup.json   what setup did, which report reads
<work_base>/baselines/<machine>_<compiler>_polaris-<sha7>_omega-<sha7>/
    build/
    omega_pr/
```

`setup` runs `polaris suite -c ocean -t omega_pr --model omega --build`
for the baseline if needed, then for the pull request. A baseline from a
second checkout is set up in a clean login shell with that checkout's load
script sourced, so its job script loads that checkout too. It then runs
`omega_ctest.py -p <pull request build>`. Last, it prints or submits three
jobs: the baseline suite, the pull request suite (`afterany` on the
baseline) and the CTests. It submits them with `polaris.job.submit_job()`,
which handles Slurm and PBS dependencies. On Frontier it adds
`--qos=normal`: the `omega_pr` suite asks for each machine's debug target,
and Frontier allows only one debug job at a time. It also adds
`--time=01:00:00`, since the GPU row's baseline ran past the suite's
30-minute wall time on Omega#524. On Aurora it adds
`-q capacity`, Aurora's queue for 1 to 16 nodes, and `-l walltime=01:00:00`,
since suites there ran past 30 minutes on slow nights. Each job waits for the
one before, because Aurora limits how many jobs a user may have queued, and
a job held on a dependency does not count. `utils/benchmark` keeps its own
copy of that code, because its driver runs outside any Polaris
environment.

A small table in the utility maps each template label to a Polaris machine,
compiler and MPI library. Another table gives the names compilers had
before Polaris renamed them, such as Aurora's `oneapi-ifx` and
`oneapi-ifxgpu` before E3SM-Project/polaris#835, so that a baseline can
run from a Polaris checkout older than the rename.

If the baseline cannot run on a row, such as an older Kokkos on a new GPU,
the requester may ask for `setup --no-baseline <reason>`. The PR suite then
runs without a baseline, and the report gives the reason.

### Implementation: Each report carries its own provenance.

#829, which fixed #827, provides the record. The Omega build template
writes `omega_source.txt` to the build directory before it runs `cmake`.
Both `polaris suite --build` and the CTest utility build through that
template. `polaris.build.source_record.read_source_record()` reads the
file. `provenance` gained `polaris git hash`, `polaris git log`,
`component git hash` and `component git log` entries. The component entries
are marked `(at setup, not build)` when the build has no record (see
{ref}`dev-provenance`).

On this branch, `_write_output_for_pull_request()` adds three lines, for
Polaris, the baseline Omega and the pull request Omega. It also adds a
`<details>` block with their logs.

`utils/omega/ctest/run_command.template` writes its summary to
`ctest_output_for_pr.md` as well as to standard output. It adds the build
directory and the Omega commit from `omega_source.txt`.

### Implementation: Results are reported as Polaris and CTest write them.

```
omega_pr_test.py report [--machine <machine>] [--notes <file>]
                        [--not-run <reason>] [--agent <name>]
```

`report` reports every row of the manifest on the machine, and on any
machine that shares its login nodes. It stops if a row has no results,
unless `--not-run` gives the reason those rows were not run.

```markdown
## Testing on aurora
<!-- omega-pr-test {...} -->

Tested the merge of E3SM-Project/Omega#553 (`<sha>`) into `develop`
(`<sha>`) as `<sha>`, against Polaris' Omega submodule (`<sha>`).

| Row | CTests | `omega_pr` | New build warnings |
|---|---|---|---|
| aurora, intel, mpich | 50 of 50 passed | 25 of 25 passed | 0 |
| aurora, intelgpu, mpich | 50 of 50 passed | 25 of 25 passed | 2 |

<details>
<summary>aurora, intel, mpich</summary>

<omega_pr_output_for_pr.md>

<ctest_output_for_pr.md>

### Build warnings

<generated by pr_test_warnings.py>

</details>

<details>...</details>

### Notes

<agent text, if any>

---

*Posted by <agent> on @<requester>'s behalf. ...*
```

`report` writes the comment to `report_<machine>.md` and prints it.
Someone reads it, and then `post` runs `gh pr comment <pr> -R
E3SM-Project/Omega --body-file report_<machine>.md`. A report written on a
machine without `gh` is copied unchanged to the requester's computer and
posted there:

```
omega_pr_test.py post --pr 553 <report_<machine>.md>
```

`post` refuses a file without a marker for that pull request, so it posts
only what `report` or `lint` wrote, on the right pull request.

`gh` is never required. Queries use it when it is installed and logged in,
and otherwise GitHub's REST API, which needs no login for a public
repository (a token in `GH_TOKEN` raises the rate limit). The requester is
the owner of `fork`. Only posting needs `gh`; without it, `post` stops,
and the comment is posted by hand.

### Implementation: The requester controls every outward-facing action.

`AGENTS.md` lists these flags. It says an agent passes one only when
the requester has allowed it in that session, and the same applies to any
`sbatch` or `qsub` the agent runs by hand. No command edits the pull request
body, and `AGENTS.md` tells testers not to edit it either.

### Implementation: Progress across machines is visible.

```
omega_pr_test.py status --pr 553
```

`status` reads the pull request's comments from GitHub and parses the
markers. It prints one line per row and one for `lint`, each with its
result, and marks as stale any
row whose `pr_head` is not the pull request's current head.

### Implementation: A matching baseline is reused wherever it was run.

The matcher lives in `polaris/baselines.py`, so that `utils/benchmark` can
later use it in place of its directory-name cache. `find_baseline()` takes
the search roots and the criteria. It returns the matching directories and
the near misses, each near miss with its reasons. `find_build_log()` does
the same for build logs. Candidates are looked for up to three levels below
each root. The suite and any config file are read from the `command` line
in `provenance`.

The config file gains `baseline_search_roots`. `setup --baseline-dir <dir>`
names a candidate directly, and the matcher still checks it. A build log
counts as complete when the source record's `clean_build` is true, which
means the build started from an empty build directory.

When `setup` submits a job, it records the job's id in a `job_id` file in
that job's work directory. For a baseline in `<work_base>/baselines` that
is not complete, `setup` looks the id up with `squeue` or `qstat`. The
answer decides whether to depend on the job or to stop.

### Implementation: Linting and the documentation build are checked once.

```
omega_pr_test.py lint --fork <fork> --branch <branch> [--notes <file>]
                      [--agent <name>]
```

`lint` reads the manifest, then finds the "lint and test docs" check run
on `pr_head`, whose id is its Actions job's. It reads the conclusions of
the job's "Run pre-commit" and "Build Sphinx Docs" steps from the Actions
API. If the job has not passed, `lint` puts `omega_dev_env` first on `PATH`
in a worktree of its own at the test commit, since the checks can change
files. It runs `pre-commit run --files <changed files>` at the root and
`make html-strict` in `components/omega/doc`, keeping both logs. Only
results from such a local run are to be posted.

The comment is headed `## Testing: lint and docs`, with one line each for
pre-commit and the documentation build. Each line says `passed` or `failed`
and whether it came from CI (with the job link) or from a local run (with
the log path).

### Implementation: Each row reports new compiler warnings.

Polaris already writes each build's output to `<build>/build_omega.log`.
`pr_test_warnings.py` reads three forms of warning:

```
<path>:<line>:<col>: warning: <message> [-W<flag>]    GCC, Clang, oneAPI
<path>(<line>): warning #<n>: <message>               NVCC, ifx
<path>:<line>:<col>:  ...  Warning: <message>         gfortran
```

It also reads `CMake Warning at <path>:<line>` blocks, using the first line
of the message. Each path is made relative to the Omega tree named in the
build's source record. Lines from `make` itself, such as
`jobserver unavailable`, are ignored.

The report's `Build warnings` section gives the count of new warnings and
how many are in files the pull request changes, then lists them in a
`<details>` block. It also gives both log paths. The marker gains
`"new_warnings": <n>`.

## Testing

### Testing and Validation: Any developer can request testing.

The first trial is Omega#553 on all four machines. A second requester then
repeats it, on at least one machine, from their own accounts.

The second trial is Omega#524, tested with a Polaris test merge of
polaris#731 and a baseline run from a second checkout without it.

### Testing and Validation: GitHub credentials are needed on one computer only.

A trial tests Chrysalis, Perlmutter and Aurora over ssh from one laptop,
with an agent on Frontier, and posts every report from the laptop. Unit
tests check that `post` refuses a file without a marker or with a marker
for another pull request.

### Testing and Validation: Every machine tests the same commits.

Unit tests build small git repositories in a temporary directory, with a
base branch, a pull request branch and an extra-merge branch. They run
`init` and the `setup` checks against these repositories. The cases include
a conflict, a tip whose parent is not the test commit, a Polaris
checkout that pins a different Omega, and a baseline checkout that is
missing or contains `polaris_commit`.

### Testing and Validation: Each report carries its own provenance.

Unit tests of `_write_output_for_pull_request()` use fixture `provenance`
files for a pull request work directory and a baseline work directory. A run
of `omega_pr` against a baseline checks the file from a real suite.

### Testing and Validation: Results are reported as Polaris and CTest write them.

Unit tests build reports from fixture `.md` files and check that both files
appear byte for byte.

### Testing and Validation: Progress across machines is visible.

Unit tests parse markers from fixture comments, including a stale report,
a not-run row and a comment from before a machine's rows shared one.

### Testing and Validation: A matching baseline is reused wherever it was run.

Unit tests search fixture work directories containing an exact match, one
near miss for each test, a run whose Omega commit was recorded only at
setup, and a run with incomplete results. They look up build logs from a
clean build and from an incremental one. The #553 trial reuses a baseline
on at least one machine.

### Testing and Validation: Linting and the documentation build are checked once.

Unit tests mock the check runs for a passed check, a failed check and a
missing check, and verify which path `lint` takes and that results from CI
are never posted. The #553 trial runs `lint` for real.

### Testing and Validation: Each row reports new compiler warnings.

Unit tests parse fixture logs taken from real builds with GCC, Clang,
oneAPI and NVCC. They check three cases: a warning that only moves lines is
not new, a second instance of an existing warning is new, and a warning in
a changed file is marked. The #553 trial checks the section on every row.

## Decisions

- **Scripts do the pinning and reporting, and agent instructions do the
  rest.** The tools that build and run, `polaris suite --build` and
  `omega_ctest.py`, are already scripts. The problems on Omega#481 were of
  two kinds. Mistakes in the handoff, such as initializing submodules by
  hand and building inside jobs, are fixed by instructions that are
  reviewed. Problems in the environment, such as Omega#572 and the
  out-of-date CTest mesh, needed judgment that a wrapper script could not
  supply. The mistakes that would go unnoticed, such as testing the wrong
  commits or retyping a result, are the ones that are scripted.
- **The test commit is the merge into the base branch, not the pull request
  head**, because the merge is what lands.
- **The test branch lives on the requester's fork.** Recreating the merge
  on each machine would avoid a push, but it could not carry conflict
  resolutions or extra merges.
- **One agent drives the machines over ssh, by default.** For Omega#481
  and the first trials of this utility, an agent ran on each machine. Each
  needed its own handoff and, to post, its own `gh` login, and the
  requester relayed between them. Driving E3SM-Unified's deployment on
  Chrysalis, Perlmutter and Aurora from one laptop over ssh needed only the
  requester's usual logins. The utility itself stays on the machines, so
  an agent on a machine runs the same commands, which Frontier still
  needs.
- **Nothing is posted without being read first.** `report` and `lint`
  only write their comments, and `post` posts one once an agent or a
  person has read it. A flag that posted straight away would let a
  comment reach the pull request unchecked.
- **Each machine gets one comment.** One tester runs all of a machine's
  rows, so its comment needs no coordination with other machines. A
  comment for each row, as the first trials posted, repeated the same
  commits and paths for every row and was hard to read.
- **Baselines are found from provenance, not from `utils/benchmark`'s
  cache or by an agent.** The cache sees only directories that `benchmark`
  named itself, and most existing baselines were set up some other way. An
  agent reading provenance by hand can miss a field that matters, such as a
  dirty tree or a commit recorded at setup rather than at build time.
- **Linting and the documentation build come from Omega's CI when it has
  passed, and are then not posted.** Running them locally every time would
  repeat the same commands on nearly the same tree, and would need an
  `omega_dev` environment on every initiator's machine. A comment repeating
  CI's passing checks adds nothing the pull request does not already show.
- **Polaris changes an Omega pull request needs are a Polaris test merge,
  and the baseline runs without them.** Omega#524 renamed the options
  every task writes, so its Polaris changes (polaris#731) cannot run the
  baseline Omega. Testing the pull request with Polaris `main` is not
  possible either.
- **The template's compiler names are mapped where Polaris differs.**
  Omega has called Aurora's compilers `intel` and `intelgpu` since
  E3SM-Project/Omega#566, and the template follows. Polaris used
  `oneapi-ifx` and `oneapi-ifxgpu` until E3SM-Project/polaris#835, so the
  utility mapped those rows until then. The names now match, and only a
  baseline from an older Polaris checkout needs its former names.