Testing an Omega pull request

The Omega pull request template asks for the CTests and the Polaris omega_pr suite to pass on each supported machine and compiler, and for the results to be recorded in Testing comments. The utility in utils/omega/pr_testing does this, run by a developer or by an AI agent. Its README.md describes the commands, and its AGENTS.md holds the instructions agents follow.

In outline, an initiator pins what to test in a branch on the requester’s fork and reports on linting and the documentation build. A tester builds and runs the baseline and the PR on each machine, then posts one comment for the machine’s rows: a line for each row, and the files Polaris and the CTests wrote, with the commits of Polaris and both Omega builds and any new compiler warnings. By default, one agent on the requester’s own computer is both, running each machine’s commands over ssh through the connections the requester opens by logging in. Frontier does not allow such connections, so an agent on Frontier tests its rows.

The baseline is the Omega commit Polaris pins in e3sm_submodules/Omega, unless the requester chooses another, develop included, with a reason that rests on evidence rather than a guess. If the PR needs Polaris changes that the baseline Omega cannot run, the PR is tested with a Polaris test merge and the baseline runs from a second Polaris checkout without those changes. An existing baseline run is reused when its provenance matches (see polaris.baselines.find_baseline()), which relies on the record of the source that Polaris’s build scripts write (see Provenance). A baseline with failed tasks is still reused, since running it again would not fix them. The report says how many tasks failed in the baseline and does not count a task as a baseline difference or a failure just because the baseline has no output to compare it with.

To start, the requester copies utils/pr_testing/example.cfg to ~/.config/polaris_pr_test.cfg on their computer and on each machine, fills it in, logs in to each machine, and asks an agent in a Polaris checkout on their computer something like:

Test Omega PR 553 with utils/omega/pr_testing, following
utils/omega/pr_testing/AGENTS.md.

init prints the prompt to give an agent on Frontier, which works out its own rows.

The modules it shares with other PR testing utilities are in utils/pr_testing.