Task Parallelism Phase A: Placement
Creation date: 2026/08/23
Contributors:
Xylar Asay-Davis
Claude
Summary
Today, when Polaris runs a step that uses MPI, it launches it across the whole allocation. That is correct when only one step runs at a time, and it is the single largest obstacle to ever running two steps at once: two steps that each believe they own the machine will either collide or, more often, queue behind one another.
Phase A gives Polaris the ability to run a step on a named part of its allocation – these nodes, these cores, these GPUs – and gives steps a way to describe what they need, in terms the batch system can act on.
Phase A does not run anything concurrently. At the end of it, Polaris still executes one step at a time and produces identical results. What changes is that the machinery for saying “run this here” exists and is known to work on every machine Polaris supports.
Most of the work is not in Polaris. Polaris does not build the command that
launches parallel work; mache does, through its ParallelSystem classes.
mache is a contract shared with other software, so the launcher change must
land there first. That change has its own design document in the mache
repository; this document describes what Polaris needs from it and what
changes on the Polaris side.
Success in Phase A means Polaris can construct a correct launch command for a
step confined to a given subset of the allocation, on Slurm and on PBS, for
CPU-only and GPU steps, and that polaris serial behaves exactly as before.
Requirements
Requirement: Confine a Step to Part of the Allocation
Date last modified: 2026/08/23
Contributors:
Xylar Asay-Davis
Claude
Polaris shall be able to launch a step so that it runs on a specified subset of the allocation’s nodes and cores, rather than on all of them.
The subset shall be expressible in terms Polaris already understands: which nodes, how many cores, and how many GPUs. Polaris shall not need to know machine-specific syntax in order to express it.
Requirement: Isolation Enforced by the Batch System
Date last modified: 2026/08/23
Contributors:
Xylar Asay-Davis
Claude
When two steps are confined to non-overlapping subsets, the batch system shall be what keeps them apart, not Polaris.
We prototyped an alternative in which Polaris pinned processes itself using CPU affinity masks. It works, but it is a weaker guarantee: a step that ignores its mask is not prevented from doing so, and the mechanism differs per machine. Measurements showed the batch system can do this properly on every supported machine, so it should.
The exception is machines running Slurm older than 20.11, such as Chrysalis, where the necessary options do not exist. There, Polaris shall fall back to explicit CPU binding, and shall record that it has done so.
Requirement: A Step’s GPU Need Is Always Stated Explicitly
Date last modified: 2026/08/23
Contributors:
Xylar Asay-Davis
Claude
A Polaris step uses no GPUs unless it declares otherwise. That premise does not change, and most steps will continue to declare nothing.
What must change is that Polaris shall state a step’s GPU need to the launcher in every case, including when it is zero. Passing nothing is not the same as passing zero: the batch system treats an unstated GPU requirement as a claim on the node’s GPUs and reserves all of them, which prevents any other step from starting. Because most Polaris steps use no GPUs, explicitly requesting none is the ordinary case rather than a special one.
A step that does want GPUs shall declare how many the step needs, not
how many each MPI rank needs. This differs from how CPU resources are
described today, where ntasks and cpus_per_task are per-rank
quantities, and the difference is not cosmetic: measurements on both GPU
machines showed the per-rank form does not confine a step at all, while a
per-step total does.
Requirement: Steps Declare Memory
Date last modified: 2026/08/23
Contributors:
Xylar Asay-Davis
Claude
A step shall be able to declare how much memory it needs, as a target and a minimum, in the same style as its existing CPU requirements.
Memory is unlike cores and GPUs in that the batch system will not keep it for us. Asking a launch for a share of the node’s memory was measured to reserve nothing, so nothing Polaris passes to a launcher will make room for a step or tell Polaris whether one fits. Memory is therefore a budget Polaris keeps itself: the only way one step’s memory is protected from another’s is that Polaris declines to start the second step. That decision is Phase B’s, and so is the accounting behind it.
What a memory figure passed to a launcher does do, on machines whose Slurm postdates the 20.11 change, is cap the launch: a step allowed 1024 MB and told to take 4 GB is killed at 960 MB on Perlmutter GPU and on Frontier, while the same launch runs to completion on Chrysalis. So such a figure is an enforced ceiling on some machines and inert on others, and is never a reservation anywhere. Whether Polaris should impose that ceiling is taken up in Phase B, and the answer there turns on the distinction the next paragraph draws.
What Phase A shall provide is the declaration, its default, and its visibility to the step – and shall keep a declared figure distinguishable from a defaulted one, because later phases treat them differently. A step that says a number has made a claim about itself and can be held to it. A step that says nothing is being estimated by the framework, and must not be held to the framework’s estimate. Anything that reads a step’s memory needs to be able to tell which it has.
A step that declares nothing shall be treated as needing memory in proportion to the cores it asked for – the node’s memory divided by its cores, times the step’s cores. This is deliberately the value that makes memory-aware packing arithmetically identical to packing on cores alone, so that introducing memory can never make Phase B schedule worse than it would have without it, and so that no per-step number has to be invented for the steps that exist today. Steps that have been measured, which is mostly the analysis work in Phase C, override the default with a real figure and are then scheduled on it.
A declared figure is therefore a ceiling as well as a claim, wherever the machine can hold a step to one, and a step author should declare a peak with margin rather than a typical value. A figure that describes what a step usually needs will kill it on the day it needs more, on the machines that enforce and not on the others.
An under-declaring step running beside others is a real failure mode, and one that did not exist when Polaris ran a step at a time. Phase B addresses it, and the declared-versus-defaulted distinction is what lets it do so without turning the framework’s own estimate into a death sentence.
Device memory needs no declaration of its own. GPUs are never divided between steps, so a step given a GPU is given that GPU’s memory with it, and the GPU count already accounts for it.
Host memory on a GPU machine is the case where the proportional default is most likely to be wrong, and a step author should expect to declare rather than rely on it. A step that wants every GPU on a node and only a handful of cores to drive them gets, by the proportional rule, only that handful’s share of the node’s memory – which is unlikely to be what it needs to stage data for the devices. Making the default depend on GPUs as well would destroy the property that makes it safe, since it is exactly its being proportional to cores that makes memory-aware packing reduce to packing on cores. The default is a floor for steps nobody has measured, not an estimate anyone should trust for a step whose memory has nothing to do with its core count.
Requirement: Resource Views Describe What a Step Can Use
Date last modified: 2026/08/23
Contributors:
Xylar Asay-Davis
Claude
The resource information given to a step shall describe the resources that step can actually use.
A step confined to one node shall be told about one node’s resources, and a step whose work may span nodes shall be told about all of the resources it was given. Telling a step that runs its work in its own process about cores on nodes it will never reach invites it to size itself wrongly; telling a step that distributes its work about one node’s worth understates what it has. The view shall describe how a step’s resources are distributed, not only how many there are, since that is the part a step sizing itself needs and the part a single number cannot carry.
Any resources withheld from a step shall be genuinely withheld, not merely subtracted from a number.
Memory shall be part of that view. It is the one resource where the number a step is told is the only thing standing between it and the node’s limit, because nothing below Polaris will stop it, and it is the number a step that sizes something for itself – a worker pool, a chunk size – has to size against. A step told its cores and its memory can do that; a step told only its cores cannot, and the natural mistake is to derive memory from cores, which is wrong in exactly the case that matters.
Requirement: A Step Says Whether Its Resources May Span Nodes
Date last modified: 2026/08/23
Contributors:
Xylar Asay-Davis
Claude
Whether a step’s resources may be drawn from more than one node shall be a property the step declares, and shall not be inferred from whether the step uses MPI.
This covers cores and GPUs together, as one property rather than two. A step that reaches other nodes reaches them by one mechanism, and that mechanism carries whatever the work needs; a step that cannot reach them cannot reach either. No case has been identified that wants the two to differ, and splitting them would invite a step to claim GPUs on nodes its cores cannot reach.
MPI and spanning are not the same question, and treating them as the same is what this requirement exists to prevent. An MPI step spans nodes because its launcher spreads its ranks. A single-process Python step that does its work in its own threads cannot span nodes, because there is no mechanism by which it would reach them. A single-process Python step that hands its work to a distributed worker pool spans nodes perfectly well – the pool is exactly the mechanism the thread-based step lacks – and it does so while remaining one process asking for one task. “Not MPI” covers the last two cases, which want opposite answers.
Such a step’s workers may use GPUs as readily as cores, and when they do, the GPUs come from the nodes the workers are on rather than from the node the step’s own process happens to occupy. Nothing about a step being single-process confines its GPUs to one node once its work is somewhere else.
Steps that do not say shall be treated as confined to a node, which is what every non-MPI step in Polaris is today. This is also the safe direction: a request that a node can satisfy is satisfiable on any allocation that could have satisfied a larger one.
Where a step cannot be given what it asked for, the existing target-and-minimum rule shall decide what happens, with the node boundary as one more thing that can make a request unsatisfiable. A step confined to a node may be reduced silently towards its target, exactly as it may be today, because a step that names a minimum has said in advance which reductions are acceptable. A step whose minimum cannot be met within a node shall be an error when the run is set up, naming what it needs, what a node holds, and the property that would let it span – not quietly reduced to what fits. This applies to a step asking for more GPUs than a node has exactly as it applies to cores.
This is deliberately the rule Polaris already follows, extended rather than replaced, and it leaves today’s steps alone. The two steps that currently ask for more cores than some machines have per node both name a minimum of one, so they are reduced as they always were. What changes is only that the bound is the allocation for a step that may span, and that a step that may not span and cannot fit says so instead of shrinking in silence.
Requirement: Portability Across Supported Machines
Date last modified: 2026/08/23
Contributors:
Xylar Asay-Davis
Claude
Placement shall work on all machines Polaris supports for parallel work: Slurm systems both older and newer than the 20.11 change in how job steps reserve resources, and PBS systems using the PALS launcher.
Where a machine cannot support placement, Polaris shall detect this and say so, rather than silently running steps on the whole allocation.
Requirement: No Change to Existing Behavior
Date last modified: 2026/08/23
Contributors:
Xylar Asay-Davis
Claude
polaris serial shall behave exactly as it does today. A step that does not
ask to be confined shall be launched as it is now.
Two of the requirements above change what a step is told or permitted rather than how it is launched, and both are constructed to leave existing steps where they are: the memory default reproduces core-only packing exactly, and the node-span rule reduces to today’s target-and-minimum behavior for every step now in Polaris. Neither should show up as a difference in a run.
This requirement is also what the migration of existing non-MPI steps onto the new fields is checked against. A step restated in different words shall receive the same resources and produce the same outputs, and any step for which that is not true has been restated wrongly.
Algorithm Design
Algorithm Design: What Placement Means to the Launcher
Date last modified: 2026/08/23
Contributors:
Xylar Asay-Davis
Claude
The launcher should accept an optional description of where a step is to run, alongside the existing task and CPU counts. In substance that description is:
the nodes the step may use;
how many cores it may use on each;
how many GPUs it needs in total, which is normally none.
All three are always present. There is no “unspecified” for GPUs, because an unspecified GPU requirement is what the batch system reads as a claim on all of them.
Each supported system can express all three, though they say it differently. On newer Slurm it is a matter of asking for exactly the resources requested rather than inheriting the job’s; on PBS with PALS it is a host list plus an explicit core list; on older Slurm, where the modern options do not exist, it is an explicit CPU mask, which is the fallback noted above.
The important design point is that Polaris should describe the placement, never the flags. Which flags implement it is the launcher’s business, and they differ enough between machines – and between Slurm versions on the same kind of machine – that leaking them into Polaris would spread machine-specific knowledge through the scheduler.
Algorithm Design: Describing Step Resources
Date last modified: 2026/08/23
Contributors:
Xylar Asay-Davis
Claude
Polaris steps today describe CPU needs per MPI rank: ntasks how many ranks,
cpus_per_task how many cores each, with min_tasks and min_cpus_per_task
giving the scheduler room to run a step smaller when resources are tight.
That target-and-minimum pattern is a good one and should be kept.
GPUs should be added as a per-step total with a matching minimum, for the reason given in the requirements, defaulting to none so that the great majority of steps need say nothing and still get an explicit “no GPUs” passed to the launcher on their behalf.
Memory follows the same target-and-minimum pattern but does not follow the same path afterwards. GPUs go to the launcher because the launcher acts on them; memory goes to the scheduler and to the step, because those are the only two things that act on it. It is worth being explicit that this is not an omission: a memory figure rendered into a launch command would be decoration, and worse than decoration, because it would suggest an enforcement that does not happen.
Memory’s default differs from the GPU default in kind. “No GPUs” is a true statement about a step that uses no GPUs. There is no equivalent true statement about memory – every step uses some – so the default has to be an assumption, and the assumption chosen is the step’s proportional share of the node: cores requested, times the node’s memory divided by the node’s cores. Its merit is not that it is accurate for any particular step; it is that a run in which every step defaults packs exactly as a run with no memory accounting at all, so the mechanism is inert until someone supplies a measured number.
An alternative was considered and rejected: converting memory into an equivalent number of cores and packing on cores alone, so that a step needing a large fraction of the node’s memory reserves a matching fraction of its cores. It has the appeal of resting on the one resource the batch system does enforce. But it over-reserves whenever a step’s ratio of memory to cores differs from the node’s, and it does so worst for steps that want much memory and few cores, which is precisely the analysis work that motivated declaring memory at all. It would also hand such a step more cores than it asked for, so a step that sizes a worker pool from its cores would respond to needing more memory by creating more workers with less memory each. Tracking memory as its own quantity avoids both, and is consistent with the decision already made for GPUs, which this design likewise declines to express in CPU-shaped terms.
GPUs need no separate treatment for non-MPI steps, and it is worth saying so
rather than leaving it to be inferred. A per-step total is already the shape
a single-process step wants, so the same gpus and min_gpus serve both
kinds of step and nothing further is required. The measurement that forced
GPUs into that shape – a per-rank count does not confine a launch – happens
to have put them where a step with no ranks can use them. Cores are the
exception rather than GPUs being an omission: they are described per rank
for historical reasons, and are the one resource a non-MPI step therefore
cannot state.
Non-MPI steps are worth calling out for that reason. A single-process Python step has no meaningful “number of ranks”; what it has is a number of cores it can use and an amount of memory it needs. Expressing that through MPI-shaped fields is how Polaris ends up telling a Python step it has 192 cores across three nodes. Non-MPI steps should describe cores and memory directly, and their GPUs through the same per-step total every other step uses.
That matters more once such a step is allowed to span nodes. A step that wants two hundred cores from a distributed pool is, in MPI-shaped terms, one task with two hundred CPUs each – a sentence no launcher can act on, since one process cannot be given two hundred cores on a node that has a hundred and twenty-eight. Written directly, as two hundred cores that may come from several nodes, it says something true and actionable. The MPI-shaped fields are not merely awkward here; they cannot express the case at all.
Algorithm Design: Reservations Are Not Always Placements
Date last modified: 2026/08/23
Contributors:
Xylar Asay-Davis
Claude
For most steps, the resources Polaris reserves and the resources it confines the step to are the same set, described once and used twice. An MPI step given ninety-six cores is launched on those ninety-six cores; reserving them and placing them are one act.
Two cases in this design separate them. Memory is reserved and never placed, because no launcher acts on it. A step that delegates its work to a distributed pool is the mirror image: what Polaris launches is a driver process needing about one core, while the cores the step claims – and the GPUs, where its workers use them – are consumed by pool workers, which are separate launches, started elsewhere, and shared with other steps. Placing two hundred cores on that driver would confine them to a process that will not use them, while the workers that will are somewhere else entirely, and placing GPUs on it would reserve devices on the one node whose work is smallest.
The distinction to carry forward is that placement is what the launcher acts on and a reservation is what the scheduler tracks. They coincide for ordinary steps. Where they do not, the scheduler’s accounting is the one that has to be right, because it is what stops the machine being oversubscribed; the placement is only ever a description of where a particular launch goes.
Phase A implements no delegation – there is no pool until Phase C, and no scheduler until Phase B. What Phase A must not do is build the two ideas as one thing, because separating them afterwards means revisiting every place that assumed a step’s cores and its launch describe the same set.
Algorithm Design: Detecting What a Machine Supports
Date last modified: 2026/08/23
Contributors:
Xylar Asay-Davis
Claude
Placement support is a property of the machine and of the version of its batch system, and it must be decided at run time rather than baked into configuration, because the same machine can change underneath us.
The launcher should determine, once per run, which placement mechanism applies, and report it. Three outcomes matter: full placement with batch-system enforcement; placement by explicit CPU binding, on older Slurm; and no placement, which Polaris should treat as “concurrency is not available here” rather than as an error.
Implementation
Implementation: The mache Side
Date last modified: 2026/08/23
Contributors:
Xylar Asay-Davis
Claude
mache.parallel.ParallelSystem.get_parallel_command() gains an optional
placement argument, and each subclass renders it appropriately. This is the
contract change, and it must land in mache before Polaris can depend on
it. Its design is Design Document: parallel placement, at
docs/design/parallel_placement.md in the mache repository. It is a
separate repository, so this cannot be a cross-reference.
That work was mache pull request #470, which adds a
ResourcePlacement type and the optional argument, renders it for Slurm
both before and after 20.11, for PBS with PALS and for a single node, adds
an optional memory cap, and tests the rendering against every shipped
machine config. Its condition for merging was that Polaris testing first
confirm the rendered commands behave as intended on real machines, so it and
Phase A unblocked each other and were worked on together. It is merged,
and released as mache 3.12.0.
The sequencing that release was held to is worth recording, because it was
deliberate and the alternative was easy to drift into. The order was:
Polaris testing confirms the rendering and measures each machine’s memory,
mache takes both, then merges and releases, Polaris pins that release, and
only then does Phase A land. Folding the memory corrections into the same
release mattered because the alternative is shipping estimates and
correcting them in a second release that nothing forces anyone to make.
That held: 3.12.0 carries the placement work, the memory cap, and the
surveyed memory_per_node figures together.
While the work was unreleased, Polaris deployed against the pull request
branch, which deploy.py already supported and which therefore needed no
change to Polaris’s deployment machinery:
./deploy.py --mache-fork xylar/mache --mache-branch parallel-placement ...
deploy.py and deploy/cli_spec.json are contract files shared with
mache and must not be edited in Polaris to accommodate anything of this
kind. That was always a temporary state, and it ends here: Phase A must not
merge while depending on an unreleased branch, and no longer has to.
Polaris shall require mache 3.12.0 or later, and shall fail clearly if an
older one is present. Those are two separate mechanisms and it is worth
saying which does what.
The version requirement belongs in the deployment pin, which is what governs
any environment built the ordinary way. The check in the running code is a
backstop for an environment that was not, and it shall test the capability
– whether this mache accepts a placement at all – rather than compare
version numbers. The capability is the thing that matters, and a version is
only a proxy for it: a proxy that would reject a development build or a fork
that can place, and accept a release that could not.
What either guards against is the reason they exist: a run that silently lost placement would appear to work while oversubscribing the machine, which is the worst failure available here – no error, wrong results, and slower than serial.
Implementation: The Polaris Side
Date last modified: 2026/08/23
Contributors:
Xylar Asay-Davis
Claude
Stepgainsgpusandmin_gpusas per-step totals. The existinggpus_per_taskshould be deprecated in favor of the total, since it does not do what its name suggests when steps run concurrently.Stepgainsmemoryandmin_memory.Stepalready carries amax_memoryattribute, documented as a placeholder for task parallelism and unused by anything. This is that placeholder being redeemed, not a second mechanism beside it, and the existing attribute should be reconciled rather than left alongside. Its units, megabytes, are worth keeping; its name is not, since what a step declares is what it needs and not a ceiling it is held to.Stepgains a way to say how many cores and how much memory a non-MPI step needs, without going through MPI task counts.Stepgains a property saying whether its resources – cores and GPUs alike – may be drawn from more than one node, false by default for a non-MPI step and true for an MPI one.constrain_resources()uses it in place of the node-sized cap it applies today: a step that may span is bounded by the allocation, and a step that may not and asks for more than a node holds is an error rather than a silent reduction. Nothing in Polaris sets the property to true in Phase A; it exists so that Phase C does not have to remove a rule.No non-MPI GPU field is added, because
gpusandmin_gpusalready are one. Deprecatinggpus_per_taskis what completes this: it is the only GPU field with a shape a non-MPI step cannot use.The non-MPI steps that exist today are moved onto the new fields, as a commit of its own at the end of the Phase A series rather than mixed into the framework change. Nothing forces this: the two spellings mean the same thing for a step confined to a node, and the framework should accept either. It is done in Phase A anyway because Phase A is the only phase whose acceptance criterion is that behavior does not change, which is precisely how a translation of this kind is checked, and because leaving both spellings live means the Phase B scheduler is the first thing to meet them – where a misread declaration shows up as a packing bug rather than as a wrong number on a page.
That migration is smaller than it sounds and needs more judgment than it sounds. Of the non-model steps that declare resources, most name one core and one task, which is the default and can simply go. Four genuinely want to state cores directly. The remainder set
ntasks=1while being MPI steps that happen to run at width one – the WOA23 steps, the topography remapping step, and the shared mapping-file step – and moving those would be wrong, sincentasksis the field that means what they mean. This has to be decided per step rather than swept, and is the reason it is its own commit and not a mechanical pass.The code that builds a step’s parallel command passes a placement through to
machewhen one has been assigned, and passes none when it has not, preserving today’s behavior.The resource information handed to a step is built from its placement, so that a confined step sees only what it was given, and includes memory alongside cores, nodes and GPUs.
The per-node memory a machine has becomes a
[parallel]configuration option inmache, beside thecores_per_nodeandgpus_per_nodethat are already there. That is a separate and much smaller change than pull request #470: it describes a machine rather than altering an interface, and it does not touchResourcePlacement. It should be a configured quantity rather than one read from the running node, both because Polaris needs it before a compute node is in hand and because what belongs in it is the memory a job may actually use, which is not what the operating system reports.Nothing about memory reaches
mache’sResourcePlacement. That type describes where a launch runs – which nodes, which cores, which GPUs – and every field in it is rendered into the launch command. Memory is not a location and does not belong there.The enforcement question that was open when this was first written has since been answered, and machines do honor a memory request, so a cap does now reach the launcher – as a separate optional argument to
get_parallel_command()rather than as a field on the placement. Keeping the two apart is deliberate: a placement still says only where a launch runs, and a cap is a different kind of statement that the signature should distinguish rather than blend.machereports separately whether a machine can enforce one, so Polaris can say in a run log that caps are not enforced here rather than quietly believe it has a safety net.Polaris passes that cap for a step that declared a memory figure, and omits it entirely for a step carrying the proportional default, which is what makes the argument optional rather than always supplied. This is the only place the declared-versus-defaulted distinction reaches outside Polaris, and it is the reason Phase A must keep the two separable rather than resolving the default eagerly into one number.
Implementation: Boundaries
Date last modified: 2026/08/23
Contributors:
Xylar Asay-Davis
Claude
Nothing in Phase A decides which subset a step should get. There is no scheduler yet; the only caller is the existing serial path, which assigns no placement. Keeping that boundary makes Phase A reviewable on its own and means a mistake in it shows up as a wrong command rather than as a wrong schedule.
Testing
Testing and Validation: Command Construction
Date last modified: 2026/08/23
Contributors:
Xylar Asay-Davis
Claude
Unit tests shall check the command built for a given placement on each supported system, including: a step confined to one node, a step spanning several, a step needing GPUs, a step needing none, and a step with no placement at all, which shall produce today’s command unchanged.
These tests need no allocation and shall run in ordinary CI.
Testing and Validation: Real Machines
Date last modified: 2026/08/23
Contributors:
Xylar Asay-Davis
Claude
The mechanisms Phase A relies on were measured on Chrysalis, Perlmutter (CPU
and GPU), Frontier and Aurora before this design was written; the results
are summarized in Task Parallelism in Polaris. Those
measurements used commands written by hand. Phase A shall be validated by
showing that the commands mache renders produce the same behavior on
each of those machines, which is a different claim and the one that gates
merging mache pull request #470.
That validation does not require the rest of Phase A. Building placements,
rendering them through get_parallel_command() and launching them
concurrently is enough to confirm the rendering, and keeping it separate
means a failure is attributable to mache rather than ambiguous between
mache and Polaris’s placement construction. It should be done first, on
all five machines.
A single confined step shall be run on each machine and shall report that it sees only the cores and GPUs it was given. This is the check that catches a placement which is constructed correctly but not honored, and it is the first thing that would break if a site changed its scheduler configuration.
One combination that neither the harness nor the unit tests reach has been checked separately, because it is the one Phase B relies on most. The harness confines synthetic payloads; the unit tests give a step a placement and inspect what would be launched. Neither runs a real step under a placement Polaris built from that step’s own declarations. That was done once, on Chrysalis, with an Omega forward step declaring four tasks of one core:
given |
rendered launch |
ran |
|---|---|---|
no placement |
|
yes |
four cores |
|
yes |
two cores |
|
yes |
The third row is the one that establishes anything. A step asking for four tasks came out as two, on a node holding sixty-four cores, which is only possible if the resources it was constrained against were the placement rather than the allocation. The second row cannot show that, because four is what the step asked for either way – a distinction worth keeping in mind if this is ever repeated, since the obvious test is the uninformative one.
The placed and unplaced runs at the same width produced bit-identical output, which is the other thing worth confirming: a placement changes where a step runs and not what it computes. This was the cpu-binding fallback rather than the reservation path, Chrysalis running Slurm 20.02, so it exercised the weaker of the two mechanisms.
Something of all this shall keep running after Phase A, but not the harness or the scaffold that produced it. Both exist to answer a question and are not part of what Polaris ships; they are removed once the question is answered, and anything worth keeping has to be moved somewhere permanent before that rather than after. Two things are worth keeping, and neither needs an allocation to be remembered.
The first is a test that needs no batch system at all. A single-node
launcher confines a launch with ordinary process affinity, which is enough
to build several disjoint placements, render each through mache, run them
at once and read back what each could see. That exercises the rendering
end to end and runs anywhere, so it belongs with the ordinary tests and
guards against a rendering regression on every commit. It cannot tell
whether a real scheduler honors a placement, which is the other tier.
The second is not a test at all. Phase A assigns no placements – the serial path passes none, deliberately – so any check here has to construct one for itself, which makes it a diagnostic rather than an observation of Polaris doing its job. From Phase B onwards every placed step has a placement to verify, and a step that checks it received what it was given costs almost nothing and runs on every machine on every run. That is the standing check on real machines, and it should replace this one rather than sit beside it.
What must not happen is an allocation-gated test that exists and is never
run. A check that requires someone to remember to run pytest inside a job
is a check that reports nothing for as long as nobody does.
While those machines are being visited, two further things should be settled, because both are cheap there and expensive to guess at. The first has been answered and its result is recorded below; the second has not. The question was whether a memory request is inert. The evidence for assuming so showed only that memory was not the cause of the serialization, which is a different claim. The test was direct – launch a step with a small allowance, have it allocate several times that, record whether it is killed – and it has been run:
machine |
1024 MB allowed, 4 GB requested |
|---|---|
Chrysalis (Slurm 20.02) |
reached 4 GB, exited 0 – not enforced |
Perlmutter GPU (Slurm 25.11) |
killed at 960 MB |
Frontier (Slurm 25.11) |
killed at 960 MB |
A memory request is real on newer Slurm and inert on older. What follows for the design is in the requirement above and in Phase B.
The second consequence that was anticipated does not occur, and can be retired rather than carried forward. By the argument that applies to GPUs, a step that says nothing about memory might have been read as claiming all of it. It is not: on both machines that enforce, four concurrent launches that said nothing about memory started within 40 ms of each other and ran their full duration. Aurora and Perlmutter CPU remain unmeasured.
What is still open, and matters more than what was closed, is whether placement itself imposes a memory ceiling. The measurement above passed an explicit allowance. Placement on newer Slurm asks for exactly the resources a step needs rather than the job’s, and memory is evidently a resource the scheduler tracks there, so a placed step may be given a share of the node’s memory it never asked for and be killed at it – with Polaris having said nothing about memory and its own accounting unaware of any limit. That would make placement impose a hidden ceiling that bites only the memory-hungry steps, which is the failure shape this design most wants to avoid. The check is a placed launch given no memory request at all, allocating hard, with an unplaced launch beside it to separate what placement causes from what the allocation causes. It should be added to the machine runs that remain.
Testing and Validation: Measuring Each Machine’s Memory
Date last modified: 2026/08/23
Contributors:
Xylar Asay-Davis
Claude
The per-node memory mache reports is a number nobody has been able to
measure. Whoever adds memory_per_node to the machine configs is not on
those machines, and the figure that matters – what a job may actually
allocate – is not what a vendor specification says, not what site
documentation says, and not what a login node reports. The values that ship
with mache will be estimates.
Polaris’s validation is the only occasion on which anyone is on all five of
those machines with a reason to look, so measuring the real values is part
of it. For each of Chrysalis, Perlmutter CPU, Perlmutter GPU, Frontier and
Aurora, a job shall record what the batch system says a node has, and the
corrections shall be returned to mache as a change to its machine configs.
Two figures are worth recording rather than one, because they answer
different questions. What the scheduler believes a node has is what a
scheduler would pack against, and is what the config option should hold: on
Slurm that is the node’s real memory as scontrol or sinfo reports it,
and on PBS it is the equivalent node attribute. What a process can actually
allocate before it is refused is the figure that decides whether the first
is honest. Where the two disagree, the smaller one is the one Polaris must
not exceed, and the disagreement is itself worth reporting.
Until a machine has been measured its value should be treated as provisional, and estimates should err low: a figure that is too high costs a job killed for exhausting a node, while one that is too low costs only work that could have been packed. Those are not comparable, so the unverified direction to be wrong in is downwards.
This is a small task with an unusual failure mode: it is easy to omit and
nothing fails when it is. A run with an over-estimated node memory looks
entirely normal until a suite happens to pack tightly enough to exhaust a
node, at which point the failure appears far from its cause and looks like a
step’s bug. Recording which machines have been measured, in the mache
configs themselves, is what makes the omission visible.
A measurement is only as good as the label on it. An early reading appeared to show Perlmutter CPU returning nodes of two different sizes within one machine, and this document briefly recorded that as a finding. It was not one. The small node was a Perlmutter GPU node: the run had been labelled from the deployment’s machine name rather than from the node it landed on, and three hardware signals agree it was misfiled – the job held four GPUs, its hyperthread siblings were 64 apart rather than 128, and its memory matched a GPU node. Perlmutter CPU is homogeneous, and its two genuine samples agree.
Aurora, however, is heterogeneous, and that is measured rather than inferred. Its scheduler was asked what every one of its 10,624 nodes has, which is a survey and not a sample: about 89% report roughly 1135 GiB, 11% report roughly 1007 GiB, and a handful sit between. The smallest is 1,030,518 MB against a median of 1,162,460 – so a figure taken from the majority would be about 13% too high for one node in nine, and a step packed against it would be over-admitted whenever it landed on a small one.
The two episodes are worth reading together, because they point the same way for opposite reasons. The Perlmutter reading was withdrawn because two samples of different machines were compared as though they were one machine. The Aurora figure stands because nothing was sampled at all – every node was asked. The rule that survives both is the one already stated: where nodes differ the smallest binds, and a single sample cannot establish the number.
The episode is worth keeping for what it says about the measurements
generally, which is that they are labelled by configuration and confirmed by
nothing. A harness that records what a deployment claims, while the node it
ran on says otherwise, will mislabel silently and produce a finding that
looks like a property of the machine. Any measurement taken this way should
cross-check the label against the hardware – GPUs present where the
configuration says there are none, or a physical core count that disagrees
with cores_per_node – and refuse to record a result it cannot vouch for.
Two precautions stand regardless, since neither depended on that finding. Where nodes differ, the smallest binds, because the figure exists to say what a caller must not exceed and a value above the smallest node over-admits whenever work lands on one – the direction that kills a job rather than wasting it. And a single sample cannot establish the number: most measurements taken so far are single samples, and the reliable method costs no allocation, being to query every node in a partition and take the minimum rather than asking whichever node one happens to be on.
A node that turns up with unexpectedly little memory shall be treated as a finding rather than a curiosity, and reported. It means a configured figure is wrong, or a machine has changed, or – as here – that something is not the machine it was thought to be. All three are things to learn from a message rather than from an exhausted node.
The configured figure plans; the allocation’s own nodes account. These are two uses with different requirements and only one of them has to be predicted in advance. Sizing a job script happens before any node has been assigned, so it can only use a configured number. Deciding what fits happens inside the allocation, where the nodes are known and can simply be asked what they have – which is better than any configured value, because it is the truth for the nodes in hand rather than an estimate of a machine.
Polaris shall therefore read the memory of the nodes it was actually given, once, at the start of a run, and the scheduler shall account against those figures rather than against the configured one. Where they disagree the disagreement shall be reported, because it means the configuration is stale or the machine has changed, and neither should be discovered silently.
This does not make the configured figure unnecessary; it makes it a planning estimate rather than a promise, which is what it can honestly be.
It also answers the class of problem the mislabelled run belongs to, rather than the specific one. A run that reads what its nodes actually report, and says so when that disagrees with what was configured, catches a stale figure, a machine that has changed, and a job that is not on the machine it was thought to be – without anyone having to anticipate which. The mislabelling above went unnoticed because nothing compared the two; the comparison is cheap and belongs in the run rather than in a later query.
Testing and Validation: No Regression
Date last modified: 2026/08/23
Contributors:
Xylar Asay-Davis
Claude
A representative suite shall be run with polaris serial and compared
against a baseline, to confirm that adding the placement machinery has not
changed behavior when placement is not used. Since Phase A adds no
concurrency, any difference in results is a bug rather than a trade-off.