Task Parallelism Phase D: Coexistence and Elasticity
Creation date: 2026/08/23
Contributors:
Xylar Asay-Davis
Claude
Summary
Phase C gives the node residents and their workers a fixed share of the allocation for the life of the run. Once the Python work is finished, they go on holding that share, and those nodes sit idle while model runs queue for resources that are right there.
Phase D makes the share change with the work. When Python work is ready, the pool grows; when it drains, it gives nodes back.
This is the phase that makes a mixed suite – model runs and analysis in one job – use the machine properly. It is also the phase most likely to be unnecessary. If suites do their analysis at the end, after the model runs, then a pool that appears late and holds resources until the job ends costs nothing. Test that before building this.
Success in Phase D means a suite mixing model runs and analysis finishes in close to the time the work itself requires, with no long stretch where part of the allocation is reserved but idle.
Requirements
Requirement: The Pool Gives Resources Back
Date last modified: 2026/08/23
Contributors:
Xylar Asay-Davis
Claude
When there is less ready Python work than the pool is sized for, the pool shall shrink and the freed resources shall become available to other steps.
When the Python work is finished entirely, the pool shall release everything.
Requirement: The Pool Grows When Needed
Date last modified: 2026/08/23
Contributors:
Xylar Asay-Davis
Claude
When ready Python work exceeds what the current pool can handle, and resources are free, the pool shall grow.
Growing shall not take resources from steps that are already running.
Requirement: Changes Are Not Constant
Date last modified: 2026/08/23
Contributors:
Xylar Asay-Davis
Claude
The pool shall not resize continuously in response to small fluctuations.
Adding and removing workers is not free. An earlier design that switched execution modes at every opportunity spent a significant fraction of its wall time doing so. The scheduler shall require a change to be worth making before making it.
Requirement: A Rule for Competing Work
Date last modified: 2026/08/23
Contributors:
Xylar Asay-Davis
Claude
When ready model steps and ready Python work both want the same free resources, Polaris shall apply a stated rule rather than whichever the loop happens to reach first.
The rule shall be simple enough to explain in a sentence and shall be recorded in the run’s event stream when it is applied, so that a schedule that looks wrong can be understood.
Requirement: Resizing Is Safe
Date last modified: 2026/08/23
Contributors:
Xylar Asay-Davis
Claude
Shrinking the pool shall not interrupt work already running on the workers being removed.
A worker shall be allowed to finish what it holds before it is stopped, and the resources it occupied shall not be offered to another step until it has actually gone.
Requirement: Elasticity Is Visible
Date last modified: 2026/08/23
Contributors:
Xylar Asay-Davis
Claude
Every resize shall be recorded, with what triggered it and what it cost, and the run summary shall report how much of the allocation was idle and for how long.
Idle time is the quantity this phase exists to reduce, so it must be measurable before and after.
Algorithm Design
Algorithm Design: Deciding the Size
Date last modified: 2026/09/09
Contributors:
Xylar Asay-Davis
Claude
The scheduler already holds the dependency graph, so it knows not only what Python work is ready now but how much is still to come. The pool’s target size should be the number of workers the ready Python work can actually use, capped by what the allocation can spare.
Resizing should happen in whole nodes. A pool spanning three nodes releasing “half a node” leaves a fragment too small to run a model step in, which is the shape of free resource that looks available and is useless.
Two guards keep this from thrashing. A change should have to be worth more than it costs, and a resize should not be followed immediately by its opposite.
The cost is not uniform, and the difference shapes the policy. Adding a worker to a node that already has a resident is a fork, measured at about a tenth of a second. Adding a node means launching a resident there, which pays the Polaris import – tens of seconds. So growing within nodes the pool already holds is nearly free, and growing onto new nodes is not. Shrinking should give up whole nodes for the same reason: a node given up has to be paid for again.
Algorithm Design: When Both Kinds of Work Are Waiting
Date last modified: 2026/08/23
Contributors:
Xylar Asay-Davis
Claude
The rule proposed is: prefer the work on the critical path, and break ties in favour of model steps.
Model steps tend to be longer and to have more work depending on them, so starting one late delays more. Python work tends to be shorter and more plentiful, which makes it good for filling gaps. The scheduler can estimate the critical path from the graph, so this is not guesswork.
This is a proposal, not a conclusion. It should be revisited once there is a real mixed suite to measure, and the alternative – simple first-ready ordering – should be measured against it rather than assumed worse.
Algorithm Design: Shrinking Without Interrupting
Date last modified: 2026/08/23
Contributors:
Xylar Asay-Davis
Claude
Shrinking should be cooperative: mark the workers to be removed as accepting no new work, wait for what they hold to finish, then stop them and return their resources to the pool. The scheduler must not count those resources as free until the workers have actually exited, or it will place a step on top of one still shutting down.
Implementation
Implementation: What Changes
Date last modified: 2026/09/09
Contributors:
Xylar Asay-Davis
Claude
Phase D changes the pool module from Phase C and the scheduler loop from Phase B, and adds nothing structurally new:
the residents gain grow and drain operations, and a notion of workers that are finishing but not yet gone;
the resource pool distinguishes resources that are free from resources that will be free shortly;
the scheduler consults a target pool size each time round the loop, and applies the competing-work rule when both kinds are ready.
Implementation: Order of Work
Date last modified: 2026/08/23
Contributors:
Xylar Asay-Davis
Claude
Shrinking should be implemented before growing. Shrinking is what recovers the idle resources that motivate the phase; growing is an optimisation on top of it, and a pool that only ever shrinks is still correct.
Testing
Testing and Validation: Resize Behavior
Date last modified: 2026/08/23
Contributors:
Xylar Asay-Davis
Claude
Tests with synthetic work shall cover: Python work draining to nothing, and the pool releasing everything; Python work appearing when the pool is at zero; a burst of work followed immediately by a lull, which shall not produce a resize followed by its opposite; and a shrink while work is still running, which shall wait for it.
Testing and Validation: Idle Time
Date last modified: 2026/08/23
Contributors:
Xylar Asay-Davis
Claude
A mixed synthetic suite – model-like steps and Python-like work, interleaved so that a fixed pool would clearly waste resources – shall be run with a fixed pool and with an elastic one. Total allocation-seconds idle shall fall, and results shall be identical.
This test is also what decides whether Phase D is worth building at all. It should be written first, run against Phase C, and the answer recorded.
Testing and Validation: Real Mixed Workload
Date last modified: 2026/08/23
Contributors:
Xylar Asay-Davis
Claude
Once a real suite exists that mixes model runs and analysis, it shall be run with a fixed and an elastic pool and the wall times compared, on at least one Slurm and one PBS machine.