Task Parallelism Phase D: Coexistence and Elasticity
Creation date: 2026/08/23
Contributors:
Xylar Asay-Davis
Claude
Summary
Phase C gives the worker pool a fixed share of the allocation for the life of the run. That is simple, and it is wrong in one specific way: once the Python work is finished, the pool goes on holding its share, and those nodes sit idle while model runs queue for resources that are right there.
Phase D makes the pool’s share change with the work. When Python work is ready, the pool grows; when it drains, the pool gives nodes back and ordinary steps use them.
This is the phase that makes a mixed suite – model runs and analysis in the same job – use the machine properly. It is also the phase most likely to be unnecessary, and that should be tested before it is built: if suites in practice do their analysis at the end, after the model runs are done, then a pool that appears late and holds resources until the job ends costs nothing, and Phase D is optimisation without a problem to solve.
Success in Phase D means a suite mixing model runs and analysis finishes in close to the time the work itself requires, with no long stretch where part of the allocation is reserved but idle.
Requirements
Requirement: The Pool Gives Resources Back
Date last modified: 2026/08/23
Contributors:
Xylar Asay-Davis
Claude
When there is less ready Python work than the pool is sized for, the pool shall shrink and the freed resources shall become available to other steps.
When the Python work is finished entirely, the pool shall release everything.
Requirement: The Pool Grows When Needed
Date last modified: 2026/08/23
Contributors:
Xylar Asay-Davis
Claude
When ready Python work exceeds what the current pool can handle, and resources are free, the pool shall grow.
Growing shall not take resources from steps that are already running.
Requirement: Changes Are Not Constant
Date last modified: 2026/08/23
Contributors:
Xylar Asay-Davis
Claude
The pool shall not resize continuously in response to small fluctuations.
Adding and removing workers is not free. An earlier design that switched execution modes at every opportunity spent a significant fraction of its wall time doing so. The scheduler shall require a change to be worth making before making it.
Requirement: A Rule for Competing Work
Date last modified: 2026/08/23
Contributors:
Xylar Asay-Davis
Claude
When ready model steps and ready Python work both want the same free resources, Polaris shall apply a stated rule rather than whichever the loop happens to reach first.
The rule shall be simple enough to explain in a sentence and shall be recorded in the run’s event stream when it is applied, so that a schedule that looks wrong can be understood.
Requirement: Resizing Is Safe
Date last modified: 2026/08/23
Contributors:
Xylar Asay-Davis
Claude
Shrinking the pool shall not interrupt work already running on the workers being removed.
A worker shall be allowed to finish what it holds before it is stopped, and the resources it occupied shall not be offered to another step until it has actually gone.
Requirement: Elasticity Is Visible
Date last modified: 2026/08/23
Contributors:
Xylar Asay-Davis
Claude
Every resize shall be recorded, with what triggered it and what it cost, and the run summary shall report how much of the allocation was idle and for how long.
Idle time is the quantity this phase exists to reduce, so it must be measurable before and after.
Algorithm Design
Algorithm Design: Deciding the Size
Date last modified: 2026/08/23
Contributors:
Xylar Asay-Davis
Claude
The scheduler already holds the dependency graph, so it knows not only what Python work is ready now but how much is still to come. The pool’s target size should be the number of workers the ready Python work can actually use, capped by what the allocation can spare.
Resizing should happen in whole nodes. A pool spanning three nodes releasing “half a node” leaves a fragment too small to run a model step in, which is the shape of free resource that looks available and is useless.
Two guards keep this from thrashing. A change should have to be worth more than it costs – the cost being one launch per node added, which measurement puts at a fraction of a second to a second – and a resize should not be followed immediately by its opposite.
Algorithm Design: When Both Kinds of Work Are Waiting
Date last modified: 2026/08/23
Contributors:
Xylar Asay-Davis
Claude
The rule proposed is: prefer the work on the critical path, and break ties in favour of model steps.
Model steps tend to be longer and to have more work depending on them, so starting one late delays more. Python work tends to be shorter and more plentiful, which makes it good for filling gaps. The scheduler can estimate the critical path from the graph, so this is not guesswork.
This is a proposal, not a conclusion. It should be revisited once there is a real mixed suite to measure, and the alternative – simple first-ready ordering – should be measured against it rather than assumed worse.
Algorithm Design: Shrinking Without Interrupting
Date last modified: 2026/08/23
Contributors:
Xylar Asay-Davis
Claude
Shrinking should be cooperative: mark the workers to be removed as accepting no new work, wait for what they hold to finish, then stop them and return their resources to the pool. The scheduler must not count those resources as free until the workers have actually exited, or it will place a step on top of one still shutting down.
Implementation
Implementation: What Changes
Date last modified: 2026/08/23
Contributors:
Xylar Asay-Davis
Claude
Phase D changes the pool module from Phase C and the scheduler loop from Phase B, and adds nothing structurally new:
the pool gains grow and drain operations, and a notion of workers that are finishing but not yet gone;
the resource pool distinguishes resources that are free from resources that will be free shortly;
the scheduler consults a target pool size each time round the loop, and applies the competing-work rule when both kinds are ready.
Implementation: Order of Work
Date last modified: 2026/08/23
Contributors:
Xylar Asay-Davis
Claude
Shrinking should be implemented before growing. Shrinking is what recovers the idle resources that motivate the phase; growing is an optimisation on top of it, and a pool that only ever shrinks is still correct.
Testing
Testing and Validation: Resize Behavior
Date last modified: 2026/08/23
Contributors:
Xylar Asay-Davis
Claude
Tests with synthetic work shall cover: Python work draining to nothing, and the pool releasing everything; Python work appearing when the pool is at zero; a burst of work followed immediately by a lull, which shall not produce a resize followed by its opposite; and a shrink while work is still running, which shall wait for it.
Testing and Validation: Idle Time
Date last modified: 2026/08/23
Contributors:
Xylar Asay-Davis
Claude
A mixed synthetic suite – model-like steps and Python-like work, interleaved so that a fixed pool would clearly waste resources – shall be run with a fixed pool and with an elastic one. Total allocation-seconds idle shall fall, and results shall be identical.
This test is also what decides whether Phase D is worth building at all. It should be written first, run against Phase C, and the answer recorded.
Testing and Validation: Real Mixed Workload
Date last modified: 2026/08/23
Contributors:
Xylar Asay-Davis
Claude
Once a real suite exists that mixes model runs and analysis, it shall be run with a fixed and an elastic pool and the wall times compared, on at least one Slurm and one PBS machine.