Design Document: mache.deploy

Summary

mache.deploy is a subpackage of mache that provides a unified, documented, and extensible mechanism for deploying combined pixi (conda packages) and spack environments for E3SM-supported software (e.g. Polaris, Compass, E3SM-Unified) on E3SM-supported HPC systems.

The primary motivation is to replace the redundant, subtly divergent, and poorly documented deployment logic currently embedded independently in these packages with a single, shared implementation. This improves maintainability, ensures feature parity across target software, and provides a scalable model for future E3SM software with mixed pixi/spack dependencies.

In this document, software such as Polaris or E3SM-Unified that uses mache.deploy is referred to as the target software.

Requirements

Date last modified: Dec 29, 2025 Contributors: Xylar Asay-Davis, Althea Denlinger

Requirement: A mechanism to begin deployment

The target software must provide a user-facing entry point to begin deployment. This mechanism cannot depend on mache already being installed, because deployment is precisely how mache is introduced.

Design resolution

Each target software provides a small, stable deploy.py script at repository root. This script:

  • Parses command-line arguments defined declaratively

  • Downloads a standalone bootstrap.py script from the mache repository

  • Executes bootstrap.py to install pixi (if needed) and mache

Requirement: A mechanism to install pixi

If pixi is not already installed, the deployment process must be able to install it automatically.

Design resolution

The standalone bootstrap.py script (downloaded from mache) handles:

  • Installing pixi into a user-specified or inferred location

  • Using pixi in a non-interactive, non-login context (no reliance on shell init)

This logic is intentionally duplicated only in bootstrap.py, which runs before mache is available.

Requirement: A mechanism to install mache

The target software must support installing:

  • A released version of mache from conda-forge (via pixi), or

  • A developer-specified fork and branch (for testing and development)

Design resolution

  • bootstrap.py creates a minimal bootstrap pixi environment

  • mache is installed into that environment either:

    • via a pixi manifest that depends on mache==<version> (from conda-forge), or

    • via cloning and installing from a fork/branch (installed into the pixi environment)

Requirement: A way for target software to specify pixi packages

The target software must define:

  • Which conda packages are installed (via pixi)

  • Version constraints

  • Variants across machines, compilers, MPI, etc.

Design resolution

The target software provides two inputs:

  • deploy/pixi.toml.j2: a Jinja2-templated pixi manifest that declaratively defines the pixi environment dependencies

  • deploy/config.yaml.j2: a Jinja2-templated YAML configuration file that defines deployment options and variants (channels, prefix, MPI selection, feature toggles, etc.)

mache.deploy renders and interprets these files to construct a pixi environment. The pixi manifest (deploy/pixi.toml.j2) is the authoritative specification of the pixi environment dependencies.

Requirement: A way for target software to specify spack packages

The target software must define:

  • Spack packages

  • Versions, variants, and compiler/MPI combinations

Design resolution

  • Spack specifications are included in deploy/config.yaml.j2

  • mache.deploy interprets and realizes these specs using spack

  • Spack logic lives entirely inside mache, not in the target software

Requirement: Support for environment variants

The target software must support multiple deployment variants, including:

  • Different machines

  • Different compilers and MPI stacks

  • Optional components (e.g., Trilinos, Albany, PETSc)

Design resolution

  • Variants are declared declaratively in deploy/config.yaml.j2

  • Users select variants via command-line options to deploy.py / mache deploy

  • Variant selection is propagated consistently to pixi, spack, modules, and environment variables

Requirement: Provide environment variables

Some target software requires environment variables to be set to function correctly.

Design resolution

  • Environment variables are specified declaratively in deploy/config.yaml.j2

  • mache.deploy generates shell load scripts that:

    • activate pixi environments

    • load spack environments

    • load system modules

    • export required environment variables

Requirement: Provide a mechanism to load environments

After deployment, users need a simple way to load the environment.

Design resolution

mache.deploy generates load scripts, for example:

  • load_<software>.sh (no toolchain-specific Spack environment)

  • load_<software>_<machine>_<compiler>_<mpi>.sh (toolchain-specific)

Including <machine> avoids filename collisions on shared filesystems where multiple machines (or machine partitions exposed as distinct machine names) share compiler and MPI naming.

These scripts encapsulate:

  • Pixi activation

  • Spack environment activation

  • Module loads

  • Environment variables

Target software documentation points users to these scripts.

Requirement: Conditional spack deployment

Allow day-to-day developers to avoid rebuilding spack, while allowing maintainers to do full shared-environment deployments when needed.

Design resolution

Mechanisms

  • deploy/config.yaml.j2 includes a default such as:

    • spack.deploy: true | false (deploy all supported spack envs, or none)

    • spack.spack_path: <path> (default checkout path for spack)

    • spack.supported: true | false (whether the target supports a “library” spack env)

    • spack.software.supported: true | false (whether the target supports a “software” spack env)

  • deploy.py exposes CLI flags such as:

    • --deploy-spack (force-enable spack deployment)

    • --no-spack (disable all spack use for a single run)

    • --spack-path <path> (temporary override for spack.spack_path)

Precedence

  1. If the user passes --no-spack, all spack use is disabled for that run.

  2. Otherwise, if the user passes --deploy-spack, spack deployment is enabled regardless of the config default.

  3. Otherwise, fall back to the config.yaml.j2 default for deployment and reuse any supported pre-existing spack environments in load scripts.

For spack checkout path resolution:

  1. If the user passes --spack-path, it takes highest priority.

  2. Otherwise, a deployment hook may set ctx.runtime['spack']['spack_path'].

  3. Otherwise, fall back to spack.spack_path in config.yaml.j2.

Rationale

  • E3SM-Unified maintainer workflow: default spack.deploy: true

  • Polaris developer workflow: default spack.deploy: false

  • Polaris maintainer workflow: run ./deploy.py --deploy-spack ... when producing or updating a shared spack environment

  • Temporary testing workflow: run ./deploy.py --spack-path /tmp/<my-spack> ... to avoid editing and accidentally committing deploy/config.yaml.j2

Desired: Skip pixi deployment (future)

Some deployments may want to reuse a shared pixi environment.

Design resolution (future)

  • Supported declaratively

  • mache.deploy would still provide load scripts and environment management

Desired: Testing target software (future)

It should be possible to validate that deployment succeeded.

Design resolution (future)

  • Target software may optionally provide post-deployment tests

  • mache.deploy can expose hooks for running these tests

Conceptual Design

Date last modified: Dec 29, 2025 Contributors: Xylar Asay-Davis, Althea Denlinger

Three-stage deployment model

Deployment proceeds in three clearly separated stages:

  1. Target software entrypoint (deploy.py)

    • Runs with system Python (minimal assumptions)

    • Parses CLI from a declarative spec

    • Downloads bootstrap.py

    • Invokes bootstrap.py

  2. Bootstrap stage (bootstrap.py)

    • Standalone script, not part of the mache package

    • Installs pixi if needed

    • Creates a bootstrap pixi environment

    • Installs mache

  3. Deployment stage (mache deploy)

    • Runs with full mache installed

    • Interprets deploy/config.yaml.j2

    • Creates pixi and spack environments

    • Generates load scripts

This separation is intentional and strictly enforced.

CLI Design and Sharing

Date last modified: Jan 26, 2026 Contributor: Xylar Asay-Davis

Problem

  • deploy.py, bootstrap.py, and mache deploy all need overlapping CLI arguments

  • Duplicating argparse definitions across repositories is fragile

  • Users expect ./deploy.py --help to be authoritative

Design resolution: cli_spec.json(.j2)

Each target software includes deploy/cli_spec.json rendered from the packaged template, plus an optional downstream-owned deploy/custom_cli_spec.json, which declaratively define:

  • Command-line flags

  • Help text

  • Destinations

  • Routing (deploy, bootstrap, run)

Usage

  • deploy.py

    • Builds its argparse interface from deploy/cli_spec.json plus optional deploy/custom_cli_spec.json

    • Forwards appropriate arguments to bootstrap.py and mache deploy run

  • mache deploy run

    • Builds its CLI from the packaged mache/deploy/templates/cli_spec.json.j2 plus optional deploy/custom_cli_spec.json

    • Uses the same source template as mache deploy init/update for the generated portion

Benefits

  • Single source of truth for user-facing CLI

  • No duplication across repositories

  • Consistent --help output

  • Safe downstream extension point for repo-specific flags

Starter Templates and Initialization

Date last modified: Jan 26, 2026 Contributor: Xylar Asay-Davis

Templates

mache.deploy provides templates for:

  • deploy.py

  • cli_spec.json.j2

  • custom_cli_spec.json

  • pins.cfg

  • config.yaml.j2.j2 (renders to deploy/config.yaml.j2)

  • pixi.toml.j2.j2 (renders to deploy/pixi.toml.j2)

  • spack.yaml.j2

  • hooks.py.j2

  • load.sh.j2

  • spack_install.bash.j2

These templates include placeholders for:

  • Software name

  • Pinned mache version

  • Minimal configuration scaffolding

mache deploy init

A command that:

  • Copies templates into a target software repository

  • Fills in required placeholders

  • Creates a minimal, working deployment setup

  • Writes: deploy.py, deploy/cli_spec.json, deploy/pins.cfg, deploy/custom_cli_spec.json, deploy/config.yaml.j2, deploy/pixi.toml.j2, deploy/spack.yaml.j2, deploy/hooks.py, and a placeholder deploy/load.sh

Updating mache versions

When a target software updates its pinned mache version:

  • deploy.py and deploy/cli_spec.json should be updated from the matching mache release

  • deploy/custom_cli_spec.json, deploy/pins.cfg, deploy/config.yaml.j2, and deploy/pixi.toml.j2 remain software-owned

The mache deploy update command updates only deploy.py and deploy/cli_spec.json.

The intended upgrade workflow is therefore:

  1. ./deploy.py --bootstrap-only --mache-version <new_version>

  2. mache deploy update --software <software> --mache-version <new_version> inside the bootstrap environment

  3. manual edit of deploy/pins.cfg so the pinned mache version matches

Package Organization

Date last modified: Jan 26, 2026 Contributor: Xylar Asay-Davis

All deployment-related assets live under:

mache/deploy/
├── bootstrap.py        # standalone script
├── cli.py              # mache deploy CLI
├── cli_spec.py         # cli_spec parsing helpers
├── conda.py            # conda/pixi helpers
├── hooks.py            # deployment hook framework
├── init_update.py      # init/update logic for target repos
├── jinja.py            # Jinja helpers
├── machine.py          # machine/deploy config helpers
├── run.py              # deployment runner (called by `mache deploy run`)
├── spack.py            # spack helpers
└── templates/
  ├── deploy.py.j2
  ├── cli_spec.json.j2
  ├── pins.cfg.j2
  ├── config.yaml.j2.j2
  ├── pixi.toml.j2.j2
  ├── spack.yaml.j2
  ├── hooks.py.j2
  ├── load.sh.j2
  └── spack_install.bash.j2

Reusable JIGSAW build/install logic now lives outside mache.deploy in:

mache/jigsaw/
├── __init__.py         # backend selection + build/install orchestration
├── recipe.yaml.j2      # rattler-build recipe template
├── linux-64.yaml.j2    # linux variant template
├── osx-64.yaml.j2      # macOS variant template
└── build.sh            # build script used by the recipe

This keeps deployment concerns clearly separated from the rest of mache.

Closing Notes

The current design:

  • Satisfies all original requirements

  • Clarifies responsibilities across stages

  • Minimizes redundancy

  • Supports both stable users and developers

  • Provides a clear path for future growth (init, update, testing)

Most importantly, it turns deployment from an ad hoc per-project liability into a shared, documented, and evolvable capability.

Testing

Date last modified: Jan 26, 2026 Contributor: Xylar Asay-Davis

Polaris

  • Full deployment on Chrysalis

  • Intel and GNU compilers

  • MPAS-Ocean pr suite (both compilers)

  • Omega omega_pr suite (both compilers)

Planned test deployment of E3SM-Unified

  • Not yet started

  • Full test deployment on Chrysalis

  • GNU compilers

  • Test run of MPAS-Analysis