Adding a New Machine to Mache

Adding an E3SM-known machine to mache requires adding a new config file, as well as updating the list of machines in discover.py.

Note

Only machines that are included in mache’s machine config list can be added to mache. This list is a copy of the E3SM cime machine config list which we try to keep up-to-date. If you wish to add a machine that is not included in this list, you must contact the E3SM-Project developers to add your machine.

For details on the automated workflow that detects upstream drift in this file and assigns follow-up work to Copilot, see Automated config_machines.xml updates.

Adding a new config file

Adding a new config file is usually straightforward if you follow the format of an existing config file.

For machines with a known E3SM inputdata location, also add an [inputdata] section with:

  • base_path: base directory for the shared E3SM inputdata tree

When the machine also appears in mache/cime_machine_config/config_machines.xml, this value should match that machine’s DIN_LOC_ROOT entry.

Parallel execution settings

Machine config files now include parallel-resource settings that are consumed by mache.parallel. At minimum, each machine should define a [parallel] section with:

  • system: one of slurm, pbs, single_node, or login

  • parallel_executable: launcher command (for example, srun --label or mpiexec --label)

Depending on the parallel system, the following options are typically required:

  • cores_per_node

  • gpus_per_node (if GPUs are available)

  • max_mpi_tasks_per_node

  • cpus_per_task_flag (primarily for PBS launchers)

  • cpu_bind, gpu_bind, mem_bind, placement (optional launcher tuning)

  • login_cores, login_gpus (for the login system)

For machines with hyperthreading, mache’s convention is that cores_per_node should normally be the physical-core count, not the hardware-thread count. Likewise, max_mpi_tasks_per_node should normally reflect the default non-hyperthreaded layout used by E3SM and most downstream software, and cpu_bind = cores is the preferred default when supported by the launcher. Downstream projects that intentionally want hyperthreading can override these settings in their own config to use hardware-thread counts and thread binding. In other words, these config fields are the hyperthreading controls rather than a dedicated boolean option.

Compiler-specific overrides can be provided in optional [parallel.<compiler>] sections, e.g. [parallel.gnu].

For machines with scheduler-target policy limits, you can also define optional sections for queue- or partition-based schedulers:

  • [queue.<name>] sections corresponding to entries in parallel.queues

  • [partition.<name>] sections corresponding to entries in parallel.partitions

  • [qos.<name>] sections corresponding to entries in parallel.qos

Supported keys are:

  • min_nodes: minimum node count for this scheduler target

  • max_nodes: maximum node count for this scheduler target (leave unset for no upper bound)

  • max_wallclock: maximum allowed wall-clock time (for example, 01:00:00)

  • max_wallclock_bins: maximum allowed wall-clock time as a function of job size, for machines whose policy varies with node count

Some machines allow longer jobs the more nodes they use. Frontier’s batch partition, for example, allows 2 hours for 1-91 nodes, 6 hours for 92-183 nodes and 12 hours above that. Describe this with max_wallclock_bins instead of max_wallclock:

[partition.batch]
min_nodes = 1
max_nodes = 9472
max_wallclock_bins = 91: 02:00:00,
                     183: 06:00:00,
                     9472: 12:00:00

Each entry is <max nodes>: <max wallclock> and the bin that applies is the first one, in increasing order of node count, whose node bound is at least the job’s node count. Continuation lines must be indented. A section may set max_wallclock or max_wallclock_bins but not both.

Downstream software can query these values with MachineInfo.get_queue_specs(), MachineInfo.get_partition_specs(), MachineInfo.get_qos_specs() or MachineInfo.get_scheduler_specs().

A scheduler target can only be requested by name (see Parallel execution with mache.parallel) if it is listed in parallel.queues, parallel.partitions or parallel.qos. The matching [queue.<name>], [partition.<name>] or [qos.<name>] section is what supplies the limits mache uses to decide whether such a request can be honored, so a target with no section can always be requested but never rejected. The first entry in each list is the machine’s default.

parallel.constraints is an availability list in the same sense – the first entry is the default and a constraint can only be requested if it appears there – but it takes no [constraint.<name>] section, since a constraint has no node-count or wall-clock limits to record.

These options are used to:

  • detect available resources on the current allocation,

  • construct launcher commands via mache.parallel, and

  • enforce machine-specific limits like max MPI tasks per node.

Adding the new machine to discover.py

You will need to amend the list of machine names in discover.py so that mache can identify the new machine via its hostname. This process is typically done using a regular expression, which is often possible whenever the machine’s hostname follows a standardized format. For example, we can identify known machines from hostnames with the following regular expressions:

'^chr-\d{4}'  # Chrysalis compute nodes with hostnames chr-0000 to chr-9999
'^compy'      # Compy nodes with hostname compy
'^dane\d{1,4}' # Dane nodes with hostnames dane0 to dane9999

In some cases, the hostname assigned to a machine is too generic to differentiate it from other machines. In these cases, we must identify the machine by its environment variables. However, this is not the recommended procedure and should only be done as a last resort. For example, we identify frontier by its LMOD_SYSTEM_NAME environment variable:

if machine is None and 'LMOD_SYSTEM_NAME' in os.environ:
    hostname = os.environ['LMOD_SYSTEM_NAME']
    if hostname == 'frontier':
        # frontier's hostname is too generic to detect, so relying on
        # LMOD_SYSTEM_NAME
        machine = 'frontier'

Note

Identifying the machine by environment variables is not recommended unless absolutely necessary.