Adding a new task
The task’s bash file
Create zppy/templates/<task-name>.bash
.
This is where the bulk of the work required for a task will be.
Most of the file is task-specific, so it is difficult to describe precise steps.
Some key parts, however, are displayed below:
#!/bin/bash {% include 'slurm_header.sh' %} {{ environment_commands }} # Turn on debug output if needed debug={{ debug }} if [[ "${debug,,}" == "true" ]]; then set -x fi # Script dir cd {{ scriptDir }} # Get jobid id=${SLURM_JOBID} # Update status file STARTTIME=$(date +%s) echo "RUNNING ${id}" > {{ prefix }}.status# Update status file and exit {% raw %} ENDTIME=$(date +%s) ELAPSEDTIME=$(($ENDTIME - $STARTTIME)) {% endraw %} echo ============================================== echo "Elapsed time: $ELAPSEDTIME seconds" echo ============================================== echo 'OK' > {{ prefix }}.status exit 0
Variables of the form {{ <variable-name> }}
can come from a number of sources:
zppy/templates/default.ini
(in the form of<variable-name> = <variable-type>(default=<default-value>)
)zppy/templates/<campaign>.cfg
(in the form of<variable-name> = <value>
); overrides values from #1The user’s configuration file (in the form of
<variable-name> = <value>
); overrides values from #1, #2
Another possible source is zppy/<task-name>.py
(in the form of c["<variable-name>"]
where c
is defined in for c in tasks
).
The task’s Python file
Create zppy/<task-name>.py
. The basic outline of the file should be similar to the
below example, adapted from zppy/e3sm_diags.py
.
import os import pprint import jinja2 from zppy.utils import checkStatus, getTasks, getYears, submitScript # ----------------------------------------------------------------------------- def <task-name>(config, scriptDir): # Initialize jinja2 template engine templateLoader = jinja2.FileSystemLoader( searchpath=config["default"]["templateDir"] ) templateEnv = jinja2.Environment(loader=templateLoader) template = templateEnv.get_template("<task-name>.bash") # --- List of <task-name> tasks --- tasks = getTasks(config, "<task-name>") if len(tasks) == 0: return # --- Generate and submit <task-name> scripts --- for c in tasks: # Loop over year sets year_sets = getYears(c["years"]) for s in year_sets: c["year1"] = s[0] c["year2"] = s[1] c["scriptDir"] = scriptDir if c["subsection"]: sub = c["subsection"] else: sub = c["grid"] prefix = "<task-name>_%s_%s_%04d-%04d" % ( sub, c["tag"], c["year1"], c["year2"], ) print(prefix) c["prefix"] = prefix scriptFile = os.path.join(scriptDir, "%s.bash" % (prefix)) statusFile = os.path.join(scriptDir, "%s.status" % (prefix)) settingsFile = os.path.join(scriptDir, "%s.settings" % (prefix)) skip = checkStatus(statusFile) if skip: continue # Create script with open(scriptFile, "w") as f: f.write(template.render(**c)) # List of dependencies dependencies = [] # Add dependencies # Example: dependencies.append(os.path.join(scriptDir, "%s_%04d-%04d.status" % (<name-of-dependency>, c["year1"], c["year2"]),)) with open(settingsFile, "w") as sf: p = pprint.PrettyPrinter(indent=2, stream=sf) p.pprint(c) p.pprint(s) if not c["dry_run"]: # Submit job jobid = submitScript( scriptFile, dependFiles=dependencies, export="NONE" ) if jobid != -1: # Update status file with open(statusFile, "w") as f: f.write("WAITING %d\n" % (jobid))
The main file
Add the task to zppy/__main__.py
:
from <task_name> import <task_name># <task-name> tasks <task-name>(config, scriptDir)
Update defaults
Add default values for parameters in zppy/templates/default.ini
.
[<task-name>] ... [[__many__]] # Only if the task supports sub-tasks ... # List the same parameters but with `default=None`
Update the tutorial
Add example values for parameters in docs/source/post.mysimulation.cfg
.
[ <task-name> ] ... # Specify parameters that would likely apply to all sub-tasks [[ <subtask-name> ]] # Only if the task supports sub-tasks ... # Specify parameters that make more sense defined in the sub-tasks
Explain new parameters
In docs/source/parameters.rst
Copy defaults from zppy/templates/default.ini
and add a comment line
above each explaining the parameter.
Update the tests
In tests/integration/test_*.cfg
Copy example values from docs/source/post.mysimulation.cfg
.
The expected files will have to be updated as well.