Accessing E3SM Data

This page describes how E3SM archives released simulation output and how to retrieve it. Each dataset page links to a table of simulations; this page explains how to use the paths and links in those tables.

How E3SM data is archived

E3SM simulation output is archived and released in the following ways. Each links to a section below describing how to retrieve it.

NERSC HPSS – Native model output is archived on the NERSC HPSS tape system in directories with world-read permissions (under /home/projects/e3sm/www/ or a user’s www directory). Because the files are world-readable, they can be retrieved either by NERSC users with the standard HPSS tools, or by anyone through the NERSC HPSS web portal at https://portal.nersc.gov/archive/home/projects/e3sm/www/ (or https://portal.nersc.gov/archive/home/<u>/<user>/www/ for a user’s www directory). Each simulation table lists the HPSS Path of each simulation and, where available, an HPSS URL to the corresponding web portal page.

Most campaign output is archived with zstash, which stores files in a series of tar files together with an index.db database listing their contents. Some datasets (e.g., the AI training datasets) instead provide an archive_contents text file listing the files in each tar archive.
ESGF – Data reformatted to CMIP conventions (and, for some campaigns, native model output) is published to the Earth System Grid Federation (ESGF). When an ESGF link is listed in a simulation table, ESGF is the preferred retrieval method.
Globus sharing – High-usage datasets, i.e., those being actively analyzed by E3SM, may also have a copy on the NERSC Community File System (CFS), shared through Globus. Simulation tables list these copies’ Data on Disk path and Globus URL. Currently, this applies to the AI training datasets. Because the data is already on disk, this is usually much faster than retrieving it from tape. These disk copies are not guaranteed to persist – the HPSS archive is the authoritative long-term copy.

Run scripts. Where available, the scripts originally used to run each simulation campaign are archived in the run_scripts directory of this repository, and are linked from each dataset page. Original run scripts will not run out-of-the-box; they are provided as guidance for setting up a similar run. For some campaigns (e.g., E3SMv2), separate reproduction scripts are also provided.

NERSC HPSS

All simulations with an HPSS Path in the simulation table can be retrieved from NERSC HPSS, either with a NERSC account or, more slowly, through the web portal without one.

With a NERSC account

This is the most efficient way to retrieve data from HPSS, and is recommended for DOE-affiliated users and anyone who needs a large volume of data. See NERSC’s account documentation for how to obtain an account.

zstash archives. Use zstash, available in the E3SM-Unified conda environment. To extract files locally on a NERSC machine:

source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_pm-cpu.sh
zstash extract --hpss=<HPSS path in table>

To list the files in an archive without extracting:

zstash ls --hpss=<HPSS path in table>

Patterns can be used to retrieve only a subset of files. For example, to retrieve the monthly land output for years 0030–0049 of v2.LR.piControl:

zstash extract --hpss=/home/projects/e3sm/www/WaterCycle/E3SMv2/LR/v2.LR.piControl "*.elm.h0.00[3-4]?-??.nc"

To transfer data to a machine other than NERSC, use the zstash Globus interface from that machine:

zstash extract --hpss=globus://nersc/<HPSS path in table>

or, equivalently, using the NERSC HPSS Globus endpoint UUID:

zstash extract --hpss=globus://9cd89cfd-6d04-11e5-ba46-22000b92c6ec/<HPSS path in table>

For more information, refer to the zstash usage documentation.

Other archives. Use the standard HPSS tools hsi and htar on a NERSC machine, or transfer the files with Globus using the NERSC HPSS endpoint. For example:

hsi ls <HPSS path in table>
hsi get <HPSS path in table>/<file>

See NERSC’s HPSS documentation for details.

Without a NERSC account

Anyone can download world-readable HPSS data through the NERSC HPSS web portal. An HPSS path such as /home/projects/e3sm/www/<...> is available at https://portal.nersc.gov/archive/home/projects/e3sm/www/<...>; the HPSS URL column of the simulation tables provides these links directly.

Files are downloaded through the web browser (or a tool such as wget or curl) one file at a time. Because the data must be staged from tape and many archives are multiple terabytes, this can be slow. Downloading only what you need is strongly recommended:

For zstash archives, first download the index.db file and query it to find which tar files contain the data you need:
```
sqlite3 index.db "SELECT tar,name,size from files;" > archive_contents.txt
```
For archives that provide an archive_contents text file, check it first to identify the tar files you need.

ESGF

Simulations with an ESGF Links entry in the simulation table can be downloaded from the Earth System Grid Federation (ESGF). Neither a NERSC account nor an ESGF login is required to download data with a wget script. ESGF has two kinds of E3SM data:

CMIP – a subset of the output, reformatted to CMIP conventions (CMOR’d) and published as part of CMIP6. The CMIP links in the tables open these datasets.
Native – native model output, as written by E3SM. Where available, the Native links in the tables open these datasets.

To download the data:

Click the CMIP or Native link in the simulation table. This opens the ESGF search page, filtered to that simulation’s datasets.
Use the search facets (e.g., variable, frequency, realm) to narrow down to the datasets you need, and add them to the Data Cart.
From the Data Cart, download a wget script and run it on the machine where you want the data:
```
bash wget-<...>.sh
```
Alternatively, for data hosted at nodes that support it, you can start a Globus transfer from the Data Cart (this requires signing in to Globus).

For more information, see the ESGF user documentation.