IO

A lot of I/O related tasks are handled internally in the step class polaris.Step. Some of the lower level functions can be called directly if need be.

Symlinks

You can create your own symlinks that aren’t input files (e.g. for a README file that the user might want to have available) using polaris.io.symlink():

from importlib.resources import path

from polaris.io import symlink


def configure(task, config):
    ...
    with path('polaris.ocean.tasks.global_ocean.files_for_e3sm', 'README') as \
            target:
        symlink(str(target), '{}/README'.format(task['work_dir']))

In this example, we get the path to a README file within polaris and make a local symlink to it in the task’s work directory. We did this with symlink() rather than add_input_file() because we want this link to be within the task’s work directory, not the step’s work directory. We must do this in configure() rather than collect() because we do not know if the task will be set up at all (or in what work directory) during collect().

Download

You can download files more directly if you need to using polaris.io.download(), though we recommend using polaris.Step.add_input_file() whenever possible because it is more flexible and takes care of more of the details of symlinking the local file and adding it as an input to the step. No current tasks use download() directly, but an example might look like this:

import os
from polaris.io import symlink, download

def setup(self):
    config = self.config
    step_dir = self.work_dir
    database_root = self.config.get('paths', 'database_root')
    download_path = os.path.join(database_root, 'mpas-ocean',
                                 'bathymetry_database')

    remote_filename = \
        'BedMachineAntarctica_and_GEBCO_2019_0.05_degree.200128.nc'
    local_filename = 'topography.nc'

    download(
        url=f'https://web.lcrc.anl.gov/public/e3sm/mpas_standalonedata/'
            f'mpas-ocean/bathymetry_database/{remote_filename}',
        config=config, dest_path=download_path)

    symlink(os.path.join(download_path, remote_filename),
            os.path.join(step_dir, local_filename))

In this example, the remote file BedMachineAntarctica_and_GEBCO_2019_0.05_degree.200128.nc gets downloaded into the bathymetry database (if it’s not already there). Then, we create a local symlink called topography.nc to the file in the bathymetry database.

Permissions

After downloading a file to a shared location, it is typically a good idea to change permissions so others can access the file and write to the directory it is stored in.

This can be accomplished with polaris.io.update_permissions(), which takes a list of one or more directories on which to update permissions, along with a group name that files should belong to. As an example, if we download a file ocean.QU.240km.151209.nc to a database ocean/omega_ctest, we would then change permissions on the database so it is readable and writable by the group identified by the [e3sm_unified]/group config option (if any) as follows:

import os
from polaris.io import download, update_permissions


machine = os.environ['POLARIS_MACHINE']

config = PolarisConfigParser()
config.add_from_package('polaris', 'default.cfg')
config.add_from_package('mache.machines', f'{machine}.cfg')
config.add_from_package('polaris.machines', f'{machine}.cfg')

database_root = config.get('paths', 'database_root')

base_url = 'https://web.lcrc.anl.gov/public/e3sm/polaris/'

filename = 'ocean.QU.240km.151209.nc'

database_path = 'ocean/omega_ctest'

download_path = os.path.join(database_root, database_path, filename)
url = f'{base_url}/{database_path}/{filename}'
download_target = download(url, download_path, config)

if config.has_option('e3sm_unified', 'group'):
    full_path = os.path.join(database_root, database_path)
    group = config.get('e3sm_unified', 'group')
    update_permissions([full_path], group)

This counts on us having set the $POLARIS_MACHINE environment variable, which would be the case if the user has sourced a polaris load script.