SCORPIO  1.7.0
Install Walk-through

This document provides specific instructions for installing SCORPIO using a variety of compilers on a few commonly used super computers. Click on the link below to go directly to the machine of interest.

  • Yellowstone (NCAR's 1.5-petaflop IBM Supercomputer)
  • Edison (A NERSC Cray XC30 Supercomputer)
  • Mira (IBM Blue Gene Supercomputer at ALCF)
  • Blue Waters (NCSA's 1.3-petaflop Cray Supercomputer)
  • Hobart (The NCAR CGD local cluster)
  • Linux with MPICH (Standard Linux box with MPICH)

Yellowstone

  1. Directory setup

    Download a copy of the SCORPIO source into a sub-directory of your working directory (refered to here as the PIO_source directory). Create another sub-directory for the build (refered to here as the PIO_build directory) and 'cd' into it.

  2. Modules

    Modules required for installation depend on your prefered compiler. Issue the commands below to set the module environment for building SCORPIO on Yellowstone.

    • Intel

      %> module reset
      %> module unload netcdf
      %> module swap intel intel/15.0.3
      %> module load git/2.3.0
      %> module load cmake/3.0.2
      %> module load netcdf-mpi/4.3.3.1
      %> module load pnetcdf/1.6.1

    • GNU

      %> module reset
      %> module unload netcdf
      %> module swap intel gnu/4.8.2
      %> module load git/2.3.0
      %> module load cmake/3.0.2
      %> module load netcdf-mpi/4.3.3.1
      %> module load pnetcdf/1.6.1

    • PGI

      %> module reset
      %> module unload netcdf
      %> module swap intel pgi/13.3
      %> module load git/2.3.0
      %> module load cmake/3.0.2
      %> module load netcdf-mpi/4.3.3.1
      %> module load pnetcdf/1.6.1

  3. Environment Variables

    The appropriate compiler wrappers must be chosen for cmake, so the environment variables CC, CXX and FC must be set as:
    CC=mpicc
    CXX=mpicxx
    FC=mpif90

  4. Build

    Building SCORPIO requires running the CMake configure and then make. In the PIO_build directory type
    %> cmake ../PIO_source/
    %> make

Edison

  1. Directory setup

    Download a copy of the SCORPIO source into a sub-directory of your working directory (refered to here as the PIO_source directory). Create another sub-directory for the build (refered to here as the PIO_build directory) and 'cd' into it.

  2. Modules

    Modules required for installation depend on your prefered compiler. Issue the commands below to set the module environment for building SCORPIO on Edison.

    • Intel

      %> module purge
      %> module load PrgEnv-intel
      %> module load craype-ivybridge
      %> module load cray-shmem
      %> module load cray-mpich
      %> module load torque
      %> module load git/2.4.6
      %> module load cmake/3.0.0
      %> module load cray-hdf5-parallel/1.8.14
      %> module load cray-netcdf-hdf5parallel/4.3.3.1
      %> module load cray-parallel-netcdf/1.6.1

    • GNU

      %> module purge
      %> module load PrgEnv-gnu
      %> module load craype-ivybridge
      %> module load cray-shmem
      %> module load cray-mpich
      %> module load torque
      %> module load git/2.4.6
      %> module load cmake/3.0.0
      %> module load cray-hdf5-parallel/1.8.14
      %> module load cray-netcdf-hdf5parallel/4.3.3.1
      %> module load cray-parallel-netcdf/1.6.1

    • Cray

      %> module purge
      %> module load PrgEnv-cray
      %> module load craype-ivybridge
      %> module load cray-shmem
      %> module load cray-mpich
      %> module swap cce cce/8.4.0.223
      %> module load torque
      %> module load git/2.4.6
      %> module load cmake/3.0.0
      %> module load cray-hdf5-parallel/1.8.14
      %> module load cray-netcdf-hdf5parallel/4.3.3.1
      %> module load cray-parallel-netcdf/1.6.1

  3. Environment Variables

    The appropriate compiler wrappers must be chosen for cmake, so the environment variables CC and FC must be set as:
    CC=cc
    CXX=CC
    FC=ftn

  4. Build

    Building SCORPIO requires running the CMake configure and then make. In the PIO_build directory type
    %> cmake -DCMAKE_VERBOSE_MAKEFILE=TRUE \
    -DPREFER_STATIC=TRUE \
    -DNetCDF_PATH=${NETCDF_DIR} \
    -DPnetCDF_PATH=${PARALLEL_NETCDF_DIR} \
    -DHDF5_PATH=${HDF5_DIR} \
    -DMPI_C_INCLUDE_PATH=${MPICH_DIR}/include \
    -DMPI_Fortran_INCLUDE_PATH=${MPICH_DIR}/include \
    -DMPI_C_LIBRARIES=${MPICH_DIR}/lib/libmpich.a \
    -DMPI_Fortran_LIBRARIES=${MPICH_DIR}/lib/libmpichf90.a \
    -DCMAKE_SYSTEM_NAME=Catamount \
    ../PIO_source/
    %> make

Mira/Cetus

  1. Directory setup

    Download a copy of the SCORPIO source into a sub-directory of your working directory (refered to here as the PIO_source directory). Create another sub-directory for the build (refered to here as the PIO_build directory) and 'cd' into it.

  2. Softenv packages and environment variables

    It is not necessary to edit your .soft file on Mira inorder to build SCORPIO. Execute the following commands to temporarily load packages into your softenv. These packages use the IBM/XL compiler.
    %> soft add +mpiwrapper-xl (or switch from the default in your softenv)
    %> soft add @ibm-compilers-2015-02
    %> soft add +cmake
    %> soft add +git

    And then set the following environment variables to add in the rest of the library paths.
    %> setenv LIBZ /soft/libraries/alcf/current/xl/ZLIB
    %> setenv HDF5 /soft/libraries/hdf5/1.8.14/cnk-xl/V1R2M2-20150213
    %> setenv NETCDF /soft/libraries/netcdf/4.3.3-f4.4.1/cnk-xl/V1R2M2-20150213
    %> setenv PNETCDF /soft/libraries/pnetcdf/1.6.1/cnk-xl/V1R2M2-20150213
    %> setenv CC /soft/compilers/wrappers/xl/mpixlc_r
    %> setenv FC /soft/compilers/wrappers/xl/mpixlf90_r

  3. Build

    Building SCORPIO requires running the CMake configure and then make. In the PIO_build directory type
    %> cmake -DPREFER_STATIC=TRUE ../PIO_source/
    %> make

Blue Waters

  1. Directory setup

    Download a copy of the SCORPIO source into a sub-directory of your working directory (refered to here as the PIO_source directory). Create another sub-directory for the build (refered to here as the PIO_build directory) and 'cd' into it.

  2. Modules

    Modules required for installation depend on your prefered compiler. Issue the commands below to set the module environment for building SCORPIO on Hobart.

    • Intel

      %> module swap PrgEnv-cray PrgEnv-intel
      %> module load torque
      %> module load git
      %> module load cmake
      %> module load cray-hdf5-parallel/1.8.14
      %> module load cray-netcdf-hdf5parallel/4.3.3.1
      %> module load cray-parallel-netcdf/1.6.1

    • PGI

      %> module swap PrgEnv-cray PrgEnv-pgi
      %> module load torque
      %> module load git
      %> module load cmake
      %> module load cray-hdf5-parallel/1.8.14
      %> module load cray-netcdf-hdf5parallel/4.3.3.1
      %> module load cray-parallel-netcdf/1.6.1

  3. Environment Variables

    The appropriate compiler wrappers must be chosen for cmake, so the environment variables CC and FC must be set as:
    CC=cc
    CXX=CC
    FC=ftn

  4. Build

    Building SCORPIO requires running the CMake configure and then make. In the PIO_build directory type
    %> cmake -DCMAKE_VERBOSE_MAKEFILE=TRUE \
    -DPREFER_STATIC=TRUE \
    -DNetCDF_PATH=${NETCDF_DIR} \
    -DPnetCDF_PATH=${PARALLEL_NETCDF_DIR} \
    -DHDF5_PATH=${HDF5_DIR} \
    -DMPI_C_INCLUDE_PATH=${MPICH_DIR}/include \
    -DMPI_Fortran_INCLUDE_PATH=${MPICH_DIR}/include \
    -DMPI_C_LIBRARIES=${MPICH_DIR}/lib/libmpich.a \
    -DMPI_Fortran_LIBRARIES=${MPICH_DIR}/lib/libmpichf90.a \
    -DCMAKE_SYSTEM_NAME=Catamount \
    ../PIO_source/
    %> make

Hobart

  1. Directory setup

    Download a copy of the SCORPIO source into a sub-directory of your working directory (refered to here as the PIO_source directory). Create another sub-directory for the build (refered to here as the PIO_build directory) and 'cd' into it.

  2. Modules

    Modules required for installation depend on your prefered compiler. Issue the commands below to set the module environment for building SCORPIO on Hobart.

    • Intel

      %> module purge
      %> module load compiler/intel/15.0.2.164
      %> module load tool/parallel-netcdf/1.6.1/intel

    • Nag

      %> module purge
      %> module load compiler/nag/6.0
      %> module load tool/parallel-netcdf/1.6.1/nag/openmpi

    • PGI

      %> module purge
      %> module load compiler/pgi/15.1
      %> module load tool/parallel-netcdf/1.6.1/pgi/mvapich2

  3. Environment Variables

    The appropriate compiler wrappers must be chosen for cmake, so the environment variables CC, CXX and FC must be set as:
    CC=mpicc
    CXX=mpicxx
    FC=mpif90

  4. Build

    Building SCORPIO requires running the CMake configure and then make. In the PIO_build directory type
    %> cmake -DNETCDF_DIR=$NETCDF_PATH -DPNETCDF_DIR=$PNETCDF_PATH ../PIO_source/
    %> make

    ParallelIO does not require Parallel netcdf to run, so if you decide to use the GNU compiler on Hobart (not described here) without the parallel-netcdf library, use the cmake configure flags:
    %> cmake -DNETCDF_DIR=$NETCDF_PATH -DWITH_PNETCDF=FALSE ../PIO_source/
    %> make

Linux with MPICH

  1. Installing MPICH

    Download from the MPICH2 downloads page. (These instructions were tested using version 3.2). Untar with: 

    tar zxf mpich-3.2.tar.gz

    Build with: 

    cd mpich-3.2 && ./configure --prefix=/usr/local && make all check
&& sudo make install

    Now you should be able to access mpicc, mpicxx, mpifort, and mpirun from the command line. (If not, make sure /usr/local/bin is in path.)

    Note that if you wish to use valgrind on your programs, you should configure MPICH like this: 

    CPPFLAGS=-I/usr/include/valgrind ./configure --prefix=/usr/local --enable-g=mem,meminit

    For this to work you must have packages valgrind and valgrind-devel installed.

  2. Installing Zlib

    Download from the zlib downloads page. (These instructions were tested using version 1.2.11). Untar with: 

    tar zxf zlib-1.2.11.tar.gz

    Build with: 

    cd zlib-1.2.11
CC=mpicc ./configure --prefix=/usr/local/zlib-1.2.11_mpich-3.2
make all check
sudo make install

  3. Installing SZlib

    Download szlib from the HDF5 szlib page. (These instructions were tested using version 2.1). Untar with: 

    tar zxf szip-2.1.tar.gz

    Build with: 

    cd slib-2.1
CC=mpicc ./configure --prefix=/usr/local/szip-2.1_mpich-3.2
make all check
sudo make install

  4. Installing pNetCDF

    Download parallel-netcdf from the parallel-netcdf download page. (These instructions were tested using version 1.8.1). Untar with: 

    tar zxf parallel-netcdf-1.8.1.tar.gz

    Build with: 

    cd parallel-netcdf-1.8.1
FC=mpifort CC=mpicc CFLAGS=-fPIC ./configure --prefix=/usr/local/pnetcdf-1.8.1_mpich-3.2
make all check
sudo make install

  5. Installing HDF5

    Download HDF5 from the HDF5 download page.

    These instructions were tested using version 1.10.1. Untar with: 

    tar zxf hdf5-1.10.1.tar.gz

    . Note that in my case I need to add /usr/local/bin to the PATH for su, because that is where mpicc is found.

    Build with: 

    cd hdf5-1.10.1
CC=mpicc ./configure --with-zlib=/usr/local/zlib-1.2.11_mpich-3.2 --with-szlib=/usr/local/szip-2.1_mpich-3.2 --prefix=/usr/local/hdf5-1.10.1_mpich-3.2 --enable-parallel
make all check
sudo PATH=$PATH:/usr/local/bin make install


  6. Installing NetCDF-4 C Library


    Download the netcdf C library from the NetCDF download
page. Untar with: 
    tar zxf netcdf-c-4.5.0-rc1.tar.gz
    Build with:

    cd netcdf-c-4.5.0-rc1
CPPFLAGS='-I/usr/local/zlib-1.2.11_mpich-3.2 -I/usr/local/szip-2.1_mpich-3.2/include -I/usr/local/hdf5-1.10.1_mpich-3.2/include' LDFLAGS='-L/usr/local/zlib-1.2.11_mpich-3.2/lib -L/usr/local/szip-2.1_mpich-3.2/lib -L/usr/local/hdf5-1.10.1_mpich-3.2/lib' CC=mpicc ./configure --enable-parallel-tests --prefix=/usr/local/netcdf-c-4.5.0-rc1_mpich-3.2
make all check
sudo make install 
    Note that you may not build netCDF with it's built-in
parallel-netCDF support, if you are also planning to use the
parallel-netCDF library with SCORPIO. For SCORPIO, parallel-netCDF must be
installed independently of netCDF.
    7. 

  7. 

    Installing NetCDF-4 Fortran Library
    Download the netcdf Fortran library from the NetCDF download
page. Untar
with: 
    tar zxf netcdf-fortran-4.4.4.tar.gz
    Build with:

    cd netcdf-fortran-4.4.4
CC=mpicc LD_LIBRARY_PATH=/usr/local/netcdf-c-4.5.0-rc1_mpich-3.2/lib FC=mpifort CPPFLAGS=-I/usr/local/netcdf-c-4.5.0-rc1_mpich-3.2/include LDFLAGS=-L/usr/local/netcdf-c-4.5.0-rc1_mpich-3.2/lib ./configure --enable-parallel-tests --prefix=/usr/local/netcdf-fortran-4.4.4_mpich-3.2
make all
sudo make install 
    Note that make check did not work because of a failure in a
test. This has been reported to the netCDF team.
    8. 

  8. 

    Installing ParallelIO Library
    Clone the ParallelIO library.
    Create a build directory and run cmake.

    cd ParallelIO
mkdir build
cd build
CC=mpicc CXX=mpicxx FC=mpifort cmake -DNetCDF_C_PATH=/usr/local/netcdf-c-4.5.0-rc1_mpich-3.2 -DNetCDF_Fortran_PATH=/usr/local/netcdf-fortran-4.4.4_mpich-3.2 -DPnetCDF_PATH=/usr/local/pnetcdf-1.8.1_mpich-3.2 -DPIO_HDF5_LOGGING=On -DPIO_USE_MALLOC=On ..
make
make check
sudo make install

    When debugging build issues, it's helpful to be able to do a clean
build from the command line, including tests:
    cd build
rm -rf * && CFLAGS='-Wall -g' CXXFLAGS='-Wall -g' FFLAGS=-g CC=mpicc CXX=mpicxx FC=mpifort cmake -DNetCDF_C_PATH=/usr/local/netcdf-c-4.5.0-rc1_mpich-3.2 -DNetCDF_Fortran_PATH=/usr/local/netcdf-fortran-4.4.4_mpich-3.2 -DPnetCDF_PATH=/usr/local/pnetcdf-1.8.1_mpich-3.2 -DPIO_HDF5_LOGGING=On -DPIO_USE_MALLOC=On .. && make VERBOSE=1 all tests check

    Note the optional CFLAGS=-g which allows the use of a debugger
with this code. Also note the optional VERBOSE=1 provided to make,
which causes the build commands to be made visible.
    Note also the -DPIO_ENABLE_LOGGING=On which is helpful for debugging but should probably not be used in production builds.
    9. 

  9. 

    Building and Running Performance Tests
    Download a decomp file from our google
code page. You can use any of those files, save them to
build/test/performance. (These instructions were tested with the first
one in the list.)
    Create a namelist file, named "pioperf.nl". Save this file in the
subdirectory (note that it is in the BUILD directory):

    build/tests/performance/

    The contents of the namelist file should look like:

    &pioperf
decompfile = "/u/sciteam/thayerca/scratch/pio_work/piodecomp30tasks01dims06.dat"
pio_typenames = 'pnetcdf'
niotasks = 30
rearrangers = 1
nvars = 2
/

    You should change the path to your decomp file to wherever you
saved it. You can add items to the list to run more tests, so, for
instance, to test all of the types of io, your pio_typenames would
look like: pio_typenames = 'pnetcdf','netcdf','netcdf4p','netcdf4c'
    HDF5 is netcdf4p, and Parallel-Netcdf is pnetcdf.
    Example to test with different numbers of tasks, you could do:
niotasks = 30,15,5
    Example to test with both of the rearranger algorithms:
rearrangers = 1,2
    Example to test with different numbers of variables:
nvars = 8,5,3,2
    Once you have your decomp file and your namelist set up, run like this:
    mpiexec -n 30 ./pioperf

    If you run the most basic performance tests (as in the first
pioperf.nl example), this script should only take a few minutes (less
than 5) to complete. The output and results will be written to your
pioperf.o$PBS_JOBID file. Adding more iotypes and rearrangers and
variables to these tests will increase the time significantly.
    10. 



Last updated: 05-16-2016







Generated on Tue Apr 8 2025 11:46:45 for SCORPIO by doxygen 1.9.1