Installing libraries and automating WRF-Chem

Backgroud

NETCDF version (netcdf/4.5.0/02-CF-17-par) on TH-2 seems something wrong. This results in error when NETCDF4 of WRF-Chem enabled (NETCDF4=1). So, I decide to compile NETCDF on my own.

Prerequisite

According to NetCDF’s instruction , we need zlib and hdf5 first. For zlib, I use the version zlib/1.2.11-icc17 originally compiled by TH-2.

HDF5

The newest version is 1.10.4.

$ ./configure --prefix=/HOME/nuist_chenq_2/WORKSPACE/xin/software/hdf5-1.10.4 --enable-fortran --enable-static-exec

But, I got this error:

checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
: command not found/WORKSPACE/xin/software/sources/hdf5-1.10.4/bin/missing: line 3:
: command not found/WORKSPACE/xin/software/sources/hdf5-1.10.4/bin/missing: line 5:
: command not found/WORKSPACE/xin/software/sources/hdf5-1.10.4/bin/missing: line 8:
: command not found/WORKSPACE/xin/software/sources/hdf5-1.10.4/bin/missing: line 13:
: command not found/WORKSPACE/xin/software/sources/hdf5-1.10.4/bin/missing: line 18:
: command not found/WORKSPACE/xin/software/sources/hdf5-1.10.4/bin/missing: line 21:
: command not found/WORKSPACE/xin/software/sources/hdf5-1.10.4/bin/missing: line 26:
'HOME/nuist_chenq_2/WORKSPACE/xin/software/sources/hdf5-1.10.4/bin/missing: line 32: syntax error near unexpected token `in
'HOME/nuist_chenq_2/WORKSPACE/xin/software/sources/hdf5-1.10.4/bin/missing: line 32: `case $1 in
configure: WARNING: 'missing' script is too old or missing
checking for a thread-safe mkdir -p... /bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking whether make supports nested variables... (cached) yes
checking whether to enable maintainer-specific portions of Makefiles... no
configure: error: cannot run /bin/sh bin/config.sub

It seems the newest version of hdf5 1.10 is not compatiable with the system.

I returned to hdf5 1.8.21 and it works fine.

$ ./configure --prefix=/HOME/nuist_chenq_2/WORKSPACE/xin/software/hdf5-1.10.4 --enable-fortran --enable-static-exec
$ make -j 8 install &> make.log&

Edit ~/.bashrc for installation of NetCDF:

module load intel-compilers/2017_update4 MPI/Intel/MPICH/3.2-icc2017-dyn zlib/1.2.11-icc17 jasper/1.900.1/01-CF-15-libpng
export CC=icc
export FC=ifort
# hdf5 and netcdf
software=/HOME/nuist_chenq_2/WORKSPACE/xin/software
export HDF5=${software}/hdf5-1.8.21
export NETCDF=${software}/netcdf-4.6.2
export PATH=${PATH}:${HDF5}/bin:${NETCDF}/bin
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH=}:${HDF5}/lib:${NETCDF}/lib
export LIBRARY_PATH=${LIBRARY_PATH}:${NETCDF}/lib
# WRF-Chem
export NETCDF4=1
export WRFIO_NCD_LARGE_FILE_SUPPORT=1
export PATH=${PATH}:${software}/byacc-20180609:${software}/flex-2.5.3/bin
export YACC='${software}/byacc-20180609/yacc -d'
export FLEX=${software}/flex-2.5.3/bin/flex
export FLEX_LIB_DIR=${software}/flex-2.5.3/lib

NetCDF-c

$ CPPFLAGS=-I${H5DIR}/include LDFLAGS=-L${H5DIR}/lib ./configure --prefix=${NCDIR}
$ make -j 8 install &> make.log&

NetCDF-fortran

$ CPPFLAGS=-I${NCDIR}/include LDFLAGS=-L${NCDIR}/lib ./configure --prefix=${NCDIR}
$ make -j 8 install &> make.log&

WRF-Chem

Configure and compile

Auto-script see AutoWRFChem-R2SMH.

$ ./autowrfchem config override
Use KPP (kinetic preprocessor)? Default is Y. [y/n]: y
24. Linux x86_64 i486 i586 i686, Xeon (SNB with AVX mods) ifort compiler with icc (dmpar)
19. Linux x86_64, Intel compiler (dmpar)
Choose the meterology type:
1: NARR
2: Other
Enter choice (1-2):1
What would you like to do?
1: Create or modify namelists
2: Show help
3: Quit
Enter 1-3: 1
What namelist would you like to load?
A empty answer will cancel.
1: Load standard templates
2: Load existing namelists
3: Modify the current namelists
4: Quit
Enter 1-4: 1
Choose what to modify or do:
1: Start/end date
2: Domain (common opts only)
3: Meterology
4: Chemistry
5: Other WRF options
6: Other WPS options
7: Check for NEI compatibility
8: Select MOZBC file
9: Display WRF options
10: Display WPS options
11: Save and exit
Enter 1-11: 11
Save the namelists?
A empty answer will cancel.
1: Write namelist regularly (namelist files and pickle)
2: Write just the namelist files (not pickle - i.e. temporary namelists)
3: Save namelists to NAMELISTS folder for later
4: Do not save
Enter 1-4: 1
Goodbye
Configuration complete!
*****************************************************************************
Configuration files generated. Run autowrfchem with 'compile' to begin compilation
*****************************************************************************

Namelists

Link data of MEGAN, MOZBC,NARR to correct path according to script.

Copy my namelists to AutoWRFChem-R2SMH/BUILD/CONFIG/NAMELISTS. Then, use the autowrfchem script to generate namelists again.

$ ./autowrfchem config namelist
What would you like to do?
1: Create or modify namelists
2: Show help
3: Quit
Enter 1-3: 1
What namelist would you like to load?
A empty answer will cancel.
1: Load standard templates
2: Load existing namelists
3: Modify the current namelists
4: Quit
Enter 1-4: 2
Select the namelist files to use. If there is a discrepancy in the domain, the WPS file
takes precedence.
Choose the WRF namelist:
A empty answer will cancel.
1: Standard template
2: namelist.input
3: namelist.input.LNOx_US
..........

Job

Because all processes of preparing data for WRF-Chem is included in autowrfchem, I just need to edit some for mpirun which is yhrun on TH-2.

prepinpt

Here’re scripts in AutoWRFChem-R2SMH/BUILD/PREPINPUT:

bestemissyear.py mozbc-chemopt-list.txt postcheck.py prepmegan prepnei_convert prepwps splitwps
metgriblist.py mozcheck.py precheck prepmozbc prepnei_intermediate pyncdf.py

Since login nodes of TH-2 are unstable and slow, I need to submit jobs to multiple computing nodes.

I want to use 72 (24*3) cores for preparing all input data.

Because autowrfchem doesn’t have the function of setting cores, all ./real.exe in scripts should be replaced by yhrun -n 72 -N 3 -p work ./real.exe.

It’s easier to use sed. I find the solution from question_1 and question_2.

$ pattern='yhrun -n 72 -N 3 -p work'
$ sed -i -- "s/.\/real.exe/${pattern} .\/real.exe/g; s/.\/geogrid.exe/${pattern} .\/geogrid.exe/g; s/.\/metgrid.exe/${pattern} .\/metgrid.exe/g" prep*

Then, create prep.sh in AutoWRFChem-R2SMH directory:

#!/bin/bash
export PATH=~/WORKSPACE/xin/anaconda3/bin:$PATH
source activate base
date
export PATH=/bin/readlink:$PATH
source readlink.sh
./autowrfchem prepinpt > 2014_prep.log
date

readlink.sh is the script to source readlink:

#!/bin/sh
export OSCPP=/WORK/app/osenv/ln1
export PATH=${PATH}:${OSCPP}/usr/local/mpi3-dynamic/bin:${OSCPP}/usr/lib64/qt-3.3/bin:${OSCPP}/usr/kerberos/sbin:${OSCPP}/usr/kerberos/bin:${OSCPP}/usr/local/bin:${OSCPP}/bin:${OSCPP}/usr/bin:${OSCPP}/usr/local/sbin:${OSCPP}/sbin:${OSCPP}/usr/libexec/gcc/x86_64-redhat-linux/3.4.6
export LD_LIBRARY_PATH=/lib64:/usr/lib64:${LD_LIBRARY_PATH}:${OSCPP}/usr/local/mpi3-dynamic/lib:${OSCPP}/usr/lib64:${OSCPP}/usr/lib64/mpich2/lib:${OSCPP}/usr/lib64/mysql:${OSCPP}/lib64:${OSCPP}/intel64:${OSCPP}/usr/libexec/gcc/x86_64-redhat-linux/3.4.6:${OSCPP}/usr/local/cuda/lib:${OSCPP}/usr/local/cuda/lib64
export GNUPLOT_PS_DIR=${OSCPP}/usr/share/gnuplot/4.2/PostScript

Now, I can submit the job:

$ yhbatch -n 72 -N 3 -p work ./prep.sh

Run

The script AutoWRFChem-R2SMH/BUILD/runwrf reads variable ntasks for running wrf.exe by usempi.

However, the submitting script only has cores and nodes.

I need to convert ntasks to cores and nodes around line 156:

elif $usempi; then
nodes=$(expr $ntasks / 24 )
runcmd="yhrun -n $ntasks -N $nodes -p work ./wrf.exe"

Then, create run.sh for running wrf.exe:

#!/bin/bash
export PATH=~/WORKSPACE/xin/anaconda3/bin:$PATH
source activate base
date
export PATH=/bin/readlink:$PATH
source readlink.sh
./autowrfchem run --ntasks=480 > run.log
date

Submit job:

$ yhbatch -n 480 -N 20 -p work ./run.sh

Enjoy TH-2 now!

Anaconda (part)

Because the autowrfchem script needs python which has netCDF4 package installed, I install anaconda on my own.

As TH-2 doesn’t have Internet, crftime and netCDF4 should be uploaded and install from source code.

Note:

Please add CC=gcc before python setup.py install, otherwise you will get error like this:

>>> from netCDF4 import Dataset as ncdat
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/HOME/nuist_chenq_2/WORKSPACE/xin/anaconda3/lib/python3.7/site-packages/netCDF4-1.4.2-py3.7-linux-x86_64.egg/netCDF4/__init__.py", line 3, in <module>
from ._netCDF4 import *
File "netCDF4/_netCDF4.pyx", line 1111, in init netCDF4._netCDF4
File "/HOME/nuist_chenq_2/WORKSPACE/xin/anaconda3/lib/python3.7/site-packages/cftime-1.0.3.4-py3.7-linux-x86_64.egg/cftime/__init__.py", line 1, in <module>
from ._cftime import utime, JulianDayFromDate, DateFromJulianDay
ImportError: /HOME/nuist_chenq_2/WORKSPACE/xin/anaconda3/lib/python3.7/site-packages/cftime-1.0.3.4-py3.7-linux-x86_64.egg/cftime/_cftime.cpython-37m-x86_64-linux-gnu.so: undefined symbol: __intel_sse2_strchr

The solution is found here:

The problem was that I used the Intel compiler without running the Intel Python distribution, or otherwise providing the Intel runtime libraries.

Powered by Hexo and Hexo-theme-hiker

Copyright © 2013 - 2018 Dreambooker All Rights Reserved.

UV : | PV :