OpenHPC - Developer Tooling
OpenHPC is an Open Source project that provides, among other things, access to current versions of many developer tools. To find out more about the project and the functionality it provides visit the community website
Usage
OpenHPC leverages lmod to handle loading multiple development environments. The below is a very quick introduction on how to use lmod. If you would like further details, please visit the lmod documentation
Using the module command
To see what modules you currently have loaded, use the module list command.
$ module list
No modules loaded
To see the available developer tools you can use the module avail command.
$ module avail
-------------------------- /opt/ohpc/pub/modulefiles --------------------------
cmake/3.13.4 gnu8/8.3.0 intel/18.0.0.128 intel/19.0.3.199 (D) prun/1.3
Where:
D: Default Module
Use "module spider" to find all possible modules.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".
In this case you can see that gnu8 and two versions of the Intel compiler are available. The (D) indicates which version of a module is loaded when no arguments are given. To load the Intel compiler, you can use the module load intel command. We can verify that version 19.0.3.199 was loaded by using icc --version.
$ module load intel
$ icc --version
icc (ICC) 19.0.3.199 20190206
Copyright (C) 1985-2019 Intel Corporation. All rights reserved.
If you decide that you no longer need the Intel compiler and instead need the gnu versions of the compiler, you can use the module swap command. In our example above we would swap the intel compiler for the gnu8 compiler, and then confirm that the proper version of gcc was loaded.
$ module swap intel gnu8
$ gcc --version
gcc (GCC) 8.3.0
Copyright (C) 2018 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Failure to use the swap option will show an error such as
$ module load gnu8
Lmod has detected the following error: You can only have one compiler module loaded at a time.
You already have intel loaded.
To correct the situation, please execute the following command:
$ module swap intel gnu8/8.3.0
While processing the following module(s):
Module fullname Module Filename
--------------- ---------------
gnu8/8.3.0 /opt/ohpc/pub/modulefiles/gnu8/8.3.0
Note that after a compiler family is loaded, you will then have access to various MPI libraries. After you’ve loaded the compiler family, use module avail again to find out what MPI libraries are available.
$ module load intel
$ module avail
-------------------------------- /opt/ohpc/pub/moduledeps/intel --------------------------------
openmpi3/3.1.3
---------------------------------- /opt/ohpc/pub/modulefiles -----------------------------------
cmake/3.13.4 gnu8/8.3.0 intel/18.0.0.128 intel/19.0.3.199 (L,D) prun/1.3
Where:
D: Default Module
L: Module is loaded
Use "module spider" to find all possible modules.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".
$ module load openmpi3
$ module avail
--------------------------- /opt/ohpc/pub/moduledeps/intel-openmpi3 ----------------------------
phdf5/1.10.4
-------------------------------- /opt/ohpc/pub/moduledeps/intel --------------------------------
openmpi3/3.1.3 (L)
---------------------------------- /opt/ohpc/pub/modulefiles -----------------------------------
cmake/3.13.4 gnu8/8.3.0 intel/18.0.0.128 intel/19.0.3.199 (L,D) prun/1.3
Where:
D: Default Module
L: Module is loaded
Use "module spider" to find all possible modules.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".
Using modules in PBS jobs
After using the module load command to load the environment that you wish to use, submit the to PBS and include the -V switch. This will make your current environment available your job. For example, I have a simple job that runs module avail after loading intel, openmpi3, and phdf5. As you can see from the job output, all of the modules were loaded on the node in which the job ran.
$ module load intel openmpi3 phdf5
$ qsub -lselect=1:ncpus=112 -V ./driver.sh
132837.iam-pbs
$ cat results.*
------------------- /opt/ohpc/pub/moduledeps/intel-openmpi3 --------------------
phdf5/1.10.4 (L)
------------------------ /opt/ohpc/pub/moduledeps/intel ------------------------
impi/2019.3.199 openmpi3/3.1.3 (L)
-------------------------- /opt/ohpc/pub/modulefiles ---------------------------
cmake/3.13.4 intel/18.0.0.128 prun/1.3
gnu8/8.3.0 intel/19.0.3.199 (L,D)
Where:
D: Default Module
L: Module is loaded
Use "module spider" to find all possible modules.
Use "module keyword key1 key2 ..." to search for all possible modules matching
any of the "keys".
The -V switch also applies to interactive jobs. To use an environment loaded by module load in your interactive job, submit it with the -V flag. For example:
$ module load intel openmpi3
$ qsub -q xeon -I -V
qsub: waiting for job 133137.iam-pbs to start
qsub: job 133137.iam-pbs ready
$ module list
Currently Loaded Modules:
1) intel/19.0.3.199 2) openmpi3/3.1.3
Requesting Installation of additional components
OpenHPC supports a wide variety of compilers and MPI versions. You can see the full list here . If you need an additional component installed, send an email to iam@intel-research.net and we’ll be happy to get it installed.