Marc2_UserGuide/A_FAQ

Frequently asked questions

How do High Performance Computers (HPC) like MaRC2 differ from Desktop PCs?

On a HPC cluster, we usually do not run CPU-intensive or windowed software programs directly (like e.g. Excel or SPSS on Windows). Instead, you may upload your software to the cluster and must reserve CPU time for deferred execution. In detail, you need to submit a startup script to the batch system, along with your resource requirements (max. runtime, main memory, number of CPU cores etc.). Depending on your requirements, it may take a few hours or even a few days till execution starts. After your software has terminated, you may download your results.

Contrary to common expectations, the CPU core speed on a HPC cluster is not extraordinary high, but comparable to common Desktop CPUs: On MaRC2, for example, the CPU cores are running at 2.3 GHz. However, performance comes through multi-core CPUs and software parallelization (multi-threading and/or grid computing). Thus each MaRC2 compute node is equipped with 4 multi-core CPUs (with 16 cores each), resulting in 64 CPU cores per compute node.

MaRC2 uses the GNU/Linux operating system instead of Microsoft Windows, thus you may only run software which has been (or may be) compiled for Linux. Please note that parallelization and non-interactive execution must be supported by your software as well - sometimes there is a special HPC version available.

When my job has finished, can I be notified by email?

Yes, just use qsub options like "-m bea" (mail will be sent after begin, end or abort) to get notification emails under your account email address. To specify a custom email address, use the -M option.

How large can my task array be?

Your task array can be of reasonably large size. However, you can run only 88x64=5632 tasks simultaneously.

My job is listed in state "Eqw". What should I do?

Use the -explain switch:

qstat -j <your job id> -explain E

qstat says that gathering scheduler information is disabled. How do I get the information I need?

Please use the following command to get scheduler information for your job:

qalter -w v <your job id>

Jobs complain about missing libmpich

When I use the compiler wrapper, ldd of the produced binary says everything is fine. On the node within batch it complains about missing libraries, e.g. libmpich.so.

Resolution: use the appropriate module(s), see example job script.

Java quits due to lack of memory

The Java Virtual Machine usually needs more RAM than configured with the -Xmx option, thus you need to provide a higher h_vmem value for qsub.

Using a simple "hello world" program and a short wall clock time (h_rt), you may experimentally figure out the amount of memory needed for a given -Xmx value. Remember to also set -Xms to the same value as -Xmx, as otherwise the Java VM will not consume all the asserted memory at startup. If this job exits without error, your h_vmem value is big enough.

Here is an example startup script (to be submitted by qsub):

#$ -S /bin/bash 
#$ -e /path/to/stderr-output-file 
#$ -o /path/to/stdout-output-file 
#$ -l h_rt=10
#$ -l h_vmem=9G
java -Xms5120m -Xmx5120m -cp /path/to/HelloWorld.jar HelloWorld

I have accidentally modified/deleted some files. Is there a backup?

The /home filesystem is backed up each night by ITSM. In order to have your files restored, please write an email to marc[at]hrz.uni-marburg.de and tell us the directory and date of the files to be restored.

What is the NUMA architecture on MaRC2?

For an introduction to NUMA, see Wikipedia article:  http://en.wikipedia.org/wiki/Non-uniform_memory_access

The following picture describes the NUMA architecture on a MaRC2 compute node (click to enlarge):

NUMA architecture on node001

What software is currently installed on MaRC2?

The following programs and libraries are available for all users of the cluster (last updated on December 12, 2016):

Directly installed on all nodes

Libraries

Version Description
ghostscript 8.70 PDF and PostScript Renderer,  http://www.ghostscript.com
gstreamer 0.10.29 Multimedia Framework,  http://gstreamer.freedesktop.org
ncurses 5.7 CLI Interface Library,  https://www.gnu.org/software/ncurses
pulseaudio 0.9.21 Sound System,  http://www.freedesktop.org/wiki/Software/PulseAudio

Compiler/Runtimes/Tools

Version Description
gcc 4.4.7 GNU Compiler Collection,  https://gcc.gnu.org
doxygen 1.6.1 Documentation Generator,  http://www.doxygen.org
git 1.7.1 Distributed Version Control System,  http://git-scm.com
subversion 1.6.11 Version Control System,  https://subversion.apache.org
openjdk 6u34 Open Source Java Development Kit,  http://openjdk.java.net
octave 3.4.3 Programming Language,  https://www.gnu.org/software/octave
perl 5.10.1 Programming Language,  https://www.perl.org
python 2.6.6 Programming Language,  http://www.python.org
ruby 1.8.7.374 Programming Language,  http://www.ruby-lang.org
sqlite3 3.6.20 Serverless SQL Database Engine,  https://sqlite.org
matlab R2012a Programming Language,  https://mathworks.com

Text editors

Version Description
vim 7.2.411 Command Based Powerful Editor,  http://www.vim.org
emacs 23.1 Shortcut Based Powerful Editor,  http://www.gnu.org/software/emacs
nano 2.0.9 Userfriendly Editor,  http://www.nano-editor.org

Additionally installed on login nodes only

Version Description
boost 1.41.0 C++ Libraries,  http://www.boost.org
lapack 3.2.1 Numerical Linear Algebra Library,  http://www.netlib.org/lapack
atlas 3.8.4 Numerical Linear Algebra Library,  http://math-atlas.sourceforge.net

Available via module environment

Libraries

Version Description
fftw 3.3.1, 3.3.4, 3.3.6-pl2 Discrete Fourier Transformation Library,  http://www.fftw.org
eigen 3.2.3 Linear Algebra Template Library,  http://eigen.tuxfamily.org
boost 1.57.0 C++ Libraries,  http://www.boost.org
ffms2 2.20.0 Wrapper Library for ffmpeg,  https://github.com/FFMS/ffms2
opencv 2.4.10 Computer Vision Machine Learning Library,  http://opencv.org

Compiler/Runtimes/Tools

Version Description
acml(gfortran/ifort/open64/pgi) 5.1.0, 5.2.0, 5.3.1 AMD Core Math Library,  http://developer.amd.com/tools-and-sdks/cpu-development/amd-core-math-library-acml
atlas 3.10.1 Numerical Linear Algebra Library,  http://math-atlas.sourceforge.net
gcc 4.4.6, 4.6.2, 4.7.2, 4.8.2, 5.2.0, 6.3.0 GNU Compiler Collection,  https://gcc.gnu.org
intelPSXE 2011SP1, 2013SP1, 2015SP2, 2016, 2017u4 Intel Compiler and Tools,  https://software.intel.com/en-us/intel-parallel-studio-xe
openBLAS 0.2.19 Numerical Linear Algebra Library,  http://www.openblas.net
openmpi(gcc/intel/open64/pgi) 1.4.3, 1.6.3, 1.6.5, 2.1.0, 2.1.1 Open Source MPI,  http://www.open-mpi.org
oraclejdk 7u25 Oracle Java Development Kit,  http://www.oracle.com/technetwork/java/javase
parastation-mpi2(gcc/intel/pgi) 5.1.5 Parastation MPI,  http://www.par-tec.com/products/parastation-mpi.html
pgi 12.2, 13.2, 15.1, 16.10-Community PGI Compiler and Tools,  http://www.pgroup.com
cmake 2.8 Open Source Build System,  http://www.cmake.org
python 2.7, 3.5 Programming Language,  http://www.python.org
R 2.15.0, 3.0.1, 3.2.0 Programming Language for Statistical Computing,  http://www.r-project.org
vampir 8.4 Performance Monitoring,  https://www.vampir.eu
ffmpeg 2.5.3 Leading Multimedia Framework,  https://www.ffmpeg.org
pigz 2.3.3 Parallel threaded gzip implementation,  http://zlib.net/pigz/
matlab R2016a, R2017a Programming Language,  https://mathworks.com

Biology programs

Version Description
beast 2.1.3 Bayesian Analysis of Molecular Sequences,  http://beast.bio.ed.ac.uk
blast 2.2.26 Local Alignment Search Tool,  http://blast.ncbi.nlm.nih.gov/Blast.cgi
blast+ 2.2.29+ BLAST Tools for NCBI C++,  http://blast.ncbi.nlm.nih.gov/Blast.cgi
bedtools 2.17.0 Genomic Analysis Tools,  https://github.com/arq5x/bedtools2
bowtie 1.0.0 DNA Sequence Aligner,  http://bowtie-bio.sourceforge.net/index.shtm
bowtie2 2.1.0 DNA Sequence Aligner,  http://bowtie-bio.sourceforge.net/bowtie2/index.shtml
bwa 0.7.5a Burrows-Wheeler Aligner,  http://bio-bwa.sourceforge.net
cmtk 2.3.0, 3.2.3 Morphometry of Biomedical Images,  http://www.nitrc.org/projects/cmtk
jellyfish 1.1.11 Approximate and Phonatic String Matching,  https://github.com/sunlightlabs/jellyfish
mrbayes 3.2.3-767 Bayesian Inference of Phylogeny,  http://mrbayes.sourceforge.net
proteinortho 5.11, 5.13 Orthology Detection Tool,  https://www.bioinf.uni-leipzig.de/Software/proteinortho
samtools 0.1.19 Sequencing Data Tools,  http://www.htslib.org
smrt-analysis 2.3.0 Assembly and variant detection analysis of sequencing data,  http://www.pacb.com/products-and-services/analytical-software/smrt-analysis/
tcoffee 10.00.r1613 Multiple Alignment Sequences Tools,  http://www.tcoffee.org
tophat 2.0.10 Read-Mapper for RNA Sequences,  http://ccb.jhu.edu/software/tophat/index.shtml
trinityrnaseq r20131110 De Novo Reconstruction of Transcriptomes,  http://trinityrnaseq.github.io/
velvet 1.2.10 De Novo Assembly Tool,  http://www.ebi.ac.uk/~zerbino/velvet

FB Chemistry (own module environment)

Application Version Type of License Validity Period Yearly Maintenance Fee
Gaussian16 A.03 Site license for source code (Linux/macOS) 2037-06-28 no
Gaussian09 C.01 Site license for source code (Linux/macOS) 2032-02-01 no
Gaussian03 E.01 Site license for source code 2027-12-12 no
Linda for parallel execution of Gaussian03 7.1a Site license for Linda object code for i386, x86_64, AIX, Tru64 2027-12-12 no
NBO 5.G Group license for AG Berger source code unlimited no
NBO 6.0 Site license for source code (Linux/macOS) unlimited no
GaussView 5.0 Single machine license for MaRC2 headnode unlimited no
GaussView 5.0 Single machine license for Annemarie headnode unlimited no
Turbomole 7.2 Department license (FB15/FB16) for Linux on x86_64 unlimited yes
ADF 2017.107 Group license for parallel execution for AG Frenking (ask for access) 2018-04-01 yes
ADF-VIEW 2017.107 Group license for AG Frenking 2018-04-01 yes
ADF 2017.107 Group license for parallel execution for AG Tonner (ask for access) 2018-08-01 no
ADF-VIEW 2017.107 Group license for AG Tonner 2018-08-01 no
Molcas 6.4sp1 Group license incl. source code unlimited no
Molpro 2015.1p22 Group licenses for AG Frenking / AG Berger (Linux/macOS) 2017-11-05 yes
VASP 5.4.4 Group licenses for source code AG Berger / AG Tonner unlimited no
Chemcraft b523a Site license 32 Bit (Windows32, macOS [Wineskin], Linux [Wine]) 2024 no
Chemcraft b523b Site license 64 Bit (Windows64) 2024 no
Intel Cluster Studio XE 2016 Update 2 Floating 2-seat license (ask for access) unlimited yes (expired)

Yearly maintenance fees will include any updates for the application as long as they are paid.

Worth to note: even if someone wants to use a software from this list which is not available to him/her according to the above terms, it's sometimes cheaper to extend an already existing license than buying a fresh independent one (e.g. Turbomole for another platform or the physics department, or extending Molpro to a four-groups license,…).

The list of directly installed software is by no means exhaustive; it includes programs and libraries that are likely to see frequent use. Programs used and installed only by particular workgroups are also partially excluded. If there is any program or library you need that is not on this list, feel free to contact us via marc@….

Attachments