Zentrum für Datenverarbeitung (ZDV)

What is Singularity

Singularity allows you to create your own environment within a container. You can install all workflows, software, and libraries you need into that container. Maybe you have heard of the popular container software 'Docker'? You could describe Singularity in a sloppy way as 'Docker for HPC-Systems'. Later you can execute the software in your container on BinAC in the same way as you execute 'normal' software, namely via the the batch system. The 'Getting Started' guide shows you how to create your own container and execute it on BinAC.

Problems Singularity can solve

Getting Started

The general workflow for Singularity container includes the following steps:

  1. Install Singularity on your own machine where you can be root (e.g. your laptop, your working station, at home, …)
  2. Create the container on your machine
  3. Upload the container to a BinAC workspace
  4. Run the container on BinAC

1. Install Singularity on your machine

You have to be root to create a new Singularity container or to change an existing Singularity container. Therefore you have to install Singularity on a machine where you can be root. Please install Singularity 2.3.1. This is the version that is also installed on BinAC. The installation for Linux is explained on the Singularity website:

singularity.lbl.gov/install-linux

Singularity runs only on Linux machines, but there is a guide to run Singularity via Vagrant on Mac and Windows:

Mac: http://singularity.lbl.gov/install-mac

Windows: http://singularity.lbl.gov/install-windows

 

2. Create the Singularity container

First, you will create an empty Singularity container. Then you will install the guest operating system and additional software into the empty Singularity container (this is called bootstrapping). You have to create and bootstrap the container on the command line:

First, create an empty container:

 

sudo singularity create --size 500 my_first_container.img

 

This will create an empty container with the name 'my_first_container.img' in your working directory. A container behaves like a normal file on your file system. That means you can copy, move, and delete the container as you do it with normal files. The container has a size of 500 Mb.

 

Second install the guest operating system and additional software:

You have to create a so-called 'Singularity definition file' for that step. This file describes what operating system will be installed in the container . You can also specify additional software that should be installed automatically in the container. For your first container, we will install CentOS 7 and install some additional packages.

Create a file called 'Singularity' with the content:

 

BootStrap: yum
OSVersion: 7
MirrorURL: http://mirror.centos.org/centos-%{OSVERSION}/%{OSVERSION}/os/$basearch/
Include: yum %post # Install some additional packages yum -y install epel-release
yum -y install cowsay %runscript # This will be executed when you run the container with singularity run echo "Hello World from inside the container" | cowsay

 

 

Then you can bootstrap the Singularity container with the following command:

 

sudo singularity bootstrap my_first_container.img Singularity

 

 

3. Upload the container to your workspace on BinAC

 

If you don't have a workspace on BinAC yet, create one:

https://www.bwhpc-c5.de/wiki/index.php/BwForCluster_BinAC_Hardware_and_Architecture#Work_Space

 

You have to copy the container into an workspace. It has to be in the workspace, because we configured Singularity such that only container in an workspace can be executed. So upload the container in your BinAC workspace using scp on Linux or WinSCP on Windows:

 

scp my_first_container.img <login_id>@login01.binac.uni-tuebingen.de:<path_to_your_workspace>

 

4. Run the Container

4.1 On your local machine

To run the container, use the singularity run command:

 

singularity run my_first_container

 

This will execute the command we have specified in the %runscript section of the Singularity definition file.

You can also run arbitrary commands inside the container with the singularity exec command:

 

 singularity exec my_first_container.img cat /etc/os-release

 

This will run the command 'cat /etc/os-release' INSIDE the container. Therefore it prints some information about the operating system INSIDE the container to your command line. You can compare the output with the one of 'cat /etc/os-release' on the login-node.

4.2 On BinAC

Singularity is not installed on the login nodes, therefore you cannot start the container on a login node. You have to submit it to the batch system.

First, write a shell script 'my_first_container_batch_job.sh' and set your workspace where the container resides:

 

#PBS -N my_first_container_batch_job
#PBS -l nodes=1:ppn=1,walltime=00:01:00 # Change into the directory where the container resides cd <path_to_workspace> singularity run my_first_container.img > my_first_container_output.txt

 

 

Submit the job:

 

qsub -q short my_first_container_batch_job.sh

 

After the job was completed the output should be in the file my_first_container_output.txt.

5. Remove the container from BinAC!!

This example has no scientific value (beside to get in touch with Singularity). So please remove the container from your workspace afterwards!

Getting the real work done!

We created and started our first Singularity Container in the 'Getting Started' guide. Now we go more in detail. We will answer some questions you might have regarding container in general or your use-case.

Can I inspect what is inside the container?

You can open a shell inside the Singularity container and inspect the file system, software, environment, etc. that lives inside the container:

 

sudo singularity shell <container_file>

 

If you want to install new software, add/remove/change files, or other things that will change the container, you have to be root:

 

sudo singularity shell -w <container_file>

 

The -w flag tells singularity that you want to change something inside the container. Without the -w flag, the container is just readable.

Can I install new software in a bootstrapped container?

You can, but just on your local machine where you can become root. Just shell into the container:

 

sudo singularity shell -w <container_file>

 

Now you can install new software inside the container as you would do it on your own local machine. If you are done, close the shell inside the container:

 

exit

 

It is wise to add changes also in the Singularity definition file such that the changes are also applied if you bootstrap a new container!

What permissions do I have during the container execution?

This is simple: You have the same permissions inside the container and outside the container. Inside the container you can do everything that you could do 'in a normal shell' on BinAC.

Can I install another guest operating system in the container?

You can! Please visit the following website that explains how to bootstrap Debian, Ubuntu, Arch Linux, …:

http://singularity.lbl.gov/docs-bootstrap

Can I also use an existing Docker container?

You can. You can bootstrap a Singularity container from an existing Docker container! It's also explained here:

http://singularity.lbl.gov/docs-bootstrap#docker

One example:

If you want to have a container with CUDA and cuDNN installed, just use one of the NVIDIA docker images (https://hub.docker.com/r/nvidia/cuda/) for the bootstrapping. The header section of the Singularity definition file is then:

 

BootStrap: docker
From: nvidia/cuda:8.0-cudnn5-devel-centos7

 

The container will then contain Centos7 as guest operating system and has Cuda 8.0 and cuDNN 5.1 installed. Afterwards you can install additional software that needs Cuda (e.g. tensorflow, caffe, ...)

How can I use GPUs with Cuda, cuDNN, etc.?

In the container you will need your own Cuda and maybe other libraries needed by your application. You cannot use the modules available on BinAC. A nice way is to bootstrap from a NVIDIA docker image (see above) and then install the software you need in the %post section of the Singularity definition file.

When you run your Singularity container on BinAC, use the –nv flag to tell Singularity that you need GPUs. It will use the GPU drivers on the BinAC. You don't need to install GPU driver in the container!

 

singularity run --nv <container_file>

 

or

 

singularity exec --nv <container_file> <command inside container>

 

How can I use my data inside the container?

Your workspaces are mounted in the container by default. Therefore you can access everey file in one of your workspaces from within the container by using the absolute path to it. Your home directory is also accessible from inside the container.

How are paths handled in Singularity Container

Because the container has its own root file system, every path starting with '/' will be resolved in your container. There are some directories, that are automatically mounted in the container. These are /tmp, /scratch, /beegfs/work/workspace/ws/<your_workspace>, /home-link.

Every change you make to these directories from inside the container WILL change the files on the BinAC file system. If your container removes your home directory, your home directory on BinAC is deleted. So beware, especially if you remove directories in more complicated scripts inside your container.

How can I use openMPI?

Coming soon.

Getting Help

There is a good user guide on the Singularity website that explains everything in more detail:

http://singularity.lbl.gov/user-guide

There is also an active Singularity Google-Group:

https://groups.google.com/a/lbl.gov/forum/#!forum/singularity

Or ask someone of the BinAC team:

hpcmasterspam prevention@uni-tuebingen.de

Tips

Container Size

Often a container is too small or too big. A too small container results in errors during bootstrapping or installing new software. A too big container results in a unnecessary long upload time to BinAC and memory usage. Therefore, it is a good idea to create a 'big' container on your local machine. After you installed the software you need, you can get the used space in the container by:

 

sudo singularity exec -w <your_container> df -h

 

Then you get the exact container size you need for your use-case. You can now create a new container with that size (+ a few Mb) and install again. Now your container is just as big as it needs to be.

Possible Problems

Because you cannot change Singularity images as normal user you have to consider one some things. Every time your application wants to write some data, the destination has to be OUTSIDE the container (in your home, workspace, /tmp). This is a especially a problem if an application tries to make directories inside the container (e.g. a cache, a temporary directory, …). You have to tell your application then either

  • not to create these directories
  • or to create them somewhere OUTSIDE the container (your home, workspace, /tmp)