Singularity

Singularity is a container system for high performance computing. It allows users to create a reproducible environment for their applications (it is even possible to containerize whole operating systems). This makes it easier to develop applications locally and later deploy them on the server, where they will work exactly the same. Especially in the context of scientific work, ensuring reproducibility allows sharing programs between researchers as containers can simply be duplicated.

If you only want to run or download an existing container, you can continue reading at Running Containers on BinAC.

In order to build custom containers you need to use a Linux system with root privileges. Tests with Linux subsystems on Windows have been unsuccessful but you can run Linux in a virtual machine.

First of all you need to install Singularity 3 on your machine (most packet managers only support older versions of Singularity):

https://www.sylabs.io/guides/3.1/user-guide/quick_start.html#quick-installation-steps

With the command

we create a mutable container with a complete CentOS installation in our current working directory. You should now find a folder called <containername> in it.

Using  exec --writable and shell --writable we can already run commands within the container, just as if we were running CentOS on our machine.

We could for example update the packet manager

and open a shell to download ClustalW2.

In the shell type:

ClustalW2 is used to calculate multiple sequence alignments for DNA and proteins.

Finally we can create an immutable image of our container:

We can execute ClustalW2 with the follwing command:

Now that you've got a feeling of how singularity works, it is time to introduce a better way to build and execute containers.

For the use of Singularity in production systems it is recommended to construct containers using definition files. Not only does it make the process less labour intensive but it also ensures that all containers built from the same definition file will be identical.

A file with this content

will lead to the same container we manually built earlier. Use

to build it.

Definition files are split into sections which represent different stages of the construction process. E.g. %post is used to list the commands that should be executed immediately after the installation of the operating system.

The only thing new are runscripts. Commands listed in this section will be triggered, when singularity is called with the argument run.

Now,

is enough to execute ClustalW2.

See https://www.sylabs.io/guides/3.0/user-guide/definition_files.html#definition-files for all the options that can be set in a definition file.

Since the Singularity-Modul is not loaded by default, we first have to locate and load it using modul avail and module load.

Next, we need to upload the container from our machine onto BinAC or download one from an external service. A simple way to upload files is scp.

Alternatively you can download existing containers form DockerHub or SingularityHub. After changing into you workspace directory

You can download a container using the schemes docker://<user>/container_name>:<tag> for DockerHub or shub://<user>/<container_name>:<tag> for SingularityHub. For example:

Some biocontainer are only available on quay.io. Use the DockerHub scheme for those: docker://quay.io/biocontainers/<container_name>:<tag>.

We will now execute our custom container we previously built. In case you didn't follow that part of the tutorial you can also find the container at docker://fbartusch/clustalw2:latest.

First we'll download and unpack small sample dataset.

After that we can compute multiple sequence alignments (in this case Homoglobin Alpha sequences of different species)

and view the results using cat

Calculating alignments on the test dataset didn't take long but long running tasks will be automatically terminated to protect the login nodes of the cluster from overload. Therefore it is necessary to hand the program execution over to a scheduler running in the background whoose job it is to allocate resources on the cluster. We can do that using batchjobs. A jobscript for our ClustalW2 example could look like this:

We can pass it to the scheduler using

In most cases singularity containers can use GPUs just like any other program built for GPU usage.

First we will start an interactive job and probe for available GPUs outside a container.

To check the availability within a container we will open a shell in the container and execute the same command there. We should get the same response.

The parameter -nv makes the NVIDIA driver available within the container.

Last but not least a small example for training a small machine learning model with containers and GPUs:

• Singularity-Dokumentation
• Singularity-Slack-Channel