Singularity
Singularity is a container platform that allows users to have full control of the environment in which their application runs. It is a safe and secure alternative to Docker, as Docker requires root access for much of its functionality, whereas Singularity allows the safe running of application bundles without requiring root privileges. It is even possible to convert Docker images to Singularity.
Our public computing systems offer Singularity at this time in order to enable the portability of highly specialized applications without some of the risks involved with other solutions such as Docker.
In order to use Singularity, you can either create an image of your application or download it from a hub. Your custom image must be created in either a Linux, Mac, or Windows environment, and then can be moved to any system on which Singularity is installed, such as our public computing systems, where it can be executed without administrative privileges.
Getting help
The most up-to-date help on Singularity comes from the command itself:
singularity help
The user guide and other helpful information can be found at Sylabs.io.
Singularity setup
No setup is necessary. Singularity is installed on all CS clusters.
You can download (pull) a container from a hub. Popular hubs are Docker Hub and Singularity Hub. You can search for a container that fits your requirements.
Pull a container from a hub
Docker Hub
Singularity can be used to pull docker images from the hub. Docker is not installed on any of the CS servers, so you would have to either run it locally or look through docker hub.
For example, to fetch a tensorflow image from Docker Hub, first find the image using docker on your own machine (as root).
$ docker search tensorflow NAME DESCRIPTION STARS OFFICIAL AUTOMATED tensorflow/tensorflow Official Docker images for the machine learn… 1828 ..........
You now have a Singularity image tensorflow.sif in your current directory for use with the singularity command.
$ singularity pull tensorflow.sif docker://tensorflow/tensorflow Using default tag: latest latest: Pulling from tensorflow/serving f22ccc0b8772: Downloading [===========> ] 6.106MB/26.71MB .........
Additional information can be found here.
Singularity hub
$ singularity search tensorflow No users found for 'tensorflow' Found 1 collections for 'tensorflow' library://emmeff/tensorflow .......... $ singularity pull library://emmeff/tensorflow/tensorflow INFO: Downloading library image 114.19 MiB / 549.83 MiB [================>--------------------------------------------------------------] 20.77% 11.40 MiB/s 00m37s ............
Working with images
In order to use singularity on the Beowulf cluster, you need to start an interactive job or submit a batch-job to the available slurm queues.
The below example will illustrate the interactive use of Singularity in an interactive bash shell.
$ salloc srun --pty $SHELL -l salloc: Granted job allocation 2885955 $ which singularity /bin/singularity $ singularity --version singularity version 3.5.2-1.1.sdl7
Shell - Run a shell within a container
$ singularity shell tensorflow_latest.sif Singularity> pwd /u/nobody
Commands within a container
Exec - Run a command within a container
$ singularity exec tensorflow_latest.sif cat /etc/os-release NAME="Ubuntu" VERSION="18.04.5 LTS (Bionic Beaver)" ID=ubuntu
Run - Run the user-defined default command within a container
$ singularity run tensorflow_latest.sif ________ _______________ ___ __/__________________________________ ____/__ /________ __ __ / _ _ \_ __ \_ ___/ __ \_ ___/_ /_ __ /_ __ \_ | /| / / _ / / __/ / / /(__ )/ /_/ / / _ __/ _ / / /_/ /_ |/ |/ / /_/ \___//_/ /_//____/ \____//_/ /_/ /_/ \____/____/|__/ You are running this container as user with ID xxxxxx and group xxx, which should map to the ID and group for your user on the Docker host. Great! Singularity containers as Slurm jobs
#!/bin/bash # #*** #*** "#SBATCH" lines must come before any non-blank, non-comment lines *** #*** # # 1 node, 1 CPU per node (total 1 CPU), wall clock time of 30 hours # #SBATCH -N 1 ## Node count #SBATCH --ntasks-per-node=1 ## Processors per node #SBATCH -t 30:00:00 ## Walltime SBATCH --gres=gpu:1 --ntasks-per-node=1 -N 1 # # send mail if the process fails #SBATCH --mail-type=fail # Remember to set your email address here instead of nobody #SBATCH --mail-user=nobody@cs.princeton.edu # singularity exec tensorflow_latest.sif cat /etc/os-release
References
https://github.com/ArangoGutierrez/Singularity-tutorial
https://hpc.nih.gov/apps/singularity.html