About the service¶
The CISM manage several compute clusters. A cluster is a made of computers (nodes) that are interconnected and appear to the user as one large machine. The cluster is accessed through a frontend where users can manage their jobs and data in their home directory. Each node has a certain number of processors, a certain amount of memory (RAM) and some local storage (scratch space). Each processor comprises several independent computing units (cores). In a hardware context, a CPU is often understood as a processor die, which you can buy from a vendor and fits into a socket on the motherboard, while in a software context, a CPU is often understood as one compute unit, a.k.a. a core. All the resources are managed by a Resource Manager/Job Scheduler to which users must submit batch jobs. The jobs enter a queue, and are allocated resources based on priorities and policies. The resource allocation is exclusive, meaning that when a job starts, it has exclusive access to the resources it requested in at submission time.
The software installed on the clusters to manage resources is called Slurm ; an introductory tutorial can be found here.
The CISM operates two clusters: Lemaitre and Manneback. Lemaitre was named in reference of Georges Lemaitre (1894–1966), a Belgian priest, astronomer and professor of physics at UCLouvain. He is seen by many as the father of the Big Bang Theory but also he is the one who brought the first supercomputer to our University.
Manneback was named after Charles Manneback (1894-1975), Professor of Physics at UCLouvain. Close friend to Georges Lemaitre, he was the lead of the FNRS-IRSIA project to build the first supercomputer in Belgium in the 50’s.
You can see Manneback (left) and Lemaitre (right) in the picture below, picture that was taken during a visit of Cornelius Lanczos (middle) at Collège des Prémontrés in 1959 (click on the image for more context).
Besides Lemaitre3 and Manneback, and thanks to various collaborations (CECI, EuroHPC, Cenaero and PRACE), CISM users can access more clusters, of multiple sizes. Clusters are organised according to their size into three categories:
The general rule is that access to a higher-up category Tier-N is conditioned on being able to demonstrate proper abilities in the Tier-(N+1) category.
Access & conditions¶
- access to Tier-2 clusters is granted automatically to every member of the University, a CÉCI account must be requested;
- access to Tier-1 clusters requires the submission of a Tier-1 project and a valid CÉCI account ;
- access to Tier-0 clusters requires responding to call for EuroHPC proposals and a valid UCLouvain portal account ;
- access to Tier-2 is free, usage is free as well but users are encouraged to contribute by requesting computing CPU.hours in their scientific project submissions to funding agencies;
- users are requested to tag their publications that were made possible thanks to the CISM/CÉCI infrastructure.
List of clusters¶
|Cluster name||Funding||Size||Hosting institution||More information|
|Zenobe||Walloon Region||Tier-1||Cenaero (Gosselies)||https://tier1.cenaero.be/fr/zenobe|
The Lemaitre cluster is a CÉCI clusters and is shared among all CÉCI users, by
contrast with Manneback, which is a UCLouvain-only machine. Consequently, all
documentation regarding Lemaitre will be found on the CÉCI documentation
website, while only Manneback is described
in this document. As the name implies this is the third generation of Lemaitre
cluster. Below is a summary of the cluster resources as output by
[dfr@lemaitre3 ~]$ sinfo "Partitions:" batch* (2days) debug (6hours) "Nodes:" #Nodes Partition CPU Cores/Slots Memory GPUs 77 batch* intel,skylake5000,5118 24 93G (null) 4 debug intel,haswell,e5-2690v4 28 63G (null) "Filesystems:" Filesystem quota $HOME 100G $CECIHOME 100.0GiB $CECITRSF 1.0TiB $GLOBALSCRATCH unlimited # 380TB available
More information available on the CÉCI website
Manneback is a cluster built with hardware acquired progressively thanks to multiple funding solutions brought by CISM users. It is configured for most aspects just like the CÉCI clusters so the CÉCI documentation mostly applies for Manneback as well.
While CÉCI clusters are mostly homogeneous, Manneback is made of several different generations of hardware.
sinfo command to learn about the available hardware on Manneback
"Partitions": Def* (5days) keira (5days) pelican (5days) cp3 (5days) cp3-gpu (5days) gpu (5days) "Nodes": #Nodes Partition Features Cores/Slots Memory GPUs 16 cp3 CascadeLake,Xeon,4214,CentOS7 48 187G 1 cp3-gpu SandyBridge,Xeon,E5-2640,CentOS7 20 63G TeslaK80:2 13 cp3 IvyBridge,Xeon,E5-2695v2,CentOS7 48 126G 2 cp3 K10,Opteron,6134,CentOS7 16 31G 1 cp3 Milan,EPYC,7452,CentOS7 128 504G 8 cp3 Rome,EPYC,7452,CentOS7 128 504G 2 cp3 Rome,EPYC,7452,Rocky8 128 503G 18 cp3 SkyLake,Xeon,4116,CentOS7 48 187G 24 Def* Haswell,Xeon,E5-2630v3,CentOS7 16 63G 8 Def* IvyBridge,Xeon,E5-2650v2,CentOS7 16 63G 13 Def* SandyBridge,Xeon,E5-2650,CentOS7 16 63G 1 Def* SandyBridge,Xeon,E5-4620,CentOS7 32 126G 1 Def* SandyBridge,Xeon,E5-4640,CentOS7 32 252G 6 Def* SkyLake,Xeon,5118,CentOS7 24 94G 1 Def* Zen,EPYC,7551,CentOS7 128 504G 1 gpu CascadeLake,Xeon,5217,Tesla,TeslaV100,CentOS7 16 377G TeslaV100:2 1 gpu CascadeLake,Xeon,5217,Tesla,TeslaV100,CentOS7 32 377G TeslaV100:2 1 gpu CascadeLake,Xeon,6244,GeForce,GeForceRTX2080T 32 376G GeForceRTX2080Ti:6 1 gpu IceLake,Xeon,6346,Tesla,TeslaA10,Rocky8 64 251G TeslaA10:4 1 gpu Milan,EPYC,7313,TeslaMIG,TeslaA100_80,CentOS7 64 252G TeslaA100_80:2 1 gpu Milan,EPYC,7313,Tesla,TeslaA100_80,CentOS7 64 252G TeslaA100_80:2 1 gpu Milan,EPYC,7313,Tesla,TeslaA100,TeslaMIG,Rock 64 251G TeslaA100:2 2 gpu Rome,EPYC,7302,Tesla,TeslaA100,CentOS7 64 504G TeslaA100:2 1 gpu Rome,EPYC,7352,GeForce,GeForceRTX3090,CentOS7 96 504G GeForceRTX3090:4 2 keira Rome,EPYC,7742,CentOS7 256 252G 4 keira Rome,EPYC,7742,CentOS7 256 504G 3 pelican IceLake,Xeon,6326,Rocky8 64 251G "Filesystems": Filesystem quota $CECIHOME 100.0GiB $CECITRSF 1.0TiB $HOME 50G $GLOBALSCRATCH unlimited
Multiple CPU vendors and generations are represented. The
Features column in the above list show a triplet with the CPU code name, the CPU family (Xeon is Intel’s server CPU brand, EPYC is AMD’s) and the CPU reference.
The code name is representative of the generation of the CPU (from older to more recent):
- Intel: Nehalem > Westmere > SandyBridge > IvyBridge > Haswell > Broadwell > SkyLake > CascadeLake > IceLake
- AMD: K10 > Zen > Rome > Milan
In your submission script, you can select one or more specific features with
--constraint= option, like
for instance to choose a compute node with an older CPU. The
--constraint option accepts quit complex expressions, you can find the documentation here.
Another distinctive feature of Manneback is that some nodes are equiped with GPUs. They are listed in the last column as a GPU name followed by the number of GPUs in each node.
As for CPUs multiple generations are available, in chronological order:
- nVidia: TeslaM10 > TeslaV100 (Volta) > GeForceRTX2080Ti (Turing) > TeslaA100, GeForceRTX3090, TeslaA10 (Ampere)
The GPU’s are considered a “generic resource” in Slurm, meaning that you
can reserve the GPU for your job with a command like
in your submission script. This would request two V100 GPUs. If you do not need to choose a particular type of GPU, you can also simply request
--gres=gpu:1 for instance for just one GPU.
--gres (documentation) option will allow you to request a number of GPUs per node. You can also use
--gpus (documentation) option
to request a total number of GPUs per job and not per node.
--gpus options do not allow as much flexibility as the
--constraint. Therefore, we also define features related to GPUs, such as
Geforce, etc. You can thus specify
to request one Tesla GPU if you do not care about the difference between a
TeslaV100 and a
Beware that the demand for GPUs is very high so jobs that request a GPU but let it sit idle for long periods will be cancelled.
As GPU resources are much scarcer that CPU resources, GPUs will impact the fairshare much more than CPUs, depending on the performance of the GPU and the ratio of CPUs to GPU on the compute nodes.
The nodes are organised into partitions, which are logical sets of nodes with either similar hardware or similar policies/configurations.
Three CPU partitions are available on Manneback: The default one (Def) which is opened to everyone,
cp3, which is open to everyone but CP3 users have a higher priority there,
keira which is open to everyone but NAPS users have a higher priority there, and GPU partitions. The GPU nodes are grouped into two GPU partitions: gpu and cp3-gpu.
Note that you can specify
#SBATCH -–partition=Def,cp3 to submit a job that will run on the
earliest available partition.
Quality of Service¶
A Quality Of Service (QOS) is a parameter that a job can request to bring specific privileges for the job, for instance a higher priority, or a longer run time. A QOS is typically granted to department/groups who participate in the funding of the Manneback hardware.
Currently, the following QOS’es are defined and available on the respective partitions for the listed groups:
||higher priority / longer jobs||
||higher priority but limited nb of jobs/job duration||
||longer job, more jobs, but preemptible (killable) jobs||
A QOS is chosen in Slurm with
`--qos=.... For instance, to use the
interactive QOS, add
#SBATCH --qos=interactive in your submission script.
Beware that QOS are not automatically granted to users; users must request access to them explicitely by email to the administrators.
keira QOS’es bring a higher priority for jobs on the partitions that their corresponding groups funded.
The configuration of the GPU partition is specific and was designed in order to make interactive jobs and production jobs run as smoothly as possible together on that partition.
By default, a job submitted to the
GPU partition will be subjected to the following restrictions: maximum two jobs running at all times, for a maximum duration of 2 days. To run more jobs and/or longer jobs, users must use the
preemptiable QOS, that will allow up to 30 5-days-long jobs. The drawback is that those jobs will be preemptible, meaning that they can potentially be stopped by Slurm to let a higher-priority job run immediatley. As a consequence,
Jobs submitted with the
preemptible QOS then be checkpoint/restart-enabled.
interactive QOS will allow 1 9-hour-long job with a very high priority per user, intended for interactive jobs running during the day.
Every user has access to a home directory with a 100GB quota.
A global scratch
/globalscratch is available, offering a 90TB space NFS mounted on all
the compute nodes. There is no quota enforced on that filesystem ; it is the responsibility of the users to remove the files that are not needed anymore by any job. This space is cleaned periodically.
The scratch space is not suitable for long-term storage of data.
Each node furthermore has a local
/scratch space. Local scratch is a smaller file system directly attached to the worker node. It is created at the beginning of a job and deleted automatically at the end. Again here no quota is enforced, but the available space is limited by the hardware.
For heavy computations, you can run a Jupyterhub service in a Slurm job allocation on any cluster and then use the nice little tool named Sshuttle tool to access it.
Sshuttle is a program that, according to its documentation, acts as a “Transparent proxy server that works as a poor man’s VPN. Forwards over ssh. Doesn’t require admin. Works with Linux and MacOS.” Installation on a Linux or MacOS laptop can be done with
dnf depending on your Linux distribution, or with
brew on MacOS, but you can also install it with
git clone from GitHub directly. See the GitHub page for details.
After you have installed Sshuttle, make sure you can access the cluster using an SSH gateway or VPN if necessary. In the following, we will assume the connection to Manneback through
gwceci is properly configured in
Next, find out in which module Jupyterhub is installed on the cluster or install it by yourself with pip. In the example, it installed in the user’s directory for the default Python module. Also find out which private network range is used on the cluster for the compute nodes. Often, it will be
10.0.0.0/8, but if you are unsure, ask the cluster admin.
Those operations will only be needed once.
Then, create a tunnel with Sshuttle. Open a terminal window that will be dedicated to that and run the following in it.
$ sshuttle -r manneback%gwceci 10.0.0.0/8 [local sudo] Password: Warning: No xauth data; using fake authentication data for X11 forwarding. client: Connected.
It will ask for your password in order to elevate privileges (
sudo). That is needed because it will modify temporarily low-level network routing. As long as it runs, the private network of the cluster will be accessible from your laptop, wherever you are connected. Leave that terminal session open for as long as you need access to the Jupyterhub service in your job. About the job, connect to the frontend (in another terminal) and submit it with
[dfr@manneback ~]$ srun --pty bash -c 'ml Python ; jupyter notebook --ip $(hostname -i)'
srun options to your liking. Once the job start, you should see something like this:
[I 15:59:54.435 NotebookApp] Serving notebooks from local directory: /auto/home/users/d/f/dfr [I 15:59:54.435 NotebookApp] The Jupyter Notebook is running at: [I 15:59:54.435 NotebookApp] http://10.3.220.101:8888/?token=4bd6038c1f0df6e51d901891b95b98ebf8e899646cc68223 [I 15:59:54.435 NotebookApp] or http://127.0.0.1:8888/?token=4bd6038c1f0df6e51d901891b95b98ebf8e899646cc68223 [I 15:59:54.435 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation). [W 15:59:54.477 NotebookApp] No web browser found: could not locate runnable browser. [C 15:59:54.478 NotebookApp] To access the notebook, open this file in a browser: file:///auto/home/users/d/f/dfr/.local/share/jupyter/runtime/nbserver-14230-open.html Or copy and paste one of these URLs: http://10.3.220.101:8888/?token=4bd6038c1f0df6e51d901891b95b98ebf8e899646cc68223 or http://127.0.0.1:8888/?token=4bd6038c1f0df6e51d901891b95b98ebf8e899646cc68223
If your terminal supports it, you can then click on the URL that starts with
http://10.. Otherwise simply copy that URL in your browser and you should see the Jupyterhub interface.
Once you are finished hit
CTRL-C in both terminals to stop everything.
^C[I 16:02:40.877 NotebookApp] interrupted Serving notebooks from local directory: /auto/home/users/d/f/dfr 0 active kernels The Jupyter Notebook is running at: http://10.3.220.101:8888/?token=4bd6038c1f0df6e51d901891b95b98ebf8e899646cc68223 or http://127.0.0.1:8888/?token=4bd6038c1f0df6e51d901891b95b98ebf8e899646cc68223 Shutdown this notebook server (y/[n])? y [C 16:02:43.185 NotebookApp] Shutdown confirmed [I 16:02:43.186 NotebookApp] Shutting down 0 kernels [dfr@manneback ~]$
^Cclient: client: Keyboard interrupt: exiting.
See it in action below:
Access and file transfer¶
Clusters are accessed with your CÉCI login and SSH key. Refer to the CÉCI documentation for more details. Access to Manneback is achieved by pointing your SSH client to
manneback.cism.ucl.ac.be with your CÉCI login and SSH key.
Do not hesitate to use the SSH configuration wizard if you are using the command line SSH client. Manneback will be automatically configured for you.
First connection warning¶
When you connect for the first time, the SSH client will ask you to confirm the identity of the remote host, you can do so by verifying that the fingerprint shown to you is among the ones listed below:
User self-assessment test¶
Users must pass a simple test to use Manneback. The test is very short (5 questions) and easy; 5 minutes should be enough. It exists only to make sure every one uses the cluster in the intended way and does not harm other users’ experience on the cluster. New users have a few weeks to pass the test voluntarily before the test is enforced upon login. This is recalled upon every login in the message of the day.
To start the test, you simply run
selftest.py. As soon as you respond correctly to the questions, the message will disappear. As a hint, each question contains a link to the corresponding section in the documentation.
About the cost¶
Access to the computing facilities is free of charge. Usage of the equipment for fundamental research is free since 2017 for most researchers with a normal usage.
Although access is free, the acquisition model of the hardware for Manneback is solely based on funding brought by users.
However, we encourage users who foresee a large use of the equipment for a significant duration to contact us and include budget for additional equipment in their project funding requests. When equipment is acquired thanks to funds brought by a specific group, the equipment is shared with all users but the funding entity can obtain exclusive reservation periods on the equipment or request specific configurations.
If the expected usage does not justify buying new equipment, but the project’s budge include computation time, the CISM can also bill the usage of the equipment for the duration of the project. Rates vary based on the funding agency (European, Federal, Regional, etc.) and the objective of the research (fundamental, applied, commercial, etc.)
Before 2017, if a research group (pôle de recherche) usage exceeded 200.000 hCPU (CPU hours), equivalent to the usage of 23 processors during a full year, the cost was computed as a function of the yearly consumption as follows:
Rate applied in 2016 related to you research group consumption in 2015: - Below 200.000 hCPU: 0 €/hCPU - Between 200.000 and 2.500.000 hCPU: 0.00114 €/hCPU - Over 2.500.000 hCPU: 0.00065 €/hCPU
For 2017 and beyond, thanks to a participation by the SGSI in the budget of the CISM, the cost will be null for the users who do not have specific funding for computational resources. Users funded by a Regional, Federal, European or Commercial project with specific needs should contact the CISM team for a quote.