Slurm Partitions (Queues)
Anvil provides different queues with varying job sizes and walltimes. There are also limits on the number of jobs queued and running on a per-user and queue basis. Queues and limits are subject to change based on the evaluation from the Early User Program.
| Queue Name | Node Type | Max Nodes per Job | Max Cores per Job | Max Duration | Max running Jobs in Queue | Max running + submitted Jobs in Queue | Charging factor |
|---|---|---|---|---|---|---|---|
| debug | regular | 2 nodes | 256 cores | 2 hrs | 1 | 2 | 1 |
| gpu-debug | gpu | 1 node | 2 gpus | 0.5 hrs | 1 | 2 | 1 |
| wholenode | regular | 16 nodes | 2,048 cores | 96 hrs | 64 | 2500 | 1 (node-exclusive) |
| wide | regular | 56 nodes | 7,168 cores | 12 hrs | 5 | 10 | 1 (node-exclusive) |
| shared | regular | 1 node | 128 cores | 96 hrs | 1280 cores | - | 1 |
| highmem | large-memory | 1 node | 128 cores | 48 hrs | 2 | 4 | 4 |
| gpu | gpu | - | - | 48 hrs | - | - | 1 |
| ai | ai | - | - | 48 hrs | - | - | 1 |
For gpu and ai queue: max of 12 GPU in use per user and max of 32 GPU in use per allocation.
Make sure to specify the desired partition when submitting your jobs (e.g. -p wholenode). If you do not specify one, the job will be directed into the default partition (shared).
If the partition is node-exclusive (e.g. the wholenode and wide queues), even if you ask for 1 core in your job submission script, your job will get allocated an entire node and would not share this node with any other jobs. Hence, it will be charged for 128 cores' worth and squeue command would show it as 128 cores, too. See SU accounting for more details.
Link to section 'Useful tools' of 'Slurm Partitions (Queues)' Useful tools
- To display all Slurm partitions and their current usage, type
showpartitionsat the command line.x-anvilusername@login03.anvil:[~] $ showpartitions Partition statistics for cluster anvil at CURRENTTIME Partition #Nodes #CPU_cores Cores_pending Job_Nodes MaxJobTime Cores Mem/Node Name State Total Idle Total Idle Resorc Other Min Max Day-hr:mn /node (GB) wholenode up 750 0 96000 512 0 92280 1 infin infinite 128 257 standard up 750 0 96000 512 0 6753 1 infin infinite 128 257 shared:* up 250 97 32000 16395 0 4997 1 infin infinite 128 257 wide up 746 0 95488 512 0 24576 1 infin infinite 128 257 highmem up 32 19 4096 2504 0 0 1 infin infinite 128 1031 debug up 17 0 2176 113 0 128 1 infin infinite 128 257 gpu up 16 0 2048 1371 0 228 1 infin infinite 128 515 ai up 21 18 2016 1814 0 0 1 infin infinite 96 1031 gpu-debug up 16 0 2048 1371 0 0 1 infin infinite 128 515 benchmarking up 1048 116 134144 20782 0 0 1 infin infinite 128 257+ profiling up 4 0 512 0 0 0 1 infin infinite 128 257 azure up 8 0 16 0 0 0 1 infin infinite 2 7 - To show the list of available constraint feature names for different node types, type
sfeaturesat the command line.x-anvilusername@login03.anvil:[~] $ sfeatures NODELIST CPUS MEMORY AVAIL_FEATURES GRES a[000-999] 128 257400 A,a (null) b[000-031] 128 1031400 B,b,testing (null) g[000-015] 128 515400 G,g,A100 gpu:4 h[000-020] 96 1031000 H,h,H100 gpu:4 login[00-07] 32 500000 login (null) ondemand 32 90000 (null) (null) z[000-007] 2 7684 Z,z,power_azure (null)