Skip to main content

Slurm Partitions (Queues)

Anvil provides different queues with varying job sizes and walltimes. There are also limits on the number of jobs queued and running on a per-user and queue basis. Queues and limits are subject to change based on the evaluation from the Early User Program.

Anvil Production Queues
Queue Name Node Type Max Nodes per Job Max Cores per Job Max Duration Max running Jobs in Queue Max running + submitted Jobs in Queue Charging factor
debug regular 2 nodes 256 cores 2 hrs 1 2 1
gpu-debug gpu 1 node 2 gpus 0.5 hrs 1 2 1
wholenode regular 16 nodes 2,048 cores 96 hrs 64 2500 1 (node-exclusive)
wide regular 56 nodes 7,168 cores 12 hrs 5 10 1 (node-exclusive)
shared regular 1 node 128 cores 96 hrs 1280 cores - 1
highmem large-memory 1 node 128 cores 48 hrs 2 4 4
gpu gpu - - 48 hrs - - 1
ai ai - - 48 hrs - - 1

For gpu and ai queue: max of 12 GPU in use per user and max of 32 GPU in use per allocation.

Make sure to specify the desired partition when submitting your jobs (e.g. -p wholenode). If you do not specify one, the job will be directed into the default partition (shared).

If the partition is node-exclusive (e.g. the wholenode and wide queues), even if you ask for 1 core in your job submission script, your job will get allocated an entire node and would not share this node with any other jobs. Hence, it will be charged for 128 cores' worth and squeue command would show it as 128 cores, too. See SU accounting for more details.

Link to section 'Useful tools' of 'Slurm Partitions (Queues)' Useful tools

  1. To display all Slurm partitions and their current usage, type showpartitions at the command line.
    x-anvilusername@login03.anvil:[~] $ showpartitions
    Partition statistics for cluster anvil at CURRENTTIME
            Partition     #Nodes     #CPU_cores  Cores_pending   Job_Nodes MaxJobTime Cores Mem/Node
            Name State Total  Idle  Total   Idle Resorc  Other   Min   Max  Day-hr:mn /node     (GB)
       wholenode    up   750     0  96000    512      0  92280     1 infin   infinite   128     257 
        standard    up   750     0  96000    512      0   6753     1 infin   infinite   128     257 
        shared:*    up   250    97  32000  16395      0   4997     1 infin   infinite   128     257 
            wide    up   746     0  95488    512      0  24576     1 infin   infinite   128     257 
         highmem    up    32    19   4096   2504      0      0     1 infin   infinite   128    1031 
           debug    up    17     0   2176    113      0    128     1 infin   infinite   128     257 
             gpu    up    16     0   2048   1371      0    228     1 infin   infinite   128     515 
              ai    up    21    18   2016   1814      0      0     1 infin   infinite    96    1031 
       gpu-debug    up    16     0   2048   1371      0      0     1 infin   infinite   128     515 
    benchmarking    up  1048   116 134144  20782      0      0     1 infin   infinite   128     257+
       profiling    up     4     0    512      0      0      0     1 infin   infinite   128     257 
           azure    up     8     0     16      0      0      0     1 infin   infinite     2       7
    
  2. To show the list of available constraint feature names for different node types, type sfeatures at the command line.
    x-anvilusername@login03.anvil:[~] $ sfeatures
    NODELIST       CPUS   MEMORY    AVAIL_FEATURES    GRES
    a[000-999]     128    257400    A,a               (null)
    b[000-031]     128    1031400   B,b,testing       (null)
    g[000-015]     128    515400    G,g,A100          gpu:4
    h[000-020]     96     1031000   H,h,H100          gpu:4
    login[00-07]   32     500000    login             (null)
    ondemand       32     90000     (null)            (null)
    z[000-007]     2      7684      Z,z,power_azure   (null)
Helpful?

Thanks for letting us know.

Please don't include any personal information in your comment. Maximum character limit is 250.
Characters left: 250
Thanks for your feedback.