Computationally intensive calculations should never be run on the head node. MERCED makes use of Slurm. Detailed information regarding Slurm is available from a number of online resources.
This section provides a brief description of the Slurm system, an overview of the specific queueing environment employed on MERCED, and a set of example submission scripts. The table below provides a brief overview of available queues and standard maximum resource limitations. The available queues are designed to balance use and accessibility to the cluster for all users.
Three of the queues – std.q, fast.q, and long.q – are distinguished from one another by the required wall time of the submitted job. The fourth available queue is the gpu.q queue. Requesting this queue is necessary for calculations that require a node with GPUs. If a submitted job does not complete before reaching the maximum wall time limit, the queue system will terminate the calculation.
In an effort to provide a balanced access to compute cycles on MERCED, the course queue resource limitations and user-level queue resource limitations become increasingly restrictive. Therefore, users are advised to use the most limiting queue that satisfies their computational needs. In other words, users should not default their work to long.q. It is essential for the best use of MERCED that users develop an awareness of their resource needs.
|
Queue |
Max Wall Time |
Max Cores |
Description |
|---|---|---|---|
|
std.q |
24 hrs. |
800 |
This queue is a general use queue. |
|
fast.q |
4 hrs. |
300 |
This queue is a queue for fast jobs. |
|
long.q |
14 days |
200 |
This queue is a queue for long-running jobs. |
|
gpu.q |
14 days |
unlimited |
This queue is used to request nodes with available GPU cards. Some care is required to ensure resource requirements are properly communicated with the scheduler when queueing jobs in the GPU queue. To use this queue, users must first request access from MERCED system administrators via email sent to helpdesk@ucmerced.edu. |
The command sbatch, which is shown below, is used to submit jobs to the queue. Additional commands to work with and monitor the queue/scheduler include those shown in the table below.
| Command | Description |
|---|---|
squeue |
reports the state of jobs or job steps. |
sinfo |
reports the state of partitions and nodes managed by Slurm |
scancel |
used to cancel a pending or running job or job step. It can also be used to send an arbitrary signal to all processes associated with a running job or job step. |
strigger |
used to set, get or view event triggers. |
The listed queue resource limitations are based on planned limitations. Queue resource limitations may vary for specific users and user groups according to node buy-in, project needs, current resource use, and/or administrative needs. Contact the system administration team for clarification or to request a variance from standard limitations.
Typical use of the queue system begins by writing a submission script that will be handed to the queue and scheduling system. The submission script is comprised of two sections: (1) a list of SGE options and resource requirements to guide the scheduler’s assignment of the job to one or more production nodes; and (2) a standard shell script that carries out the user desired calculation.
To submit a submission script call my_job.sub to the queue, one uses the following command:
sbatch my_job.sub
#!/bin/bash #SBATCH --mail-user=UCMercedNetID@ucmerced.edu #SBATCH --mail-type=ALL #SBATCH --nodes=1 #SBATCH --ntasks=20 #SBATCH --partition std.q #SBATCH --mem=96G #SBATCH --time=0-00:15:00 # 15 minutes #SBATCH --output=test1.qlog #SBATCH --job-name=test1 #SBATCH --export=ALL # This submission file will run a simple set of commands. All stdout will # be captured in test1.qlog (as specified in the Slurm command --output above). # This job file uses a shared-memory parallel environment and requests 20 # cores (--ntasks option) on a single node(--nodes option). This job will also run a global #script called # run. For more info on this script, cat /usr/local/bin/merced_node_print. # whoami pwd uptime hostname datetime
