Accessing the Pinnacles Cluster

Once the account is being created on Pinnacles, the cluster can be accessed on Mac and Linux where you can use a built-in terminal application; on Windows you can use WSL or Moba XTERM to open a terminal and type the following command:

ssh <YourUCMercedNetID>@login.pinnacles.ucmerced.edu

Note: you have to use UC, Merced campus network to connect to Pinnacles cluster. When accessing the research clusters from an off-campus location, please ensure you are logged into UC Merced VPN to access your research cluster

Refer to HPC Wiki for any questions that you may have regarding Pinnacles Cluster. If you have additional questions, please submit a general request/consultation here.

System Configuration:

Compute nodes: Compute nodes are where actual jobs are run. There are three types of compute nodes on Pinnacles.

  1. 40 Regular memory (RM) nodes with 256GB RAM
  2. 4 Big memory nodes (bigmem) with 1TB RAM
  3. 8 GPU nodes with NVIDIA A100 GPUs

All nodes are interconnected via HDR InfiniBand w/ RDMA for fast (100Gbits/s) and low latency (sub ms) data transfer.

 

RM node

bigmem node

Number of nodes

40

4

CPU

2 Intel 28-Core Xeon Gold 6330

2x Intel 28-Core Xeon Gold 6330

RAM

256 GB

1TB

Node-local storage

1TB NVMe Data Center Solid State Drive (SSD)

1TB NVMe Data Center Solid State Drive (SSD)

Network

ConnectX-6 VPI adapter card, HDR 100 InfiniBand
 (100Gb/s) and 100GbE, single-port QSFP56, PCIe3/4 x16 Slot

ConnectX-6 VPI adapter card, HDR 100 InfiniBand
 (100Gb/s) and 100GbE, single-port QSFP56, PCIe3/4 x16 Slot

GPU nodes: Pinnacles GPU nodes provide exceptional performance and scalability for deep learning and accelerated computing.

GPU nodes

Number

8

GPUs per node

2x NVIDIA Tesla A100 PCIe v4 40GB HBM2 Passive Single GPU

CPU

2x Intel 28-Core Xeon Gold 6330

RAM

256GB

Node-local storage

1TB M.2 NVMe Data Center Solid State Drive (110mm)

Network

ConnectX-6 VPI adapter card, HDR-100 IB (100Gb/s) and 100GbE, single-port QSFP56, PCIe3/4 x16 Slot

Data storage:

Pinnacles has ~92TB NFS Fast Scratch Storage space for accessing large data with low latency and 1.5PB of usable long-term storage.

File system:

Users will be placed at the HOME directory once login to Pinnacles. Each user has 30GB storage space under the “/home/<UCMID>”.

For larger storage access, users have access to “/home/<UCMID>/data” and “/home/<UCMID>/scratch” folders, each directory will have 256 GB storage space. 

It is recommended that if users submit jobs that require large storage space or generate large output, please perform jobs under the “/home/<UCMID>/scratch” directory, where it can handle larger size of temporary IOs for the job. However, files under “/home/<UCMID>/scratch” directory will be deleted every 30 days based on the file last modification date, and users are responsible for their own data. Data will not be retrieved after purging.

For the data storage, each user has 256 GB storage space under “/home/<UCMID>/data”, and long-term data storage can be placed there, and the “/home/<UCMID>/data” directory will not be deleted.