Compute nodes: Compute nodes are where actual jobs are run. There are three types of compute nodes on Pinnacles.
1. 40 Regular memory (RM) nodes with 256GB RAM
2. 4 Big memory nodes (bigmem) with 1TB RAM
3. 8 GPU nodes with NVIDIA A100 GPUs
All nodes are interconnected via HDR InfiniBand w/ RDMA for fast (100Gbits/s) and low latency (sub ms) data transfer.
|
|
RM node |
bigmem node |
|
Number of nodes |
40 |
4 |
|
CPU |
2 Intel 28-Core Xeon Gold 6330 |
2x Intel 28-Core Xeon Gold 6330 |
|
RAM |
256 GB |
1TB |
|
Node-local storage |
1TB NVMe Data Center Solid State Drive (SSD) |
1TB NVMe Data Center Solid State Drive (SSD) |
|
Network |
ConnectX-6 VPI adapter card, HDR 100 InfiniBand (100Gb/s) and 100GbE, single-port QSFP56, PCIe3/4 x16 Slot |
ConnectX-6 VPI adapter card, HDR 100 InfiniBand (100Gb/s) and 100GbE, single-port QSFP56, PCIe3/4 x16 Slot |
GPU nodes: Pinnacles GPU nodes provide exceptional performance and scalability for deep learning and accelerated computing.
|
GPU nodes |
|
|
Number |
8 |
|
GPUs per node |
2x NVIDIA Tesla A100 PCIe v4 40GB HBM2 Passive Single GPU |
|
CPU |
2x Intel 28-Core Xeon Gold 6330 |
|
RAM |
256GB |
|
Node-local storage |
1TB M.2 NVMe Data Center Solid State Drive (110mm) |
|
Network |
ConnectX-6 VPI adapter card, HDR-100 IB (100Gb/s) and 100GbE, single-port QSFP56, PCIe3/4 x16 Slot |
Data storage:
Pinnacles has ~92TB NFS Fast Scratch Storage space for accessing large data with low latency and 1.5PB of usable long-term storage.
File system:
Users will be placed at the HOME directory once login to Pinnacles. Each user has 30GB storage space under the “/home/<UCMID>”.
For larger storage access, users have access to “/home/<UCMID>/data" and “/home/<UCMID>/scratch” folders, each directory will have 256 GB storage space.
It is recommended that if users submit jobs that require large storage space or generate large output, please perform jobs under the “/home/<UCMID>/scratch” directory, where it can handle larger size of temporary IOs for the job. However, files under “/home/<UCMID>/scratch” directory will be deleted every 30 days based on the file last modification date, and users are responsible for their own data. Data will not be retrieved after purging.
For the data storage, each user has 256 GB storage space under “/home/<UCMID>/data”, and long-term data storage can be placed there, and the “/home/<UCMID>/data” directory will not be deleted.
