Storage

All MCSR systems share home directories. Home directories are stored on a DDN network-attached storage appliance. It is attached to MCSR systems via Infiniband.

The storage appliance is in a high-performance, high-redundancy RAID configuration. All user data is also backed up.

Quotas

Home directories are subject to a 600GB quota. Temporary excursions to as much as 800GB are tolerated for up to seven days. Quotas can be increased to 1TB upon request. Storage needs above 1TB will require funding.

Temporary Files

All systems also share a /scratch area for large temporary files. Each user has their own directory inside the /scratch area. This area is stored on the same device as the home directories. Users should cleanup after themselves in their PBS scripts. This area is NOT backed up, and is subject to being purged as needed. It is subject to a separate 600GB quota.

All nodes have /tmp directories backed by on-node disks for temporary files, but they vary greatly in size. When creating files inside /tmp, please create a directory structure that looks like /tmp/$USER.$PBS_JOBID. This can be done using the following command:
mkdir -p /tmp/$USER.$PBS_JOBID

This will allow our scripts to easily clean up after failed jobs.

/ptmp Directories

In the past, all MCSR users had a /ptmp directory in addition to their home directory for large files. During a 2017 upgrade these directories were combined. Existing users had their ptmp directories copied into their home directory and a symbolic link was setup in /ptmp so that existing scripts would continue to work. Users created since the upgrade have no trace of a ptmp directory.

Permissions

Mostly for historical reasons, our default settings allow users to see each other’s files. We have considered changing this in the past and have decided against it. If this is an issue for you, you should change the permissions on your existing files and change your default umask so that new files will be created with stricter permissions. Contact MCSR staff for assistance.