MSR GPU Servers

Lambda Labs Workstations with:

1. AMD Ryzen Threadripper PRO 5995WX with 64 Cores
2. 1 TB of RAM
3. Two NVIDIA RTX 6000 Ada GPUs with 48 GB Ram
4. Disks: a 4TB and a 14TB NVME Drive

The GPU servers are available at

• sheep.mech.northwestern.edu
• lamb.mech.northwestern.edu

• You should assume that data on the GPU servers can be deleted without notice at the end of each quarter
• There is no redundancy on the server so backup everything that is important to you
• In addition to your home directory you have access to /data/users/netID, which provides additional storage.
  • Each user is limited to 150GB in their /home/netID directory
  • Each netID has an additional 850GB in their /data/users/netID directory
  • Use quota -vs to see how much disk space you are using relative to your quotas.

1. Use tmux to persist your session across ssh logins
   • Otherwise, what you run in an ssh session (e.g., a long ML training) will exit when ssh disconnects
2. If doing RL training, run the simulator in headless mode on the server.
   • There is no GUI on the server, so everything will be via the command line.
   • A useful workflow is to test small batches on laptop, then run on server
   • Be sure to save checkpoints so you can train a little, see what is happening, then continue training.
3. Use rsync to synchronize files between the server and your computer
   • rsync can operate directly over ssh by specifying user@server:/path/to/file as either a source or destination
   • Be careful, rsync can delete files. I recommend never using it directly on your home directory but rather a sub-directory
   • If you completely destroy your home directory the hidden files in /etc/skel have the default configuration and you can ssh-copy-id to gain passwordless access again.
4. It is possible to run a jupyter notebook on the server and connect to it via your computer's web browser.
5. You can use python3 virtual environments for python development
   • However, most packages you would need (if starting from scratch) are already pre-installed
6. Many C++ and Python packages can be placed directly in a colcon workspace and compiled from scratch
7. If pre-existing software has dependencies that differ from our system (as is likely) your quickest path to success is a docker container.
8. If you are coding your own script, your quickest path to success is to use the versions of the pre-installed packages that are on the server and on your laptop.
9. There are no limits on GPU or CPU usage, but be mindful of other students by coordinating when you will use each machine and GPU.

Some useful software on the GPU machines includes:

1. Ubuntu 24.04 LTS (using the Ubuntu Minimal installation.
2. ROS 2 Jazzy (and image pipeline, moveit2, simulator, nav2, rtabmap, and others).
3. Lambda Stack (provides PyTorch, TensorFlow, etc.)
4. Docker

• If the preinstalled software is not sufficient (wrong versions etc) then you should make a docker image to do what you need
• All users can access Docker only in rootless mode. Docker rootless mode is already set up for you.
• When interpreting instructions for docker on the internet:
  • Skip anything related to installing docker, nvidia drivers, or the nvidia container toolkit.
  • If the instructions involve sudo omit the sudo. Most commands should work but if they don't that means you can't (and likely do not actually need to) run it.
  • If the instructions involve systemctl <SOMETHING> docker substitute systemctl --user <SOMETHING> docker instead.
    • The docker daemon must be running. It should start automatically but use systemctl --user start docker to start it if it is not.
  • If the instructions involve --privileged you can't do it (but there is likely another way to do what you need to do).
• To use the nvidia gpus in a docker container add the following flags to the docker run command: --runtime=nvidia --gpus all
  • For example docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi
• For more information see Rootless Docker and NVIDIA Docker Rootless Mode
• By default your docker images are stored in /data/users/<netID>/docker
  • Use docker image ls to see downloaded docker images and the amount of disk space they are using
  • Use docker image rm to delete a docker image.

1. tank.mech.northwestern.edu
2. 26TB of RAIDZ2 Storage
3. Regular snapshots are taken so that you can restore your work at multiple points.
4. sftp and rsync access only, no shell accounts on this server.
   • There are multiple sftp clients, the standard sftp command is rather basic.
   • An alternative is lftp

1. Customize ./code/backup.bash to back up your home directory

   #!/usr/bin/bash
   rsync -aH \
       --info=progress2 --no-inc-recursive -h \
       --delete \
       --exclude='.zfs/' \
       --exclude='.cache/' \
       --exclude='.ssh/id_ed25519' \
       "$HOME/" \
       "<netid>@tank.mech.northwestern.edu:$(hostname -s)/"
   

2. You will need to replace <netid> with your netid
3. By default I exclude some local directories that don't belong on the server and your private ssh key
4. Add more directories to exclude by specifying additional --exclude lines
5. By default your $HOME/ directory is backed up. You could also use a different directory depending on your needs
6. Restoration can be accomplished by reversing the source and destination.

• Browse snapshots under ~/.zfs/snapshots using sftp
  • Snapshots allow you to see versions of your files on the backup server as of a specific date
• Snapshots are taken frequently (every 15 minutes), hourly, daily, monthly and yearly.
• Generally you may want to download individual files (using sftp) but you can also copy files back to your machine from snapshots using rsync.