NYU HPC how-to quick start instructions #2
Conversation
ch2/compute.md
Outdated
|
|
||
| ## Accessing HPC and First Time Login | ||
|
|
||
| * Raise a request to get access to HPC using this link - https://identity.it.nyu.edu/identityiq/home.jsf. This step creates your Unix id and a user profile on HPC. |
There was a problem hiding this comment.
This needs to be accessed from on-campus or via VPN, right? Probably worth documenting.
| For Windows and other SSH Config options please see [[1]](https://sites.google.com/nyu.edu/nyu-hpc/accessing-hpc?authuser=0 ) in the document links above. | ||
| * Next, you can logon to the HPC login nodes using yout Net ID and password using iTerm (on Mac) or PuTTy (on Windows)- | ||
| ```ssh <Net ID>@greene.hpc.nyu.edu``` |
There was a problem hiding this comment.
This can actually be simplified if you add
User <Net ID>
to .ssh/config for the hpc hosts. Then you can just do ssh greene.hpc.nyu.edu
| * Next, you can logon to the HPC login nodes using yout Net ID and password using iTerm (on Mac) or PuTTy (on Windows)- | ||
| ```ssh <Net ID>@greene.hpc.nyu.edu``` | ||
| **Note:** If you are not on the NYU network, you will need to connect to VPN. |
There was a problem hiding this comment.
There's also the gateway tunnel if you're not on the VPN. Maybe we can add a section at the end for this?
| NYU HPC uses SLURM to submit a job to the CPU or GPU clusters. A job can be a batch (e.g., training your neural network on GPUs as a long running background thread/process), or an interactive job (e.g., running a Jupyter notebook) | ||
| **Note**: You will need a project account code to run any SLURM commands (on the CPU or GPU clusters). This code is of the format "pr_###_<function>". Your project PI/Supervisor should be able to create a new project on the HPC management portal and give you a code. |
There was a problem hiding this comment.
I don't think this is strictly necessary. RIT is used to manage priority access, but if you don't include it, things will still run (just at lower priority) -- right?
There was a problem hiding this comment.
The HPC doc says RIT projects are enforced by the Slurm scheduler - https://sites.google.com/nyu.edu/nyu-hpc/hpc-projects - so I just assumed we needed one.
ch2/compute.md
Outdated
| ``` | ||
| $ ./singrw # Assuming you have already created a container image at the location, start it in read-write mode. | ||
| $ pip install jupyterlab |
There was a problem hiding this comment.
probably this should be in your conda rather than pip, right?
Also a good idea to include links to configuring a conda kernel in jupyter
There was a problem hiding this comment.
I usually use pip from inside the conda environment (which pip points to the python installation within my conda env). But using conda here makes sense.
| $ ./singrw # Assuming you have already created a container image at the location, start it in read-write mode. | ||
| $ pip install jupyterlab | ||
| $ jupyter lab --no-browser -port=9000 -ip=0.0.0.0 | ||
| ``` |
There was a problem hiding this comment.
actually this all seems like not something we should document / encourage, and rather we should direct folks to use OOD for launching notebook instances.
Maybe @tomxi has some useful things to contribute here?
Co-authored-by: Brian McFee <[email protected]>
2 participants
Added quick start instructions for new joiners to get started with NYU HPC.
See my fork for a readable version of the markdown - https://github.com/pkamath2/marl-handbook/blob/hpc_updates/ch2/compute.md