Research Guides: Computer Science: Compute Resources

New Compute Resource

NYU's High Speed Research Network department has an alpha version of Kubernetes available for the NYU community to test out. More information here.

Compute Resources

At some point in your computer science career, you will probably need access to some medium or high performance computing infrastructure. You would use the HPC when you need to deal with some data that is too large to be dealt with on your local machine.

HPC infrastructure is basically a bunch of clusters (computers) to which people submit jobs (scripts) and wait for them to be first in the queue, then run. Runs last from minutes to days, depending on the size of the input data. Typically you have to use the command line to access clusters. So a typical workflow might look like this:

SSH (securely log-in) into a cluster (if you don't know how to use the command line, we have an introduction to that).
Copy or transfer data onto the storage layer of the cluster. These are typically distributed, which means that data is stored across multiple nodes to be accessed by compute resources.
Run some data clean-up operations via the command line, through python scripts, R scripts, bash, etc. These operations are typically about formatting the data similarly so analysis pipelines run correctly, filtering out bad/unnecessary data, or validating data.
Run some analysis pipeline via the command line, through python scripts, R scripts, bash, etc. Batch processing is one method that allows researchers to analyze large data – this consists of breaking up the data into smaller pieces, scheduling each piece on a different compute node, and calculating the final result after running some job (analysis script).
Visualize what happened during the analysis – make some graph, time series, etc. that helps others understand what happened during the analysis stage.

NYU Courant

Computing Resources are available to anyone with a CIMs account. Courant accounts are available to all faculty, researchers, administrative staff, and graduate students affiliated with CIMS, the Math and CS departments. CIMS accounts can also be given to collaborators, visitors, and students who are otherwise ineligible (eg, from other departments) but are enrolled in Math or CS graduate classes, or working with a member of Courant's faculty. CIMS accounts may also be given to undergraduate students registered for certain classes. See more info about how to obtain an account. They also maintain backups and provide some storage for these resources.

Computer labs in Warren Weaver Hall
The Courant Institute of Mathematical Sciences maintains computer labs where anyone with a valid Courant account can use any one of the many Linux or Windows machines located in these rooms.
Compute servers
Courant has a number of systems that can be used for CPU intensive jobs and programming assignments for courses and research. These machines are restricted to logins from within the Courant network, so if you are coming from outside of CIMS, you will have to first login to one of the remote access servers, then use ssh to get to them.
Desktop computing
The Courant Institute of Mathematical Sciences provides desktop computers in the labs, in offices, and as supplies are available unsupported machines for home use. The labs house a combination of Linux and Windows systems that are all tied to our computing resources network. The labs are restricted to people who have accounts with CIMS, and therefore a general NYU NetID does not allow acces to these machines. Accounts are generally given to Math and CS Graduate students, or may be given through a course.
Licensed software
There are specially licensed software for use on CIMs servers and machines.
MySQL database hosting
Courant offers users the ability to set up MySql databases on its central database server, warehouse.cims.nyu.edu
Scanners, copiers, DVD burners
Version control server
The CIMS systems group operates an instance of Apache to serve subversion and mercurial repositories. If you don't intend to use subversion or mercurial with colloborators (like, you want to use Git instead) then there is no need to use the server. You may simply setup a repository in your home directory and access it through the local filesystem or over ssh.
Web hosting
You can set up a web page by simply placing it under the directory /web/$USER (where $USER is your CIMS username). If you have access to another document hierarchy, such as a research group or administrative page, the same instructions apply except for the directory path. All filesystems are mounted on all the webservers and the webservers are configured to treat a user's /web directory as if it were part of the document hierarchy. Note that HTML files outside your /web hierarchy will not be accessible via the web server.

NYU HPC

NYU Research Technology's High Performance Computing department maintains computing infrastructure that is available to the NYU community for research and teaching and learning. The HPC team also provides classes (live and online) and support for using the clusters.

There is also the Secure Research Data Environment service, which is available on a case-by-case basis. This is a custom secure research environment for you to store sensitive data and analyze it.

The compute resources offered by the HPC department include:

Greene - HPC Cluster
- The Greene cluster is named after Greene street in SoHo, a neighborhood in Lower Manhattan in New York City, near NYU. The cluster has also "green" characteristics, such as most of the cluster nodes are water cooled and it is deployed in a power efficient data center.
- General purpose cluster suitable for the majority of computing and data analytics tasks, such numerical simulations and AI.
- Uses SLURM for job submission
- Detailed hardware specs and software specs
Cloud - there are a few options for those who need cloud infrastructure for their work:

NYU HPC Data Transfer Guides

The HPC team has also put together documentation on how to transfer large amounts of data to and from the NYU HPC:

Globus
A Globus Endpoint is a data transfer location, a location where data can be moved to or from using Globus transfer, sync and sharing service. An endpoint can either be a personal endpoint (on a user’s personal computer) or a server endpoint (located on a server, for use by multiple users).
Google Drive
rclone - rsync for cloud storage, is a command line program to sync files and directories to and from cloud storage systems such as Google Drive, Amazon Drive, S3, B2 etc. rclone is available on Prince cluster.
Local Computer
To copy data between your workstation and the NYU HPC clusters, you must set up and start an SSH tunnel on the workstation. There are instructions for this for Windows, Mac and Linux workstations.

Computer Science

Ask your librarian!

CC