Data Science

NYU's High Speed Research Network department has an alpha version of Kubernetes available for the NYU community to test out. More information here: https://k8s-docs.hsrn.nyu.edu/

At some point in your data science career, you will probably need access to some medium or high performance computing infrastructure. You would use the HPC when you need to deal with some data that is too large to be dealt with on your local machine.

HPC infrastructure is basically a bunch of clusters (computers) to which people submit jobs (scripts) and wait for them to be first in the queue, then run. Runs last from minutes to days, depending on the size of the input data. Typically you have to use the command line to access clusters. So a typical workflow might look like this:

1. SSH (securely log-in) into a cluster (if you don't know how to use the command line, we have an introduction to that).
2. Copy or transfer data onto the storage layer of the cluster. These are typically distributed, which means that data is stored across multiple nodes to be accessed by compute resources.
3. Run some data clean-up operations via the command line, through python scripts, R scripts, bash, etc. These operations are typically about formatting the data similarly so analysis pipelines run correctly, filtering out bad/unnecessary data, or validating data.
4. Run some analysis pipeline via the command line, through python scripts, R scripts, bash, etc. Batch processing is one method that allows researchers to analyze large data – this consists of breaking up the data into smaller pieces, scheduling each piece on a different compute node, and calculating the final result after running some job (analysis script).
5. Visualize what happened during the analysis – make some graph, time series, etc. that helps others understand what happened during the analysis stage.

NYU Research Technology's High Performance Computing department maintains computing infrastructure that is available to the NYU community for research and teaching and learning. The HPC team also provides classes (live and online) and support for using the clusters.

Below is a table showing the different compute infrastructure offered. There is also the Secure Research Data Environment service, which is  available on a case-by-case basis. This is a custom secure research environment for you to store sensitive data and analyze it.

The HPC team has also put together documentation on how to transfer large amounts of data to and from the NYU HPC: