Merge pull request #134 from NYU-RTS/cloud_begin_port

Cloud section first pass

Author: s-sajid-ali

_typos.toml

@@ -1,3 +1,8 @@
 [default.extend-words]
 # Scientific Software
 namd = "namd"
+
+[default.extend-identifiers]
+# Technical terms
+SerDe = "SerDe"
+JsonSerDe = "JsonSerDe"

docs/cloud/01_getting_started/01_intro.md

@@ -0,0 +1,4 @@
+# Start here!
+
+We facilitate access to cloud computing resources (GCP) for research, HPC Bursting, visualization, data analysis (hadoop) and host an OpenShift Kubernetes cluster on-prem. Please proceed to the relevant section to know more about the offerings and how you could harness them.
+

docs/cloud/01_getting_started/_category_.json

@@ -0,0 +1,3 @@
+{
+  "label": "Getting Started"
+}

docs/cloud/02_gcp_self_managed/01_intro.md

@@ -0,0 +1,51 @@
+# Google Cloud Platform for Research
+
+[internet2-gcp]: https://internet2.edu/services/google-cloud-platform/
+[waive-egress]: https://cloud.google.com/billing/docs/how-to/data-transfer-waiver
+[nih-strides]: https://cloud.nih.gov/
+[internet2-cc]: https://internet2.edu/services/cloud-connect/
+[google-for-edu]: https://cloud.google.com/edu?hl=en
+[gcp-get-started]: https://cloud.google.com/docs/get-started/
+[gcp-credits]: https://cloud.google.com/edu/researchers?hl=en
+
+We facilitate access to Google Cloud Platform (GCP) for your research projects. NYU is a member of the [Internet2 Net+ GCP program][internet2-gcp] that allows for community-negotiated GCP terms that provides NYU researchers with benefits that include, amongst other things, price discounts, [waivers for data egress fees][waive-egress], and [NIH STRIDES][nih-strides] initiative benefits.
+
+In addition to the I2 Net+ GCP benefits, NYU scholars can enjoy significant discounts in using GCP resources in their research project through a 3 year commitment (started in 2019) NYU made in GCP. The NYU network connects with GCP via Partner Interconnect using [Internet2 Cloud Connect (I2CC)][internet2-cc] service.
+
+## Why work with the NYU Research Cloud team to deploy your research project on GCP?
+
+NYU researchers who work with the Research Cloud team to deploy projects on GCP may benefit from the following:
+-   Discounted rates, lower GCP project cost:  Through NYU's participation in the Internet2 Net+ agreement, as well as the 3-year commitment NYU made to using GCP, GCP projects enjoy discounted rates, lowering the cost of the project. The exact discounts depend on the GCP service used in the research project and can vary between 5% and 25%. Free data egress is usually included.
+-   GCP Expertise and project setup: The NYU research cloud team can work closely with researchers to Identity and Access Management, groups access, establish billing, etc.
+-   GCP Support cases: Given proper permissions the NYU Research Cloud team can open support cases on behalf of the researchers or directly discuss GCP project issues with GCP experts
+-   Cost monitoring, billing alerts, and spending reports: The Research Cloud team has access to additional tools that can provide cost monitoring, switch between billing options and provide spending reports.
+-   Research project funding options: The NYU research cloud team can help with providing invoices, internal fund transfers, establishing GCP Billing ids, etc. that can enable NYU researchers to pay for GCP services.
+
+## Getting Started with GCP
+There are a number of ways you can get started with using GCP in research projects:
+-   To start using GCP resources in a research project, request a consultation with the Research Cloud team (via email research-cloud-support@nyu.edu). The Research Cloud team can advise on ways to set up your research project, available discounts, etc.
+-   If a project involves Data Science and Machine Learning, consult with the DS3 team before starting your project on GCP.
+:::tip
+Creating a GCP project using your NYU account (NetID@nyu.edu) will place the project under the nyu.edu organization, an environment managed by the NYU Research Cloud team. However, your project doesn't automatically qualify for the price discounts and benefits that NYU has negotiated with GCP.
+:::
+:::warning
+For non-NYU work on GCP, please use your personal, non-NYU, Google email when creating GCP projects.
+:::
+
+The NYU Research Cloud team does not currently offer training on how to deploy and utilize resources on GCP in research projects or teaching. If you are new to GCP and you want to learn the GCP fundamentals or if you want to learn how to perform specific tasks on GCP (obtain skill badges), please consider the following resources:
+-   Through the [Google For Education][google-for-edu] program, GCP offers training credits and discounts to Students, Faculty, and IT Admins. To apply for training credits and discounts, please [click here](https://services.google.com/fb/forms/googlecloudskillsbooststudenttrainingcreditsapplication/).
+-   [Getting started with Google Cloud Platform][gcp-get-started] offers quick starts and sample projects on GCP.
+
+## How can I fund my research project on GCP?
+### GCP Free Tier
+Apply for credits using your NYU account (https://cloud.google.com/free/). After credits expire, if you would like to switch to another type of funding and are approved to do so, we will modify your project so it can use other funds:
+-   https://edu.google.com/programs/credits/research/?modal_active=none
+-   https://edu.google.com/programs/?modal_active=none
+### Sources of funding for GCP project
+Please consider options below and explore other options which may exist for your specific field.
+-   [Google Cloud research credits][gcp-credits]
+-   NIH STRIDES
+-   NSF CloudBank
+-   GCP seed grant through NYU HPC
+-   Departmental funds
+-   Apply using this form [GCP project request form](https://docs.google.com/forms/d/e/1FAIpQLSfyIxlsqBXtYlVgE8j_MZ1030HKdkTPWphXwwBUM5g6Lhi44w/viewform)

docs/cloud/02_gcp_self_managed/02_nih_strides.md

@@ -0,0 +1,23 @@
+# NIH Strides
+
+[nih-strides]: https://cloud.nih.gov/
+
+The NIH Science and Technology Research Infrastructure for Discovery, Experimentation, and Sustainability ([STRIDES][nih-strides]) Initiative allows NIH to explore the use of cloud environments to streamline NIH data use by partnering with commercial providers (GCP/AWS). The NIH STRIDES Initiative:
+-   Provides biomedical researchers with access to advanced, cost-effective cloud computing infrastructure, tools, and resources
+-   Enable researchers to work collaboratively in the cloud by establishing an interconnected ecosystem of biomedical research data
+-   Equip researchers with emerging cloud solutions for data management & computation to enable experimentation and innovation
+
+The benefits of participating in NIH Strides program include:
+-   Pre-negotiated favorable pricing for cloud services
+-   Access to training to help researchers harness the power of the cloud
+-   Receive opportunities for professional service engagements to help drive success
+-   Receive guidance for best practices in areas such as data storage, governance, and controlled access
+
+## Enrolling to the NIH Strides initiative
+
+NYU has enrolled in the NIH Strides initiative in December 2020 by signing an agreement with Carahsoft, GCP's billing and administrative partner. Thus NIH-funded NYU researchers with an active NIH award may take advantage of the STRIDES Initiative for their NIH-funded research projects. The NYU RTS team works closely with Burwood Group, a GCP reseller, to provide access to GCP resources for NYU researchers who are approved to participate in the NIH STRIDES initiative. NYU researchers who wish to participate must follow the steps outlined below.
+
+
+## Contacts
+-   For general questions about Research Cloud/GCP please email the NYU HPC Research Cloud team: hpc-cloud@nyu.edu
+-   To learn more about the NIH Strides Initiative, email the team at strides@nih.gov

docs/cloud/02_gcp_self_managed/_category_.json

@@ -0,0 +1,3 @@
+{
+  "label": "GCP self-managed projects"
+}

docs/cloud/03_hpc_bursting_to_cloud/01_intro.md

@@ -0,0 +1,61 @@
+# HPC Bursting
+
+[gcp-cost-calculator]: https://cloudpricingcalculator.appspot.com/
+[bursting-form]: https://sites.google.com/nyu.edu/nyu-hpc/hpc-systems/cloud-computing/hpc-bursting-to-cloud/hpc-bursting-request-form?authuser=0
+
+HPC may provide bursting capabilities to researchers or classes, in some cases, in order to augment the available resources. Bursting is ideal for when you need a large amount of resources for a very short period of time. The way that bursting is made possible is by running a scalable SLURM cluster in the Google Cloud Platform (GCP), which is separate from the on-premise HPC clusters.
+
+Bursting is not available to all users and requires advanced approval. In order to get access to these capabilities, please contact hpc@nyu.edu to check your eligibility. Please let us know the amount of storage, total CPUs, Memory, GPU, the number of days you require access, and the estimated total CPU/GPU hours you will use. For reference, please review the [GCP cost calculator][gcp-cost-calculator]. Please send a copy of your cost calculation to hpc@nyu.edu as well.
+
+:::tip
+To request access to the HPC Bursting capabilities, [please complete this form][bursting-form].
+:::
+
+## Running a Bursting Job
+:::note
+This is not public, only per request of eligible classes or researchers
+:::
+```sh
+ssh <NetID>@greene.hpc.nyu.edu
+```
+ssh to the class on GCP (burst login node) - anyone can login but you can only submit jobs if you have approval
+```sh
+ssh burst 
+```
+Start an interactive job
+```sh
+srun --account=hpc --partition=interactive --pty /bin/bash
+```
+If you got an error "Invalid account or account/partition combination specified" it means your account is not approved to use cloud bursting.
+
+Once your files are copied to the bursting instance you can run a batch job from the interactive session.
+
+## Access to Slurm Partitions
+In the example above the partition "interactive" is used. You can list current partitions by running command
+```sh
+sinfo
+```
+
+However, approval is required to submit jobs to the partitions. Partitions are set up by the resources available to a job, such as the number of CPU, amount of memory, and number of GPUs. Please email hpc@nyu.edu to request access to a specific partition or create a new partition (e.g. 10 CPUs and 64 GB Memory) for more optimal cost/performance of your job.
+
+### Storage
+
+Torch's `/home` and `/scratch` are mounted (available) at login node of bursting setup.
+
+Compute node however, do have independent `/home` and `/scratch`.  These `/home` and `/scratch` mounts are persistent, are available from any compute node and independent from `/home` and `/scratch` at Torch.
+
+User may need to copy data from Torch's `/home` or `/scratch` to GCP mounted `/home` or `/scratch`
+
+When you run a bursting job the compute nodes will not see those file mounts. This means that you need to copy data to the burst instance.
+
+The file systems are independent, so you must copy data to the GCP location.
+
+To copy data, you must first start an interactive job. Once started, you can copy your data using scp from the HPC Data Transfer Nodes (greene-dtn). Below is the basic setup to copy files from Torch to your home directory while you are in an interactive bursting job:
+```sh
+scp <NetID>@greene-dtn.hpc.nyu.edu:/path/to/files /home/<NetID>/
+```
+
+
+### Current Limits
+
+20,000 CPUs available at any given time for all active bursting users

docs/cloud/03_hpc_bursting_to_cloud/02_visualization.md

@@ -0,0 +1,65 @@
+# Visualization Workstations
+
+[vnc-clients]: https://help.ubuntu.com/community/VNC/Clients
+
+The burst cluster includes a partition (`nvgrid`) that can be used to run graphical applications on NVIDIA GPUs for visualization purposes.  You can use this partition by following the instructions below.
+
+
+Add the following to your SSH config file (~/.ssh/config) on your local workstation so that you can log into the burst login node directly:
+```sh
+Host burst
+  HostName burst.hpc.nyu.edu
+  User <NetID>
+  ProxyJump <NetID>@greene.hpc.nyu.edu
+  ProxyJump <NetID>@burst.hpc.nyu.edu
+  StrictHostKeyChecking no
+  UserKnownHostsFile /dev/null
+  LogLevel ERROR
+```
+
+Log into the burst login node by running `ssh <NetID>@burst` while on-campus or connected to the VPN.  Run the following command on the login node to request an interactive command line session:
+```sh
+srun --account=hpc --partition=nvgrid --gres=gpu:p100:1 --pty /bin/bash
+```
+
+When your interaction session is active, run the following command to start the VNC (remote desktop) server.  If this is the first time you’ve used a visualization node, you will be prompted to set a password to use when you access your remote session:
+```sh
+/opt/TurboVNC/bin/vncserver
+```
+
+Note the hostname of the node that you are running on.  This hostname is displayed in the NODELIST column of the output from the squeue command:
+```
+[NetID@b-23-1 ~]$ squeue
+             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
+…
+             92727    nvgrid     bash  jp6546  R       2:55      1 b-23-1
+```
+
+In another terminal on your local machine, run the following command:
+```
+ssh -N -L 5901:<Hostname>:5901 <NetID>@burst
+```
+This command will ensure that you can connect to the remote desktop service from your local computer.
+
+
+If you do not already have a VNC remote desktop client installed on your computer, you will need to install one.  A list of VNC clients available for various platforms can be found [here][vnc-clients].  Note that Mac OS X comes with a built-in VNC client, which is accessible from the Finder by navigating to Go → Connect to Server and then typing vnc:// at the beginning of the server field.
+
+Within your VNC client, connect to `localhost:5901` (`vnc://localhost:5901` on Mac OS).
+
+You should now be presented with a desktop environment. If you are using any OpenGL-based applications that are started from a terminal, be sure to type vglrun before the command name in order to ensure that the application uses the GPU.
+
+After your first time using the `nvgrid` partition, you can start the remote desktop server non-interactively using the following batch script (although you will need to remember the password that you set in step 3).  Note that the sleep command should have the length of time that you want the server to run (in seconds) after it (3600 seconds for 1 hour in the example below).
+```sh
+#!/bin/bash
+
+#SBATCH --gres=gpu:p100:1
+#SBATCH --partition=nvgridk
+#SBATCH --account=hpc
+#SBATCH --job-name=vnc
+#SBATCH --time=1:00
+#SBATCH --output=slurm_%j.out
+
+/opt/TurboVNC/bin/vncserver
+
+sleep 3600
+```

docs/cloud/03_hpc_bursting_to_cloud/_category_.json

@@ -0,0 +1,3 @@
+{
+  "label": "HPC bursting to cloud"
+}

docs/cloud/04_dataproc/01_intro.md

@@ -0,0 +1,40 @@
+# Dataproc
+
+[gcs]: https://cloud.google.com/storage?hl=en
+[bigquery]: https://cloud.google.com/bigquery?hl=en
+[bigtable]: https://cloud.google.com/bigtable?hl=en
+[data-locality]: https://en.wikipedia.org/wiki/Locality_of_reference
+[yarn-ui]: https://dataproc.hpc.nyu.edu/yarn/
+
+
+Dataproc is a cloud-based Hadoop distribution that is managed by Google.  Google administers updates to Dataproc so that it is kept current.  Google also packages and maintains additional software that can be run on top of Hadoop. Additionally Dataproc includes other cloud-specific features, such as the ability to automatically add/remove nodes depending upon how busy the cluster is (autoscaling).  It can also use object storage ([GCS][gcs]) or [BigQuery][bigquery] as an alternative to HDFS, and provides integration with [BigTable][bigtable] using HBase interfaces.
+
+### What is Hadoop?
+
+Hadoop is an open-source software framework for storing and processing big data in a distributed/parallel fashion on large clusters of commodity hardware. At its core, Hadoop strives to increase processing speed by increasing [data locality][data-locality] (i.e., it moves computation to servers where the data is located).  There are three components to Hadoop: HDFS (the Hadoop Distributed File System), the Hadoop implementation of MapReduce, and YARN (Yet Another Resource Negotiator; a scheduler).
+
+### Autoscaling
+
+NYU Dataproc is configured to be as cloud-agnostic as possible, and still uses HDFS and non-proprietary HBase components.  It does, however, leverage autoscaling.  This means that if the cluster hasn't been used for a while, it might take a while for resources to become available (typically 3-5 minutes) as nodes are turned on and NodeManagers register with YARN.  During this time, the following warning message will appear and indicate that the cluster is at capacity and resources are not available:
+```sh
+WARN org.apache.spark.scheduler.cluster.YarnScheduler: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
+```
+This warning will go away after new nodes have been added by autoscaling and more resources are available for YARN applications.  If it does not go away after more than 10 minutes, please contact the HPC team.  Autoscaling is actively monitored, but a duration of more than 10 minutes may indicate a failure of  Dataproc's monitoring infrastructure.
+
+Currently, NYU Dataproc's autoscaling is configured so that the cluster will have between 3 and 43 nodes depending upon demand.  The number of nodes that are currently active can be seen in the YARN web UI.  Additionally, percentage of cluster capacity that is used can be seen on the Scheduler page in the lefthand menu in the [YARN web UI][yarn-ui].
+
+## Accessing the NYU Dataproc Hadoop Cluster
+
+Access to the NYU Dataproc cluster is granted via your NYU Google account.  If you are in a class that uses Dataproc, your instructor or TA will request Dataproc access for your NetID.  
+
+Once this is granted you can log in by navigating to https://dataproc.hpc.nyu.edu/ssh in your web browser.  After you’ve reached this page you will have access to a browser-based terminal interface where you can run Hadoop commands.  
+
+If you are having difficulty connecting to a terminal, please make sure that you are not logged in to a non-NYU Google account by clicking the icon displayed in the upper right corner in the Gmail web interface (see [here](https://support.google.com/accounts/answer/1721977)).  If you are using Google Chrome, you may also need to switch to your NYU account profile using the instructions [here](https://support.google.com/chrome/answer/2364824).
+
+If you continue to have difficulties connecting via https://dataproc.hpc.nyu.edu/ssh, you can also log in by navigating to https://shell.cloud.google.com/ and running the following command in the terminal that appears:
+```sh
+gcloud compute ssh nyu-dataproc-m --project=hpc-dataproc-19b8 --zone=us-central1-f
+```
+You may need to authorize Google to log in to Dataproc after running the above command.
+
+Once logged in, your username will be of the form NetID_nyu_edu rather than just your NetID.