Merge pull request #207011 from srajtiwari/patch-2

Use a single VM for end-to-end processing

Author: Stacyrch140

articles/orbital/satellite-imagery-with-orbital-ground-station.md

@@ -1,6 +1,6 @@
 ---
-title: Collect and process Aqua satellite payload - Azure Orbital
-description: An end-to-end walk-through of using the Azure Orbital Ground Station (AOGS) to capture and process satellite imagery.
+title: Collect and process Aqua satellite data - Azure Orbital
+description: An end-to-end walk-through of using the Azure Orbital Ground Station (AOGS) to capture and process Aqua satellite imagery.
 ms.service: orbital
 author: EliotSeattle
 ms.author: eliotgra
@@ -9,39 +9,40 @@ ms.date: 07/13/2022
 ms.custom: template-overview 
 ---
 
-# Tutorial: Collect and process Aqua satellite payload using Azure Orbital Ground Station (AOGS)
+# Tutorial: Collect and process Aqua satellite data using Azure Orbital Ground Station (AOGS)
 
 This article is a comprehensive walk-through showing how to use the [Azure Orbital Ground Station (AOGS)](https://azure.microsoft.com/services/orbital/) to capture and process satellite imagery. It introduces the AOGS and its core concepts and shows how to schedule contacts. The article also steps through an example in which we collect and process NASA Aqua satellite data in an Azure virtual machine (VM) using NASA-provided tools.
 
-Aqua is a polar-orbiting spacecraft launched by NASA in 2002. Data from all science instruments aboard Aqua is downlinked to the Earth using direct broadcast over the X-band in near real-time. More information about Aqua can be found on the [Aqua Project Science](https://aqua.nasa.gov/) website. With AOGS, we can capture the Aqua broadcast when the satellite is within line of sight of a ground station.
+Aqua is a polar-orbiting spacecraft launched by NASA in 2002. Data from all science instruments aboard Aqua is downlinked to the Earth using direct broadcast over the X-band in near real-time. More information about Aqua can be found on the [Aqua Project Science](https://aqua.nasa.gov/) website. 
 
-A *contact* is time reserved at an orbital ground station to communicate with a satellite. During the contact, the ground station orients its antenna towards Aqua and captures the broadcast payload. The captured data is sent to an Azure VM as a data stream that is processed using the [RT-STPS](http://directreadout.sci.gsfc.nasa.gov/index.cfm?section=technology&page=NISGS&subpage=NISFES&sub2page=RT-STPS&sub3Page=overview) (Real-Time Software Telemetry Processing System) provided by [Direct Readout Laboratory](http://directreadout.sci.gsfc.nasa.gov/) at NASA to generate a level 0 product. Further processing of level 0 data is done using IPOPP (International Planetary Observation Processing Package) tool, also provided by DRL.
+Using AOGS, we capture the Aqua broadcast when the satellite is within line of sight of a ground station by scheduling a *contact*. A *contact* is time reserved at a ground station to communicate with a satellite. During the contact, the ground station orients its antenna towards Aqua and captures the direct broadcast data. The captured data is sent to an Azure VM as a data stream and processed using the [Real-Time Software Telemetry Processing System](http://directreadout.sci.gsfc.nasa.gov/index.cfm?section=technology&page=NISGS&subpage=NISFES&sub2page=RT-STPS&sub3Page=overview)(RT-STPS) tool provided by the [Direct Readout Laboratory](http://directreadout.sci.gsfc.nasa.gov/)(DRL) which generates a Level-0 product. This Level-0 product is processed further using DRL's [International Planetary Observation Processing Package](https://directreadout.sci.gsfc.nasa.gov/?id=dspContent&cid=68)(IPOPP) tool to produce higher level products.
 
-In this tutorial, you'll follow these steps to process the Aqua data stream:
+In this tutorial, we will follow these steps to collect and process Aqua data:
 
 > [!div class="checklist"]
-> * [Prerequisites](#step-1-prerequisites).
-> * [Process RAW data using RT-STPS](#step-2-process-raw-data-using-rt-stps).
-> * [Prepare a virtual machine (processor-vm) to process higher level products](#step-3-prepare-a-virtual-machine-processor-vm-to-create-higher-level-products).
-> * [Create higher level products using IPOPP](#step-4-create-higher-level-products-using-ipopp).
+> * [Schedule a contact and collect Aqua direct broadcast data using AOGS](#step-1-schedule-a-contact-and-collect-aqua-direct-broadcast-data-using-aogs).
+> * [Process Aqua direct broadcast data using RT-STPS](#step-2-process-aqua-direct-broadcast-data-using-rt-stps).
+> * [Create higher level products using IPOPP](#step-3-create-higher-level-products-using-ipopp).
 
 Optional setup steps for capturing the ground station telemetry are included in the [Appendix](#appendix).
 
-## Step 1: Prerequisites
+## Step 1: Schedule a contact and collect Aqua direct broadcast data using AOGS
 
-You must first follow the steps listed in [Tutorial: Downlink data from NASA's AQUA public satellite](downlink-aqua.md). 
+Follow the steps listed in [Tutorial: Downlink data from NASA's AQUA public satellite](downlink-aqua.md) to schedule a contact with Aqua using AOGS and collect the direct broadcast data on an Azure VM for further processing. 
 
 > [!NOTE]
 > In the section [Prepare a virtual machine (VM) to receive the downlinked AQUA data](downlink-aqua.md#prepare-your-virtual-machine-vm-and-network-to-receive-aqua-data), use the following values:
 >
 >   - **Name:** receiver-vm
 >   - **Operating System:** Linux (CentOS Linux 7 or higher)
->   - **Size:** Standard_D8_v5 or higher
+>   - **Size:** Standard_D8s_v5 or higher
 >   - **IP Address:** Ensure that the VM has at least one standard public IP address
 
-## Step 2: Process RAW data using RT-STPS
+At the end of this step, you should have the raw direct broadcast saved as ```.bin``` files under the ```~/aquadata``` folder on the receiver-vm. 
 
-The [Real-time Software Telemetry Processing System (RT-STPS)](https://directreadout.sci.gsfc.nasa.gov/?id=dspContent&cid=69) is NASA-provided software for processing the raw Aqua payload. The steps below cover installation of RT-STPS on the receiver-vm, and production of level-0 PDS files for the Aqua payload captured in the previous step. 
+## Step 2: Process Aqua direct broadcast data using RT-STPS
+
+The [Real-time Software Telemetry Processing System](https://directreadout.sci.gsfc.nasa.gov/?id=dspContent&cid=69)(RT-STPS) is a NASA-provided software for processing Aqua direct broadcast data. The steps below cover installation of RT-STPS Verson 6.0 on the receiver-vm, and production of Level-0 Production Data Set(PDS) files from the data collected in the previous step. 
 
 Register with the [NASA DRL](https://directreadout.sci.gsfc.nasa.gov/) to download the RT-STPS installation package.
 
@@ -87,49 +88,49 @@ rm ./data/*
 ls -la ./data
 ```
 
-### Process RAW Aqua data
+### Create Level-0 product
 
-Run rt-stps in batch mode to process the previously captured Aqua data (.bin files).
+Run rt-stps in batch mode to process the ```.bin``` file collected in Step 1
 
 ```console
 cd ~/software/rt-stps
 ./bin/batch.sh ./config/aqua.xml ~/aquadata/raw-2022-05-29T0957-0700.bin
 ```
 
-That command creates level-0 product (.PDS files) in the ```~/software/rt-stps/data``` directory.
-[AzCopy](../storage/common/storage-use-azcopy-v10.md) the level-0 files to a storage container:
-
-```console
-azcopy sync ~/software/rt-stps/data/ "https://[account].blob.core.windows.net/[container]/[path/to/directory]?[SAS]"
-```
+This command produces Level-0 Production Data Set (```.pds```) files under the ```~/software/rt-stps/data``` directory.
 
-Download the level-0 PDS files from this storage container for further processing in later steps. 
-
-## Step 3: Prepare a virtual machine (processor-vm) to create higher level products
+## Step 3: Create higher level products using IPOPP
 
 [International Planetary Observation Processing Package (IPOPP)](https://directreadout.sci.gsfc.nasa.gov/?id=dspContent&cid=68) is another NASA-provided software to process Aqua Level-0 data into higher level products.
-In the steps below, you'll process the Aqua PDS files downloaded from the Azure Storage container in the previous step. 
-Because IPOPP has higher system requirements than RT-STPS, it should be run on a bigger VM called the "processor-vm".
+In the steps below, you'll process the Level-0 data generated in the previous step using IPOPP.
+
+> [!NOTE]
+> Due to potential resource contention, DRL recommends installing RT-STPS and IPOPP on separate machines. But for this tutorial, we install both on the our receiver-vm because we don't run them at the same time. For production workloads, please follow sizing and isolation recommendations in the user guides available on the DRL website. 
 
-[Create a virtual machine(VM)](../virtual-machines/linux/quick-create-portal.md) within the virtual network above. Ensure that this VM has the following specifications:
+### Attach a data disk to the receiver-vm
 
--  **Name:** processor-vm
--  **Size:** Standard D16ds v5
--  **Operating System:** Linux (CentOS Linux 7 or higher)
--  **Disk:** 2 TB Premium SSD data disk
+IPOPP installation and subsequent generation of products requires more disk space and I/O throughput than what is available on the receiver-vm by default. 
+To provide more disk space and throughput, attach a 1TB premium data disk to the receiver-vm by following steps in [Attach a data disk to a Linux VM](../virtual-machines/linux/attach-disk-portal.md)
 
-Create a file system on the data disk:
+### Create a file system on the data disk
 
 ```console
-sudo fdisk /dev/sdc
-sudo mkfs -t ext4 /dev/sdc1
-sudo mount /dev/sdc1 /datadrive
+lsblk -o NAME,HCTL,SIZE,MOUNTPOINT | grep -i "sd"
+sudo parted /dev/sdb --script mklabel gpt mkpart xfspart xfs 0% 100% 
+sudo mkfs.xfs /dev/sdb1
+sudo partprobe /dev/sdb1 
+sudo mkdir /datadrive
+sudo mount /dev/sdb1 /datadrive
+sudo chown azureuser:azureuser /datadrive
 ```
+> [!NOTE]
+> To ensure that the datadrive is mounted automatically after every reboot, please refer to [Attach a data disk to a Linux VM](../virtual-machines/linux/attach-disk-portal.md#mount-the-disk) for instructions on how to add an entry to ```/etc/fstab```
 
-IPOPP installation requires using a browser to sign on to the DRL website to download the installation script. This script must be run from the same host that it was downloaded to. IPOPP configuration also requires a GUI. Therefore, we install a full desktop and a vnc server to enable running GUI applications.
 
 ### Install Desktop and VNC Server
 
+IPOPP installation requires using a browser to sign on to the DRL website to download the installation script. This script must be run from the same host that it was downloaded to. The subsequent IPOPP configuration also requires a GUI. Therefore, we install a full desktop and a vnc server to enable running GUI applications on the receiver-vm.
+
 ```console
 sudo yum install tigervnc-server
 sudo yum groups install "GNOME Desktop"
@@ -145,7 +146,7 @@ Enter a password when prompted.
 Port forward the vncserver port (5901) over ssh:
 
 ```console
-ssh -L 5901:localhost:5901 azureuser@processor-vm
+ssh -L 5901:localhost:5901 azureuser@receiver-vm
 ```
 
 Download the [TightVNC](https://www.tightvnc.com/download.php) viewer and connect to ```localhost:5901``` and enter the vncserver password entered in the previous step.  You should see the GNOME desktop running on the VM. 
@@ -170,6 +171,8 @@ cd $INSTALL_DIR
 
 This script will download \~35G and will take 1 hour or more.
 
+Alternatively, you can upload your installation binaries to a container in Azure Storage and download them to the receiver-vm using [AzCopy](../storage/common/storage-use-azcopy-v10.md)
+
 ### Install IPOPP
 
 ```console
@@ -178,7 +181,7 @@ chmod -R 755 $INSTALL_DIR/IPOPP
 $INSTALL_DIR/IPOPP/install_ipopp.sh -installdir $INSTALL_DIR/drl -datadir $INSTALL_DIR/data -ingestdir $INSTALL_DIR/data/ingest
 ```
 
-### Install patches
+### Install IPOPP patches
 
 ```console
 $INSTALL_DIR/drl/tools/install_patch.sh $PATCH_FILE_NAME
@@ -195,16 +198,12 @@ $INSTALL_DIR/drl/tools/services.sh status
 $INSTALL_DIR/drl/tools/spa_services.sh status
 ```
 
-## Step 4: Create higher level products using IPOPP
-
-Before we can create level-1 and level-2 products from the PDS files, we need to configure IPOPP. 
-
-### Configure the IPOPP service using the dashboard
+### Configure IPOPP services using its dashboard
 
-IPOPP must be configured with the dashboard GUI. To start the dashboard, first port forward the vncserver port (5901) over ssh:
+Before we can create Level-1 and Level-2 products from the Level-0 PDS files generated by rt-stps, we need to configure IPOPP. IPOPP must be configured with its dashboard GUI. To start the dashboard, first port forward the vncserver port (5901) over ssh:
 
 ```console
-ssh -L 5901:localhost:5901 azureuser@processor-vm
+ssh -L 5901:localhost:5901 azureuser@receiver-vm
 ```
 
 Using the TightVNC client, connect to localhost:5901 and enter the vncserver password. On the virtual machine desktop, open a new terminal and start the dashboard:
@@ -236,15 +235,13 @@ cd /datadrive/ipopp
 
 ## Ingest data for processing
 
-Download the PDS files generated by the RT-STPS tool from your storage container to the IPOPP ingest directory configured during installation.
+Copy the Level-0 PDS files generated by RT-STPS to the IPOPP ingest directory for further processing.
 
 ```console
-azcopy cp
-"https://[account].blob.core.windows.net/[container]/[path/to/directory]?[SAS]"
-"/datadrive/ipopp/drl/data/dsm/ingest" --recursive=true
+cp ~/software/rt-stps/data/* /datadrive/ipopp/drl/data/dsm/ingest/.
 ```
 
-Run the IPOPP ingest to create the products configured in the dashboard. 
+Run IPOPP ingest to create the products configured in the dashboard. 
 
 ```
 /datadrive/ipopp/drl/tools/ingest_ipopp.sh