Skip to content

Commit 3a83abe

Browse files
denreadenrea
authored
Merge pull request #215746 from srajtiwari/patch-4
Use resized OS disk instead of an attached managed disk for IPOPP
2 parents 73d75ca + 5abe455 commit 3a83abe

File tree

1 file changed

+110
-181
lines changed

1 file changed

+110
-181
lines changed

‎articles/orbital/satellite-imagery-with-orbital-ground-station.md

Lines changed: 110 additions & 181 deletions
Original file line numberDiff line numberDiff line change
@@ -20,15 +20,18 @@ Using AOGS, we capture the Aqua broadcast when the satellite is within line of s
2020
In this tutorial, we will follow these steps to collect and process Aqua data:
2121

2222
> [!div class="checklist"]
23-
> * [Schedule a contact and collect Aqua direct broadcast data using AOGS](#step-1-schedule-a-contact-and-collect-aqua-direct-broadcast-data-using-aogs).
24-
> * [Process Aqua direct broadcast data using RT-STPS](#step-2-process-aqua-direct-broadcast-data-using-rt-stps).
25-
> * [Create higher level products using IPOPP](#step-3-create-higher-level-products-using-ipopp).
23+
> * [Use AOGS to schedule and contact and collect Aqua data](#step-1-use-aogs-to-schedule-a-contact-and-collect-aqua-data).
24+
> * [Install NASA DRL tools](#step-2-install-nasa-drl-tools).
25+
> * [Create Level-0 product using RT-STPS](#step-3-create-level-0-product-using-rt-stps).
26+
> * [Create higher level products using IPOPP](#step-4-create-higher-level-products-using-ipopp).
2627
27-
Optional setup steps for capturing the ground station telemetry are included in the [Appendix](#appendix).
28+
Optional setup steps for capturing the ground station telemetry are included the guide on [receiving real-time telemetry from the ground stations](receive-real-time-telemetry.md).
2829

29-
## Step 1: Schedule a contact and collect Aqua direct broadcast data using AOGS
30+
## Step 1: Use AOGS to schedule a contact and collect Aqua data
3031

31-
Follow the steps listed in [Tutorial: Downlink data from NASA's AQUA public satellite](downlink-aqua.md) to schedule a contact with Aqua using AOGS and collect the direct broadcast data on an Azure VM for further processing.
32+
Execute steps listed in [Tutorial: Downlink data from NASA's AQUA public satellite](downlink-aqua.md)
33+
34+
The above tutorial provides a walkthrough for scheduling a contact with Aqua and collecting the direct broadcast data on an Azure VM.
3235

3336
> [!NOTE]
3437
> In the section [Prepare a virtual machine (VM) to receive the downlinked AQUA data](downlink-aqua.md#prepare-your-virtual-machine-vm-and-network-to-receive-aqua-data), use the following values:
@@ -38,233 +41,159 @@ Follow the steps listed in [Tutorial: Downlink data from NASA's AQUA public sate
3841
> - **Size:** Standard_D8s_v5 or higher
3942
> - **IP Address:** Ensure that the VM has at least one standard public IP address
4043
41-
At the end of this step, you should have the raw direct broadcast saved as ```.bin``` files under the ```~/aquadata``` folder on the receiver-vm.
42-
43-
## Step 2: Process Aqua direct broadcast data using RT-STPS
44-
45-
The [Real-time Software Telemetry Processing System](https://directreadout.sci.gsfc.nasa.gov/?id=dspContent&cid=69)(RT-STPS) is a NASA-provided software for processing Aqua direct broadcast data. The steps below cover installation of RT-STPS Verson 6.0 on the receiver-vm, and production of Level-0 Production Data Set(PDS) files from the data collected in the previous step.
46-
47-
Register with the [NASA DRL](https://directreadout.sci.gsfc.nasa.gov/) to download the RT-STPS installation package.
48-
49-
Transfer the installation binaries to the receiver-vm:
50-
51-
```console
52-
ssh azureuser@receiver-vm 'mkdir -p ~/software'
53-
scp RT-STPS_6.0*.tar.gz azureuser@receiver-vm:~/software/.
54-
```
44+
At the end of this step, you should have the raw direct broadcast data saved as ```.bin``` files under the ```~/aquadata``` folder on the ```receiver-vm```.
5545

56-
Alternatively, you can upload your installation binaries to a container in Azure Storage and download them to the receiver-vm using [AzCopy](../storage/common/storage-use-azcopy-v10.md)
46+
## Step 2: Install NASA DRL tools
47+
> [!NOTE]
48+
> Due to potential resource contention, DRL recommends installing RT-STPS and IPOPP on separate machines. But for this tutorial, we install both tools on the ```receiver-vm``` because we don't run them at the same time. For production workloads, please follow sizing and isolation recommendations in the user guides available on the DRL website.
5749
58-
### Install rt-stps
50+
### Increase OS disk size on the receiver-vm
5951

60-
```console
61-
sudo yum install java-11-openjdk
62-
cd ~/software
63-
tar -xzvf RT-STPS_6.0.tar.gz
64-
cd ./rt-stps
65-
./install.sh
66-
```
52+
The default disk space allocated to the OS disk of an Azure VM is not sufficient for installing NASA DRL tools. Follow the steps below to increase the size of the OS disk on the ```receiver-vm``` to 1TB.
6753

68-
### Install rt-stps patches
54+
### [Portal](#tab/portal2)
6955

70-
```console
71-
cd ~/software
72-
tar -xzvf RT-STPS_6.0_PATCH_1.tar.gz
73-
tar -xzvf RT-STPS_6.0_PATCH_2.tar.gz
74-
tar -xzvf RT-STPS_6.0_PATCH_3.tar.gz
75-
cd ./rt-stps
76-
./install.sh
77-
```
78-
79-
### Validate install
56+
1. Open the [portal](https://portal.azure.com).
57+
1. Navigate to your virtual machine.
58+
1. On the **Overview** page, select **Stop**.
59+
1. On the **Disks** page, select the OS disk.
60+
1. On the **Disk** pane, navigate to **Size + performance** page.
61+
1. Select **Premium SSD(locally redundant storage)** from the **Disk SKU** dropdown.
62+
1. Select the **P30** Disk Tier (1024GB).
63+
1. Select **Save**.
64+
1. Navigate back to **Virtual Machine** pane.
65+
1. On the **Overview** page, select **Start**
66+
---
67+
On the receiver-vm, verify that the root partition now has 1TB available
8068

81-
```console
82-
cd ~/software
83-
tar -xzvf RT-STPS_6.0_testdata.tar.gz
84-
cd ~/software/rt-stps
85-
rm ./data/*
86-
./bin/batch.sh config/npp.xml ./testdata/input/rt-stps_npp_testdata.dat
87-
# Verify that files exist
88-
ls -la ./data
69+
```bash
70+
lsblk -o NAME,HCTL,SIZE,MOUNTPOINT
8971
```
90-
91-
### Create Level-0 product
92-
93-
Run rt-stps in batch mode to process the ```.bin``` file collected in Step 1
72+
This should show ~1TB allocated to the root ```/``` mountpoint.
9473

9574
```console
96-
cd ~/software/rt-stps
97-
./bin/batch.sh ./config/aqua.xml ~/aquadata/raw-2022-05-29T0957-0700.bin
75+
NAME HCTL SIZE MOUNTPOINT
76+
sda 0:0:0:0 1T
77+
├─sda1 500M /boot
78+
├─sda2 1023G /
79+
├─sda14 4M
80+
└─sda15 495M /boot/efi
9881
```
9982

100-
This command produces Level-0 Production Data Set (```.pds```) files under the ```~/software/rt-stps/data``` directory.
101-
102-
## Step 3: Create higher level products using IPOPP
103-
104-
[International Planetary Observation Processing Package (IPOPP)](https://directreadout.sci.gsfc.nasa.gov/?id=dspContent&cid=68) is another NASA-provided software to process Aqua Level-0 data into higher level products.
105-
In the steps below, you'll process the Level-0 data generated in the previous step using IPOPP.
106-
107-
> [!NOTE]
108-
> Due to potential resource contention, DRL recommends installing RT-STPS and IPOPP on separate machines. But for this tutorial, we install both on the our receiver-vm because we don't run them at the same time. For production workloads, please follow sizing and isolation recommendations in the user guides available on the DRL website.
109-
110-
### Attach a data disk to the receiver-vm
111-
112-
IPOPP installation and subsequent generation of products requires more disk space and I/O throughput than what is available on the receiver-vm by default.
113-
To provide more disk space and throughput, attach a 1TB premium data disk to the receiver-vm by following steps in [Attach a data disk to a Linux VM](../virtual-machines/linux/attach-disk-portal.md)
114-
115-
### Create a file system on the data disk
116-
117-
```console
118-
lsblk -o NAME,HCTL,SIZE,MOUNTPOINT | grep -i "sd"
119-
sudo parted /dev/sdb --script mklabel gpt mkpart xfspart xfs 0% 100%
120-
sudo mkfs.xfs /dev/sdb1
121-
sudo partprobe /dev/sdb1
122-
sudo mkdir /datadrive
123-
sudo mount /dev/sdb1 /datadrive
124-
sudo chown azureuser:azureuser /datadrive
125-
```
126-
> [!NOTE]
127-
> To ensure that the datadrive is mounted automatically after every reboot, please refer to [Attach a data disk to a Linux VM](../virtual-machines/linux/attach-disk-portal.md#mount-the-disk) for instructions on how to add an entry to ```/etc/fstab```
128-
129-
13083
### Install Desktop and VNC Server
131-
132-
IPOPP installation requires using a browser to sign on to the DRL website to download the installation script. This script must be run from the same host that it was downloaded to. The subsequent IPOPP configuration also requires a GUI. Therefore, we install a full desktop and a vnc server to enable running GUI applications on the receiver-vm.
133-
134-
```console
84+
Using NASA DRL tools requires support for running GUI applications. To enable this, install desktop tools and vncserver on the `receiver-vm`:
85+
```bash
13586
sudo yum install tigervnc-server
13687
sudo yum groups install "GNOME Desktop"
13788
```
138-
13989
Start VNC server:
140-
141-
```console
90+
```bash
14291
vncsever
14392
```
144-
Enter a password when prompted.
145-
146-
Port forward the vncserver port (5901) over ssh:
93+
Enter a password when prompted.
14794

148-
```console
95+
### Remotely access the VM Desktop
96+
Port forward the vncserver port (5901) over SSH to your local machine:
97+
```bash
14998
ssh -L 5901:localhost:5901 azureuser@receiver-vm
15099
```
100+
1. On your local machine, download and install [TightVNC Viewer](https://www.tightvnc.com/download.php).
101+
1. Start the TightVNC Viewer and connect to ```localhost:5901```.
102+
1. Enter the vncserver password you entered in the previous step.
103+
1. You should see the GNOME Desktop that is running on the VM in the VNC viewer window.
151104

152-
Download the [TightVNC](https://www.tightvnc.com/download.php) viewer and connect to ```localhost:5901``` and enter the vncserver password entered in the previous step. You should see the GNOME desktop running on the VM.
153-
154-
Start a new terminal, and start the Firefox browser
155-
156-
```console
157-
firefox
158-
```
105+
### Download RT-STPS and IPOPP installation files
106+
From the GNOME Desktop, go to **Applications** > **Internet** > **Firefox** to start a browser.
159107

160-
[Log on the DRL website](https://directreadout.sci.gsfc.nasa.gov/loginDRL.cfm?cid=320&type=software) and download the downloader script.
108+
Log on to the [NASA DRL](https://directreadout.sci.gsfc.nasa.gov/?id=dspContent&cid=325&type=software) website and download the **RT-STPS** installation files and the **IPOPP downloader script** under software downloads. The downloaded files will land under ~/Downloads.
161109

162-
Run the downloader script from the ```/datadrive/ipopp``` directory because
163-
the home directory isn't large enough to hold the downloaded content.
110+
Alternatively, you can download the installation files on your local machine first and then upload to a container in Azure Storage. Then use [AzCopy](../storage/common/storage-use-azcopy-v10.md) to download to your ```receiver-vm```.
164111

165-
```console
166-
INSTALL_DIR=/datadrive/ipopp
167-
cp ~/Downloads/downloader_DRL-IPOPP_4.1.sh $INSTALL_DIR
168-
cd $INSTALL_DIR
169-
./downloader_DRL-IPOPP_4.1.sh
112+
### Install RT-STPS
113+
```bash
114+
tar -xvzf ~/Downloads/RT-STPS_7.0.tar.gz --directory ~/
115+
tar -xvzf ~/Downloads/RT-STPS_7.0_testdata.tar.gz --directory ~/
116+
cd ~/rt-stps
117+
./install.sh
170118
```
171-
172-
This script will download \~35G and will take 1 hour or more.
173-
174-
Alternatively, you can upload your installation binaries to a container in Azure Storage and download them to the receiver-vm using [AzCopy](../storage/common/storage-use-azcopy-v10.md)
119+
Validate your RT-STPS install by processing the test data supplied with the installation:
120+
```bash
121+
cd ~/rt-stps
122+
./bin/batch.sh config/jpss1.xml ./testdata/input/rt-stps_jpss1_testdata.dat
123+
```
124+
Verify that output files exist in the data folder:
125+
```bash
126+
ls -la ~/data/
127+
```
128+
This completes the RT-STPS installation.
175129

176130
### Install IPOPP
177-
178-
```console
179-
tar -xvzf DRL-IPOPP_4.1.tar.gz --directory $INSTALL_DIR
180-
chmod -R 755 $INSTALL_DIR/IPOPP
181-
$INSTALL_DIR/IPOPP/install_ipopp.sh -installdir $INSTALL_DIR/drl -datadir $INSTALL_DIR/data -ingestdir $INSTALL_DIR/data/ingest
131+
Run the IPOPP downloader script to download the IPOPP installation files.
132+
```bash
133+
cd ~/Downloads
134+
./downloader_DRL-IPOPP_4.1.sh
135+
tar -xvzf ~/Downloads/DRL-IPOPP_4.1.tar.gz --directory ~/
136+
cd ~/IPOPP
137+
./install_ipopp.sh
182138
```
183139

184-
### Install IPOPP patches
140+
### Configure and start IPOPP services
141+
IPOPP services are configured using its Dashboard GUI.
185142

186-
```console
187-
$INSTALL_DIR/drl/tools/install_patch.sh $PATCH_FILE_NAME
188-
```
189-
### Start IPOPP services
190-
191-
```console
192-
$INSTALL_DIR/drl/tools/services.sh start
193-
```
194-
### Verify service status
143+
[Go to the VM Desktop](#remotely-access-the-vm-desktop) and start a new terminal under **Applications** > **Utilities** > **Terminal**
195144

145+
Start the IPOPP dashboard from the terminal:
146+
```bash
147+
~/drl/tools/dashboard.sh
196148
```
197-
$INSTALL_DIR/drl/tools/services.sh status
198-
$INSTALL_DIR/drl/tools/spa_services.sh status
199-
```
149+
IPOPP starts in the process monitoring mode. Switch to **Configuration Mode** by the using the menu option.
200150

201-
### Configure IPOPP services using its dashboard
151+
Enable the following under the **EOS** tab:
152+
* gbad
153+
* MODISL1DB l0l1aqua
154+
* MODISL1DB l1atob
155+
* IMAPP
202156

203-
Before we can create Level-1 and Level-2 products from the Level-0 PDS files generated by rt-stps, we need to configure IPOPP. IPOPP must be configured with its dashboard GUI. To start the dashboard, first port forward the vncserver port (5901) over ssh:
157+
Switch back to **Process Monitoring** mode using the menu option.
204158

205-
```console
206-
ssh -L 5901:localhost:5901 azureuser@receiver-vm
159+
Start IPOPP services:
160+
```bash
161+
~/drl/tools/services.sh start
162+
~/drl/tools/services.sh status
207163
```
164+
This completes the IPOPP installation and configuration.
208165

209-
Using the TightVNC client, connect to localhost:5901 and enter the vncserver password. On the virtual machine desktop, open a new terminal and start the dashboard:
166+
## Step 3: Create Level-0 product using RT-STPS
210167

211-
```console
212-
cd /datadrive/ipopp
213-
./drl/tools/dashboard.sh & 
168+
Run rt-stps in batch mode to process the ```.bin``` file collected in Step 1
169+
```bash
170+
cd ~/rt-stps
171+
./bin/batch.sh ./config/aqua.xml ~/aquadata/raw-2022-05-29T0957-0700.bin
214172
```
173+
This command produces Level-0 Production Data Set (```.pds```) files under the ```~/rt-stps/data``` directory.
215174

216-
1. IPOPP Dashboard starts in process monitoring mode. Switch to **Configuration Mode** by using the menu option. 
217-
218-
2. Aqua related products can be configured from EOS tab in configuration mode. Disable all other tabs. We're interested in the MODIS Aerosol L2 (MOD04) product, which is produced by IMAPP SPA. Therefore, enable the following in the **EOS** tab: 
219-
220-
- gbad 
221-
222-
- MODISL1DB l0l1aqua 
223-
224-
- MODISL1DB l1atob 
225-
226-
- IMAPP 
227-
228-
3. After updating the configuration, switch back to **Process Monitoring** mode using the menu. All tiles will be in OFF mode initially. 
229-
230-
4. When prompted, save changes to the configuration.  
231-
232-
5. Click **Start Services** in the action menu. Note that **Start Services** is only enabled in process monitoring mode.  
233-
234-
6. Click **Check IPOPP Services** in action menu to validate.
175+
## Step 4: Create higher level products using IPOPP
235176

236177
## Ingest data for processing
237178

238-
Copy the Level-0 PDS files generated by RT-STPS to the IPOPP ingest directory for further processing.
179+
Copy the PDS files generated by RT-STPS in the previous step to the IPOPP ingest directory for further processing.
239180

240-
```console
241-
cp ~/software/rt-stps/data/* /datadrive/ipopp/drl/data/dsm/ingest/.
181+
```bash
182+
cp ~/rt-stps/data/* ~/drl/data/dsm/ingest/.
242183
```
243-
244184
Run IPOPP ingest to create the products configured in the dashboard. 
245-
246-
```
247-
/datadrive/ipopp/drl/tools/ingest_ipopp.sh
185+
```bash
186+
~/drl/tools/ingest_ipopp.sh
248187
```
249-
250188
You can watch the progress in the dashboard.
251-
189+
```bash
190+
~/drl/tools/dashboard.sh
252191
```
253-
/datadrive/ipopp/drl/tools/dashboard.sh
192+
IPOPP will produce output products in the following directory:
193+
```bash
194+
cd ~/drl/data/pub/gsfcdata/aqua/modis/
254195
```
255196

256-
IPOPP will produce output products in the following directories:
257-
258-
```
259-
/datadrive/ipopp/drl/data/pub/gsfcdata/aqua/modis/level[0,1,2] 
260-
```
261-
262-
## Appendix
263-
264-
### Capture ground station telemetry
265-
266-
Follow steps here to [receive real-time telemetry from the ground stations](receive-real-time-telemetry.md).
267-
268197
## Next steps
269198

270199
For an end-to-end implementation that involves extracting, loading, transforming, and analyzing spaceborne data by using geospatial libraries and AI models with Azure Synapse Analytics, see:

0 commit comments

Comments
 (0)