You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: content/blog/production-ready-object-detection-model-training-workflow-with-hpe-machine-learning-development-environment.md
+13-13Lines changed: 13 additions & 13 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -81,7 +81,7 @@ Now that you have your Docker engine installed and the PyTorch Container running
81
81
82
82
This notebook walks you through each step required to train a model using containers from the NGC catalog. We chose the GPU optimized PyTorch container as an example. The basics of working with docker containers apply to all NGC containers.
83
83
84
-
We will show you how to:
84
+
Here, I will show you how to:
85
85
86
86
* Download the Xview Dataset
87
87
* How to convert labels to coco format
@@ -92,25 +92,25 @@ Let's get started!
92
92
93
93
- - -
94
94
95
-
## Pre-reqs, set up Jupyter notebook environment using NGC container
95
+
## Pre-reqs, set up Jupyter Notebook environment using NGC container
96
96
97
-
### Execute docker run to create NGC environment for data preparation
97
+
### Execute Docker run to create NGC environment for data preparation
98
98
99
-
Make sure to map host directory to docker directory, we will use the host directory again to
99
+
Make sure to map host directory to Docker directory. You will use the host directory again to do the following:
100
100
101
101
*`docker run --gpus all --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 -v /home/ubuntu:/home/ubuntu -p 8008:8888 -it nvcr.io/nvidia/pytorch:21.11-py3 /bin/bash`
102
102
103
-
### Run Jupyter notebook command within Docker container to access it on your local browser
103
+
### Run Jupyter Notebook command within Docker container to access it on your local browser
The dataset we will be using is from the [DIUx xView 2018 Challenge](https://challenge.xviewdataset.org) by U.S. National Geospatial-Intelligence Agency (NGA). You will need to [create an account](https://challenge.xviewdataset.org/welcome), agree to the terms and conditions, and download the dataset manually.
112
112
113
-
You can also [download the dataset](https://challenge.xviewdataset.org/data-download).
113
+
You can also [download the dataset](https://challenge.xviewdataset.org/data-download).
114
114
115
115
```python
116
116
# run pip install to get the SAHI library
@@ -146,9 +146,9 @@ You can also [download the dataset](https://challenge.xviewdataset.org/data-down
146
146
--out_dir xview_dataset/train_images_rgb/
147
147
```
148
148
149
-
## 2. How to convert labels to coco format
149
+
## 2. How to convert labels to COCO format
150
150
151
-
Here we run a script to convert the dataset labels from .geojson format to COCO format. [Read more details about the COCO format at this link.](https://www.immersivelimit.com/tutorials/create-coco-annotations-from-scratch)
151
+
Run a script to convert the dataset labels from .geojson format to COCO format. [Read more details about the COCO format at this link.](https://www.immersivelimit.com/tutorials/create-coco-annotations-from-scratch)
152
152
153
153
The result will be two files (in COCO formal) generated `train.json` and `val.json`
154
154
@@ -165,7 +165,7 @@ The result will be two files (in COCO formal) generated `train.json` and `val.js
165
165
166
166
## 3. Slicing/Tiling the Dataset
167
167
168
-
Here we are using the SAHI library to slice our large satellite images. Satellite images can be up to 50k^2 pixels in size, which wouldn't fit in GPU memory. We alleviate this problem by slicing the image.
168
+
Here, you will be using the SAHI library to slice our large satellite images. Satellite images can be up to 50k^2 pixels in size, which wouldn't fit in GPU memory. You can alleviate this problem by slicing the image.
## 4. Upload to s3 bucket to support distributed training
182
+
## 4. Upload to S3 bucket to support distributed training
183
183
184
-
We will now upload our exported data to a publically accessible S3 bucket. This will enable for a large scale distributed experiment to have access to the dataset without installing the dataset on device.
184
+
We will now upload our exported data to a publically accessible AWS S3 bucket. This will enable for a large scale distributed experiment to have access to the dataset without installing the dataset on device.
185
185
View [Determined Documentation](<* https://docs.determined.ai/latest/training/load-model-data.html#streaming-from-object-storage>) and [AWS instructions](<* https://codingsight.com/upload-files-to-aws-s3-with-the-aws-cli/>) to learn how to upload your dataset to an S3 bucket. Review the `S3Backend` class in `data.py`
186
186
187
187
Once you create an S3 bucket that is publically accessible, here are example commands to upload the preprocessed dataset to S3:
@@ -191,7 +191,7 @@ Once you create an S3 bucket that is publically accessible, here are example com
191
191
192
192
Our satellite imagery data is in an S3 bucket and is prepped for distributed training, so now we can progress to model training and inference via the NGC Container.
193
193
194
-
# Part 3: End-to-End Example training object detection model using NVIDIA Pytorch Container from NGC
194
+
# Part 3: End-to-End example training object detection model using NVIDIA PyTorch Container from NGC
0 commit comments