Merge branch 'main' of github.com:markveillette/sevir_challenges into main

Author: markveilletteLL

README.md

@@ -1,17 +1,53 @@
 # sevir_challenges
-A collection of tasks and baseline models for the SEVIR weather dataset
-
+A collection of challenges and baseline models for the SEVIR weather dataset.
 
 ## Obtaining SEVIR data
 
-To obtian the dataset used in each of the challenges, run the following command
+The challenges in this repo are based on the [SEVIR weather dataset](https://proceedings.neurips.cc//paper/2020/hash/fa78a16157fed00d7a80515818432169-Abstract.html).  This dataset is made of up sequences of weather imagery sampled and aligned across radar and satellite.   It was constucted as a benchmark dataset to support algorithm development in meterology. For a detailed tutorial on this dataset, see [the SEVIR tutorial.](https://nbviewer.jupyter.org/github/MIT-AI-Accelerator/eie-sevir/blob/master/examples/SEVIR_Tutorial.ipynb)
+
+SEVIR is currently available for download from the [AWS Open Data registry](https://registry.opendata.aws/sevir/).  In total, the dataset is approximately 1TB in size, however smaller samples of the full dataset are provided for selected challenges (see `s3://sevir/data/processed/`).  To construct larger datasets, you can download SEVIR using one of the following methods:
+
+### Using AWS CLI
+
+If you have [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html), you can download SEVIR using the 
 
-To download, install AWS CLI, and download all of SEVIR (~1TB) to your current directory run
 ```
 aws s3 sync --no-sign-request s3://sevir .
 ```
-Each of the benchmarks in this repo use a subset of the full SEVIR dataset.
 
+To download only a specific modalitiy, e.g. `vil`, you can instead run
+
+```
+aws s3 cp --no-sign-request s3://sevir/CATALOG.csv CATALOG.csv
+aws s3 sync --no-sign-request s3://sevir/data/vil .
+```
+
+### Using `boto3` moduels
+
+Using the python `boto3` modules (`conda install boto3`) you can obtain SEVIR data by first connecting to the S3 bucket
+
+```python
+import boto3
+from botocore.handlers import disable_signing
+resource = boto3.resource('s3')
+resource.meta.client.meta.events.register('choose-signer.s3.*', disable_signing)
+bucket=resource.Bucket('sevir')
+```
+
+Then, get a list of files using
+
+```
+objs=bucket.objects.filter(Prefix='')
+print([o.key for o in objs])
+```
+
+Finally, download files of interest from this list, e.g.
+
+```pthon
+bucket.download_file('CATALOG.csv','/home/data/SEVIR/CATALOG.csv')
+bucket.download_file('data/vil/2017/SEVIR_VIL_STORMEVENTS_2017_0701_1231.h5','/home/data/SEVIR/data/vil/2017/SEVIR_VIL_STORMEVENTS_2017_0701_1231.h5')
+#... etc
+```
 
 
 

radar_nowcasting/README.md

@@ -0,0 +1,6 @@
+# Radar Nowcast Challenge
+
+The radar nowcast challenge is to generate future radar imagery given previous radar and satellite imagery as input.
+
+
+This challenge is still in development.  See [RadarNowcastChallenge notebook](RadarNowcastBenchmarks.ipynb) for a description of the datasets, problem, baseline model, and metrics.

synthetic_radar/README.md

@@ -0,0 +1,3 @@
+# Synthetic Weather Radar Challenge
+
+COMING SOON