Data Loader Improvements

[jamesward]

• Fake Data Loader: Generates a dataset that provides us obvious predictions
• Sample Data Loader: Selects a given number of past months and picks a given number of records from that dataset

Improvements to the current dataloader:

• Parametrize whether to use fake or sampled data
• For sampled data, parametrize past months, and number of records

[adamwhat]

Some other considerations:

1. Raw data is stored in S3. The dataset is 25GB per year across multiple years. In the sample data loader, how do we plan to initialize/download the data into local? I can also look into streaming solutions to sample data.

2. For both fake data and sample data loader, should we save the result in disk/S3?

3. For the fake data, we can specify the following parameters:

4. time range

5. number of clusters

6. demand distribution per clusters (or we can assume it's uniform)

7. We also want to incorporate weather as a feature. I will look into data sources for weather.

[adamwhat]

Another thing we want to explore is unit testing. We are running into some problems with testing in the PIO engine. I think this part is a good start to incorporate the unit test framework and act as an example for other components.

@jamesward do you have any recommendations for the unit testing framework?

[jamesward]

1. The demo data loader currently caches to the local filesystem: https://github.com/jamesward/koober/blob/master/demo-data/src/main/scala/DemoData.scala#L26-L45
   But we will probably want to make sure we have enough space and if not maybe resort to streaming. This should be pretty straightforward with Akka Streams.
2. It would be nice to not have to resample the data. Probably easiest right now to store the sampled data locally. Then down the road we can either automate the storage to S3 or just do it manually.
3. Sounds good.

For unit testing I think we can use the demo data loader as a library in the pio project. That'll be one nice thing about having this in Scala.

[jamesward]

It is unfortunate that these aren't zipped: https://github.com/toddwschneider/nyc-taxi-data/blob/master/raw_data_urls.txt

I wonder if we should zip them and put them back on S3. Or maybe work with the provider to do that. Seems like that'd save some serious money.