TODO: Local tests for streaming jobs #6
Description
I have already incorporated some of the items from https://github.com/Gamesight/aws-glue-local-scala to ensure that we can locally compile this, and verify compilation before updated the s3 resources in the action, but there are some additional handling pieces that we need to cover. I want to be able to run this locally.
- Research mocking Kinesis streams. I think there is some solid Java tooling already out there for this, but I need to look it up.
- Set test files
- I could potentially set up a test bucket for output and read the existing stream. This is a reasonable option if mocking proves to be difficult. The project already assumes that the IoT device is sending the MQTT packet
- Figure out how to test without starting a never-ending job. The biggest thing to test is whether or not it correctly reads 100 seconds worth of the stream and joins properly. We don't want this to continually wait for more data. I may need to add some additional boilerplate to the job that short-circuits after a single "batch"