create README for future readers

Author: nepeat

README.md

@@ -0,0 +1,55 @@
+# placedump
+
+Toolset to make a best effort attempt to grab all r/place (2022) WebSocket messages, pixels, and canvases.
+
+## Layout
+
+Documenting layout which powered archival and parsing for the last 1.5 days of r/place.
+
+### `dump.py`
+* Connects to Reddit's live servers and subscribes to all r/place canvas changes and config changes
+* Saves all messages to a Redis stream for later parsing
+* Pushes all messagesto a Redit PubSub channel
+
+### `pixel_watcher.py`
+* Connects to Reddit's live servers to query pixel statuses in bulk to save on HTTP requests and data.
+* Fetches pixels in bulk from a Redis set `queue:pixels`
+* Spawns Celery task `pixels.update_pixel` for every pixel result from Reddit
+
+### Celery
+* Main job queue for message processing
+* Redis used for job result storage and queuing
+
+### Celery: Tasks
+* `parse.parse_message` 
+    - Parse GraphQL responses from the live WS
+    - Canvas responses updated the current canvas ID
+    - All pixel updates spawns a `pixels.download_url` task
+* `pixels.download_url`
+    - Downloads URL passed into function
+    - Uploads URL and result straight into Backblaze B2
+    - Spawns `pixels.get_non_transparent` task
+    - Adds URL into database
+* `pixels.update_pixel`
+    - Postgres upsert insert for the Pixel table
+    - Very inefficient but it did the job!
+    - 192 processes were needed to handle the workload for this task
+    - Certainly could have been improved with batching
+* `pixels.get_pixel`
+    - Gets a single pixel from Reddit and calls `pixels.update_pixel`
+    - Very inefficient
+    - Guaranteed rate limit!
+* `pixels.get_non_transparent`
+    - Thanks Stack Overflow. https://stackoverflow.com/questions/60051941/find-the-coordinates-in-an-image-where-a-specified-colour-is-detected
+    - Parses a PNG passed as bytes over Celery and adds all pixels to Redis set `queue:pixels`
+    - Returns all pixels as a list.
+
+## dev runbook
+```
+# forever loop alias
+run_forever() { while :; do "$@"; sleep 1; done }
+
+# deploy dump stack
+export DOCKER_HOST=unix:///tmp/docker.sock
+docker stack deploy -c docker-compose.yml --with-registry-auth placedump
+```