I've made heavy use of AI coding for this. I've tested that it works but do be warned, jank may lie ahead.
This also does require a Linux environment as I save fits to a ram disk at /dev/shm. This prevents writing over 900 gb of fits to my ssd.
Additionally, this is not fast. Caltech may even block me at some point since I'm scraping so much data. I've yet to download everything. I'll post an update with the final file size if it finishes.
This scrapes the images of the plates available from Caltech and converts them to a normalized, 8 bit png file. I am using cropping about 1% of value off the top and bottom for more robust scaling.
There is a bit of dataloss doing all this but it makes the file SIGNIFICANTLY smaller. Worth it for what I plan to do.
This also does save portions of the fits header file but not everything. Just the stuff I needed. Add keys to line 77 if you need a specific header value saved.
The application is split into two main components:
src/main.rs: The main executable that orchestrates the downloading and processing of images.src/downloader.rs: A module that provides a reusable function to download files from a URL with progress indication.
- Prerequisites: Ensure you have Rust and Cargo installed.
- Build: Navigate to the project root and run
cargo build --release. - Execute: Run the compiled binary with
cargo run.
The application will create a poss_1 directory in the project root containing the processed images and header data.