9.2.25

[atambay37]

From @mihow (thank you!)

We had a successful in-person meeting in Seattle on Tuesday! Here are my notes on what we covered.

• Reviewed, deployed and tested the job handler fix - which seems to fix a critical issue with processing medium-sized jobs! Jobs are already processing much faster. Failing after about ~2000 images instead of ~200.
• Carlos showed me a new debugger technique in VSCode that works in our docker compose stack. I would like to share this demo with the team. Should absolutely speed up backend development.
• Discussed how researchers tend to work in R Studio (most common) & Jupyter Notebook for developing models and doing analysis. We ideated how to make the Antenna fit into that workflow. What’s the easiest way for them to process data from Antenna and post-back results to Antenna from their existing workspace?
• Reviewed current approach for distributed data processing in Antenna
• Discussed new “pull” approach with intermediate API. And how to simply the processing services while also allowing them to be reliable and easy for researchers to implement
• Reviewed the example processing service with the zero shot detector, for use as a general template, and making development more fun.
• Deeper dive into the existing production processing service (the ADC).
• Deeper dive into the Antenna API, discussed where to put new endpoints for the pull queue approach.
• Discussed some fast ways to prototype the pull approach using a worker/agent in the ADC and a simple endpoint in Antenna for retrieving image URLs.
• Looked at TeamCity

Some todos that came out of the day:

• Carlos will break up the pull queue API approach into smaller tasks and decide what to put into our repo vs. their own repo.
• Michael will work on reviving the shared staging environment