10.21.2025

[atambay37]

• Prioritization of Collaborator Requests: Francis and Carlos discussed prioritizing feature development for Natural Resources Canada, the main financial contributor, to support their upcoming presentation and ongoing collaboration.
  • Collaborator Funding Influence: Francis explained that Natural Resources Canada is the only collaborator providing financial support, so their feature requests are prioritized to ensure continued and potentially expanded collaboration.
  • Use Case Alignment: Francis clarified that while the main use case for the antenna platform is not invasive species monitoring, the technical overlap allows the platform to be adapted for this purpose without significant changes.
• Field Data Collection and Upload Challenges: Francis described technical and logistical difficulties faced by collaborators in uploading field data, including unreliable internet connections and the need for manual SD card handling.
  • Trap Hardware Variants: Francis outlined different camera trap models used by collaborators, ranging from large, long-term field systems to smaller, more portable units, all requiring manual SD card retrieval and data upload.
  • Data Upload Pain Points: Francis noted that many users struggle with uploading data to S3 buckets due to technical constraints, such as unstable internet connections in remote locations, leading to incomplete data transfers.
  • Alternative Data Transfer Methods: Francis mentioned that, in some cases, collaborators have resorted to physically mailing hard drives with collected data when online uploads fail.
• Cloud Infrastructure Cost Estimation and Configuration: Carlos and Michael discussed requirements for estimating cloud infrastructure costs, including region selection, user activity metrics, and database sizing, to inform deployment planning.
  • Cloud Region Selection: Carlos asked whether to use the Canada or US East region for deployment, and Michael and Francis explained that Canadian researchers often require data residency in Canada, but US regions offer more resources; the group agreed to align compute resources with data location for efficiency.
  • User Activity Metrics: Carlos requested daily active and concurrent user numbers, which Michael provided from Google Analytics, reporting approximately 14 unique users per day and an estimated 3-5 concurrent users.
  • Database Sizing: Carlos inquired about the current database size to configure RDS appropriately, and Michael and Francis confirmed that the latest database snapshot is around 10 GB, exceeding the smallest RDS instance capacity.
• Task Queue and Celery Deployment Decisions: Carlos and Michael evaluated options for running Celery task queues, considering whether to containerize Celery or migrate to Amazon-managed queues, and agreed to initially keep Celery with Redis for simplicity.
  • Celery vs. Amazon Queues: Carlos asked if Celery should be replaced with Amazon-managed queues for cost and maintenance benefits, but the group decided to retain Celery in containers for the initial deployment to minimize code changes.
  • Redis as Backend: Michael explained that Celery currently uses Redis for both task queue management and Django caching, and discussed the pros and cons of separating these functions for improved troubleshooting and reliability.
  • Migration Path: Carlos and Michael agreed to first validate the system with the current setup and consider migrating to Amazon SQS or separating Redis instances in the future for optimization.
• Database Inspection and Debugging Tools: Carlos asked Michael for recommendations on tools to inspect and modify the database, and Michael described using Django admin, debug toolbar, DBeaver, and the Django shell for various inspection and debugging tasks.
  • Django Admin and Debug Toolbar: Michael recommended using Django admin for quick CRUD operations and the debug toolbar for inspecting queries and debugging endpoints during development.
  • DBeaver for Database Access: Michael suggested DBeaver as a versatile database client for direct inspection of Postgres tables, noting its support for multiple relational databases and ease of use after initial setup.
  • Django Shell and ORM: Michael highlighted the Django shell as a preferred tool for interacting with the database using ORM syntax, allowing for efficient querying and object manipulation without direct SQL.
• Processing Service and Job Queue Architecture Changes: Carlos and Michael discussed architectural changes to support both push-based and queue-based processing services, including making the processing service endpoint optional and updating pipeline registration and status mechanisms.
  • Distinguishing Job Types: Carlos raised the need to distinguish between jobs that use a processing service endpoint and those that are queued, proposing to make the processing service endpoint field nullable to indicate queue-based jobs.
  • Pipeline Registration Updates: Michael and Carlos agreed that for queue-based jobs, pipelines should be registered via a new API endpoint rather than by querying the processing service, requiring changes to the registration workflow.
  • Online/Offline Status Handling: The group discussed propagating processing service online/offline status to pipelines, using last communication timestamps to determine availability and inform users in the UI.
  • Support for Interactive and Batch Processing: Carlos and Michael considered how to support both interactive (push) and batch (queue) processing for the same pipeline, potentially requiring separate endpoints or pipeline definitions.
• Pull Request Coordination and Feature Integration: Carlos, Michael, and Francis coordinated on merging ongoing feature branches, discussed the state of draft pull requests, and reviewed Vanessa's recent changes to the pipeline processing logic.
  • Draft Pull Requests: Carlos reported that their work on the consumer and Pytorch features is in draft PRs, aiming to merge incrementally to avoid large, unmanageable changes.
  • Vanessa's Pipeline Changes: Michael described Vanessa's refactoring of the pipeline loop to queue tasks per image using Celery, improving reliability and enabling easier transition between push and queue-based processing.
  • Merge Conflict Avoidance: Carlos agreed to review Vanessa's changes to ensure compatibility and avoid merge conflicts with their own ongoing work.

Follow-up tasks:

• Meeting Scheduling: Request to move the recurring meeting to 15 minutes after the hour to avoid overlap with previous meetings. (Carlos)
• Cost Estimate Evaluation: Determine which data set and region to use for benchmarking and cost evaluation, specifically considering proximity to the Victoria data center. (Carlos)
• User Analytics Access: Provide Google Analytics data to Carlos to determine daily active and concurrent users for cost estimation. (Michael)
• Database Size Confirmation: Confirm the latest database snapshot size for configuring RDS, ensuring it aligns with the 10 GB figure mentioned. (Michael)
• Celery Queue Transition: Test the current setup by keeping Celery and Redis as managed services, and plan to validate before considering a transition to Amazon SQS for task queuing. (Carlos, Michael)
• Job Management System Update: Make the processing service endpoint URL nullable and implement an API endpoint for V2 processing services to register pipelines and update status. (Carlos)
• Authentication for External Processing Services: Implement a mechanism to provide tokens for external processing services to register and authenticate with the system. (Carlos)
• PR Review and Merge Coordination: Review and coordinate merging of draft PRs for consumer and Pytorch updates, ensuring minimal disruption and timely integration. (Carlos, Michael)
• Collision Check for PRs: Check for potential code collisions between Carlos's and Vanessa's PRs before merging to avoid integration issues. (Carlos)