refactor: Decouple Streamlit Frontend from Backend with FastAPI Microservice & Enhance Newsletter System

[doepking]

This issue proposes a major architectural refactor to decouple the application into a distinct Streamlit frontend and a FastAPI backend microservice. This will improve scalability, separation of concerns, and allow for more advanced features, starting with a robust newsletter subscription and tracking system.

Part 1: Architectural Refactor

• Create a new FastAPI application to serve as the backend microservice.

• Migrate all business logic from the Streamlit app (utils.py, newsletter.py, etc.) to the FastAPI service. This includes:

  • Gemini API interactions and function calling.
  • All database operations (CRUD functions).

• The Streamlit application will be refactored to act as a pure client, making API calls to the FastAPI backend for all data and logic.

Part 2: Newsletter Subscription & Management

• The FastAPI service will expose the following endpoints for newsletter management:

  • POST /newsletter/subscribe: Creates or updates a user's subscription.
  • POST /newsletter/unsubscribe: Marks a user as unsubscribed.
  • GET /newsletter/preferences/{user_email}: Retrieves a user's current subscription status.

• The Streamlit frontend will be updated with a new UI section for users to manage their newsletter subscription.

Part 3: Newsletter Tracking & Automation

• Implement a 1x1 tracking pixel endpoint in the FastAPI service:
  • GET /track/open/{log_id}: This endpoint will be embedded in newsletter emails to track open rates by updating the opened_at field in the SentNewsletterLog.
• Create a new endpoint for automated sending:
  • POST /newsletter/send-daily: This endpoint will fetch all subscribed users and trigger the newsletter generation and sending process for each.
• This endpoint is designed to be called by an external scheduler (e.g., Google Cloud Scheduler) to automate daily newsletters.