Conversation
Add database schema, entities, and CRUD operations for the AI meeting recording and transcription integration: Database Migrations: - Add user_integrations table for encrypted API credentials - Add ai_privacy_level enum and meeting_url to coaching_relationships - Add meeting_recordings, transcriptions, transcript_segments tables - Add ai_suggested_items table for AI-detected actions/agreements Entity Definitions: - user_integrations (Google OAuth, Recall.ai, AssemblyAI credentials) - meeting_recordings (Recall.ai bot tracking) - transcriptions (AssemblyAI transcript data) - transcript_segments (speaker-diarized utterances) - ai_suggested_items (pending AI suggestions) - Enums: ai_privacy_level, meeting_recording_status, transcription_status, sentiment, ai_suggestion_type, ai_suggestion_status Entity API: - CRUD operations for user_integration, meeting_recording, transcription, and ai_suggested_item modules Other: - AES-256-GCM encryption utilities for API key storage - Config additions for external service credentials - Update coaching_relationships with meeting_url and ai_privacy_level Relates to: refactor-group/refactor-platform-fe#146
…se 3) Add gateway clients for Recall.ai, AssemblyAI, and Google OAuth with configurable base URLs. Create integration controller endpoints for API key verification and OAuth flow handling.
Add start/stop recording endpoints that integrate with Recall.ai bot for meeting capture. Add webhook handler for receiving recording status updates from Recall.ai.
- Add transcript_segment entity_api module with CRUD operations - Add AssemblyAI webhook handler for transcription callbacks - Create transcription controller with transcript/segments/summary endpoints - Update router with transcription routes - Export transcript_segment and transcription modules from domain
- Add ai_suggestion_controller with endpoints for suggestion management - Implement accept flow that creates Actions or Agreements from suggestions - Implement dismiss flow to mark suggestions as dismissed - Export ai_suggested_item module from domain - Add routes for GET /coaching_sessions/:id/ai-suggestions - Add routes for POST /ai-suggestions/:id/accept and /dismiss
- Add nested response structures for Recall.ai video URL extraction (recordings[0].media_shortcuts.video_mixed.data.download_url) - Add backend polling for Recall.ai bot status when frontend requests - Fix AssemblyAI webhook field name mismatch (id vs transcript_id) - Fetch full transcript from AssemblyAI API on webhook notification (webhooks are notifications only, don't include transcript data) - Add comprehensive debug logging for troubleshooting
- Add serde(rename_all = "snake_case") to AiPrivacyLevel enum - Add serde(default) to UpdateParams fields for optional deserialization - Add Default derive to UpdateParams struct
- Add action item extraction after transcription completes - Create AI suggestion records for each detected action item - Use keyword-based extraction (will, going to, need to, should, etc.) - Log extraction results for debugging
- Add LeMUR API client for extracting actions/agreements from transcripts - Implement dual AI privacy levels (coach + coachee must both consent) - Add manual extraction endpoints for actions and agreements - Fix Recall.ai JSON deserialization for null values - Fix serde serialization to use lowercase enum variants - Upgrade API error logs from WARN to ERROR level - Silence verbose SQLx/SeaORM query logs - Add auto_approve_ai_suggestions field to user integrations - Add stated_by, assigned_to, source_segment fields to AI suggestions
- Add POST /coaching_relationships/:id/create-google-meet endpoint - Automatically refresh expired Google OAuth tokens - Update relationship with new meeting URL after creation - Remove x-version header requirement from OAuth endpoints (browser redirects can't set headers)
- Add GOOGLE_OAUTH_SUCCESS_REDIRECT_URI config option - Update OAuth callback to use full frontend URL instead of relative path - Fixes redirect going to backend port instead of frontend
- Add WebhookConfig and WebhookEvents structs to CreateBotRequest - Configure bot to send bot.status_change, bot.done, and recording.done events - This enables receiving webhooks when recording completes instead of only polling
- Replace speaker labels (A, B) with actual coach/coachee names in transcript segments using word-count heuristic for speaker identification - Add speaker context to LeMUR summary prompt for accurate attribution - Extract due dates from natural language phrases in action items - Prepend assignee name as [Name] prefix when it matches coach/coachee - Run action extraction as background task (HTTP 202) to prevent timeouts - Add pre-calculated date lookup table for relative date accuracy - Strip markdown code blocks from LeMUR JSON responses
Log filtering with Trace-level bypass is now implemented in PR #216 with additional unit tests and improved architecture.
jhodapp
left a comment
There was a problem hiding this comment.
This looks really great Caleb, thanks for such a well thought out and simple plan. I could follow almost all of it pretty easily, which is a sign of a great architecture and design.
For testing this before you have a UI on top, I think you'll want to have a set of test sessions that have already been recorded and transcribed. You could create a few manual triggers that use my new crate to launch a meeting, add a bot, hold a meeting (we could record some of our meetings for example), and then you can have a way of specifying which recorded meeting to analyze after the fact. You may have already thought of this, but I'm just offering it as a suggestion in case you haven't gotten this far in your testing plan.
| ├── ZoomProvider | ||
| └── TeamsProvider | ||
|
|
||
| RecordingBotProvider (Meeting bots) |
There was a problem hiding this comment.
What do you think of calling this MeetingBotProvider instead which may allow for these bots to be responsible for other kinds of things?
| ├── OpenAiProvider | ||
| └── ClaudeProvider | ||
|
|
||
| WebhookHandler (Event processing) |
There was a problem hiding this comment.
Is your intention for this become a high-level universal webhook handler for the entire backend that could be used not only by this AI feature but also anything else?
There was a problem hiding this comment.
I didn't consider that yet. This, as of right now, is specific to the AI recording abstraction
There was a problem hiding this comment.
Alright. We can take the approach that I did to new vs existing auth token handling. We can design for a generalized solution here without going overboard, and if it makes sense to backport the existing JWT token handling to this, we can do that. Same for existing webhook handling with TipTap's API (if we do in fact have some, I can't remember for sure).
| ```rust | ||
| /// Configuration for OAuth authentication | ||
| #[derive(Debug, Clone)] | ||
| pub struct OAuthConfig { |
There was a problem hiding this comment.
Perhaps we should extract OAuth token handling into a separate crate that you can use here and I can use for my meeting link/creation manager crate that I'm working on. What do you think - worth designing to handle my use cases and yours and design this together?
Perhaps this becomes a generalized AuthenticationProvider trait that can have different implementations like GoogleOAuth, ZoomOAuth, Assembly, Recall, etc.
| /// A meeting space/room created on the platform | ||
| #[derive(Debug, Clone, Serialize, Deserialize)] | ||
| pub struct MeetingSpace { | ||
| pub id: String, | ||
| pub meeting_url: String, | ||
| pub meeting_code: Option<String>, | ||
| pub platform: String, | ||
| pub metadata: HashMap<String, String>, | ||
| } | ||
|
|
||
| /// Configuration for creating a meeting space | ||
| #[derive(Debug, Clone, Default)] | ||
| pub struct MeetingSpaceConfig { | ||
| pub title: Option<String>, | ||
| pub description: Option<String>, | ||
| pub start_time: Option<DateTime<Utc>>, | ||
| pub duration_minutes: Option<u32>, | ||
| pub is_public: bool, | ||
| } |
There was a problem hiding this comment.
Perhaps these can be types that live in my new meeting manager crate?
| #[async_trait] | ||
| pub trait MeetingPlatformProvider: Send + Sync { | ||
| /// Get the authorization URL for OAuth flow | ||
| fn get_authorization_url(&self, state: &str) -> SdkResult<String>; | ||
|
|
||
| /// Exchange authorization code for access tokens | ||
| async fn exchange_code(&self, code: &str) -> SdkResult<OAuthTokens>; | ||
|
|
||
| /// Refresh an expired access token | ||
| async fn refresh_token(&self, refresh_token: &str) -> SdkResult<OAuthTokens>; | ||
|
|
||
| /// Get user information using an access token | ||
| async fn get_user_info(&self, access_token: &str) -> SdkResult<PlatformUser>; | ||
|
|
||
| /// Verify if an access token is still valid | ||
| async fn verify_token(&self, access_token: &str) -> SdkResult<bool>; | ||
|
|
||
| /// Create a new meeting space | ||
| async fn create_meeting_space( | ||
| &self, | ||
| access_token: &str, | ||
| config: Option<MeetingSpaceConfig> | ||
| ) -> SdkResult<MeetingSpace>; | ||
|
|
||
| /// Get the platform identifier (e.g., "google_meet", "zoom") | ||
| fn platform_id(&self) -> &str; | ||
| } |
There was a problem hiding this comment.
I'm curious about the implementation plan here and why these are generally marked as async and looking like being readied to work across multiple threads?
| pub input_tokens: u32, | ||
| pub output_tokens: u32, |
There was a problem hiding this comment.
u32 should be more than enough at about 4.2 billion, but is it worth making the u64 for future-proofing just in case? Or am I overthinking this?
| pub extract_actions: bool, | ||
| pub extract_agreements: bool, | ||
| pub generate_summary: bool, | ||
| pub custom_prompt: Option<String>, |
There was a problem hiding this comment.
What is the intended use of this field?
| } | ||
|
|
||
| #[derive(Debug, Clone, Serialize, Deserialize)] | ||
| pub struct Participant { |
There was a problem hiding this comment.
Will this get correlated to a User model type?
| fn provider_id(&self) -> &str; | ||
|
|
||
| /// Verify API credentials are valid | ||
| async fn verify_credentials(&self) -> SdkResult<bool>; |
There was a problem hiding this comment.
Same comment as the other Provider above with this same function, does it belong here?
| } | ||
| ``` | ||
|
|
||
| ### 7. Workflow Orchestrator (Optional) |
There was a problem hiding this comment.
Is this meant to be like a state machine of the entire process? Why's it marked as optional?
Update trait naming in abstraction layer design document to better reflect that these bots serve multiple purposes beyond just recording (transcription, note-taking, etc.). Changes include: - Rename trait from RecordingBotProvider to MeetingBotProvider - Update MeetingWorkflow field from recording_provider to bot_provider - Update controller examples to use bot_provider naming - Update mock test examples to use BotProvider naming
Update crate naming in abstraction layer design document. The term "SDK" typically refers to vendor-specific libraries, whereas this is a trait-based abstraction layer. Changes include: - Rename crate from meeting-ai-sdk to meeting-ai - Update Cargo.toml package name - Update directory structure references - Update import statements in code examples - Change "SDK" to "abstraction layer" in open questions section
Add comprehensive inline documentation (3-5 lines each) to all traits, types, and methods in the meeting-ai-abstraction-layer implementation plan. Enhanced documentation covers: - SdkError variants with handling guidance and use cases - MeetingPlatformProvider OAuth flow and token management - MeetingBotProvider lifecycle, artifacts, and monitoring strategies - TranscriptionProvider features, status tracking, and cost considerations - AiAnalysisProvider capabilities, token usage, and confidence scoring - WebhookEvent types with security and idempotency requirements - WorkflowOrchestrator state machine and resumption patterns Each comment now includes: - Purpose and functionality - When and how to use - Important considerations and edge cases - Relationships to other system components This improves developer experience by making design decisions explicit and reducing ambiguity during implementation of the abstraction layer.
Replace "SDK" references with more accurate terminology: - Title: "Meeting AI SDK" → "Meeting AI Abstraction Layer" - Error type: Describe as "universal error type" abstracting providers - Code comments: Use "trait types" instead of "SDK types" This clarifies that we're building an abstraction layer with traits, not a standalone SDK, which better reflects the hybrid architecture approach (standalone crate + domain implementations).
Remove all SDK-specific type aliases in favor of standard Rust types: - SdkError → Error - SdkResult<T> → Result<T, Error> - Remove SdkResult type alias entirely This simplifies the API by using idiomatic Rust error handling patterns and removes unnecessary abstraction layers. All trait methods now return standard Result types, making the interface more familiar to Rust developers. Changes applied to: - Core error type definition - All trait method signatures (5 traits, 25+ methods) - Implementation examples (RecallAiProvider, SkribbyProvider) - Test mock examples - Documentation references Updated 32 locations across the implementation plan.
…ion system Replace hard-coded Action/Agreement types with flexible ExtractedResource trait that supports any application-defined resource types. This makes the crate completely domain-agnostic while maintaining type safety. Key changes: - Add ExtractedResource trait for application-defined types - Replace ExtractedAction/ExtractedAgreement with generic approach - Update AnalysisResult to use type-erased HashMap for flexibility - Add get_resources<T>() for type-safe resource extraction - Replace extract_actions/extract_agreements flags with generic resource_types - Add comprehensive application integration example showing SeaORM usage - Add Serialization/Deserialization error variants This design supports any domain: coaching sessions, medical consultations, sales calls, project meetings, etc. Applications define their own resource types by implementing ExtractedResource trait.
Defines architecture for two new crates: - meeting-auth: Single source of truth for all authentication (OAuth, API keys) - meeting-platform: Thin layer for platform-specific meeting operations Key features: - Custom RetryAfterPolicy for proper 429 rate limit handling - Per-user refresh locks for token management - Atomic token updates for rotating refresh tokens - Production schema design (oauth_connections, api_credentials tables) Zoom integration deferred to future phase; Google Meet is the initial target.
Key architectural changes: - OAuth provider implementations (GoogleOAuthProvider, etc.) now live in meeting-auth, not meeting-manager - MeetingClient trait does NOT extend OAuthProvider (clean separation) - meeting-manager receives tokens from meeting-auth, handles only meeting ops - Renamed meeting-platform to meeting-manager (better reflects purpose) Updated all diagrams, sequence flows, and implementation phases to reflect the cleaner separation of concerns.
calebbourg
left a comment
There was a problem hiding this comment.
@jhodapp I took a first pass at this. Overall I think it looks really great, very nice job!!
The main changes I would request are around naming and organization of the code. Apart from that, I don't have any major thoughts at this point. I think it's looking really good and will provide an excellent base to start with and we can make tweaks as we execute the actual implementation.
I have not had a chance to really delve into the database related things. I'd like to do that with a fresh pair of eyes.
| ### Location | ||
|
|
||
| ``` | ||
| meeting-auth/ |
There was a problem hiding this comment.
For any construct that has Token in it's name (TokenManager, TokenStorage, etc.), can we place those in a /token directory and remove the Token from the naming? Ex. Manager. We can then move tokens.rs code into /token/mod.rs (I'll defer to you on whether to use the singular or plural of tokens. It should probably be consistent though whatever we choose)
When we import these constructs we can then import them like this
use token::storage::Storage as TokenStorageor
use tokens::storage::Storage as TokensStorageHaving Token be part of the actual definition name, I think, is redundant.
There was a problem hiding this comment.
Excellent suggestion, I prefer the explicit approach as well instead of classifying by entity name prefix.
| fn auth_method(&self) -> AuthMethod; | ||
| fn authenticate(&self, request: RequestBuilder) -> RequestBuilder; | ||
| async fn verify_credentials(&self) -> Result<bool, AuthError>; | ||
| fn provider_id(&self) -> &str; |
There was a problem hiding this comment.
I think we do want to be able to tell what provider type is being used (e.g. for contextual error messages, etc), and I refined this to a more strongly-typed approach in commit 3f75998
| } | ||
|
|
||
| /// Result of a token refresh operation | ||
| pub struct TokenRefreshResult { |
There was a problem hiding this comment.
We might be able to remove Token from this name as well and infer it from it's module placement depending on how everything else is structured
| /// Trait for OAuth providers - implemented by platform-specific providers | ||
| #[async_trait] | ||
| pub trait OAuthProvider: Send + Sync { | ||
| fn provider_id(&self) -> &str; |
| /// Trait for storing provider credentials (API keys) | ||
| #[async_trait] | ||
| pub trait CredentialStorage: Send + Sync { | ||
| async fn store(&self, user_id: &str, provider_id: &str, credentials: CredentialData) -> Result<(), StorageError>; |
There was a problem hiding this comment.
Would it be correct to assume that we would back this via a seaORM entity and store these in postgres?
There was a problem hiding this comment.
Yes, exactly. The domain crate will implement CredentialStorage via ApiCredentialStorage, backed by a SeaORM entity mapped to the api_credentials PostgreSQL table with encryption at rest. I've added a clarifying note to the trait definition pointing to the implementation details in commit 1ad4d21.
| use meeting_auth::TokenManager; | ||
|
|
||
| /// Meeting space returned after creation | ||
| pub struct MeetingSpace { |
There was a problem hiding this comment.
Similar to above, I think this should be just called Space and be imported using as
use meeting_manager::meeting::Space as MeetingSpace| ### Main Client Interface | ||
|
|
||
| ```rust | ||
| use meeting_auth::{TokenManager, TokenStorage, OAuthProvider}; |
There was a problem hiding this comment.
This can be done as
use meeting_auth::{ Token as TokenManager, Storage as TokenStorage, OAuthProvider};There was a problem hiding this comment.
Addressed from previous commits.
Remove concepts already handled by meeting-auth and meeting-manager crates to eliminate duplication between implementation plans. Changes: - Remove MeetingPlatformProvider trait (replaced by OAuthProvider and MeetingClient in meeting-auth/meeting-manager) - Remove OAuth types: OAuthConfig, OAuthTokens, PlatformUser - Remove MeetingSpace/MeetingSpaceConfig types (now in meeting-manager) - Rename MeetingBotProvider to RecordingBotProvider for clarity - Remove verify_webhook() from WebhookHandler trait (use meeting-auth's WebhookValidator instead) - Update MeetingWorkflow to remove meeting creation responsibilities - Remove MeetingCreated state from WorkflowState enum - Update directory structure to remove meeting_platform.rs - Add dependencies on meeting-auth and meeting-manager crates - Update implementation examples to use meeting-auth utilities (ApiKeyAuth, AuthenticatedClientBuilder, RetryAfterPolicy) - Add scope notes clarifying this plan focuses on AI operations only - Add cross-reference to meeting-and-auth-abstraction-layers.md The meeting-ai crate now focuses exclusively on: - RecordingBotProvider (bot deployment and recording) - TranscriptionProvider (speech-to-text) - AiAnalysisProvider (LLM-powered analysis) - WebhookHandler (event processing) - ExtractedResource system (domain-agnostic resource extraction)
…ase patterns Per Caleb's suggestion, reorganize token types into oauth/token/ module: - token::Storage, token::Manager, token::Tokens (no redundant Token prefix) Align error handling with existing codebase patterns: - Crate-level errors (ManagerError, StorageError) stay simple - Domain layer provides From impls to convert to domain::Error - Web layer handles HTTP status code mapping automatically This follows the established flow: crate error → domain::Error → web::Error → HTTP response
3f75998 to
ac29a01
Compare
Replace stringly-typed provider_id() method with type-safe provider() method returning OAuthProviderKind enum, matching the pattern already used for ApiKeyProvider in the ProviderAuth trait.
Add implementation notes to Storage and CredentialStorage traits explaining they will be implemented in the domain crate using SeaORM entities mapped to oauth_connections and api_credentials tables, with encryption at rest via domain::encryption.
- Rename MeetingSpace/MeetingConfig to Space/Config (DRY principle: module path provides context) - Add import examples showing aliasing for external code clarity - Move hardcoded API URLs to module-level constants - Use const_format::concatcp! for compile-time URL construction
- Define contextual error types for both meeting-auth and meeting-manager - Add new ExternalErrorKind variants: Authentication, RateLimited - Show complete From implementations mapping crate errors to domain::Error - Add error flow summary tables showing HTTP status mappings - Follow existing pattern: crate error → domain::Error → web::Error → HTTP Error types follow codebase naming conventions and map to appropriate HTTP status codes via the existing web layer error handling.
Thanks for the great review @calebbourg. I addressed all of your feedback, and I also updated and made the new error types and handling part of the plan more explicit in commit 8a5382a. |
jhodapp
left a comment
There was a problem hiding this comment.
A couple of other simple thoughts about naming.
| ### Alternative Providers | ||
|
|
||
| **Meeting Bots:** | ||
| - Recall.ai ($1,000/month + $1/hour) |
There was a problem hiding this comment.
This doesn't seem right.
| ├── DeepgramProvider | ||
| └── WhisperProvider | ||
|
|
||
| AiAnalysisProvider (Action extraction, summaries) |
There was a problem hiding this comment.
What do you think of dropping "Ai" from the name? Would we ever have a non-AI analysis provider?
Summary
Complete AI-powered meeting recording integration that enables automatic transcription and action item extraction from coaching sessions.
Key Features
Technical Changes
meeting_recordings,transcription_segments,ai_suggested_items,user_integrationsCreateAiSuggestionstruct to comply with clippy's parameter limitsTest Plan