feat: add distributed embedding table

[oliverb123]

We want to use this for more high frequency data.

[greptile-apps]

Additional Comments (1)

1. ee/hogai/llm_traces_summaries/tools/embed_summaries.py, line 82 (link)

   logic: After the migration, new embeddings will be written to sharded_posthog_document_embeddings, but this query still references the old posthog_document_embeddings table. This will cause new embeddings to not be found.

_{6 files reviewed, 1 comment}

_{Edit Code Review Agent Settings | Greptile}

[Copilot]

Pull request overview

This PR migrates the document embeddings table from a single replicated architecture to a distributed sharded architecture to support higher frequency data ingestion.

• Introduces sharded table architecture with separate read and write distributed tables
• Updates all references to use the new distributed table for reads
• Provides migration to transition from old to new table structure while preserving existing data

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
products/error_tracking/backend/embedding.py	Adds sharded and distributed table definitions with cityHash64 sharding key; updates table SQL generation functions to create sharded architecture
posthog/hogql_queries/test/test_document_embeddings_query_runner.py	Updates test queries to use distributed_posthog_document_embeddings table
posthog/hogql/database/schema/document_embeddings.py	Updates schema references to point to distributed table for HogQL queries
posthog/clickhouse/schema.py	Adds imports for new distributed and writable table SQL functions
posthog/clickhouse/migrations/0182_shard_document_embeddings.py	Migration script that drops old MV/Kafka/writable tables and creates new sharded infrastructure
posthog/clickhouse/migrations/max_migration.txt	Updates migration tracker to 0182
posthog/clickhouse/test/snapshots/test_schema.ambr	Auto-generated snapshot updates reflecting new table schemas

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.