-
Notifications
You must be signed in to change notification settings - Fork 1.9k
refactor: Remove deprecated queue events bus functionality #9815
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
2 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -24,3 +24,4 @@ rust/cubesql/profile.json | |
| .cubestore | ||
| .env | ||
| .vimspector.json | ||
| .claude/settings.local.json | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,137 @@ | ||
| # CLAUDE.md | ||
|
|
||
| This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. | ||
|
|
||
| ## Repository Overview | ||
|
|
||
| Cube is a semantic layer for building data applications. This is a monorepo containing the complete Cube ecosystem including: | ||
| - Cube backend server and core components | ||
| - Client libraries for JavaScript/React/Vue/Angular | ||
| - Database drivers for various data sources | ||
| - Documentation site | ||
| - Rust components (CubeSQL, CubeStore) | ||
|
|
||
| ## Development Commands | ||
|
|
||
| **Note: This project uses Yarn as the package manager.** | ||
|
|
||
| ### Core Build Commands | ||
| ```bash | ||
| # Build all packages | ||
| yarn build | ||
|
|
||
| # Run TypeScript compilation across all packages | ||
| yarn tsc | ||
|
|
||
| # Watch mode for TypeScript compilation | ||
| yarn tsc:watch | ||
|
|
||
| # Clean build artifacts | ||
| yarn clean | ||
|
|
||
| # Run linting across all packages | ||
| yarn lint | ||
|
|
||
| # Fix linting issues | ||
| yarn lint:fix | ||
|
|
||
| # Lint package.json files | ||
| yarn lint:npm | ||
| ``` | ||
|
|
||
| ### Testing Commands | ||
| ```bash | ||
| # Run tests (most packages have individual test commands) | ||
| yarn test | ||
|
|
||
| # Test individual packages | ||
| cd packages/cubejs-[package-name] | ||
| yarn test | ||
| ``` | ||
|
|
||
| ### Documentation Development | ||
| The documentation is in `/docs` directory: | ||
| ```bash | ||
| cd docs | ||
| yarn dev # Start development server | ||
| yarn build # Build for production | ||
| ``` | ||
|
|
||
| ## Architecture Overview | ||
|
|
||
| ### Monorepo Structure | ||
| - **`/packages`**: All JavaScript/TypeScript packages managed by Lerna | ||
| - Core packages: `cubejs-server-core`, `cubejs-schema-compiler`, `cubejs-query-orchestrator` | ||
| - Client libraries: `cubejs-client-core`, `cubejs-client-react`, etc. | ||
| - Database drivers: `cubejs-postgres-driver`, `cubejs-bigquery-driver`, etc. | ||
| - API layer: `cubejs-api-gateway` | ||
| - **`/rust`**: Rust components including CubeSQL (SQL interface) and CubeStore (distributed storage) | ||
| - **`/docs`**: Next.js documentation site | ||
| - **`/examples`**: Example implementations and recipes | ||
|
|
||
| ### Key Components | ||
| 1. **Schema Compiler**: Compiles data models into executable queries | ||
| 2. **Query Orchestrator**: Manages query execution, caching, and pre-aggregations | ||
| 3. **API Gateway**: Provides REST, GraphQL, and SQL APIs | ||
| 4. **CubeSQL**: Postgres-compatible SQL interface (Rust) | ||
| 5. **CubeStore**: Distributed OLAP storage engine (Rust) | ||
|
|
||
| ### Package Management | ||
| - Uses Yarn workspaces with Lerna for package management | ||
| - TypeScript compilation is coordinated across packages | ||
| - Jest for unit testing with package-specific configurations | ||
|
|
||
| ## Testing Approach | ||
|
|
||
| ### Unit Tests | ||
| - Most packages have Jest-based unit tests in `/test` directories | ||
| - TypeScript packages use `jest.config.js` with TypeScript compilation | ||
| - Snapshot testing for SQL compilation and query planning | ||
|
|
||
| ### Integration Tests | ||
| - Driver-specific integration tests in `/packages/cubejs-testing-drivers` | ||
| - End-to-end tests in `/packages/cubejs-testing` | ||
| - Docker-based testing environments for database drivers | ||
|
|
||
| ### Test Commands | ||
| ```bash | ||
| # Individual package testing | ||
| cd packages/[package-name] | ||
| yarn test | ||
|
|
||
| # Driver integration tests (requires Docker) | ||
| cd packages/cubejs-testing-drivers | ||
| yarn test | ||
| ``` | ||
|
|
||
| ## Development Workflow | ||
|
|
||
| 1. **Making Changes**: Work in individual packages, changes are coordinated via Lerna | ||
| 2. **Building**: Use `yarn tsc` to compile TypeScript across all packages | ||
| 3. **Testing**: Run relevant tests for modified packages | ||
| 4. **Linting**: Ensure code passes `yarn lint` before committing | ||
|
|
||
| ## Common File Patterns | ||
|
|
||
| - `*.test.ts/js`: Jest unit tests | ||
| - `jest.config.js`: Jest configuration per package | ||
| - `tsconfig.json`: TypeScript configuration (inherits from root) | ||
| - `CHANGELOG.md`: Per-package changelogs maintained by Lerna | ||
| - `src/`: Source code directory | ||
| - `dist/`: Compiled output (not committed) | ||
|
|
||
| ## Important Notes | ||
|
|
||
| - This is documentation for the old Cube docs site structure (the existing `/docs/CLAUDE.md` refers to the documentation site) | ||
| - The main Cube application development happens in `/packages` | ||
| - For data model changes, focus on `cubejs-schema-compiler` package | ||
| - For query execution changes, focus on `cubejs-query-orchestrator` package | ||
| - Database connectivity is handled by individual driver packages | ||
|
|
||
| ## Key Dependencies | ||
|
|
||
| - **Lerna**: Monorepo management and publishing | ||
| - **TypeScript**: Primary language for most packages | ||
| - **Jest**: Testing framework | ||
| - **Rollup**: Bundling for client libraries | ||
| - **Docker**: Testing environments for database drivers |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,131 @@ | ||
| # CLAUDE.md | ||
|
|
||
| This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. | ||
|
|
||
| ## Package Overview | ||
|
|
||
| The Query Orchestrator is a multi-stage querying engine that manages query execution, caching, and pre-aggregations in Cube. It receives pre-aggregation SQL queries and executes them in exact order, ensuring up-to-date data structure and freshness. | ||
|
|
||
| ## Development Commands | ||
|
|
||
| **Note: This project uses Yarn as the package manager.** | ||
|
|
||
| ```bash | ||
| # Build the package | ||
| yarn build | ||
|
|
||
| # Build with TypeScript compilation | ||
| yarn tsc | ||
|
|
||
| # Watch mode for development | ||
| yarn watch | ||
|
|
||
| # Run all tests (unit + integration) | ||
| yarn test | ||
|
|
||
| # Run only unit tests | ||
| yarn unit | ||
|
|
||
| # Run only integration tests | ||
| yarn integration | ||
|
|
||
| # Run CubeStore integration tests specifically | ||
| yarn integration:cubestore | ||
|
|
||
| # Run linting | ||
| yarn lint | ||
|
|
||
| # Fix linting issues | ||
| yarn lint:fix | ||
| ``` | ||
|
|
||
| ## Architecture Overview | ||
|
|
||
| ### Core Components | ||
|
|
||
| The Query Orchestrator consists of several interconnected components: | ||
|
|
||
| 1. **QueryOrchestrator** (`src/orchestrator/QueryOrchestrator.ts`): Main orchestration class that coordinates query execution and manages drivers | ||
| 2. **QueryCache** (`src/orchestrator/QueryCache.ts`): Handles query result caching with configurable cache drivers | ||
| 3. **QueryQueue** (`src/orchestrator/QueryQueue.js`): Manages query queuing and background processing | ||
| 4. **PreAggregations** (`src/orchestrator/PreAggregations.ts`): Manages pre-aggregation building and loading | ||
| 5. **DriverFactory** (`src/orchestrator/DriverFactory.ts`): Creates and manages database driver instances | ||
|
|
||
| ### Cache and Queue Driver Architecture | ||
|
|
||
| The orchestrator supports multiple backend drivers: | ||
| - **Memory**: In-memory caching and queuing (development) | ||
| - **CubeStore**: Distributed storage engine (production) | ||
| - **Redis**: External Redis-based caching (legacy, being phased out) | ||
|
|
||
| Driver selection logic in `QueryOrchestrator.ts:detectQueueAndCacheDriver()`: | ||
| - Explicit configuration via `cacheAndQueueDriver` option | ||
| - Environment variables (`CUBEJS_CACHE_AND_QUEUE_DRIVER`) | ||
| - Auto-detection: Redis if `CUBEJS_REDIS_URL` exists, CubeStore for production, Memory for development | ||
|
|
||
| ### Query Processing Flow | ||
|
|
||
| 1. **Query Submission**: Queries enter through QueryOrchestrator | ||
| 2. **Cache Check**: QueryCache checks for existing results | ||
| 3. **Queue Management**: QueryQueue handles background execution | ||
| 4. **Pre-aggregation Processing**: PreAggregations component manages rollup tables | ||
| 5. **Result Caching**: Results stored via cache driver for future requests | ||
|
|
||
| ### Pre-aggregation System | ||
|
|
||
| The pre-aggregation system includes: | ||
| - **PreAggregationLoader**: Loads pre-aggregation definitions | ||
| - **PreAggregationPartitionRangeLoader**: Handles partition range loading | ||
| - **PreAggregationLoadCache**: Manages loading cache for pre-aggregations | ||
|
|
||
| ## Testing Structure | ||
|
|
||
| ### Unit Tests (`test/unit/`) | ||
| - `QueryCache.test.ts`: Query caching functionality | ||
| - `QueryQueue.test.ts`: Queue management and processing | ||
| - `QueryOrchestrator.test.js`: Main orchestrator logic | ||
| - `PreAggregations.test.js`: Pre-aggregation management | ||
|
|
||
| ### Integration Tests (`test/integration/`) | ||
| - `cubestore/`: CubeStore-specific integration tests | ||
| - Tests real database interactions and queue processing | ||
|
|
||
| ### Test Abstractions | ||
| - `QueryCache.abstract.ts`: Shared test suite for cache implementations | ||
| - `QueryQueue.abstract.ts`: Shared test suite for queue implementations | ||
|
|
||
| ## Key Design Patterns | ||
|
|
||
| ### Queue Processing Architecture | ||
| The DEVELOPMENT.md file contains detailed sequence diagrams showing: | ||
| - Queue interaction with CubeStore via specific queue commands (`QUEUE ADD`, `QUEUE RETRIEVE`, etc.) | ||
| - Background query processing with heartbeat management | ||
| - Result handling and cleanup | ||
|
|
||
| ### Driver Factory Pattern | ||
| - `DriverFactory` type enables pluggable database drivers | ||
| - `DriverFactoryByDataSource` supports multi-tenant scenarios | ||
| - Separation between external (user data) and internal (cache/queue) drivers | ||
|
|
||
| ### Error Handling | ||
| - `ContinueWaitError`: Signals when queries should continue waiting | ||
| - `TimeoutError`: Handles query timeout scenarios | ||
| - Proper cleanup and resource management across all components | ||
|
|
||
| ## Configuration | ||
|
|
||
| Key configuration options in `QueryOrchestratorOptions`: | ||
| - `externalDriverFactory`: Database driver for user data | ||
| - `cacheAndQueueDriver`: Backend for caching and queuing | ||
| - `queryCacheOptions`: Cache-specific settings | ||
| - `preAggregationsOptions`: Pre-aggregation configuration | ||
| - `rollupOnlyMode`: When enabled, only serves pre-aggregated data | ||
| - `continueWaitTimeout`: Timeout for waiting operations | ||
|
|
||
| ## Development Notes | ||
|
|
||
| - Uses TypeScript with relaxed strict settings (`tsconfig.json`) | ||
| - Inherits linting rules from `@cubejs-backend/linter` | ||
| - Jest configuration extends base repository config | ||
| - Docker Compose setup for integration testing | ||
| - Coverage reports generated in `coverage/` directory |
11 changes: 0 additions & 11 deletions
11
packages/cubejs-query-orchestrator/src/orchestrator/BaseQueueEventsBus.ts
This file was deleted.
Oops, something went wrong.
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.