Skip to content

Commit d4f1648

Browse files
committed
refactor: Remove deprecated queue events bus functionality
The queue event bus was deprecated and is no longer needed. This completes the removal of the deprecated queue events bus system across the entire codebase.
1 parent 4c0fef7 commit d4f1648

File tree

15 files changed

+269
-114
lines changed

15 files changed

+269
-114
lines changed

.gitignore

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,3 +24,4 @@ rust/cubesql/profile.json
2424
.cubestore
2525
.env
2626
.vimspector.json
27+
.claude/settings.local.json

CLAUDE.md

Lines changed: 137 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,137 @@
1+
# CLAUDE.md
2+
3+
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
4+
5+
## Repository Overview
6+
7+
Cube is a semantic layer for building data applications. This is a monorepo containing the complete Cube ecosystem including:
8+
- Cube backend server and core components
9+
- Client libraries for JavaScript/React/Vue/Angular
10+
- Database drivers for various data sources
11+
- Documentation site
12+
- Rust components (CubeSQL, CubeStore)
13+
14+
## Development Commands
15+
16+
**Note: This project uses Yarn as the package manager.**
17+
18+
### Core Build Commands
19+
```bash
20+
# Build all packages
21+
yarn build
22+
23+
# Run TypeScript compilation across all packages
24+
yarn tsc
25+
26+
# Watch mode for TypeScript compilation
27+
yarn tsc:watch
28+
29+
# Clean build artifacts
30+
yarn clean
31+
32+
# Run linting across all packages
33+
yarn lint
34+
35+
# Fix linting issues
36+
yarn lint:fix
37+
38+
# Lint package.json files
39+
yarn lint:npm
40+
```
41+
42+
### Testing Commands
43+
```bash
44+
# Run tests (most packages have individual test commands)
45+
yarn test
46+
47+
# Test individual packages
48+
cd packages/cubejs-[package-name]
49+
yarn test
50+
```
51+
52+
### Documentation Development
53+
The documentation is in `/docs` directory:
54+
```bash
55+
cd docs
56+
yarn dev # Start development server
57+
yarn build # Build for production
58+
```
59+
60+
## Architecture Overview
61+
62+
### Monorepo Structure
63+
- **`/packages`**: All JavaScript/TypeScript packages managed by Lerna
64+
- Core packages: `cubejs-server-core`, `cubejs-schema-compiler`, `cubejs-query-orchestrator`
65+
- Client libraries: `cubejs-client-core`, `cubejs-client-react`, etc.
66+
- Database drivers: `cubejs-postgres-driver`, `cubejs-bigquery-driver`, etc.
67+
- API layer: `cubejs-api-gateway`
68+
- **`/rust`**: Rust components including CubeSQL (SQL interface) and CubeStore (distributed storage)
69+
- **`/docs`**: Next.js documentation site
70+
- **`/examples`**: Example implementations and recipes
71+
72+
### Key Components
73+
1. **Schema Compiler**: Compiles data models into executable queries
74+
2. **Query Orchestrator**: Manages query execution, caching, and pre-aggregations
75+
3. **API Gateway**: Provides REST, GraphQL, and SQL APIs
76+
4. **CubeSQL**: Postgres-compatible SQL interface (Rust)
77+
5. **CubeStore**: Distributed OLAP storage engine (Rust)
78+
79+
### Package Management
80+
- Uses Yarn workspaces with Lerna for package management
81+
- TypeScript compilation is coordinated across packages
82+
- Jest for unit testing with package-specific configurations
83+
84+
## Testing Approach
85+
86+
### Unit Tests
87+
- Most packages have Jest-based unit tests in `/test` directories
88+
- TypeScript packages use `jest.config.js` with TypeScript compilation
89+
- Snapshot testing for SQL compilation and query planning
90+
91+
### Integration Tests
92+
- Driver-specific integration tests in `/packages/cubejs-testing-drivers`
93+
- End-to-end tests in `/packages/cubejs-testing`
94+
- Docker-based testing environments for database drivers
95+
96+
### Test Commands
97+
```bash
98+
# Individual package testing
99+
cd packages/[package-name]
100+
yarn test
101+
102+
# Driver integration tests (requires Docker)
103+
cd packages/cubejs-testing-drivers
104+
yarn test
105+
```
106+
107+
## Development Workflow
108+
109+
1. **Making Changes**: Work in individual packages, changes are coordinated via Lerna
110+
2. **Building**: Use `yarn tsc` to compile TypeScript across all packages
111+
3. **Testing**: Run relevant tests for modified packages
112+
4. **Linting**: Ensure code passes `yarn lint` before committing
113+
114+
## Common File Patterns
115+
116+
- `*.test.ts/js`: Jest unit tests
117+
- `jest.config.js`: Jest configuration per package
118+
- `tsconfig.json`: TypeScript configuration (inherits from root)
119+
- `CHANGELOG.md`: Per-package changelogs maintained by Lerna
120+
- `src/`: Source code directory
121+
- `dist/`: Compiled output (not committed)
122+
123+
## Important Notes
124+
125+
- This is documentation for the old Cube docs site structure (the existing `/docs/CLAUDE.md` refers to the documentation site)
126+
- The main Cube application development happens in `/packages`
127+
- For data model changes, focus on `cubejs-schema-compiler` package
128+
- For query execution changes, focus on `cubejs-query-orchestrator` package
129+
- Database connectivity is handled by individual driver packages
130+
131+
## Key Dependencies
132+
133+
- **Lerna**: Monorepo management and publishing
134+
- **TypeScript**: Primary language for most packages
135+
- **Jest**: Testing framework
136+
- **Rollup**: Bundling for client libraries
137+
- **Docker**: Testing environments for database drivers

packages/cubejs-api-gateway/src/SubscriptionServer.ts

Lines changed: 0 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,6 @@ const methodParams: Record<string, string[]> = {
1212
meta: [],
1313
subscribe: ['query', 'queryType'],
1414
unsubscribe: [],
15-
'subscribe.queue.events': []
1615
};
1716

1817
const calcMessageLength = (message: unknown) => Buffer.byteLength(
@@ -149,8 +148,6 @@ export class SubscriptionServer {
149148
}
150149

151150
public async disconnect(connectionId: string) {
152-
const authContext = await this.subscriptionStore.getAuthContext(connectionId);
153-
await this.apiGateway.unSubscribeQueueEvents({ context: authContext, connectionId });
154151
await this.subscriptionStore.cleanupSubscriptions(connectionId);
155152
}
156153

packages/cubejs-api-gateway/src/gateway.ts

Lines changed: 0 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -2067,21 +2067,6 @@ class ApiGateway {
20672067
}
20682068
}
20692069

2070-
public async subscribeQueueEvents({ context, signedWithPlaygroundAuthSecret, connectionId, res }) {
2071-
if (this.enforceSecurityChecks && !signedWithPlaygroundAuthSecret) {
2072-
throw new CubejsHandlerError(
2073-
403,
2074-
'Forbidden',
2075-
'Only for signed with playground auth secret'
2076-
);
2077-
}
2078-
return (await this.getAdapterApi(context)).subscribeQueueEvents(connectionId, res);
2079-
}
2080-
2081-
public async unSubscribeQueueEvents({ context, connectionId }) {
2082-
return (await this.getAdapterApi(context)).unSubscribeQueueEvents(connectionId);
2083-
}
2084-
20852070
public async subscribe({
20862071
query, context, res, subscribe, subscriptionState, queryType, apiType
20872072
}) {

packages/cubejs-backend-shared/src/env.ts

Lines changed: 0 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2090,9 +2090,6 @@ const variables: Record<string, (...args: any) => any> = {
20902090
livePreview: () => get('CUBEJS_LIVE_PREVIEW')
20912091
.default('true')
20922092
.asBoolStrict(),
2093-
preAggregationsQueueEventsBus: () => get('CUBEJS_PRE_AGGREGATIONS_QUEUE_EVENTS_BUS')
2094-
.default('false')
2095-
.asBoolStrict(),
20962093
externalDefault: () => get('CUBEJS_EXTERNAL_DEFAULT')
20972094
.default('true')
20982095
.asBoolStrict(),

packages/cubejs-base-driver/src/queue-driver.interface.ts

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -52,7 +52,6 @@ export interface QueueDriverOptions {
5252
continueWaitTimeout: number,
5353
orphanedTimeout: number,
5454
heartBeatTimeout: number,
55-
getQueueEventsBus?: any,
5655
processUid?: string;
5756
}
5857

Lines changed: 131 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,131 @@
1+
# CLAUDE.md
2+
3+
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
4+
5+
## Package Overview
6+
7+
The Query Orchestrator is a multi-stage querying engine that manages query execution, caching, and pre-aggregations in Cube. It receives pre-aggregation SQL queries and executes them in exact order, ensuring up-to-date data structure and freshness.
8+
9+
## Development Commands
10+
11+
**Note: This project uses Yarn as the package manager.**
12+
13+
```bash
14+
# Build the package
15+
yarn build
16+
17+
# Build with TypeScript compilation
18+
yarn tsc
19+
20+
# Watch mode for development
21+
yarn watch
22+
23+
# Run all tests (unit + integration)
24+
yarn test
25+
26+
# Run only unit tests
27+
yarn unit
28+
29+
# Run only integration tests
30+
yarn integration
31+
32+
# Run CubeStore integration tests specifically
33+
yarn integration:cubestore
34+
35+
# Run linting
36+
yarn lint
37+
38+
# Fix linting issues
39+
yarn lint:fix
40+
```
41+
42+
## Architecture Overview
43+
44+
### Core Components
45+
46+
The Query Orchestrator consists of several interconnected components:
47+
48+
1. **QueryOrchestrator** (`src/orchestrator/QueryOrchestrator.ts`): Main orchestration class that coordinates query execution and manages drivers
49+
2. **QueryCache** (`src/orchestrator/QueryCache.ts`): Handles query result caching with configurable cache drivers
50+
3. **QueryQueue** (`src/orchestrator/QueryQueue.js`): Manages query queuing and background processing
51+
4. **PreAggregations** (`src/orchestrator/PreAggregations.ts`): Manages pre-aggregation building and loading
52+
5. **DriverFactory** (`src/orchestrator/DriverFactory.ts`): Creates and manages database driver instances
53+
54+
### Cache and Queue Driver Architecture
55+
56+
The orchestrator supports multiple backend drivers:
57+
- **Memory**: In-memory caching and queuing (development)
58+
- **CubeStore**: Distributed storage engine (production)
59+
- **Redis**: External Redis-based caching (legacy, being phased out)
60+
61+
Driver selection logic in `QueryOrchestrator.ts:detectQueueAndCacheDriver()`:
62+
- Explicit configuration via `cacheAndQueueDriver` option
63+
- Environment variables (`CUBEJS_CACHE_AND_QUEUE_DRIVER`)
64+
- Auto-detection: Redis if `CUBEJS_REDIS_URL` exists, CubeStore for production, Memory for development
65+
66+
### Query Processing Flow
67+
68+
1. **Query Submission**: Queries enter through QueryOrchestrator
69+
2. **Cache Check**: QueryCache checks for existing results
70+
3. **Queue Management**: QueryQueue handles background execution
71+
4. **Pre-aggregation Processing**: PreAggregations component manages rollup tables
72+
5. **Result Caching**: Results stored via cache driver for future requests
73+
74+
### Pre-aggregation System
75+
76+
The pre-aggregation system includes:
77+
- **PreAggregationLoader**: Loads pre-aggregation definitions
78+
- **PreAggregationPartitionRangeLoader**: Handles partition range loading
79+
- **PreAggregationLoadCache**: Manages loading cache for pre-aggregations
80+
81+
## Testing Structure
82+
83+
### Unit Tests (`test/unit/`)
84+
- `QueryCache.test.ts`: Query caching functionality
85+
- `QueryQueue.test.ts`: Queue management and processing
86+
- `QueryOrchestrator.test.js`: Main orchestrator logic
87+
- `PreAggregations.test.js`: Pre-aggregation management
88+
89+
### Integration Tests (`test/integration/`)
90+
- `cubestore/`: CubeStore-specific integration tests
91+
- Tests real database interactions and queue processing
92+
93+
### Test Abstractions
94+
- `QueryCache.abstract.ts`: Shared test suite for cache implementations
95+
- `QueryQueue.abstract.ts`: Shared test suite for queue implementations
96+
97+
## Key Design Patterns
98+
99+
### Queue Processing Architecture
100+
The DEVELOPMENT.md file contains detailed sequence diagrams showing:
101+
- Queue interaction with CubeStore via specific queue commands (`QUEUE ADD`, `QUEUE RETRIEVE`, etc.)
102+
- Background query processing with heartbeat management
103+
- Result handling and cleanup
104+
105+
### Driver Factory Pattern
106+
- `DriverFactory` type enables pluggable database drivers
107+
- `DriverFactoryByDataSource` supports multi-tenant scenarios
108+
- Separation between external (user data) and internal (cache/queue) drivers
109+
110+
### Error Handling
111+
- `ContinueWaitError`: Signals when queries should continue waiting
112+
- `TimeoutError`: Handles query timeout scenarios
113+
- Proper cleanup and resource management across all components
114+
115+
## Configuration
116+
117+
Key configuration options in `QueryOrchestratorOptions`:
118+
- `externalDriverFactory`: Database driver for user data
119+
- `cacheAndQueueDriver`: Backend for caching and queuing
120+
- `queryCacheOptions`: Cache-specific settings
121+
- `preAggregationsOptions`: Pre-aggregation configuration
122+
- `rollupOnlyMode`: When enabled, only serves pre-aggregated data
123+
- `continueWaitTimeout`: Timeout for waiting operations
124+
125+
## Development Notes
126+
127+
- Uses TypeScript with relaxed strict settings (`tsconfig.json`)
128+
- Inherits linting rules from `@cubejs-backend/linter`
129+
- Jest configuration extends base repository config
130+
- Docker Compose setup for integration testing
131+
- Coverage reports generated in `coverage/` directory

packages/cubejs-query-orchestrator/src/orchestrator/BaseQueueEventsBus.ts

Lines changed: 0 additions & 11 deletions
This file was deleted.

0 commit comments

Comments
 (0)