Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -24,3 +24,4 @@ rust/cubesql/profile.json
.cubestore
.env
.vimspector.json
.claude/settings.local.json
137 changes: 137 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,137 @@
# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## Repository Overview

Cube is a semantic layer for building data applications. This is a monorepo containing the complete Cube ecosystem including:
- Cube backend server and core components
- Client libraries for JavaScript/React/Vue/Angular
- Database drivers for various data sources
- Documentation site
- Rust components (CubeSQL, CubeStore)

## Development Commands

**Note: This project uses Yarn as the package manager.**

### Core Build Commands
```bash
# Build all packages
yarn build

# Run TypeScript compilation across all packages
yarn tsc

# Watch mode for TypeScript compilation
yarn tsc:watch

# Clean build artifacts
yarn clean

# Run linting across all packages
yarn lint

# Fix linting issues
yarn lint:fix

# Lint package.json files
yarn lint:npm
```

### Testing Commands
```bash
# Run tests (most packages have individual test commands)
yarn test

# Test individual packages
cd packages/cubejs-[package-name]
yarn test
```

### Documentation Development
The documentation is in `/docs` directory:
```bash
cd docs
yarn dev # Start development server
yarn build # Build for production
```

## Architecture Overview

### Monorepo Structure
- **`/packages`**: All JavaScript/TypeScript packages managed by Lerna
- Core packages: `cubejs-server-core`, `cubejs-schema-compiler`, `cubejs-query-orchestrator`
- Client libraries: `cubejs-client-core`, `cubejs-client-react`, etc.
- Database drivers: `cubejs-postgres-driver`, `cubejs-bigquery-driver`, etc.
- API layer: `cubejs-api-gateway`
- **`/rust`**: Rust components including CubeSQL (SQL interface) and CubeStore (distributed storage)
- **`/docs`**: Next.js documentation site
- **`/examples`**: Example implementations and recipes

### Key Components
1. **Schema Compiler**: Compiles data models into executable queries
2. **Query Orchestrator**: Manages query execution, caching, and pre-aggregations
3. **API Gateway**: Provides REST, GraphQL, and SQL APIs
4. **CubeSQL**: Postgres-compatible SQL interface (Rust)
5. **CubeStore**: Distributed OLAP storage engine (Rust)

### Package Management
- Uses Yarn workspaces with Lerna for package management
- TypeScript compilation is coordinated across packages
- Jest for unit testing with package-specific configurations

## Testing Approach

### Unit Tests
- Most packages have Jest-based unit tests in `/test` directories
- TypeScript packages use `jest.config.js` with TypeScript compilation
- Snapshot testing for SQL compilation and query planning

### Integration Tests
- Driver-specific integration tests in `/packages/cubejs-testing-drivers`
- End-to-end tests in `/packages/cubejs-testing`
- Docker-based testing environments for database drivers

### Test Commands
```bash
# Individual package testing
cd packages/[package-name]
yarn test

# Driver integration tests (requires Docker)
cd packages/cubejs-testing-drivers
yarn test
```

## Development Workflow

1. **Making Changes**: Work in individual packages, changes are coordinated via Lerna
2. **Building**: Use `yarn tsc` to compile TypeScript across all packages
3. **Testing**: Run relevant tests for modified packages
4. **Linting**: Ensure code passes `yarn lint` before committing

## Common File Patterns

- `*.test.ts/js`: Jest unit tests
- `jest.config.js`: Jest configuration per package
- `tsconfig.json`: TypeScript configuration (inherits from root)
- `CHANGELOG.md`: Per-package changelogs maintained by Lerna
- `src/`: Source code directory
- `dist/`: Compiled output (not committed)

## Important Notes

- This is documentation for the old Cube docs site structure (the existing `/docs/CLAUDE.md` refers to the documentation site)
- The main Cube application development happens in `/packages`
- For data model changes, focus on `cubejs-schema-compiler` package
- For query execution changes, focus on `cubejs-query-orchestrator` package
- Database connectivity is handled by individual driver packages

## Key Dependencies

- **Lerna**: Monorepo management and publishing
- **TypeScript**: Primary language for most packages
- **Jest**: Testing framework
- **Rollup**: Bundling for client libraries
- **Docker**: Testing environments for database drivers
3 changes: 0 additions & 3 deletions packages/cubejs-api-gateway/src/SubscriptionServer.ts
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,6 @@ const methodParams: Record<string, string[]> = {
meta: [],
subscribe: ['query', 'queryType'],
unsubscribe: [],
'subscribe.queue.events': []
};

const calcMessageLength = (message: unknown) => Buffer.byteLength(
Expand Down Expand Up @@ -149,8 +148,6 @@ export class SubscriptionServer {
}

public async disconnect(connectionId: string) {
const authContext = await this.subscriptionStore.getAuthContext(connectionId);
await this.apiGateway.unSubscribeQueueEvents({ context: authContext, connectionId });
await this.subscriptionStore.cleanupSubscriptions(connectionId);
}

Expand Down
15 changes: 0 additions & 15 deletions packages/cubejs-api-gateway/src/gateway.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2067,21 +2067,6 @@ class ApiGateway {
}
}

public async subscribeQueueEvents({ context, signedWithPlaygroundAuthSecret, connectionId, res }) {
if (this.enforceSecurityChecks && !signedWithPlaygroundAuthSecret) {
throw new CubejsHandlerError(
403,
'Forbidden',
'Only for signed with playground auth secret'
);
}
return (await this.getAdapterApi(context)).subscribeQueueEvents(connectionId, res);
}

public async unSubscribeQueueEvents({ context, connectionId }) {
return (await this.getAdapterApi(context)).unSubscribeQueueEvents(connectionId);
}

public async subscribe({
query, context, res, subscribe, subscriptionState, queryType, apiType
}) {
Expand Down
3 changes: 0 additions & 3 deletions packages/cubejs-backend-shared/src/env.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2090,9 +2090,6 @@ const variables: Record<string, (...args: any) => any> = {
livePreview: () => get('CUBEJS_LIVE_PREVIEW')
.default('true')
.asBoolStrict(),
preAggregationsQueueEventsBus: () => get('CUBEJS_PRE_AGGREGATIONS_QUEUE_EVENTS_BUS')
.default('false')
.asBoolStrict(),
externalDefault: () => get('CUBEJS_EXTERNAL_DEFAULT')
.default('true')
.asBoolStrict(),
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,6 @@ export interface QueueDriverOptions {
continueWaitTimeout: number,
orphanedTimeout: number,
heartBeatTimeout: number,
getQueueEventsBus?: any,
processUid?: string;
}

Expand Down
131 changes: 131 additions & 0 deletions packages/cubejs-query-orchestrator/CLAUDE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,131 @@
# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## Package Overview

The Query Orchestrator is a multi-stage querying engine that manages query execution, caching, and pre-aggregations in Cube. It receives pre-aggregation SQL queries and executes them in exact order, ensuring up-to-date data structure and freshness.

## Development Commands

**Note: This project uses Yarn as the package manager.**

```bash
# Build the package
yarn build

# Build with TypeScript compilation
yarn tsc

# Watch mode for development
yarn watch

# Run all tests (unit + integration)
yarn test

# Run only unit tests
yarn unit

# Run only integration tests
yarn integration

# Run CubeStore integration tests specifically
yarn integration:cubestore

# Run linting
yarn lint

# Fix linting issues
yarn lint:fix
```

## Architecture Overview

### Core Components

The Query Orchestrator consists of several interconnected components:

1. **QueryOrchestrator** (`src/orchestrator/QueryOrchestrator.ts`): Main orchestration class that coordinates query execution and manages drivers
2. **QueryCache** (`src/orchestrator/QueryCache.ts`): Handles query result caching with configurable cache drivers
3. **QueryQueue** (`src/orchestrator/QueryQueue.js`): Manages query queuing and background processing
4. **PreAggregations** (`src/orchestrator/PreAggregations.ts`): Manages pre-aggregation building and loading
5. **DriverFactory** (`src/orchestrator/DriverFactory.ts`): Creates and manages database driver instances

### Cache and Queue Driver Architecture

The orchestrator supports multiple backend drivers:
- **Memory**: In-memory caching and queuing (development)
- **CubeStore**: Distributed storage engine (production)
- **Redis**: External Redis-based caching (legacy, being phased out)

Driver selection logic in `QueryOrchestrator.ts:detectQueueAndCacheDriver()`:
- Explicit configuration via `cacheAndQueueDriver` option
- Environment variables (`CUBEJS_CACHE_AND_QUEUE_DRIVER`)
- Auto-detection: Redis if `CUBEJS_REDIS_URL` exists, CubeStore for production, Memory for development

### Query Processing Flow

1. **Query Submission**: Queries enter through QueryOrchestrator
2. **Cache Check**: QueryCache checks for existing results
3. **Queue Management**: QueryQueue handles background execution
4. **Pre-aggregation Processing**: PreAggregations component manages rollup tables
5. **Result Caching**: Results stored via cache driver for future requests

### Pre-aggregation System

The pre-aggregation system includes:
- **PreAggregationLoader**: Loads pre-aggregation definitions
- **PreAggregationPartitionRangeLoader**: Handles partition range loading
- **PreAggregationLoadCache**: Manages loading cache for pre-aggregations

## Testing Structure

### Unit Tests (`test/unit/`)
- `QueryCache.test.ts`: Query caching functionality
- `QueryQueue.test.ts`: Queue management and processing
- `QueryOrchestrator.test.js`: Main orchestrator logic
- `PreAggregations.test.js`: Pre-aggregation management

### Integration Tests (`test/integration/`)
- `cubestore/`: CubeStore-specific integration tests
- Tests real database interactions and queue processing

### Test Abstractions
- `QueryCache.abstract.ts`: Shared test suite for cache implementations
- `QueryQueue.abstract.ts`: Shared test suite for queue implementations

## Key Design Patterns

### Queue Processing Architecture
The DEVELOPMENT.md file contains detailed sequence diagrams showing:
- Queue interaction with CubeStore via specific queue commands (`QUEUE ADD`, `QUEUE RETRIEVE`, etc.)
- Background query processing with heartbeat management
- Result handling and cleanup

### Driver Factory Pattern
- `DriverFactory` type enables pluggable database drivers
- `DriverFactoryByDataSource` supports multi-tenant scenarios
- Separation between external (user data) and internal (cache/queue) drivers

### Error Handling
- `ContinueWaitError`: Signals when queries should continue waiting
- `TimeoutError`: Handles query timeout scenarios
- Proper cleanup and resource management across all components

## Configuration

Key configuration options in `QueryOrchestratorOptions`:
- `externalDriverFactory`: Database driver for user data
- `cacheAndQueueDriver`: Backend for caching and queuing
- `queryCacheOptions`: Cache-specific settings
- `preAggregationsOptions`: Pre-aggregation configuration
- `rollupOnlyMode`: When enabled, only serves pre-aggregated data
- `continueWaitTimeout`: Timeout for waiting operations

## Development Notes

- Uses TypeScript with relaxed strict settings (`tsconfig.json`)
- Inherits linting rules from `@cubejs-backend/linter`
- Jest configuration extends base repository config
- Docker Compose setup for integration testing
- Coverage reports generated in `coverage/` directory

This file was deleted.

Loading
Loading