|
| 1 | +# CLAUDE.md and GEMINI.md |
| 2 | + |
| 3 | +This file provides guidance to Claude Code (claude.ai/code) and also Gemini CLI (https://github.com/google-gemini/gemini-cli) when working with code in this repository. |
| 4 | + |
| 5 | +# CoCalc Source Repository |
| 6 | + |
| 7 | +- This is the source code of CoCalc in a Git repository |
| 8 | +- It is a complex JavaScript/TypeScript SaaS application |
| 9 | +- CoCalc is organized as a monorepository (multi-packages) in the subdirectory "./packages" |
| 10 | +- The packages are managed as a pnpm workspace in "./packages/pnpm-workspace.yaml" |
| 11 | + |
| 12 | +## Code Style |
| 13 | + |
| 14 | +- Everything is written in TypeScript code |
| 15 | +- Indentation: 2-spaces |
| 16 | +- Run `prettier -w [filename]` after modifying a file (ts, tsx, md, json, ...) to format it correctly. |
| 17 | +- All .js and .ts files are formatted by the tool prettier |
| 18 | +- Add suitable types when you write code |
| 19 | +- Follow DRY principles! |
| 20 | +- TypeScript: prefer `var1 ?? var2` for fallbacks. only use `var1 || var2` in explicit or-chains or when necessary. |
| 21 | +- Variable name styles are `camelCase` for local and `FOO_BAR` for global variables. React Components and Classes are `FooBar`. If you edit older code not following these guidelines, adjust this rule to fit the file's style. |
| 22 | +- Some older code is JavaScript or CoffeeScript, which will be translated to TypeScript |
| 23 | +- Use ES modules (import/export) syntax, not CommonJS (require) |
| 24 | +- Organize the list of imports in such a way: installed npm packages are on top, newline, then are imports from @cocalc's code base. Sorted alphabetically. |
| 25 | +- **Colors**: Always use the `COLORS` dictionary from `@cocalc/util/theme` for all color values. Never hardcode colors like `#f0f0f0` or `rgb(...)`. Import with `import { COLORS } from "@cocalc/util/theme";` and use predefined constants like `COLORS.GRAY_M`, `COLORS.GRAY_L`, `COLORS.GRAY_LL`, etc. |
| 26 | +- **Backend Logging**: Use `getLogger` from `@cocalc/project/logger` for logging in backend code. Do NOT use `console.log`. Example: `const L = getLogger("module:name").debug;` |
| 27 | + |
| 28 | +## Development Commands |
| 29 | + |
| 30 | +### Essential Commands |
| 31 | + |
| 32 | +- `pnpm build-dev` - Build all packages for development |
| 33 | +- `pnpm clean` - Clean all `node_modules` and `dist` directories |
| 34 | +- `pnpm test` - Run full test suite |
| 35 | +- `pnpm depcheck` - Check for dependency issues |
| 36 | +- `python3 ./scripts/check_npm_packages.py` - Check npm package consistency across packages |
| 37 | +- `prettier -w [filename]` to format the style of a file after editing it |
| 38 | +- After creating a file, run `git add [filename]` to start tracking it |
| 39 | + |
| 40 | +### Package-Specific Commands |
| 41 | + |
| 42 | +- `cd packages/[package] && pnpm build` - Build and compile a specific package |
| 43 | + - For packages/next and packages/static, run `cd packages/[package] && pnpm build-dev` |
| 44 | +- `cd packages/[package] && pnpm test` - Run tests for a specific package |
| 45 | +- To typecheck the frontend, it is best to run `cd packages/static && pnpm build` - this implicitly compiles the frontend and reports TypeScript errors |
| 46 | +- **IMPORTANT**: When modifying packages like `util` that other packages depend on, you must run `pnpm build` in the modified package before typechecking dependent packages |
| 47 | + |
| 48 | +### Development |
| 49 | + |
| 50 | +- **IMPORTANT**: Always run `prettier -w [filename]` immediately after editing any .ts, .tsx, .md, or .json file to ensure consistent styling |
| 51 | +- After TypeScript or `*.tsx` changes, run `pnpm build` in the relevant package directory |
| 52 | + - When editing the frontend, run `pnpm build-dev` in `packages/static` (this implicitly builds the frontend) |
| 53 | + |
| 54 | +## Architecture Overview |
| 55 | + |
| 56 | +### Package Structure |
| 57 | + |
| 58 | +CoCalc is organized as a monorepo with key packages: |
| 59 | + |
| 60 | +- **frontend** - React/TypeScript frontend application using Redux-style stores and actions |
| 61 | +- **backend** - Node.js backend services and utilities |
| 62 | +- **hub** - Main server orchestrating the entire system |
| 63 | +- **database** - PostgreSQL database layer with queries and schema |
| 64 | +- **util** - Shared utilities and types used across packages |
| 65 | +- **comm** - Communication layer including WebSocket types |
| 66 | +- **conat** - CoCalc's container/compute orchestration system |
| 67 | +- **sync** - Real-time synchronization system for collaborative editing |
| 68 | +- **project** - Project-level services and management |
| 69 | +- **static** - Static assets and build configuration |
| 70 | +- **next** - Next.js server components |
| 71 | + |
| 72 | +### Key Architectural Patterns |
| 73 | + |
| 74 | +#### Frontend Architecture |
| 75 | + |
| 76 | +- **Redux-style State Management**: Uses custom stores and actions pattern (see `packages/frontend/app-framework/actions-and-stores.ts`) |
| 77 | +- **TypeScript React Components**: All frontend code is TypeScript with proper typing |
| 78 | +- **Modular Store System**: Each feature has its own store/actions (AccountStore, BillingStore, etc.) |
| 79 | +- **WebSocket Communication**: Real-time communication with backend via WebSocket messages |
| 80 | +- **Authentication Waiting**: When frontend code needs to wait for user authentication, use `redux.getStore("account").async_wait({ until: () => store.get_account_id() != null, timeout: 0 })` to wait indefinitely until authentication completes |
| 81 | +- **Conat DKV Usage**: For key-value storage with real-time sync, use `webapp_client.conat_client.dkv({ account_id, name: "store-name" })` to get a distributed key-value store that syncs across sessions |
| 82 | + |
| 83 | +#### Backend Architecture |
| 84 | + |
| 85 | +- **PostgreSQL Database**: Primary data store with sophisticated querying system |
| 86 | +- **WebSocket Messaging**: Real-time communication between frontend and backend |
| 87 | +- **Conat System**: Container orchestration for compute servers |
| 88 | +- **Event-Driven Architecture**: Extensive use of EventEmitter patterns |
| 89 | +- **Microservice-like Packages**: Each package handles specific functionality |
| 90 | +- **Database Access**: Use `getPool()` from `@cocalc/database/pool` for direct database queries in hub/backend code. Example: `const pool = getPool(); const { rows } = await pool.query('SELECT * FROM table WHERE id = $1', [id]);` |
| 91 | +- **Hub Migration Functions**: Migration functions in hub should be designed to run once at startup, use batch processing with delays between batches to avoid database saturation |
| 92 | + |
| 93 | +#### Communication Patterns |
| 94 | + |
| 95 | +- **WebSocket Messages**: Primary communication method (see `packages/comm/websocket/types.ts`) |
| 96 | +- **Database Queries**: Structured query system with typed interfaces |
| 97 | +- **Event Emitters**: Inter-service communication within backend |
| 98 | +- **REST-like APIs**: Some HTTP endpoints for specific operations |
| 99 | +- **API Schema**: API endpoints in `packages/next/pages/api/v2/` use Zod schemas in `packages/next/lib/api/schema/` for validation. These schemas must be kept in harmony with the TypeScript types sent from frontend applications using `apiPost` (in `packages/next/lib/api/post.ts`) or `api` (in `packages/frontend/client/api.ts`). When adding new fields to API requests, both the frontend types and the API schema validation must be updated. |
| 100 | +- **Conat Frontend → Hub Communication**: CoCalc uses a custom distributed messaging system called "Conat" for frontend-to-hub communication: |
| 101 | + 1. **Frontend ConatClient** (`packages/frontend/conat/client.ts`): Manages WebSocket connection to hub, handles authentication, reconnection, and provides API interfaces |
| 102 | + 2. **Core Protocol** (`packages/conat/core/client.ts`): NATS-like pub/sub/request/response messaging with automatic chunking, multiple encoding formats (MsgPack, JSON), and delivery confirmation |
| 103 | + 3. **Hub API Structure** (`packages/conat/hub/api/`): Typed interfaces for different services (system, projects, db, purchases, jupyter) that map function calls to conat subjects |
| 104 | + 4. **Message Flow**: Frontend calls like `hub.projects.setQuotas()` → ConatClient.callHub() → conat request to subject `hub.account.{account_id}.api` → Hub API dispatcher → actual service implementation |
| 105 | + 5. **Authentication**: Each conat request includes account_id and is subject to permission checks at the hub level |
| 106 | + 6. **Subjects**: Messages are routed using hierarchical subjects like `hub.account.{uuid}.{service}` or `project.{uuid}.{compute_server_id}.{service}` |
| 107 | + |
| 108 | +### Key Technologies |
| 109 | + |
| 110 | +- **TypeScript**: Primary language for all new code |
| 111 | +- **React**: Frontend framework |
| 112 | +- **PostgreSQL**: Database |
| 113 | +- **Node.js**: Backend runtime |
| 114 | +- **WebSockets**: Real-time communication |
| 115 | +- **pnpm**: Package manager and workspace management |
| 116 | +- **Jest**: Testing framework |
| 117 | +- **SASS**: CSS preprocessing |
| 118 | + |
| 119 | +### Database Schema |
| 120 | + |
| 121 | +- Comprehensive schema in `packages/util/db-schema` |
| 122 | +- Query abstractions in `packages/database/postgres/` |
| 123 | +- Type-safe database operations with TypeScript interfaces |
| 124 | + |
| 125 | +### Testing |
| 126 | + |
| 127 | +- **Jest**: Primary testing framework |
| 128 | +- **ts-jest**: TypeScript support for Jest |
| 129 | +- **jsdom**: Browser environment simulation for frontend tests |
| 130 | +- Test files use `.test.ts` or `.spec.ts` extensions |
| 131 | +- Each package has its own jest.config.js |
| 132 | + |
| 133 | +### Import Patterns |
| 134 | + |
| 135 | +- Use absolute imports with `@cocalc/` prefix for cross-package imports |
| 136 | +- Example: `import { cmp } from "@cocalc/util/misc"` |
| 137 | +- Type imports: `import type { Foo } from "./bar"` |
| 138 | +- Destructure imports when possible |
| 139 | + |
| 140 | +### Development Workflow |
| 141 | + |
| 142 | +1. Changes to TypeScript require compilation (`pnpm build` in relevant package) |
| 143 | +2. Database must be running before starting hub |
| 144 | +3. Hub coordinates all services and should be restarted after changes |
| 145 | +4. Use `pnpm clean && pnpm build-dev` when switching branches or after major changes |
| 146 | +5. **IMPORTANT**: After any frontend code changes, run `pnpm build-dev` in the `packages/static` directory to compile the frontend |
| 147 | + |
| 148 | +# Workflow |
| 149 | + |
| 150 | +- Be sure to build when you're done making a series of code changes |
| 151 | +- Prefer running single tests, and not the whole test suite, for performance |
| 152 | + |
| 153 | +## Git Workflow |
| 154 | + |
| 155 | +- Never modify a file when in the `master` or `main` branch |
| 156 | +- All changes happen through feature branches, which are pushed as pull requests to GitHub |
| 157 | +- When creating a new file, run `git add [filename]` to track the file. |
| 158 | +- Prefix git commits with the package and general area. e.g. 'frontend/latex: ...' if it concerns latex editor changes in the packages/frontend/... code. |
| 159 | +- When pushing a new branch to Github, track it upstream. e.g. `git push --set-upstream origin feature-foo` for branch "feature-foo". |
| 160 | + |
| 161 | + |
| 162 | +## React-intl / Internationalization (i18n) |
| 163 | + |
| 164 | +CoCalc uses react-intl for internationalization with SimpleLocalize as the translation platform. |
| 165 | + |
| 166 | +### Architecture Overview |
| 167 | + |
| 168 | +- **Library**: Uses `react-intl` library with `defineMessages()` and `defineMessage()` |
| 169 | +- **Default Language**: English uses `defaultMessage` directly - no separate English translation files |
| 170 | +- **Supported Languages**: 19+ languages including German, Chinese, Spanish, French, Italian, Dutch, Russian, Japanese, Portuguese, Korean, Polish, Turkish, Hebrew, Hindi, Hungarian, Arabic, and Basque |
| 171 | +- **Translation Platform**: SimpleLocalize with OpenAI GPT-4o for automatic translations |
| 172 | + |
| 173 | +### Translation ID Naming Convention |
| 174 | + |
| 175 | +Translation IDs follow a hierarchical pattern: `[directory].[subdir].[filename].[aspect].[label|title|tooltip|...]` |
| 176 | + |
| 177 | +Examples: |
| 178 | + |
| 179 | +- `labels.masked_files` - for common UI labels |
| 180 | +- `account.sign-out.button.title` - for account sign-out dialog |
| 181 | +- `command.generic.force_build.label` - for command labels |
| 182 | + |
| 183 | +### Usage Patterns |
| 184 | + |
| 185 | +- **TSX Components**: `<FormattedMessage id="..." defaultMessage="..." />` |
| 186 | +- **Data Structures**: `defineMessage({id: "...", defaultMessage: "..."})` |
| 187 | +- **Programmatic Use**: `useIntl()` hook + `intl.formatMessage()` |
| 188 | +- **Non-React Contexts**: `getIntl()` function |
| 189 | + |
| 190 | +### Translation Workflow |
| 191 | + |
| 192 | +**For new translation keys:** |
| 193 | + |
| 194 | +1. Add the translation to source code (e.g., `packages/frontend/i18n/common.ts`) |
| 195 | +2. Run `pnpm i18n:extract` - updates `extracted.json` from source code |
| 196 | +3. Run `pnpm i18n:upload` - sends new strings to SimpleLocalize |
| 197 | +4. New keys are automatically translated to all languages |
| 198 | +5. Run `pnpm i18n:download` - fetches translations |
| 199 | +6. Run `pnpm i18n:compile` - compiles translation files |
| 200 | + |
| 201 | +**For editing existing translation keys:** |
| 202 | +Same flow as above, but **before 3. i18n:upload**, delete the key. Only new keys are auto-translated. `pnpm i18n:delete [id]`. |
| 203 | + |
| 204 | +### Translation File Structure |
| 205 | + |
| 206 | +- `packages/frontend/i18n/README.md` - detailed documentation |
| 207 | +- `packages/frontend/i18n/common.ts` - shared translation definitions (labels, menus, editor, jupyter, etc.) |
| 208 | +- `packages/frontend/i18n/extracted.json` - auto-extracted messages from source code |
| 209 | +- `packages/frontend/i18n/trans/[locale].json` - downloaded translations from SimpleLocalize |
| 210 | +- `packages/frontend/i18n/trans/[locale].compiled.json` - compiled translation files for runtime |
| 211 | +- `packages/frontend/i18n/index.ts` - exports and locale loading logic |
| 212 | + |
| 213 | +# Ignore |
| 214 | + |
| 215 | +- Ignore files covered by `.gitignore` |
| 216 | +- Ignore everything in `node_modules` or `dist` directories |
| 217 | +- Ignore all files not tracked by Git, unless they are newly created files |
| 218 | + |
| 219 | +# Important Instruction Reminders |
| 220 | + |
| 221 | +- Do what has been asked; nothing more, nothing less |
| 222 | +- NEVER create files unless they're absolutely necessary for achieving your goal |
| 223 | +- ALWAYS prefer editing an existing file to creating a new one |
| 224 | +- REFUSE to modify files when the git repository is on the `master` or `main` branch |
| 225 | +- NEVER proactively create documentation files (`*.md`) or README files. Only create documentation files if explicitly requested by the User |
| 226 | +- when modifying a file with a copyright banner at the top, make sure to fix/add the current year to indicate the copyright year |
0 commit comments