Implement conversation logging and fine-tuning dataset generation

.gitignore

@@ -14,6 +14,7 @@ mock/
 # Builds
 bin/
 *.vsix
+.roo-logs/*
 
 # Local prompts and rules
 /local-prompts
@@ -45,3 +46,5 @@ logs
 .qodo/
 .vercel
 .roo/mcp.json
+**/.roo-logs/*
+**/finetuning-datasets/

.rooignore

@@ -1 +1,2 @@
 .env
+.roo-logs/

package.json

@@ -23,7 +23,9 @@
 		"changeset:version": "cp CHANGELOG.md src/CHANGELOG.md && changeset version && cp -vf src/CHANGELOG.md .",
 		"knip": "knip --include files",
 		"update-contributors": "node scripts/update-contributors.js",
-		"evals": "docker compose -f packages/evals/docker-compose.yml --profile server --profile runner up --build --scale runner=0"
+		"evals": "docker compose -f packages/evals/docker-compose.yml --profile server --profile runner up --build --scale runner=0",
+		"sft": "ts-node scripts/create-finetuning-data.ts",
+		"sft:openai": "ts-node scripts/create-finetuning-data.ts --openai"
 	},
 	"devDependencies": {
 		"@changesets/cli": "^2.27.10",
@@ -40,7 +42,8 @@
 		"prettier": "^3.4.2",
 		"rimraf": "^6.0.1",
 		"turbo": "^2.5.3",
-		"typescript": "^5.4.5"
+		"typescript": "^5.4.5",
+		"@types/node": "^20.11.25"
 	},
 	"lint-staged": {
 		"*.{js,jsx,ts,tsx,json,css,md}": [

scripts/README.md

@@ -0,0 +1,124 @@
+# Scripts
+
+This directory contains various scripts for automating tasks, managing the repository, and assisting with development workflows.
+
+## Available Scripts
+
+### `create-finetuning-data.ts`
+
+Processes conversation logs from `.roo-logs` and transforms them into a supervised fine-tuning (SFT) dataset. These datasets are essential for training AI models like Google's Gemini or OpenAI's GPT models to improve their performance on specific tasks within Roo Code.
+
+The script reads the `.jsonl` log files generated by `ConversationLogger.ts`, processes the turns of conversation, and formats them into the required structure for the specified provider.
+
+**Usage:**
+
+- **Via `npx`:**
+    ```bash
+    npx ts-node scripts/create-finetuning-data.ts [flags]
+    ```
+- **Via `npm` script:**
+
+    ```bash
+    # For Gemini (default)
+    npm run sft -- [flags]
+
+    # For OpenAI
+    npm run sft:openai -- [flags]
+    ```
+
+**Flags:**
+
+| Flag                       | Description                                                           | Default               | Example                                    |
+| -------------------------- | --------------------------------------------------------------------- | --------------------- | ------------------------------------------ |
+| `--provider <name>`        | The AI provider to format the data for (`gemini` or `openai`).        | `gemini`              | `--provider openai`                        |
+| `--sessionId <session_id>` | The ID of a specific session to process.                              | (none)                | `--sessionId session_1678886400000_abc123` |
+| `--depth <number>`         | The number of most recent log files to process.                       | (all)                 | `--depth 5`                                |
+| `--input <path>`           | The path to the directory containing the log files.                   | `.roo-logs`           | `--input ./path/to/logs`                   |
+| `--output <path>`          | The path to the directory where the generated datasets will be saved. | `finetuning-datasets` | `--output ./path/to/datasets`              |
+
+---
+
+### `install-vsix.js`
+
+Automates uninstalling any existing version of the Roo Code extension and installing a newly built `.vsix` package. This is useful for testing local builds.
+
+**Usage:**
+
+- **Via `node`:**
+    ```bash
+    node scripts/install-vsix.js [flags]
+    ```
+- **Via `npm` script:**
+    ```bash
+    npm run install:vsix -- [flags]
+    ```
+
+**Flags:**
+
+| Flag                 | Description                                                          | Default | Example                      |
+| -------------------- | -------------------------------------------------------------------- | ------- | ---------------------------- |
+| `-y`                 | Skip the confirmation prompt and install automatically.              | (none)  | `npm run install:vsix -- -y` |
+| `--editor=<command>` | The editor command to use (`code`, `cursor`, `code-insiders`, etc.). | `code`  | `--editor=cursor`            |
+
+---
+
+### `find-missing-i18n-key.js`
+
+Scans the codebase to find any internationalization (i18n) keys that are used in the source code but are missing from the locale files. This helps ensure that all text displayed to the user can be translated.
+
+**Usage:**
+
+```bash
+node scripts/find-missing-i18n-key.js [flags]
+```
+
+**Flags:**
+
+| Flag              | Description                                                 | Default | Example              |
+| ----------------- | ----------------------------------------------------------- | ------- | -------------------- |
+| `--locale=<name>` | Only check for a specific locale (e.g., `de`, `fr`).        | (all)   | `--locale=de`        |
+| `--file=<name>`   | Only check a specific translation file (e.g., `chat.json`). | (all)   | `--file=common.json` |
+| `--help`          | Display help information.                                   | (none)  | `--help`             |
+
+---
+
+### `find-missing-translations.js`
+
+Compares non-English locale files against the English source files to identify any missing translations. This ensures that all locales are up-to-date with the source language.
+
+**Usage:**
+
+```bash
+node scripts/find-missing-translations.js [flags]
+```
+
+**Flags:**
+
+| Flag              | Description                                                          | Default | Example            |
+| ----------------- | -------------------------------------------------------------------- | ------- | ------------------ |
+| `--locale=<name>` | Only check a specific locale (e.g., `de`, `fr`).                     | (all)   | `--locale=fr`      |
+| `--file=<name>`   | Only check a specific translation file (e.g., `chat.json`).          | (all)   | `--file=chat.json` |
+| `--area=<name>`   | The area to check (`core` for backend, `webview` for UI, or `both`). | `both`  | `--area=webview`   |
+| `--help`          | Display help information.                                            | (none)  | `--help`           |
+
+---
+
+### `update-contributors.js`
+
+Fetches the latest list of contributors from the GitHub repository and updates the `README.md` files (both root and localized versions).
+
+**Usage:**
+
+```bash
+node scripts/update-contributors.js
+```
+
+---
+
+### `bootstrap.mjs`
+
+Ensures that the correct version of `pnpm` is installed and then runs `pnpm install`. This script is intended for new contributors to ensure a consistent development environment setup.
+
+**Usage:**
+
+This script is run automatically during `pnpm install` via the `preinstall` hook in `package.json`.