Implement conversation logging and fine-tuning dataset generation #5430
Conversation
…n and configuration updates
- Implemented `create-finetuning-data.ts` to parse log files and convert them into a fine-tuning dataset format. - Added functionality to read from a specified input directory and output the dataset to a JSONL file. - Introduced interfaces for log entries and Gemini message structures to facilitate data processing. - Enhanced error handling for malformed log entries and directory reading. - Updated `tsconfig.json` to include necessary TypeScript configurations for the new script.
Add script to convert conversation logs into valid supervised fine-tuning data for Gemini models. Fixes message ordering for tool calls and ensures proper structure compliance with Google Cloud documentation.
…versationLogger integration
… and add session filtering options
…ore to include logging README
dosubot
bot
added
size:XL
![ellipsis-dev[bot]](https://avatars.githubusercontent.com/in/64358?s=60&v=4){kind=small}
| const workspaceRoot = vscode.workspace.workspaceFolders?.[0]?.uri.fsPath | ||
| if (workspaceRoot) { | ||
| const logger = new ConversationLogger(workspaceRoot) | ||
| context.globalState.update("conversationLogger", logger) |
There was a problem hiding this comment.
Consider not storing a non-serializable ConversationLogger instance in context.globalState (line 96). Global state is meant for simple serializable data. Perhaps store the logger in a disposable or module-level variable instead.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| * @returns The new session ID. | ||
| */ | ||
| public startNewSession(): string { | ||
| this.sessionId = this.generateSessionId() |
Check failure
Code scanning / CodeQL
Insecure randomness High
hannesrudolph
added
the
Issue/PR - Triage
…ipt for fine-tuning
…dow for non-thinking models
hannesrudolph
added
PR - Needs Preliminary Review
and removed
Issue/PR - Triage
|
Thank you for your contribution. To ensure we can properly review and integrate your work, could you please provide some more context? The scope isn't entirely clear, and we couldn't find an associated issue that this pull request addresses. If this is intended to fix a bug, please feel free to open an issue to discuss the problem and solution so we can figure out if this PR fixes the problem. If this is a new feature, then please submit a detailed feature proposal. This will help us understand the problem you're solving and how your contribution fits into the project. We appreciate you taking the time to contribute to Roo Code! |
github-project-automation
bot
moved this from PR [Needs Prelim Review]
to Done
in Roo Code Roadmap
3 participants
Introduce a ConversationLogger service for enhanced logging capabilities and add a script to create fine-tuning datasets from conversation logs. Update configurations and error handling to support new features, while ensuring proper logging and dataset formats for Gemini models.
Important
Introduces
ConversationLoggerfor logging conversations and updatescreate-finetuning-data.tsto generate fine-tuning datasets for Gemini and OpenAI, with comprehensive tests and configuration updates.ConversationLoggerinConversationLogger.tsto log user messages, AI responses, and tool calls for fine-tuning datasets..roo-logsdirectory.create-finetuning-data.tsto process logs into Gemini and OpenAI fine-tuning datasets.--geminiand--openaiflags for format selection.ConversationLogger.spec.tsto validate logging and dataset generation.package.jsonto include new logging settings and dependencies.This description was created by
for 6a58d28. You can customize this summary. It will automatically update as commits are pushed.