Switch to Qwen3-Embedding-4B and setup sklearn.model_selection.GroupKFold to sold character mapping, overfitting problem

[rjurney]

No description provided.

[Copilot]

Pull Request Overview

This PR switches the model from multilingual-e5-base to Qwen3-Embedding-4B for embedding generation and addresses overfitting issues by implementing GroupKFold for character mapping. The changes include adding required dependencies and updating training configuration.

• Switch embedding model to Qwen3-Embedding-4B
• Add flash-attn dependency installation for the new model
• Increase batch size from 32 to 128 to leverage the new model's capabilities

Reviewed Changes

Copilot reviewed 2 out of 3 changed files in this pull request and generated no comments.

File	Description
pyproject.toml	Adds packaging and ninja dependencies required for Qwen3-Embedding-4B
README.md	Updates model reference, adds flash-attn installation instructions, and increases batch size

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[gemini-code-assist]

Summary of Changes

Hello @rjurney, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request primarily focuses on enhancing the project's embedding capabilities by migrating from the "intfloat/multilingual-e5-base" model to the more advanced "Qwen3-Embedding-4B". This transition involves updating installation instructions to include "flash-attn" and adding "ninja" as a core dependency, alongside an adjustment to the training batch size for improved efficiency. The overarching aim, as suggested by the PR title, is to tackle character mapping and overfitting challenges, although the specific "GroupKFold" implementation mentioned is not part of the current patch.

Highlights

• Embedding Model Update: Switched the primary embedding model from "intfloat/multilingual-e5-base" to "Qwen/Qwen3-Embedding-4B" in the training configuration.
• New Dependencies: Introduced "flash-attn" and "ninja" as new project dependencies to support the "Qwen3-Embedding-4B" model, with "flash-attn" requiring a direct "pip install" due to build specifics.
• Training Parameter Adjustment: Increased the training batch size from 32 to 128 in the "eridu train" command, likely to optimize performance with the new embedding model.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request switches the model to Qwen/Qwen3-Embedding-4B and updates dependencies accordingly. The changes in README.md and pyproject.toml seem appropriate for this switch. However, the pull request appears to be incomplete. The title mentions setting up sklearn.model_selection.GroupKFold to solve an overfitting problem (I assume 'sold' was a typo for 'solve'), but there are no code changes in this pull request that implement this functionality. Please either update the pull request title to only reflect the model switch, or add the missing commits that implement the GroupKFold logic before this can be merged.