Skip to content

Add Classification SFT recipe#153

Open
yi-fireworks wants to merge 6 commits intomainfrom
yi/classification-sft
Open

Add Classification SFT recipe#153
yi-fireworks wants to merge 6 commits intomainfrom
yi/classification-sft

Conversation

@yi-fireworks
Copy link

@yi-fireworks yi-fireworks commented Dec 15, 2025

Add Classification SFT recipe
This PR introduces a new recipe in learn/finetuning/classification-sft that demonstrates a complete workflow for Supervised Fine-Tuning (SFT) on Fireworks AI.

Minimal Steps/Scripts:

  • End-to-end classification task: Classifying support tickets into billing/hardware/software.
  • Data Generation: Script to generate synthetic training (90%) and validation (10%) datasets.
  • Robust Training Script: Handles dataset uploads, training configuration, and auto-cleanup of previous runs.
  • Validation Integration: Configures the SFT job to use a validation set for tracking loss metrics.
  • Deployment Best Practices: Includes updated deployment commands with min-replica-count 1 for immediate warm start.
  • Post-Training Evaluation: Includes calc_metrics.py to calculate one-vs-rest Precision, Recall, and F1 scores on the validation set.
  • Cleanup Utility: Provides a robust cleanup.py script to tear down datasets, models, and deployments.

The recipe is structured with a centralized config.py for easy customization and includes a comprehensive README.md guiding users through the entire process. This serves as a "Quick Start" template for users looking to fine-tune classifiers on the platform.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be better to use BuildSDK instead of calling the commands.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc: @dphuang2

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this the best way to do it or should we be using eval protocol? cc: @benjibc

@@ -0,0 +1,91 @@
import json
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It may be better to put all this code into an ipython notebook that can be structured as an interactive tutorial.

@yi-fireworks yi-fireworks force-pushed the yi/classification-sft branch 2 times, most recently from 6917a19 to e345469 Compare December 16, 2025 19:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

Comments