Major update: Add QLoRA & multi-resolution packing support #26
Conversation
- Fix issues in previous LoRA implementation - Implement custom LoRA interface based on PEFT library - Add proper injection, saving, loading, and unloading functions for LoRA adapters - Expose LoRA utility functions in package exports
- Implement multi-resolution packing for CogView4 to improve training efficiency - Add QLoRA support for both CogView and CogVideo models - Refactor trainers to fix training bugs and optimize computation pipeline - Update dataset utilities and fine-tuning base components This update significantly improves model training efficiency and flexibility.
|
We will follow up with a comparison of resize training, packing training, and QLoRA+packing training effects on the pixelart dataset. |
OleehyO
requested review from
zRzRzRzRzRzRzR and
zhangch9
and removed request for
zRzRzRzRzRzRzR
|
|
prompt：A black and white photograph of a stone carving on a wall, featuring a grotesque face with a beard and a crown-like structure above it. The carving is set within a stone archway, and the image has a vintage appearance with a faded and aged look. original: 
|
|
prompt: a beautiful photorealistic painting of cemetery urbex unfinished building building industrial architecture nature abandoned by thomas cole, nature extraterrestial tron forest darkacademia thermal vision futuristic tokyo, archdaily, wallpaper, highly detailed, trending on artstation. original: 
|
|
prompt: doom eternal, evangelion, game concept art, veins and worms, muscular, crustacean exoskeleton, chiroptera head, chiroptera ears, mecha, ferocious, fierce, hyperrealism, fine details, artstation, cgsociety, zbrush, no background original: 
|
|
TLDR; Based on pixelart, it can be observed that: 1) packing has a faster convergence speed than resize; 2) packing yields better generation quality than resize, reflected in the details and sharpness of the generated images (results from the resize approach tend to be blurry); 3) although qlora can reduce the hardware threshold for fine-tuning, it significantly impacts model quality on the pixelart dataset (resulting in grid-like images) |
Description
This PR introduces significant changes to enable:
Key improvements:
Notes
Multi-resolution packing currently supports only the cogview series.
For cogvideo, QLoRA offers limited benefits due to activation-dominated memory usage and is typically suitable only for batch size = 1 scenarios.
On the pixelart dataset, QLoRA fine-tuning may negatively impact final generation quality.
Changes include modifications to CogView4 attention processor in diffusers to support packed multi-resolution sequences (see this PR).