AutoFP8 to llmcompressor migration for FP8 quantization

[a-ys]

Description

Migrates existing FP8 quantization functionality to llmcompressor from AutoFP8. The existing quantization recipe is fp8 weights and activation quantization, with static activation scales. Uses cnn-dailymail for calibration, defaulting to 512 samples at 2048 seq len.

Installed the previous version of llm-compressor (0.3.1) due to an issue with compatibility between llm-compressor (0.4.0) and transformers 2.5.2, which is the current version in djl-serving.

TODOs

• Update CI tests in lmi-distro to verify these changes.
• Remove AutoFP8 dependency from container.
• Future feature support (prioritization tbd)
  • MoE model quantization. (May run with exiting code, but will not ignore the correct layers)
  • KV cache quantization with static scales.
  • Calibration dataset selection
  • AWQ migration (not yet supported in llmcompressor)

Type of change

• New feature (non-breaking change which adds functionality)

Checklist:

• Please add the link of Integration Tests Executor run with related tests.
• Have you manually built the docker image and verify the change?
• Have you run related tests? Check how to set up the test environment here; One example would be pytest tests.py -k "TestCorrectnessLmiDist" -m "lmi_dist"
• Have you added tests that prove your fix is effective or that this feature works?
• Has code been commented, particularly in hard-to-understand areas?
• Have you made corresponding changes to the documentation?

Feature/Issue validation/testing

• Tested quantization of tinyllama with llmcompressor and serving with v14 v2 preview container through Neo workflow.

[siddvenk]

does this work for both lmi-dist (vllm 0.6.3.post1) and vanilla vllm (0.7.0)?

[siddvenk]

merging this for now - lgtm. let's validate this again with the vllm update for both lmi-dist/vllm