LLAMA_3.1 finetune?

[SicariusSicariiStuff]

Hi,
I read there are still some bugs with the new llama 3.1 (8b, i can't sell my house to train a 405b) because of its rope and other context window issues.

Can it be trained successfully in the current version of axolotl, or do I need to change something specific for it to work?

[tridungduong-unsw]

Same issue, I've tried to fine-tune LLAMA 3.1, but it doesn't work with logs as below:

ValueError: rope_scaling must be a dictionary with two fields, type and factor, got {'factor': 8.0, 'low_freq_factor': 1.0, 'high_freq_factor': 4.0, 'original_max_position_embeddings': 8192, 'rope_type': 'llama3'}

[leemgs]

From my experience, you need to update the existing Transformers to fine-tune Llama 3.1.

[winglian]

latest transformers should work