Qualcomm AI Engine Direct - GA Static Olmo-1b

[DannyYuyang-quic]

Summary:

• e2e script for GA Static OLMo-1b
• perf 16a4w block quant token rate in kv mode: ~= 63 tokens/sec(SM8750)
• acc: PPL ~= (fp: 8.735 -> htp: 9.945) in wikitext dataset
• add model params file & model weight converter
• add workaround pass for LayerNorm without weight & unitest
• fix layernorm op builder & fix layernorm quant annotator

Test plan

python examples/qualcomm/oss_scripts/llama/llama.py -b build-android -s ${SERIAL_NUM} -m SM8750 --decoder_model olmo-1b --model_mode kv --max_seq_len 1024 --prompt "Simply put, the theory of relativity states that" --tasks wikitext --limit 1 --eval_perplexity

cc: @haowhsu-quic

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/14065

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure

As of commit c869d51 with merge base 2845fd3 ():

NEW FAILURE - The following job has failed:

• Lint / lintrunner / linux-job (gh)
  >>> Lint for examples/models/olmo/convert_weights.py:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[github-actions]

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

[DannyYuyang-quic]

Hi @cccclai,

This PR enables OLMo-1B from the GA list in the static version.
The perplexity looks promising, but there's a concern regarding its behavior as a base model(not instructed or fine-tuned). Specifically, it does well for the first ~20 tokens, but after that, it tends to repeat the prompt

model_id = "allenai/OLMo-1B-hf"
from transformers import AutoTokenizer, AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained(model_id).eval()
hf_tokenizer = AutoTokenizer.from_pretrained(model_id)
inputs = hf_tokenizer(prompt, return_tensors="pt")
output_ids = model.generate(
      **inputs,
      max_length=1024,
      eos_token_id=hf_tokenizer.eos_token_id,
      do_sample=False,
)
print(hf_tokenizer.decode(output_ids[0], skip_special_tokens=True))

Output:

Simply put, the theory of relativity states that the speed of light is the same in all directions.
The speed of light is the same in all directions.
The speed of light is the same in all directions. The speed of light is the same in all directions.
The speed of light is the same in all directions. 
....
....

Due to this repetitive behavior, I’ve created this as a draft PR.
I’d like to check: do you think it’s appropriate to enable this kind of model?

[cccclai]

I think it is worth enabling it first and improving the accuracy later. If the accuracy isn't great, maybe let's have the default recipe to be 8w weight for now, just so users can get reasonable results from it.

[facebook-github-bot]

@cccclai has imported this pull request. If you are a Meta employee, you can view this in D81938220.

[cccclai]

Also did you apply spinquant and seqmse with this model?

[cccclai]

cc: @rohan if you have bandwidth on this

[haowhsu-quic]

Hi @cccclai, I think the FP model itself already keeps repeating. Could we switch to olmo2-1B-instruct? maybe it would perform better.

[cccclai]

Hi @cccclai, I think the FP model itself already keeps repeating. Could we switch to olmo2-1B-instruct? maybe it would perform better.

I see, does this model work better? https://huggingface.co/allenai/OLMo-2-0425-1B

[haowhsu-quic]

Hi @cccclai, I think the FP model itself already keeps repeating. Could we switch to olmo2-1B-instruct? maybe it would perform better.

I see, does this model work better? https://huggingface.co/allenai/OLMo-2-0425-1B

We'll try both and see which one is more feasible. I think instruct version is empirically better than vanilla one.

[github-actions]

Looks like this PR hasn't been updated in a while so we're going to go ahead and mark this as Stale.
Feel free to remove the Stale label if you feel this was a mistake.
If you are unable to remove the Stale label please contact a maintainer in order to do so.
If you want the bot to never mark this PR stale again, add the no-stale label.
Stale pull requests will automatically be closed after 30 days of inactivity.