Skip to content

common: add config presets for falcon #14638

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 3 commits into from
Closed

common: add config presets for falcon #14638

Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
e99a47f
common: add config presets for falcon
0xs1d Jul 11, 2025
9127109
common: add config presets for smollm3
0xs1d Jul 12, 2025
c15d54a
Merge branch 'ggml-org:master' into pr-10392
0xs1d Jul 12, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions common/arg.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3423,5 +3423,21 @@ common_params_context common_params_parser_init(common_params & params, llama_ex
}
).set_examples({LLAMA_EXAMPLE_SERVER}));

add_opt(common_arg(
{"--fim-falcon-7b-spec"},
Copy link
Member

@ggerganov ggerganov Jul 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This identifier is not correct. It indicates:

  • A Fill-In-the-Middle (FIM) model
  • Speculative-decoding enabled (i.e. target + draft model)

You can call it --chat-falcon-7b

string_format("use quantized Falcon 7B model (note: can download weights from the internet)"),
[](common_params & params) {
params.model.hf_repo = "maddes8cht/tiiuae-falcon-7b-gguf";
Copy link
Member

@ggerganov ggerganov Jul 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Presets should only use models hosted on ggml-org.

Copy link
Author

@0xs1d 0xs1d Jul 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ahh ok got it. I see that falcon is not in ggml-org, I will add another model then

params.model.hf_file = "tiiuae-falcon-7b-Q6_K.gguf";
Copy link
Member

@ggerganov ggerganov Jul 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We use Q8_0 format in the presets.

params.n_gpu_layers = 40;
params.flash_attn = true;
params.n_ubatch = 1024;
params.n_batch = 1024;
params.n_ctx = 2048;
params.n_cache_reuse = 256;
}
).set_examples({LLAMA_EXAMPLE_SERVER}));


return ctx_arg;
}