Skip to content

Add LLaDA 8b Diffusion model #14771

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 12 commits into from
Jul 31, 2025
Merged

Add LLaDA 8b Diffusion model #14771

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
bef6c2d
Add support for Llada-8b: diffusion model
am17an Jul 19, 2025
0fa8b86
Add README
am17an Jul 19, 2025
267a09d
Fix README and convert_hf_to_gguf
am17an Jul 20, 2025
812bc38
convert_hf_to_gguf.py: address review comments
am17an Jul 22, 2025
6bb0093
Make everything in a single example
am17an Jul 26, 2025
3e7efcb
Remove model-specific sampling
am17an Jul 28, 2025
a50547c
Remove unused argmax
am17an Jul 31, 2025
e864a49
Remove braced initializers, improve README.md a bit
am17an Jul 31, 2025
9691f4e
Add diffusion specific gguf params in set_vocab, remove setting rope_…
am17an Jul 31, 2025
57201cc
Remove adding the mask token
am17an Jul 31, 2025
a326b13
Move add_add_bos_token to set_vocab
am17an Jul 31, 2025
ac3f91f
use add_bool in gguf_writer.py
am17an Jul 31, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions common/common.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -283,8 +283,8 @@ struct common_params {

struct common_params_sampling sampling;
struct common_params_speculative speculative;
struct common_params_vocoder vocoder;
struct common_params_diffusion diffusion;
struct common_params_vocoder vocoder;
struct common_params_diffusion diffusion;

struct common_params_model model;

Expand Down
12 changes: 3 additions & 9 deletions convert_hf_to_gguf.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2949,9 +2949,6 @@ def get_vocab_base(self) -> tuple[list[str], list[int], str]:
def set_vocab(self):
self._set_vocab_gpt2()

self.gguf_writer.add_add_bos_token(True)
self.gguf_writer.add_diffusion_shift_logits(False)

def set_gguf_parameters(self):
super().set_gguf_parameters()
self._try_set_pooling_type()
Expand Down Expand Up @@ -2979,13 +2976,10 @@ def set_gguf_parameters(self):

# LLaDA models use non-causal attention for diffusion, similar to Dream
self.gguf_writer.add_causal_attention(False)
# Handle RoPE scaling similar to LlamaModel and Dream

# Add LLaDA-specific parameters
mask_token_id = self.hparams.get("mask_token_id")

if mask_token_id is not None:
self.gguf_writer.add_mask_token_id(mask_token_id)
# LLaDA specific parameters
self.gguf_writer.add_add_bos_token(True)
self.gguf_writer.add_diffusion_shift_logits(False)

@staticmethod
def permute(weights: Tensor, n_head: int, n_head_kv: int | None):
Expand Down
Loading