Direct Softmax output, like get_logits?

[Pa-Sto]

Hey everybody,

I have seen in other huggingface implementation, that you can get the logits from the generate() function. Is there an easy way in the mlx package to do the same?

[awni]

Not yet but there is an open PR #806 to return the log probabilities from generate. Is that what you are looking for?

[Pa-Sto]

Yes, it would be great to see n number of potential embeddings for every output, not only the most likley.

[fblissjr]

@awni once we have the logprobs, we should be able to better integrate projects such as outlines, correct? I made an attempt to get a fork of a fork of it working, but it was hacky at best.

Curious if there's others in the community that have a strong opinion on this - I've been thinking it might make more sense to make a super simple DSL around the logprobs and constrained decoding rather than a full fledged outlines integration, which should allow for both to work in the end.

Been bogged down in a new role and haven't been able to give as much attention here, but the WWDC yesterday validated all the itme and energy and passion spent on this project by @awni and team and the rest of the community here. Can clearly see the future and where it's headed for on-device and hybrid compute (and a new type of AI-native cloud).

[awni]

@awni once we have the logprobs, we should be able to better integrate projects such as outlines, correct? I made an attempt to get a fork of a fork of it working, but it was hacky at best.

I assume so. Right now the server HTTP API returns the top 10 log probs. The generate python function just returns the full vector of log probs. Let me know if you think anything needs to change or if that should work.