Add inference snippets for image-text-to-text

[Wauplin]

This PR adds inference snippets for image-text-to-text models, say meta-llama/Llama-3.2-11B-Vision-Instruct for example 😄

I've tested all three examples locally and they work as expected :)

[julien-c]

Suggested change

	process.stdout.write(chunk.choices[0]?.delta?.content || "");
	process.stdout.write(chunk.choices[0]?.delta?.content);

[mishig25]

maybe good to show an example with pillow.Image.open and use that image's base64 str representation so that users can get an example where they can load local images

from PIL import Image
import requests
from io import BytesIO
import base64

image_url = "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
response = requests.get(image_url)

image = Image.open(BytesIO(response.content))

# Convert the image to a byte array in PNG format
buffered = BytesIO()
image.save(buffered, format="PNG")

# Encode this byte array to base64
img_base64 = base64.b64encode(buffered.getvalue())

# Print the base64 string
print(img_base64.decode())

maybe the snippet would become too long. I will let you decide

[julien-c]

hmm i'd say this would complicate a bit too much (no strong opinion though)

Note that this is for remote inference not local usage

[mishig25]

Note that this is for remote inference not local usage

yes, I meant more like: remote inference using local img file (otherwise, to use the snippet, user needs to upload their image and get its url)

[Wauplin]

do you know if it's possible to have several snippets by returning a list, same as for code snippets ?

[mishig25]

do you know if it's possible to have several snippets by returning a list, same as for code snippets ?

for inference snippet, not possible right now. So suggest that:

1. we merge this PR as it is with only image url example
2. mayeb unify on moon-side to have inference snippet to be able to have a list like code snippets. If so, we can re-iterate and add an example with a local image

[Wauplin]

@coyotte508 @mishig25 thanks for the feedback. I addressed above the comment to use "conversational" tag. Otherwise let's merge this and come back to it when multiple inference snippets can be provided. Can I have a final review before merging?

[mishig25]

nit, so in playground, we use snippet like this to match it as much as possible to OAI format/spec:

from huggingface_hub import InferenceClient

client = InferenceClient(api_key="YOUR_HF_TOKEN")

messages = [
	{ "role": "user", "content": "Tell me a story" }
]

output = client.chat.completions.create(
        model="mistralai/Mistral-7B-Instruct-v0.3", 
	messages=messages, 
	stream=True, 
	temperature=0.5,
	max_tokens=1024,
	top_p=0.7
)

the specific changes are:

1. use api_key rather than token
2. declare model inside completions.create rather than InferenceClient(

I will let you decide

[mishig25]

this comment maybe applies to text conv snippet as well

[mishig25]

if we decide to change it, lets handle in subseq PR

[Wauplin]

I'll use the same convention then. I've addressed it in e4c6cba for both the text-generation and text-image-to-text snippets.

[mishig25]

github is showing that model is indendted wrongly?

[Wauplin]

github is showing that model is indendted wrongly?

Well well well, looks like it yes. Addressed in 0f8452c

[mishig25]

lgtm! again

[Wauplin]

Thanks! Sorry about the back and forth 😬

[mishig25]

trigerred https://github.com/huggingface/huggingface.js/actions/runs/11107930206 so that we can get it in moon