|
| 1 | +# How-To: Use attachments in prompts with Ragbits |
| 2 | + |
| 3 | +This guide will walk you through defining and using prompts in Ragbits that accept attachments as input. |
| 4 | +It covers handling single and multiple attachment inputs, incorporating conditionals in prompt templates based on the presence of attachments, and using such prompts with an LLM. |
| 5 | + |
| 6 | +Attachment types currently supported include standard image formats (such as JPEG, PNG) and PDF documents. |
| 7 | + |
| 8 | +## How to define a prompt with an attachment input |
| 9 | + |
| 10 | +The attachment is represented by the `prompt.Attachment` class. |
| 11 | +It can be initialized in multiple ways depending on the source of the file data. |
| 12 | +The class supports both binary data and URLs, with optional MIME type specification. |
| 13 | + |
| 14 | +```python |
| 15 | +from ragbits.core.prompt import Attachment |
| 16 | + |
| 17 | +file_bytes = Attachment(data=b"file_bytes") |
| 18 | +image_url = Attachment(url="http://image.jpg") |
| 19 | +pdf_url = Attachment(url="http://document.pdf") |
| 20 | +file_with_url_and_mime = Attachment(url="http://address.pl/file_with_no_extension", mime_type="jpeg") |
| 21 | +``` |
| 22 | + |
| 23 | +To define a prompt that takes an attachment as input, create a Pydantic model representing the input structure. |
| 24 | +The model should include a field for the attachment that holds an instance of `prompt.Attachment` class - its name does not matter. |
| 25 | + |
| 26 | +To pass multiple attachments, just define multiple fields of type `Attachment` or a single field that is a list of `Attachment` instances. |
| 27 | + |
| 28 | + |
| 29 | +```python |
| 30 | +import asyncio |
| 31 | +from pydantic import BaseModel |
| 32 | +from ragbits.core.prompt import Attachment, Prompt |
| 33 | +from ragbits.core.llms.litellm import LiteLLM |
| 34 | + |
| 35 | +class EmployeeOnboardingInput(BaseModel): |
| 36 | + """ |
| 37 | + Input model for employee onboarding files. |
| 38 | + """ |
| 39 | + headshot: Attachment |
| 40 | + contract: Attachment |
| 41 | + documents: list[Attachment] |
| 42 | + |
| 43 | + |
| 44 | +class EmployeeOnboardingPrompt(Prompt): |
| 45 | + """ |
| 46 | + A prompt to process employee onboarding files. |
| 47 | + """ |
| 48 | + |
| 49 | + user_prompt = "Review the employee onboarding files and provide feedback." |
| 50 | + |
| 51 | + |
| 52 | +async def main(): |
| 53 | + llm = LiteLLM("gpt-4o") |
| 54 | + |
| 55 | + headshot = Attachment(data=b"<your_photo_here>") |
| 56 | + contract = Attachment(data=b"<your_contract_here>") |
| 57 | + documents = [ |
| 58 | + Attachment(data=b"<your_document_1_here>"), |
| 59 | + Attachment(data=b"<your_document_2_here>"), |
| 60 | + ] |
| 61 | + prompt = EmployeeOnboardingPrompt( |
| 62 | + EmployeeOnboardingInput(headshot=headshot, contract=contract, documents=documents) |
| 63 | + ) |
| 64 | + response = await llm.generate(prompt) |
| 65 | + print(response) |
| 66 | + |
| 67 | + |
| 68 | +asyncio.run(main()) |
| 69 | +``` |
| 70 | + |
| 71 | +## Using conditionals in templates |
| 72 | + |
| 73 | +Sometimes, you may want to modify the prompt based on whether an attachment is provided. Jinja conditionals can help achieve this. |
| 74 | + |
| 75 | +```python |
| 76 | +import asyncio |
| 77 | +from pydantic import BaseModel |
| 78 | +from ragbits.core.prompt import Attachment, Prompt |
| 79 | +from ragbits.core.llms.litellm import LiteLLM |
| 80 | + |
| 81 | +class QuestionWithOptionalPhotoInput(BaseModel): |
| 82 | + """ |
| 83 | + Input model that optionally includes a photo. |
| 84 | + """ |
| 85 | + question: str |
| 86 | + reference_photo: Attachment | None = None |
| 87 | + |
| 88 | + |
| 89 | +class QuestionWithPhotoPrompt(Prompt[QuestionWithOptionalPhotoInput]): |
| 90 | + """ |
| 91 | + A prompt that considers whether a photo is provided. |
| 92 | + """ |
| 93 | + |
| 94 | + system_prompt = """ |
| 95 | + You are a knowledgeable assistant providing detailed answers. |
| 96 | + If a photo is provided, use it as a reference for your response. |
| 97 | + """ |
| 98 | + |
| 99 | + user_prompt = """ |
| 100 | + User asked: {{ question }} |
| 101 | + {% if reference_photo %} |
| 102 | + Here is a reference photo: {{ reference_photo }} |
| 103 | + {% else %} |
| 104 | + No photo was provided. |
| 105 | + {% endif %} |
| 106 | + """ |
| 107 | + |
| 108 | + |
| 109 | +async def main(): |
| 110 | + llm = LiteLLM("gpt-4o") |
| 111 | + input_with_photo = QuestionWithOptionalPhotoInput( |
| 112 | + question="What animal do you see in this photo?", reference_photo=Attachment(data=b"<your_photo_here>") |
| 113 | + ) |
| 114 | + input_without_photo = QuestionWithOptionalPhotoInput(question="What is the capital of France?") |
| 115 | + |
| 116 | + print(await llm.generate(QuestionWithPhotoPrompt(input_with_photo))) |
| 117 | + print(await llm.generate(QuestionWithPhotoPrompt(input_without_photo))) |
| 118 | + |
| 119 | + |
| 120 | +asyncio.run(main()) |
| 121 | +``` |
0 commit comments