Add few-shot prompting with RAG and LLM pattern

[ArlindNocaj]

Is your feature request related to a problem? Please describe.
Fine-tuning and customizing blueprints requires a lot of effort. Few-Shot prompting could allow to improve results based on historic documents.

Describe the solution you'd like
Add few-shot prompting approach in which we utilize a set of historic document/json groundtruth pairs for learning on the fly how to perform the extraction.

Main Tasks:

• implement either knowledge_base or other similarity function to identify a most similar document from a corpus

  • a) S3 vector store with normal embeddings, or multi-modal embeddings (or multi-vector embeddings like ColQwen)
  • b) Bedrock Knowledge base
  • c) FAISS in memory database
  • d) simple tf-idf metrics using the Textract or BDA standard output results

• Implement new Pattern 4: Few-shot context engineering with LLM

• When extracting information search for other relevant documents and take the top3 for few-shot prompting

• Run Image to JSON using structured output of LLM, e.g. Nova model with constrained decoding

• Add the top3 docs to the context of the inference.

• We can also allow a hook in the lambda to have custom way of identifying relevant documents.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Fine-tuning approach like: Pattern 3, or https://github.com/aws-samples/sample-for-multi-modal-document-to-json-with-sagemaker-ai

Additional context
Add any other context or screenshots about the feature request here.

[rstrahan]

@ArlindNocaj Do you have any data or examples that demonstrate the efficacy of this approach, or specific use cases that would raise the priority of implementing it?

[ArlindNocaj]

I have at various feedback from people who are following the few-shot prompting approach.

Feedback 1: Has been doing various PoCs and few-shot prompting with RAG seemed to provide quiet good results. Case: Purchase Order processing, improved results to 95%.

Feedback 2: Another Customer told me that they also already implemented it (12 months ago already), and that the great thing about the approach is that if you have a document where you want to improve you can just add an annotated sample with direct impact of result improvement, so easy mechanisms to add feedback to the system.

The main use case is basically where you have high variation of document types, e.g. 100 of vendors of invoices, purchase order or others.

Yes you can always do fine-tuning, but that requires more effort and cannot just be adjusted as needed. E.g. if a single document or vendor is not working in few-shot with RAG you can just add a sample of this vendor, and if you provide the sample results for that vendor most likely will improve.

For fine-tuning much more effort would be required when adding a specific vendor only, e.g. to retrain retest etc.

Note that if normal zero-shot prompting is a thing, than few-shot prompting is just the next level of that!

[rstrahan]

Feedback 2: Another Customer told me that they also already implemented it (12 months ago already), and that the great thing about the approach is that if you have a document where you want to improve you can just add an annotated sample with direct impact of result improvement, so easy mechanisms to add feedback to the system.
Note that if normal zero-shot prompting is a thing, than few-shot prompting is just the next level of that!

Yeah, but to be clear, we already support (multi-modal) few-shot prompting for both classification and extraction, in pattern 2.
The distinction here is that we're proposing a dynamic few shot mechanism, where the examples are not static but instead dynamically selected from a curated set of samples, to be relevant to the document being processed, right?

So I'm not asking about the efficacy of few shot prompting in general, but I am asking if we have evidence of efficacy that dynamic few shot mechanism work well, and more clarity on when they work well.. ie in what circumstances would we recommend this feature be used?

[ArlindNocaj]

Absolutely!
Having just a few hard coded examples is hard to scale for 100 vendors.
Its much more scalable and customizable to allow for a dynamic process to get the few-shot samples.

Some examples would be:

• vendor might already be known and we pull other documents in that direction
• or more general similarities purely based on the document content and metadata would allow for a very powerfull mechanism. We would sample top k relevant documents, that are different enough to provide good few-shot sample coverage.

[rstrahan]

Hi @ArlindNocaj - Can you update on the status?