👮 Agent adversary expectations

[hejny]

Check the result of LLM with LLM.

Strategy how to do it

1. Generate and check with same model and same prompt - compare results
2. Generate and check using the same rules
3. Try the same model with a different temperature
4. Try the same model with a different system message or prompt.
5. Try different models from the same supplier
6. Try different models (GPT 🆚 Claude)
7. Combine with anomaly detaction 🔴 Anomaly detection #40

• Make some syntax in Promptbook to describe simple and advanced cases in one prompt template
• Maybe use Personas 🤼 Personas #22 - two different Personas can be adversaries.
• Can it be (human) language agnostic?

Simple case

Write a poem about {topic}.

## Rules

- {rules}

-> {poem}

This is a poem about {topic}, did it follow the given rules?

## The poem

> {poem}

## Rules

- {rules}
- Fill in percentage from 0% to 100% of how the rules are followed.

Advanced case

Write a poem about {topic}

## Rules

- {rules}

-> {poem1}

Write a poem about {topic}

## Rules

- Be creative
- Rules

-> {poem2}

Write a poem about {topic}

## Rules

- Be precise
- {rules}

-> {poem3}

Pick a poem about {topic}

## Rules

- Pick the best poem of given 3
- {rules}

## Poem 1

> {poem1}

## Poem 2

> {poem2}

## Poem 3

> {poem3}

-> {poem}

[hejny]

What we tried

(Describe our experience here)

[hejny]

🔎 Existing solutions

(Do some research / discussion)

• Has anyone else had the same problem?
• Has a project come up with the solution
• Is there a research paper about it?
• Is there an article, video, podcast about it?
• Is this even a good idea?

[hejny]

https://youtube.com/shorts/C3uI06xOGSE?si=vjN722OqPiWS9HBO