AI Application Journey

AI Applications using LLM have to deal with non-deterministic behavior.

First we write Proof-Of-Concept to validate that AI can be useful. Then we usually see some inconsistent behavior of LLM. We capture that in our automated test. Because LLM output is different each time we ran test we are exposing difficult cases for LLM by running test multiple times. And we observe that test passes certain percentage of runs.

Example

Now we have, for example test that passes 70%. Since we have a test our code can detect the failure and retry in production. At the same time we want to iterate on the prompt and try increase success rate beyond 70%.