Retry logic

[paulz]

Should retry logic for OpenAI be a part of CAT?

[paulz]

No, because CAT is agnostic to LLM in general, this retry are specific to OpenAI.
#21

[paulz]

@tkersey brought up today, that #24 is not high priority to merge because retry logic is not part of CAT.

[paulz]

Retry is needed because on example 7 of CAT example app we see that CI starts to fail:
#14

And on example 8 we can add retry decorator to our LLM call:
https://github.com/thisisartium/continuous-alignment-testing/pull/24/files#diff-9dabcf39fa43f95dc3febaed2addc2d98e1614f8e61af00c939d92582cbc8f8aR123-R129

[paulz]

Does retry logic belongs to "Alignment Testing"?

May be it's better suited to "Alignment Fixing" framework?

[paulz]

It is hard to box the 2 years experience from AI Engineering into a framework.

There are multiple pieces of discovery, one of them is retry logic.

We are learning that we need separate retry-able errors and that's hard as not all errors are known yet.
It is simple, but not always easy to do.
In our example we should an easy to reproduce retry for OpenAI ConnectionError:

continuous-alignment-testing/examples/team_recommender/tests/example_8_retry_network/test_retry_response_generation.py

Lines 122 to 128 in ec9c1c1

	@retry(
	max_attempts=4,
	exceptions=(openai.APIConnectionError,),
	initial_delay=10,
	backoff_factor=2.0,
	logger_name="openai.api",
	)

[paulz]

question: where we add that decorator?

explorations:

• at pytest test, e.g. test_allocations
• extract generate responses function and organize tests to use generated responses