(Distributed) Rule Caching #37
Description
Is your feature request related to a problem? Please describe.
Some LLM model providers do not allow a predefined seed and/or the particular implementations render models nondeterministic. This may occasionally cause variations in LLM-based Vulcan condition responses, leading to slightly different decisions. While it is best practice to replace such conditions where determinism is required with a set of computed rules, this may not always be easily achievable.
Furthermore, speed and cost are a factor when LLM-based conditions are invoked repeatedly. And in some distributed use-cases, entire decision subgraphs could be reused between instances of Vulcan.
Describe the solution you'd like
Implemented a distributed cache solution for rule evaluation and LLM invocations. Ideally support an external distributed cache for sharing and reuse among Vulcan instances.
Also investigate implementing the cache using embeddings and similarity searching to fuzzy-match on logically identical LLM sessions.
Describe alternatives you've considered
Attempted to specify seed with OpenAI models but discovered this is only available on some models, and may still not provide repeatable responses.
Additional context
References: