[DERCBOT-919] Add question condensing LLM and prompt template

[assouktim]

Use Case / Functionnal

Let the user configure in it's RAG chain the condensation prompt, this prompt is used to reformulate user's query using the latest messages in the dialog history including chatbot messaged. It's use to contextualize the user query.

For each bot using Rag Settings we can also now configure :

• Debugging of the rag messages that can be enabled for each call or disabled using debugEnabled
• the maximum number of retrieved document from the vector db using maxDocumentsRetrieved, earlier it was only possible using a specific VectorDB setting configuration in the studio.
• the number of message used from the dialog history to contextualise the user query using maxMessagesFromHistory

Technical changes

This pull request includes significant changes to the BotAdminService and related models, primarily focused on enhancing the RAG (Retrieval-Augmented Generation) configuration and sentence generation functionalities. The most important changes include the addition of new settings for question condensing and answering, updates to the data transfer objects (DTOs), and modifications to the validation and deletion processes.

Enhancements to RAG Configuration:

• Added new settings for questionCondensingLlmSetting, questionCondensingPrompt, questionAnsweringLlmSetting, and questionAnsweringPrompt in BotRAGConfigurationDTO to support more granular control over RAG processes. [1] [2] [3]
• Updated the deletion logic in BotAdminService to handle the new LLM settings for question condensing and answering. [1] [2]

Sentence Generation Updates:

• Introduced prompt as a new field in BotSentenceGenerationConfigurationDTO to allow customizable prompts for sentence generation. [1] [2] [3]
• Modified CompletionService to utilize the new prompt field for generating sentences. [1] [2]

Validation and Testing:

• Enhanced the validation logic in RAGValidationService to include checks for both question condensing and answering LLM settings. [1] [2] [3]
• Updated test cases in RAGServiceTest and RAGValidationServiceTest to reflect the new configuration fields and validation logic. [1] [2] [3] [4] [5]

These changes collectively improve the flexibility and robustness of the BotAdminService, particularly in handling complex RAG configurations and sentence generation scenarios.

Breaking changes

• tock_gen_ai_orchestrator_rag_debug_enabled is no longer used. Debug mode can only be activated directly from the RAG configuration.
• tock_gen_ai_orchestrator_dialog_number_messages is no longer used. The number of dialog messages to be taken can be specified directly from the RAG configuration.
• The VectorStoreSettingBase::k attribute is removed, as this information is now carried by the RAG configuration.

[Benvii]

Ok for me but some breaking changes need to be documented as we don't have any issue put them on the PR description the release script will be able to catch them.

Also we need to open an issue on tock helm repository to reference the environment variable changes.

Some conflicting files, rebase need to be done.

[Benvii]

LGTM, thanks 😀

[Benvii]

Sorry changes need to be done as the documentation as moved, can you check it please ?

[Benvii]

Doc is also present in french please update it, sorry I forgot it during the first review

[Benvii]

This doc doesn't exist at this location anymore I don't understand why it's not generating a conflict can you update the new file FR and EN ?

[Benvii]

Thanks for the changes

[github-actions]

Website is published: https://doc.tock.ai/tock/feature/dercbot-919/

[github-actions]

🚀 Cleanup completed for PR and associated branch: feature/dercbot-919