-
+1. Run `azd auth login` if you have not logged in recently.
+2. Start the server:
-Why do we need to break up the PDFs into chunks when Azure Cognitive Search supports searching large documents?
+You can only run a development server locally **after** having successfully run the `azd up` command. If you haven't yet, follow the [deploying](#deploying) steps above. -Chunking allows us to limit the amount of information we send to OpenAI due to token limits. By breaking up the content, it allows us to easily find potential chunks of text that we can inject into OpenAI. The method of chunking we use leverages a sliding window of text such that sentences that end one chunk will start the next. This allows us to reduce the chance of losing the context of the text. -
-
+ ```shell
+ ./app/start.ps1
+ ```
-### Troubleshooting
+ Linux/Mac:
-Here are the most common failure scenarios and solutions:
+ ```shell
+ ./app/start.sh
+ ```
-1. The subscription (`AZURE_SUBSCRIPTION_ID`) doesn't have access to the Azure OpenAI service. Please ensure `AZURE_SUBSCRIPTION_ID` matches the ID specified in the [OpenAI access request process](https://aka.ms/oai/access).
+ VS Code: Run the "VS Code Task: Start App" task.
-1. You're attempting to create resources in regions not enabled for Azure OpenAI (e.g. East US 2 instead of East US), or where the model you're trying to use isn't enabled. See [this matrix of model availability](https://aka.ms/oai/models).
+It's also possible to enable hotloading or the VS Code debugger.
+See more tips in [the local development guide](docs/localdev.md).
-1. You've exceeded a quota, most often number of resources per region. See [this article on quotas and limits](https://aka.ms/oai/quotas).
+## Using the app
-1. You're getting "same resource name not allowed" conflicts. That's likely because you've run the sample multiple times and deleted the resources you've been creating each time, but are forgetting to purge them. Azure keeps resources for 48 hours unless you purge from soft delete. See [this article on purging resources](https://learn.microsoft.com/azure/cognitive-services/manage-resources?tabs=azure-portal#purge-a-deleted-resource).
+- In Azure: navigate to the Azure WebApp deployed by azd. The URL is printed out when azd completes (as "Endpoint"), or you can find it in the Azure portal.
+- Running locally: navigate to 127.0.0.1:50505
+
+Once in the web app:
-1. You see `CERTIFICATE_VERIFY_FAILED` when the `prepdocs.py` script runs. That's typically due to incorrect SSL certificates setup on your machine. Try the suggestions in this [StackOverflow answer](https://stackoverflow.com/questions/35569042/ssl-certificate-verify-failed-with-python3/43855394#43855394).
+- Try different topics in chat or Q&A context. For chat, try follow up questions, clarifications, ask to simplify or elaborate on answer, etc.
+- Explore citations and sources
+- Click on "settings" to try different options, tweak prompts, etc.
+
+## Clean up
+
+To clean up all the resources created by this sample:
+
+1. Run `azd down`
+2. When asked if you are sure you want to continue, enter `y`
+3. When asked if you want to permanently delete the resources, enter `y`
+
+The resource group and all the resources will be deleted.
+
+## Guidance
+
+You can find extensive documentation in the [docs](docs/README.md) folder:
+
+- Deploying:
+ - [Troubleshooting deployment](docs/deploy_troubleshooting.md)
+ - [Debugging the app on App Service](docs/appservice.md)
+ - [Deploying with azd: deep dive and CI/CD](docs/azd.md)
+ - [Deploying with existing Azure resources](docs/deploy_existing.md)
+ - [Deploying from a free account](docs/deploy_lowcost.md)
+ - [Enabling optional features](docs/deploy_features.md)
+ - [All features](docs/deploy_features.md)
+ - [Login and access control](docs/login_and_acl.md)
+ - [GPT-4 Turbo with Vision](docs/gpt4v.md)
+ - [Reasoning](docs/reasoning.md)
+ - [Private endpoints](docs/deploy_private.md)
+ - [Sharing deployment environments](docs/sharing_environments.md)
+- [Local development](docs/localdev.md)
+- [Customizing the app](docs/customization.md)
+- [Data ingestion](docs/data_ingestion.md)
+- [Evaluation](docs/evaluation.md)
+- [Safety evaluation](docs/safety_evaluation.md)
+- [Monitoring with Application Insights](docs/monitoring.md)
+- [Productionizing](docs/productionizing.md)
+- [Alternative RAG chat samples](docs/other_samples.md)
+
+### Resources
+
+- [📖 Docs: Get started using the chat with your data sample](https://learn.microsoft.com/azure/developer/python/get-started-app-chat-template?toc=%2Fazure%2Fdeveloper%2Fai%2Ftoc.json&bc=%2Fazure%2Fdeveloper%2Fai%2Fbreadcrumb%2Ftoc.json&tabs=github-codespaces)
+- [📖 Blog: Revolutionize your Enterprise Data with ChatGPT: Next-gen Apps w/ Azure OpenAI and AI Search](https://techcommunity.microsoft.com/blog/azure-ai-services-blog/revolutionize-your-enterprise-data-with-chatgpt-next-gen-apps-w-azure-openai-and/3762087)
+- [📖 Docs: Azure AI Search](https://learn.microsoft.com/azure/search/search-what-is-azure-search)
+- [📖 Docs: Azure OpenAI Service](https://learn.microsoft.com/azure/cognitive-services/openai/overview)
+- [📖 Docs: Comparing Azure OpenAI and OpenAI](https://learn.microsoft.com/azure/cognitive-services/openai/overview#comparing-azure-openai-and-openai/)
+- [📖 Blog: Access Control in Generative AI applications with Azure AI Search](https://techcommunity.microsoft.com/blog/azure-ai-services-blog/access-control-in-generative-ai-applications-with-azure-ai-search/3956408)
+- [📺 Talk: Quickly build and deploy OpenAI apps on Azure, infused with your own data](https://www.youtube.com/watch?v=j8i-OM5kwiY)
+- [📺 Talks: AI Chat App Hack series](https://www.youtube.com/playlist?list=PL5lwDBUC0ag6_dGZst5m3G72ewfwXLcXV)
+
+### Getting help
+
+This is a sample built to demonstrate the capabilities of modern Generative AI apps and how they can be built in Azure.
+For help with deploying this sample, please post in [GitHub Issues](/issues). If you're a Microsoft employee, you can also post in [our Teams channel](https://aka.ms/azai-python-help).
+
+This repository is supported by the maintainers, _not_ by Microsoft Support,
+so please use the support mechanisms described above, and we will do our best to help you out.
-1. After running `azd up` and visiting the website, you see a '404 Not Found' in the browser. Wait 10 minutes and try again, as it might be still starting up. Then try running `azd deploy` and wait again. If you still encounter errors with the deployed app, consult these [tips for debugging Flask app deployments](http://blog.pamelafox.org/2023/06/tips-for-debugging-flask-deployments-to.html)
-and file an issue if the error logs don't help you resolve the issue.
+### Note
+
+>Note: The PDF documents used in this demo contain information generated using a language model (Azure OpenAI Service). The information contained in these documents is only for demonstration purposes and does not reflect the opinions or beliefs of Microsoft. Microsoft makes no representations or warranties of any kind, express or implied, about the completeness, accuracy, reliability, suitability or availability with respect to the information contained in this document. All rights reserved to Microsoft.
diff --git a/SECURITY.md b/SECURITY.md
new file mode 100644
index 0000000000..388e9ad471
--- /dev/null
+++ b/SECURITY.md
@@ -0,0 +1,41 @@
+
+
+## Security
+
+Microsoft takes the security of our software products and services seriously, which includes all source code repositories managed through our GitHub organizations, which include [Microsoft](https://github.com/Microsoft), [Azure](https://github.com/Azure), [DotNet](https://github.com/dotnet), [AspNet](https://github.com/aspnet), [Xamarin](https://github.com/xamarin), and [our GitHub organizations](https://opensource.microsoft.com/).
+
+If you believe you have found a security vulnerability in any Microsoft-owned repository that meets [Microsoft's definition of a security vulnerability](How can we upload additional PDFs without redeploying everything?
+ Windows: -To upload more PDFs, put them in the data/ folder and run `./scripts/prepdocs.sh` or `./scripts/prepdocs.ps1`. To avoid reuploading existing docs, move them out of the data folder. You could also implement checks to see whats been uploaded before; our code doesn't yet have such checks. -{query_text}
Conversations:
" + msg_to_display.replace('\n', '
')} + # If retrieval mode includes vectors, compute an embedding for the query + vectors: list[VectorQuery] = [] + if use_vector_search: + vectors.append(await self.compute_text_embedding(query_text)) - def get_messages_from_history(self, system_prompt: str, model_id: str, history: Sequence[dict[str, str]], user_conv: str, few_shots = [], max_tokens: int = 4096) -> []: - message_builder = MessageBuilder(system_prompt, model_id) + results = await self.search( + top, + query_text, + search_index_filter, + vectors, + use_text_search, + use_vector_search, + use_semantic_ranker, + use_semantic_captions, + minimum_search_score, + minimum_reranker_score, + use_query_rewriting, + ) - # Add examples to show the chat what responses we want. It will try to mimic any responses and make sure they match the rules laid out in the system message. - for shot in few_shots: - message_builder.append_message(shot.get('role'), shot.get('content')) + # STEP 3: Generate a contextual and content specific answer using the search results and chat history + text_sources = self.get_sources_content(results, use_semantic_captions, use_image_citation=False) - user_content = user_conv - append_index = len(few_shots) + 1 + extra_info = ExtraInfo( + DataPoints(text=text_sources), + thoughts=[ + self.format_thought_step_for_chatcompletion( + title="Prompt to generate search query", + messages=query_messages, + overrides=overrides, + model=self.chatgpt_model, + deployment=self.chatgpt_deployment, + usage=chat_completion.usage, + reasoning_effort="low", + ), + ThoughtStep( + "Search using generated search query", + query_text, + { + "use_semantic_captions": use_semantic_captions, + "use_semantic_ranker": use_semantic_ranker, + "use_query_rewriting": use_query_rewriting, + "top": top, + "filter": search_index_filter, + "use_vector_search": use_vector_search, + "use_text_search": use_text_search, + }, + ), + ThoughtStep( + "Search results", + [result.serialize_for_results() for result in results], + ), + ], + ) + return extra_info - message_builder.append_message(self.USER, user_content, index=append_index) + async def run_agentic_retrieval_approach( + self, + messages: list[ChatCompletionMessageParam], + overrides: dict[str, Any], + auth_claims: dict[str, Any], + ): + minimum_reranker_score = overrides.get("minimum_reranker_score", 0) + search_index_filter = self.build_filter(overrides, auth_claims) + top = overrides.get("top", 3) + max_subqueries = overrides.get("max_subqueries", 10) + results_merge_strategy = overrides.get("results_merge_strategy", "interleaved") + # 50 is the amount of documents that the reranker can process per query + max_docs_for_reranker = max_subqueries * 50 + + response, results = await self.run_agentic_retrieval( + messages=messages, + agent_client=self.agent_client, + search_index_name=self.search_index_name, + top=top, + filter_add_on=search_index_filter, + minimum_reranker_score=minimum_reranker_score, + max_docs_for_reranker=max_docs_for_reranker, + results_merge_strategy=results_merge_strategy, + ) - for h in reversed(history[:-1]): - if h.get("bot"): - message_builder.append_message(self.ASSISTANT, h.get('bot'), index=append_index) - message_builder.append_message(self.USER, h.get('user'), index=append_index) - if message_builder.token_length > max_tokens: - break + text_sources = self.get_sources_content(results, use_semantic_captions=False, use_image_citation=False) - messages = message_builder.messages - return messages + extra_info = ExtraInfo( + DataPoints(text=text_sources), + thoughts=[ + ThoughtStep( + "Use agentic retrieval", + messages, + { + "reranker_threshold": minimum_reranker_score, + "max_docs_for_reranker": max_docs_for_reranker, + "results_merge_strategy": results_merge_strategy, + "filter": search_index_filter, + }, + ), + ThoughtStep( + f"Agentic retrieval results (top {top})", + [result.serialize_for_results() for result in results], + { + "query_plan": ( + [activity.as_dict() for activity in response.activity] if response.activity else None + ), + "model": self.agent_model, + "deployment": self.agent_deployment, + }, + ), + ], + ) + return extra_info diff --git a/app/backend/approaches/chatreadretrievereadvision.py b/app/backend/approaches/chatreadretrievereadvision.py new file mode 100644 index 0000000000..f8aaf3c37d --- /dev/null +++ b/app/backend/approaches/chatreadretrievereadvision.py @@ -0,0 +1,222 @@ +from collections.abc import Awaitable +from typing import Any, Callable, Optional, Union, cast + +from azure.search.documents.aio import SearchClient +from azure.storage.blob.aio import ContainerClient +from openai import AsyncOpenAI, AsyncStream +from openai.types.chat import ( + ChatCompletion, + ChatCompletionChunk, + ChatCompletionMessageParam, + ChatCompletionToolParam, +) + +from approaches.approach import DataPoints, ExtraInfo, ThoughtStep +from approaches.chatapproach import ChatApproach +from approaches.promptmanager import PromptManager +from core.authentication import AuthenticationHelper +from core.imageshelper import fetch_image + + +class ChatReadRetrieveReadVisionApproach(ChatApproach): + """ + A multi-step approach that first uses OpenAI to turn the user's question into a search query, + then uses Azure AI Search to retrieve relevant documents, and then sends the conversation history, + original user question, and search results to OpenAI to generate a response. + """ + + def __init__( + self, + *, + search_client: SearchClient, + blob_container_client: ContainerClient, + openai_client: AsyncOpenAI, + auth_helper: AuthenticationHelper, + chatgpt_model: str, + chatgpt_deployment: Optional[str], # Not needed for non-Azure OpenAI + gpt4v_deployment: Optional[str], # Not needed for non-Azure OpenAI + gpt4v_model: str, + embedding_deployment: Optional[str], # Not needed for non-Azure OpenAI or for retrieval_mode="text" + embedding_model: str, + embedding_dimensions: int, + embedding_field: str, + sourcepage_field: str, + content_field: str, + query_language: str, + query_speller: str, + vision_endpoint: str, + vision_token_provider: Callable[[], Awaitable[str]], + prompt_manager: PromptManager, + ): + self.search_client = search_client + self.blob_container_client = blob_container_client + self.openai_client = openai_client + self.auth_helper = auth_helper + self.chatgpt_model = chatgpt_model + self.chatgpt_deployment = chatgpt_deployment + self.gpt4v_deployment = gpt4v_deployment + self.gpt4v_model = gpt4v_model + self.embedding_deployment = embedding_deployment + self.embedding_model = embedding_model + self.embedding_dimensions = embedding_dimensions + self.embedding_field = embedding_field + self.sourcepage_field = sourcepage_field + self.content_field = content_field + self.query_language = query_language + self.query_speller = query_speller + self.vision_endpoint = vision_endpoint + self.vision_token_provider = vision_token_provider + self.prompt_manager = prompt_manager + self.query_rewrite_prompt = self.prompt_manager.load_prompt("chat_query_rewrite.prompty") + self.query_rewrite_tools = self.prompt_manager.load_tools("chat_query_rewrite_tools.json") + self.answer_prompt = self.prompt_manager.load_prompt("chat_answer_question_vision.prompty") + # Currently disabled due to issues with rendering token usage in the UI + self.include_token_usage = False + + async def run_until_final_call( + self, + messages: list[ChatCompletionMessageParam], + overrides: dict[str, Any], + auth_claims: dict[str, Any], + should_stream: bool = False, + ) -> tuple[ExtraInfo, Union[Awaitable[ChatCompletion], Awaitable[AsyncStream[ChatCompletionChunk]]]]: + seed = overrides.get("seed", None) + use_text_search = overrides.get("retrieval_mode") in ["text", "hybrid", None] + use_vector_search = overrides.get("retrieval_mode") in ["vectors", "hybrid", None] + use_semantic_ranker = True if overrides.get("semantic_ranker") else False + use_query_rewriting = True if overrides.get("query_rewriting") else False + use_semantic_captions = True if overrides.get("semantic_captions") else False + top = overrides.get("top", 3) + minimum_search_score = overrides.get("minimum_search_score", 0.0) + minimum_reranker_score = overrides.get("minimum_reranker_score", 0.0) + filter = self.build_filter(overrides, auth_claims) + + vector_fields = overrides.get("vector_fields", "textAndImageEmbeddings") + send_text_to_gptvision = overrides.get("gpt4v_input") in ["textAndImages", "texts", None] + send_images_to_gptvision = overrides.get("gpt4v_input") in ["textAndImages", "images", None] + + original_user_query = messages[-1]["content"] + if not isinstance(original_user_query, str): + raise ValueError("The most recent message content must be a string.") + + # Use prompty to prepare the query prompt + query_messages = self.prompt_manager.render_prompt( + self.query_rewrite_prompt, {"user_query": original_user_query, "past_messages": messages[:-1]} + ) + tools: list[ChatCompletionToolParam] = self.query_rewrite_tools + + # STEP 1: Generate an optimized keyword search query based on the chat history and the last question + chat_completion: ChatCompletion = await self.openai_client.chat.completions.create( + messages=query_messages, + # Azure OpenAI takes the deployment name as the model name + model=self.chatgpt_deployment if self.chatgpt_deployment else self.chatgpt_model, + temperature=0.0, # Minimize creativity for search query generation + max_tokens=100, + n=1, + tools=tools, + seed=seed, + ) + + query_text = self.get_search_query(chat_completion, original_user_query) + + # STEP 2: Retrieve relevant documents from the search index with the GPT optimized query + + # If retrieval mode includes vectors, compute an embedding for the query + vectors = [] + if use_vector_search: + if vector_fields == "textEmbeddingOnly" or vector_fields == "textAndImageEmbeddings": + vectors.append(await self.compute_text_embedding(query_text)) + if vector_fields == "imageEmbeddingOnly" or vector_fields == "textAndImageEmbeddings": + vectors.append(await self.compute_image_embedding(query_text)) + + results = await self.search( + top, + query_text, + filter, + vectors, + use_text_search, + use_vector_search, + use_semantic_ranker, + use_semantic_captions, + minimum_search_score, + minimum_reranker_score, + use_query_rewriting, + ) + + # STEP 3: Generate a contextual and content specific answer using the search results and chat history + text_sources = [] + image_sources = [] + if send_text_to_gptvision: + text_sources = self.get_sources_content(results, use_semantic_captions, use_image_citation=True) + if send_images_to_gptvision: + for result in results: + url = await fetch_image(self.blob_container_client, result) + if url: + image_sources.append(url) + + messages = self.prompt_manager.render_prompt( + self.answer_prompt, + self.get_system_prompt_variables(overrides.get("prompt_template")) + | { + "include_follow_up_questions": bool(overrides.get("suggest_followup_questions")), + "past_messages": messages[:-1], + "user_query": original_user_query, + "text_sources": text_sources, + "image_sources": image_sources, + }, + ) + + extra_info = ExtraInfo( + DataPoints(text=text_sources, images=image_sources), + [ + ThoughtStep( + "Prompt to generate search query", + query_messages, + ( + {"model": self.chatgpt_model, "deployment": self.chatgpt_deployment} + if self.chatgpt_deployment + else {"model": self.chatgpt_model} + ), + ), + ThoughtStep( + "Search using generated search query", + query_text, + { + "use_semantic_captions": use_semantic_captions, + "use_semantic_ranker": use_semantic_ranker, + "use_query_rewriting": use_query_rewriting, + "top": top, + "filter": filter, + "vector_fields": vector_fields, + "use_text_search": use_text_search, + }, + ), + ThoughtStep( + "Search results", + [result.serialize_for_results() for result in results], + ), + ThoughtStep( + "Prompt to generate answer", + messages, + ( + {"model": self.gpt4v_model, "deployment": self.gpt4v_deployment} + if self.gpt4v_deployment + else {"model": self.gpt4v_model} + ), + ), + ], + ) + + chat_coroutine = cast( + Union[Awaitable[ChatCompletion], Awaitable[AsyncStream[ChatCompletionChunk]]], + self.openai_client.chat.completions.create( + model=self.gpt4v_deployment if self.gpt4v_deployment else self.gpt4v_model, + messages=messages, + temperature=overrides.get("temperature", 0.3), + max_tokens=1024, + n=1, + stream=should_stream, + seed=seed, + ), + ) + return (extra_info, chat_coroutine) diff --git a/app/backend/approaches/promptmanager.py b/app/backend/approaches/promptmanager.py new file mode 100644 index 0000000000..82941b4176 --- /dev/null +++ b/app/backend/approaches/promptmanager.py @@ -0,0 +1,31 @@ +import json +import pathlib + +import prompty +from openai.types.chat import ChatCompletionMessageParam + + +class PromptManager: + + def load_prompt(self, path: str): + raise NotImplementedError + + def load_tools(self, path: str): + raise NotImplementedError + + def render_prompt(self, prompt, data) -> list[ChatCompletionMessageParam]: + raise NotImplementedError + + +class PromptyManager(PromptManager): + + PROMPTS_DIRECTORY = pathlib.Path(__file__).parent / "prompts" + + def load_prompt(self, path: str): + return prompty.load(self.PROMPTS_DIRECTORY / path) + + def load_tools(self, path: str): + return json.loads(open(self.PROMPTS_DIRECTORY / path).read()) + + def render_prompt(self, prompt, data) -> list[ChatCompletionMessageParam]: + return prompty.prepare(prompt, data) diff --git a/app/backend/approaches/prompts/ask_answer_question.prompty b/app/backend/approaches/prompts/ask_answer_question.prompty new file mode 100644 index 0000000000..7ff73d232f --- /dev/null +++ b/app/backend/approaches/prompts/ask_answer_question.prompty @@ -0,0 +1,42 @@ +--- +name: Ask +description: Answer a single question (with no chat history) using solely text sources. +model: + api: chat +sample: + user_query: What does a product manager do? + text_sources: + - "role_library.pdf#page=29: The Manager of Product Management will collaborate with internal teams, such as engineering, sales, marketing, and finance, as well as external partners, suppliers, and customers to ensure successful product execution. Responsibilities: · Lead the product management team and provide guidance on product strategy, design, development, and launch. · Develop and implement product life-cycle management processes. · Monitor and analyze industry trends to identify opportunities for new products. · Develop product marketing plans and go-to-market strategies. · Research customer needs and develop customer-centric product roadmaps. · Collaborate with internal teams to ensure product execution and successful launch. · Develop pricing strategies and cost models. · Oversee product portfolio and performance metrics. · Manage product development budget. · Analyze product performance and customer feedback to identify areas for improvement. Qualifications: · Bachelor's degree in business, engineering, or a related field. · At least 5 years of experience in product management. · Proven track record of successful product launches." + - "role_library.pdf#page=23: Company: Contoso Electronics Location: Anywhere Job Type: Full-Time Salary: Competitive, commensurate with experience Job Summary: The Senior Manager of Product Management will be responsible for leading the product management team at Contoso Electronics. This role includes developing strategies, plans and objectives for the product management team and managing the day-to-day operations. The Senior Manager of Product Management will be responsible for the successful launch of new products and the optimization of existing products. Responsibilities: · Develop and implement product management strategies, plans and objectives to maximize team performance. · Analyze competitive landscape and market trends to develop product strategies. · Lead the product management team in the development of product plans, roadmaps and launch plans. · Monitor the performance of product management team, analyze results and implement corrective action as needed. · Manage the product lifecycle, including product development, launch, and end of life. · Ensure product features and benefits meet customer requirements. · Establish and maintain relationships with key customers, partners, and vendors." + - "role_library.pdf#page=28: · 7+ years of experience in research and development in the electronics sector. · Proven track record of successfully designing, testing, and optimizing products. · Experience leading a team of researchers and engineers. · Excellent problem-solving and analytical skills. · Ability to work in a fast-paced environment and meet tight deadlines.· Knowledge of industry trends, technologies, and regulations. · Excellent communication and presentation skills. Manager of Product Management Job Title: Manager of Product Management, Contoso Electronics Job Summary: The Manager of Product Management is responsible for overseeing the product management team, driving product development and marketing strategy for Contoso Electronics. This individual will be accountable for the successful launch of new products and the implementation of product life-cycle management processes. The Manager of Product Management will collaborate with internal teams, such as engineering, sales, marketing, and finance, as well as external partners, suppliers, and customers to ensure successful product execution." +--- +system: +{% if override_prompt %} +{{ override_prompt }} +{% else %} +You are an intelligent assistant helping Contoso Inc employees with their healthcare plan questions and employee handbook questions. +Use 'you' to refer to the individual asking the questions even if they ask with 'I'. +Answer the following question using only the data provided in the sources below. +Each source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. +If you cannot answer using the sources below, say you don't know. Use below example to answer +{{ injected_prompt }} +{% endif %} + +user: +What is the deductible for the employee plan for a visit to Overlake in Bellevue? + +Sources: +info1.txt: deductibles depend on whether you are in-network or out-of-network. In-network deductibles are $500 for employee and $1000 for family. Out-of-network deductibles are $1000 for employee and $2000 for family. +info2.pdf: Overlake is in-network for the employee plan. +info3.pdf: Overlake is the name of the area that includes a park and ride near Bellevue. +info4.pdf: In-network institutions include Overlake, Swedish and others in the region. + +assistant: +In-network deductibles are $500 for employee and $1000 for family [info1.txt] and Overlake is in-network for the employee plan [info2.pdf][info4.pdf]. + +user: +{{ user_query }} +Sources: +{% for text_source in text_sources %} +{{ text_source }} +{% endfor %} diff --git a/app/backend/approaches/prompts/ask_answer_question_vision.prompty b/app/backend/approaches/prompts/ask_answer_question_vision.prompty new file mode 100644 index 0000000000..25ab9656a7 --- /dev/null +++ b/app/backend/approaches/prompts/ask_answer_question_vision.prompty @@ -0,0 +1,31 @@ +--- +name: Ask with vision +description: Answer a single question (with no chat history) using both text and image sources. +model: + api: chat +--- +system: +{% if override_prompt %} +{{ override_prompt }} +{% else %} +You are an intelligent assistant helping analyze the Annual Financial Report of Contoso Ltd., The documents contain text, graphs, tables and images. +Each image source has the file name in the top left corner of the image with coordinates (10,10) pixels and is in the format SourceFileName:
{query_text}
Prompt:
" + '\n\n'.join([str(message) for message in messages])} + # If retrieval mode includes vectors, compute an embedding for the query + vectors: list[VectorQuery] = [] + if use_vector_search: + vectors.append(await self.compute_text_embedding(q)) + + results = await self.search( + top, + q, + filter, + vectors, + use_text_search, + use_vector_search, + use_semantic_ranker, + use_semantic_captions, + minimum_search_score, + minimum_reranker_score, + use_query_rewriting, + ) + + text_sources = self.get_sources_content(results, use_semantic_captions, use_image_citation=False) + + return ExtraInfo( + DataPoints(text=text_sources), + thoughts=[ + ThoughtStep( + "Search using user query", + q, + { + "use_semantic_captions": use_semantic_captions, + "use_semantic_ranker": use_semantic_ranker, + "use_query_rewriting": use_query_rewriting, + "top": top, + "filter": filter, + "use_vector_search": use_vector_search, + "use_text_search": use_text_search, + }, + ), + ThoughtStep( + "Search results", + [result.serialize_for_results() for result in results], + ), + ], + ) + + async def run_agentic_retrieval_approach( + self, + messages: list[ChatCompletionMessageParam], + overrides: dict[str, Any], + auth_claims: dict[str, Any], + ): + minimum_reranker_score = overrides.get("minimum_reranker_score", 0) + search_index_filter = self.build_filter(overrides, auth_claims) + top = overrides.get("top", 3) + max_subqueries = overrides.get("max_subqueries", 10) + results_merge_strategy = overrides.get("results_merge_strategy", "interleaved") + # 50 is the amount of documents that the reranker can process per query + max_docs_for_reranker = max_subqueries * 50 + + response, results = await self.run_agentic_retrieval( + messages, + self.agent_client, + search_index_name=self.search_index_name, + top=top, + filter_add_on=search_index_filter, + minimum_reranker_score=minimum_reranker_score, + max_docs_for_reranker=max_docs_for_reranker, + results_merge_strategy=results_merge_strategy, + ) + + text_sources = self.get_sources_content(results, use_semantic_captions=False, use_image_citation=False) + + extra_info = ExtraInfo( + DataPoints(text=text_sources), + thoughts=[ + ThoughtStep( + "Use agentic retrieval", + messages, + { + "reranker_threshold": minimum_reranker_score, + "max_docs_for_reranker": max_docs_for_reranker, + "results_merge_strategy": results_merge_strategy, + "filter": search_index_filter, + }, + ), + ThoughtStep( + f"Agentic retrieval results (top {top})", + [result.serialize_for_results() for result in results], + { + "query_plan": ( + [activity.as_dict() for activity in response.activity] if response.activity else None + ), + "model": self.agent_model, + "deployment": self.agent_deployment, + }, + ), + ], + ) + return extra_info diff --git a/app/backend/approaches/retrievethenreadvision.py b/app/backend/approaches/retrievethenreadvision.py new file mode 100644 index 0000000000..a021537c52 --- /dev/null +++ b/app/backend/approaches/retrievethenreadvision.py @@ -0,0 +1,181 @@ +from collections.abc import Awaitable +from typing import Any, Callable, Optional + +from azure.search.documents.aio import SearchClient +from azure.storage.blob.aio import ContainerClient +from openai import AsyncOpenAI +from openai.types.chat import ( + ChatCompletionMessageParam, +) + +from approaches.approach import Approach, DataPoints, ExtraInfo, ThoughtStep +from approaches.promptmanager import PromptManager +from core.authentication import AuthenticationHelper +from core.imageshelper import fetch_image + + +class RetrieveThenReadVisionApproach(Approach): + """ + Simple retrieve-then-read implementation, using the AI Search and OpenAI APIs directly. It first retrieves + top documents including images from search, then constructs a prompt with them, and then uses OpenAI to generate an completion + (answer) with that prompt. + """ + + def __init__( + self, + *, + search_client: SearchClient, + blob_container_client: ContainerClient, + openai_client: AsyncOpenAI, + auth_helper: AuthenticationHelper, + gpt4v_deployment: Optional[str], + gpt4v_model: str, + embedding_deployment: Optional[str], # Not needed for non-Azure OpenAI or for retrieval_mode="text" + embedding_model: str, + embedding_dimensions: int, + embedding_field: str, + sourcepage_field: str, + content_field: str, + query_language: str, + query_speller: str, + vision_endpoint: str, + vision_token_provider: Callable[[], Awaitable[str]], + prompt_manager: PromptManager, + ): + self.search_client = search_client + self.blob_container_client = blob_container_client + self.openai_client = openai_client + self.auth_helper = auth_helper + self.embedding_model = embedding_model + self.embedding_deployment = embedding_deployment + self.embedding_dimensions = embedding_dimensions + self.embedding_field = embedding_field + self.sourcepage_field = sourcepage_field + self.content_field = content_field + self.gpt4v_deployment = gpt4v_deployment + self.gpt4v_model = gpt4v_model + self.query_language = query_language + self.query_speller = query_speller + self.vision_endpoint = vision_endpoint + self.vision_token_provider = vision_token_provider + self.prompt_manager = prompt_manager + self.answer_prompt = self.prompt_manager.load_prompt("ask_answer_question_vision.prompty") + # Currently disabled due to issues with rendering token usage in the UI + self.include_token_usage = False + + async def run( + self, + messages: list[ChatCompletionMessageParam], + session_state: Any = None, + context: dict[str, Any] = {}, + ) -> dict[str, Any]: + q = messages[-1]["content"] + if not isinstance(q, str): + raise ValueError("The most recent message content must be a string.") + + overrides = context.get("overrides", {}) + seed = overrides.get("seed", None) + auth_claims = context.get("auth_claims", {}) + use_text_search = overrides.get("retrieval_mode") in ["text", "hybrid", None] + use_vector_search = overrides.get("retrieval_mode") in ["vectors", "hybrid", None] + use_semantic_ranker = True if overrides.get("semantic_ranker") else False + use_query_rewriting = True if overrides.get("query_rewriting") else False + use_semantic_captions = True if overrides.get("semantic_captions") else False + top = overrides.get("top", 3) + minimum_search_score = overrides.get("minimum_search_score", 0.0) + minimum_reranker_score = overrides.get("minimum_reranker_score", 0.0) + filter = self.build_filter(overrides, auth_claims) + + vector_fields = overrides.get("vector_fields", "textAndImageEmbeddings") + send_text_to_gptvision = overrides.get("gpt4v_input") in ["textAndImages", "texts", None] + send_images_to_gptvision = overrides.get("gpt4v_input") in ["textAndImages", "images", None] + + # If retrieval mode includes vectors, compute an embedding for the query + vectors = [] + if use_vector_search: + if vector_fields == "textEmbeddingOnly" or vector_fields == "textAndImageEmbeddings": + vectors.append(await self.compute_text_embedding(q)) + if vector_fields == "imageEmbeddingOnly" or vector_fields == "textAndImageEmbeddings": + vectors.append(await self.compute_image_embedding(q)) + + results = await self.search( + top, + q, + filter, + vectors, + use_text_search, + use_vector_search, + use_semantic_ranker, + use_semantic_captions, + minimum_search_score, + minimum_reranker_score, + use_query_rewriting, + ) + + # Process results + text_sources = [] + image_sources = [] + if send_text_to_gptvision: + text_sources = self.get_sources_content(results, use_semantic_captions, use_image_citation=True) + if send_images_to_gptvision: + for result in results: + url = await fetch_image(self.blob_container_client, result) + if url: + image_sources.append(url) + + messages = self.prompt_manager.render_prompt( + self.answer_prompt, + self.get_system_prompt_variables(overrides.get("prompt_template")) + | {"user_query": q, "text_sources": text_sources, "image_sources": image_sources}, + ) + + chat_completion = await self.openai_client.chat.completions.create( + model=self.gpt4v_deployment if self.gpt4v_deployment else self.gpt4v_model, + messages=messages, + temperature=overrides.get("temperature", 0.3), + max_tokens=1024, + n=1, + seed=seed, + ) + + extra_info = ExtraInfo( + DataPoints(text=text_sources, images=image_sources), + [ + ThoughtStep( + "Search using user query", + q, + { + "use_semantic_captions": use_semantic_captions, + "use_semantic_ranker": use_semantic_ranker, + "use_query_rewriting": use_query_rewriting, + "top": top, + "filter": filter, + "vector_fields": vector_fields, + "use_vector_search": use_vector_search, + "use_text_search": use_text_search, + }, + ), + ThoughtStep( + "Search results", + [result.serialize_for_results() for result in results], + ), + ThoughtStep( + "Prompt to generate answer", + messages, + ( + {"model": self.gpt4v_model, "deployment": self.gpt4v_deployment} + if self.gpt4v_deployment + else {"model": self.gpt4v_model} + ), + ), + ], + ) + + return { + "message": { + "content": chat_completion.choices[0].message.content, + "role": chat_completion.choices[0].message.role, + }, + "context": extra_info, + "session_state": session_state, + } diff --git a/app/backend/chat_history/__init__.py b/app/backend/chat_history/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/app/backend/chat_history/cosmosdb.py b/app/backend/chat_history/cosmosdb.py new file mode 100644 index 0000000000..764278bc76 --- /dev/null +++ b/app/backend/chat_history/cosmosdb.py @@ -0,0 +1,239 @@ +import os +import time +from typing import Any, Union + +from azure.cosmos.aio import ContainerProxy, CosmosClient +from azure.identity.aio import AzureDeveloperCliCredential, ManagedIdentityCredential +from quart import Blueprint, current_app, jsonify, make_response, request + +from config import ( + CONFIG_CHAT_HISTORY_COSMOS_ENABLED, + CONFIG_COSMOS_HISTORY_CLIENT, + CONFIG_COSMOS_HISTORY_CONTAINER, + CONFIG_COSMOS_HISTORY_VERSION, + CONFIG_CREDENTIAL, +) +from decorators import authenticated +from error import error_response + +chat_history_cosmosdb_bp = Blueprint("chat_history_cosmos", __name__, static_folder="static") + + +@chat_history_cosmosdb_bp.post("/chat_history") +@authenticated +async def post_chat_history(auth_claims: dict[str, Any]): + if not current_app.config[CONFIG_CHAT_HISTORY_COSMOS_ENABLED]: + return jsonify({"error": "Chat history not enabled"}), 400 + + container: ContainerProxy = current_app.config[CONFIG_COSMOS_HISTORY_CONTAINER] + if not container: + return jsonify({"error": "Chat history not enabled"}), 400 + + entra_oid = auth_claims.get("oid") + if not entra_oid: + return jsonify({"error": "User OID not found"}), 401 + + try: + request_json = await request.get_json() + session_id = request_json.get("id") + message_pairs = request_json.get("answers") + first_question = message_pairs[0][0] + title = first_question + "..." if len(first_question) > 50 else first_question + timestamp = int(time.time() * 1000) + + # Insert the session item: + session_item = { + "id": session_id, + "version": current_app.config[CONFIG_COSMOS_HISTORY_VERSION], + "session_id": session_id, + "entra_oid": entra_oid, + "type": "session", + "title": title, + "timestamp": timestamp, + } + + message_pair_items = [] + # Now insert a message item for each question/response pair: + for ind, message_pair in enumerate(message_pairs): + message_pair_items.append( + { + "id": f"{session_id}-{ind}", + "version": current_app.config[CONFIG_COSMOS_HISTORY_VERSION], + "session_id": session_id, + "entra_oid": entra_oid, + "type": "message_pair", + "question": message_pair[0], + "response": message_pair[1], + } + ) + + batch_operations = [("upsert", (session_item,))] + [ + ("upsert", (message_pair_item,)) for message_pair_item in message_pair_items + ] + await container.execute_item_batch(batch_operations=batch_operations, partition_key=[entra_oid, session_id]) + return jsonify({}), 201 + except Exception as error: + return error_response(error, "/chat_history") + + +@chat_history_cosmosdb_bp.get("/chat_history/sessions") +@authenticated +async def get_chat_history_sessions(auth_claims: dict[str, Any]): + if not current_app.config[CONFIG_CHAT_HISTORY_COSMOS_ENABLED]: + return jsonify({"error": "Chat history not enabled"}), 400 + + container: ContainerProxy = current_app.config[CONFIG_COSMOS_HISTORY_CONTAINER] + if not container: + return jsonify({"error": "Chat history not enabled"}), 400 + + entra_oid = auth_claims.get("oid") + if not entra_oid: + return jsonify({"error": "User OID not found"}), 401 + + try: + count = int(request.args.get("count", 10)) + continuation_token = request.args.get("continuation_token") + + res = container.query_items( + query="SELECT c.id, c.entra_oid, c.title, c.timestamp FROM c WHERE c.entra_oid = @entra_oid AND c.type = @type ORDER BY c.timestamp DESC", + parameters=[dict(name="@entra_oid", value=entra_oid), dict(name="@type", value="session")], + partition_key=[entra_oid], + max_item_count=count, + ) + + pager = res.by_page(continuation_token) + + # Get the first page, and the continuation token + sessions = [] + try: + page = await pager.__anext__() + continuation_token = pager.continuation_token # type: ignore + + async for item in page: + sessions.append( + { + "id": item.get("id"), + "entra_oid": item.get("entra_oid"), + "title": item.get("title", "untitled"), + "timestamp": item.get("timestamp"), + } + ) + + # If there are no more pages, StopAsyncIteration is raised + except StopAsyncIteration: + continuation_token = None + + return jsonify({"sessions": sessions, "continuation_token": continuation_token}), 200 + + except Exception as error: + return error_response(error, "/chat_history/sessions") + + +@chat_history_cosmosdb_bp.get("/chat_history/sessions/
") - -class HtmlCallbackHandler (BaseCallbackHandler): - html: str = "" - - def get_and_reset_log(self) -> str: - result = self.html - self.html = "" - return result - - def on_llm_start( - self, serialized: Dict[str, Any], prompts: List[str], **kwargs: Any - ) -> None: - """Print out the prompts.""" - self.html += "LLM prompts:
" + "
".join(ch(prompts)) + "
" - - def on_llm_end(self, response: LLMResult, **kwargs: Any) -> None: - """Do nothing.""" - pass - - def on_llm_error(self, error: Exception, **kwargs: Any) -> None: - self.html += f"LLM error: {ch(error)}
" - - def on_chain_start( - self, serialized: Dict[str, Any], inputs: Dict[str, Any], **kwargs: Any - ) -> None: - """Print out that we are entering a chain.""" - class_name = serialized["name"] - self.html += f"Entering chain: {ch(class_name)}
" - - def on_chain_end(self, outputs: Dict[str, Any], **kwargs: Any) -> None: - """Print out that we finished a chain.""" - self.html += "Finished chain
" - - def on_chain_error(self, error: Exception, **kwargs: Any) -> None: - self.html += f"Chain error: {ch(error)}
" - - def on_tool_start( - self, - serialized: Dict[str, Any], - input_str: str, - color: Optional[str] = None, - **kwargs: Any, - ) -> None: - """Print out the log in specified color.""" - pass - - def on_tool_end( - self, - output: str, - color: Optional[str] = None, - observation_prefix: Optional[str] = None, - llm_prefix: Optional[str] = None, - **kwargs: Any, - ) -> None: - """If not the final action, print out observation.""" - self.html += f"{ch(observation_prefix)}
{ch(output)}
{ch(llm_prefix)}
" - - def on_tool_error(self, error: Exception, **kwargs: Any) -> None: - self.html += f"Tool error: {ch(error)}
" - - def on_text( - self, - text: str, - color: Optional[str] = None, - **kwargs: Optional[str], - ) -> None: - """Run when agent ends.""" - self.html += f"{ch(text)}
" - - def on_agent_action( - self, - action: AgentAction, - color: Optional[str] = None, - **kwargs: Any) -> Any: - self.html += f"{ch(action.log)}
" - - def on_agent_finish( - self, finish: AgentFinish, color: Optional[str] = None, **kwargs: Any - ) -> None: - """Run on agent end.""" - self.html += f"{ch(finish.log)}
" diff --git a/app/backend/load_azd_env.py b/app/backend/load_azd_env.py new file mode 100644 index 0000000000..2f2db6aa8f --- /dev/null +++ b/app/backend/load_azd_env.py @@ -0,0 +1,29 @@ +import json +import logging +import os +import subprocess + +from dotenv import load_dotenv + +logger = logging.getLogger("scripts") + + +def load_azd_env(): + """Get path to current azd env file and load file using python-dotenv""" + result = subprocess.run("azd env list -o json", shell=True, capture_output=True, text=True) + if result.returncode != 0: + raise Exception("Error loading azd env") + env_json = json.loads(result.stdout) + env_file_path = None + for entry in env_json: + if entry["IsDefault"]: + env_file_path = entry["DotEnvPath"] + if not env_file_path: + raise Exception("No default azd env file found") + loading_mode = os.getenv("LOADING_MODE_FOR_AZD_ENV_VARS") or "override" + if loading_mode == "no-override": + logger.info("Loading azd env from %s, but not overriding existing environment variables", env_file_path) + load_dotenv(env_file_path, override=False) + else: + logger.info("Loading azd env from %s, which may override existing environment variables", env_file_path) + load_dotenv(env_file_path, override=True) diff --git a/app/backend/lookuptool.py b/app/backend/lookuptool.py deleted file mode 100644 index 93c3b87638..0000000000 --- a/app/backend/lookuptool.py +++ /dev/null @@ -1,22 +0,0 @@ -import csv -from pathlib import Path -from typing import Optional, Union - -from langchain.agents import Tool -from langchain.callbacks.manager import Callbacks - - -class CsvLookupTool(Tool): - data: dict[str, str] = {} - - def __init__(self, filename: Union[str, Path], key_field: str, name: str = "lookup", - description: str = "useful to look up details given an input key as opposite to searching data with an unstructured question", - callbacks: Callbacks = None): - super().__init__(name, self.lookup, description, callbacks=callbacks) - with open(filename, newline='') as csvfile: - reader = csv.DictReader(csvfile) - for row in reader: - self.data[row[key_field]] = "\n".join([f"{i}:{row[i]}" for i in row]) - - def lookup(self, key: str) -> Optional[str]: - return self.data.get(key, "") diff --git a/app/backend/main.py b/app/backend/main.py new file mode 100644 index 0000000000..0f2914a483 --- /dev/null +++ b/app/backend/main.py @@ -0,0 +1,12 @@ +import os + +from app import create_app +from load_azd_env import load_azd_env + +# WEBSITE_HOSTNAME is always set by App Service, RUNNING_IN_PRODUCTION is set in main.bicep +RUNNING_ON_AZURE = os.getenv("WEBSITE_HOSTNAME") is not None or os.getenv("RUNNING_IN_PRODUCTION") is not None + +if not RUNNING_ON_AZURE: + load_azd_env() + +app = create_app() diff --git a/app/backend/prepdocs.py b/app/backend/prepdocs.py new file mode 100644 index 0000000000..f03baac0dc --- /dev/null +++ b/app/backend/prepdocs.py @@ -0,0 +1,473 @@ +import argparse +import asyncio +import logging +import os +from typing import Optional, Union + +from azure.core.credentials import AzureKeyCredential +from azure.core.credentials_async import AsyncTokenCredential +from azure.identity.aio import AzureDeveloperCliCredential, get_bearer_token_provider +from rich.logging import RichHandler + +from load_azd_env import load_azd_env +from prepdocslib.blobmanager import BlobManager +from prepdocslib.csvparser import CsvParser +from prepdocslib.embeddings import ( + AzureOpenAIEmbeddingService, + ImageEmbeddings, + OpenAIEmbeddingService, +) +from prepdocslib.fileprocessor import FileProcessor +from prepdocslib.filestrategy import FileStrategy +from prepdocslib.htmlparser import LocalHTMLParser +from prepdocslib.integratedvectorizerstrategy import ( + IntegratedVectorizerStrategy, +) +from prepdocslib.jsonparser import JsonParser +from prepdocslib.listfilestrategy import ( + ADLSGen2ListFileStrategy, + ListFileStrategy, + LocalListFileStrategy, +) +from prepdocslib.parser import Parser +from prepdocslib.pdfparser import DocumentAnalysisParser, LocalPdfParser +from prepdocslib.strategy import DocumentAction, SearchInfo, Strategy +from prepdocslib.textparser import TextParser +from prepdocslib.textsplitter import SentenceTextSplitter, SimpleTextSplitter + +logger = logging.getLogger("scripts") + + +def clean_key_if_exists(key: Union[str, None]) -> Union[str, None]: + """Remove leading and trailing whitespace from a key if it exists. If the key is empty, return None.""" + if key is not None and key.strip() != "": + return key.strip() + return None + + +async def setup_search_info( + search_service: str, + index_name: str, + azure_credential: AsyncTokenCredential, + use_agentic_retrieval: Union[bool, None] = None, + azure_openai_endpoint: Union[str, None] = None, + agent_name: Union[str, None] = None, + agent_max_output_tokens: Union[int, None] = None, + azure_openai_searchagent_deployment: Union[str, None] = None, + azure_openai_searchagent_model: Union[str, None] = None, + search_key: Union[str, None] = None, +) -> SearchInfo: + search_creds: Union[AsyncTokenCredential, AzureKeyCredential] = ( + azure_credential if search_key is None else AzureKeyCredential(search_key) + ) + if use_agentic_retrieval and azure_openai_searchagent_model is None: + raise ValueError("Azure OpenAI SearchAgent model must be specified when using agentic retrieval.") + + return SearchInfo( + endpoint=f"https://{search_service}.search.windows.net/", + credential=search_creds, + index_name=index_name, + agent_name=agent_name, + agent_max_output_tokens=agent_max_output_tokens, + use_agentic_retrieval=use_agentic_retrieval, + azure_openai_endpoint=azure_openai_endpoint, + azure_openai_searchagent_model=azure_openai_searchagent_model, + azure_openai_searchagent_deployment=azure_openai_searchagent_deployment, + ) + + +def setup_blob_manager( + azure_credential: AsyncTokenCredential, + storage_account: str, + storage_container: str, + storage_resource_group: str, + subscription_id: str, + search_images: bool, + storage_key: Union[str, None] = None, +): + storage_creds: Union[AsyncTokenCredential, str] = azure_credential if storage_key is None else storage_key + return BlobManager( + endpoint=f"https://{storage_account}.blob.core.windows.net", + container=storage_container, + account=storage_account, + credential=storage_creds, + resourceGroup=storage_resource_group, + subscriptionId=subscription_id, + store_page_images=search_images, + ) + + +def setup_list_file_strategy( + azure_credential: AsyncTokenCredential, + local_files: Union[str, None], + datalake_storage_account: Union[str, None], + datalake_filesystem: Union[str, None], + datalake_path: Union[str, None], + datalake_key: Union[str, None], +): + list_file_strategy: ListFileStrategy + if datalake_storage_account: + if datalake_filesystem is None or datalake_path is None: + raise ValueError("DataLake file system and path are required when using Azure Data Lake Gen2") + adls_gen2_creds: Union[AsyncTokenCredential, str] = azure_credential if datalake_key is None else datalake_key + logger.info("Using Data Lake Gen2 Storage Account: %s", datalake_storage_account) + list_file_strategy = ADLSGen2ListFileStrategy( + data_lake_storage_account=datalake_storage_account, + data_lake_filesystem=datalake_filesystem, + data_lake_path=datalake_path, + credential=adls_gen2_creds, + ) + elif local_files: + logger.info("Using local files: %s", local_files) + list_file_strategy = LocalListFileStrategy(path_pattern=local_files) + else: + raise ValueError("Either local_files or datalake_storage_account must be provided.") + return list_file_strategy + + +def setup_embeddings_service( + azure_credential: AsyncTokenCredential, + openai_host: str, + openai_model_name: str, + openai_service: Union[str, None], + openai_custom_url: Union[str, None], + openai_deployment: Union[str, None], + openai_dimensions: int, + openai_api_version: str, + openai_key: Union[str, None], + openai_org: Union[str, None], + disable_vectors: bool = False, + disable_batch_vectors: bool = False, +): + if disable_vectors: + logger.info("Not setting up embeddings service") + return None + + if openai_host != "openai": + azure_open_ai_credential: Union[AsyncTokenCredential, AzureKeyCredential] = ( + azure_credential if openai_key is None else AzureKeyCredential(openai_key) + ) + return AzureOpenAIEmbeddingService( + open_ai_service=openai_service, + open_ai_custom_url=openai_custom_url, + open_ai_deployment=openai_deployment, + open_ai_model_name=openai_model_name, + open_ai_dimensions=openai_dimensions, + open_ai_api_version=openai_api_version, + credential=azure_open_ai_credential, + disable_batch=disable_batch_vectors, + ) + else: + if openai_key is None: + raise ValueError("OpenAI key is required when using the non-Azure OpenAI API") + return OpenAIEmbeddingService( + open_ai_model_name=openai_model_name, + open_ai_dimensions=openai_dimensions, + credential=openai_key, + organization=openai_org, + disable_batch=disable_batch_vectors, + ) + + +def setup_file_processors( + azure_credential: AsyncTokenCredential, + document_intelligence_service: Union[str, None], + document_intelligence_key: Union[str, None] = None, + local_pdf_parser: bool = False, + local_html_parser: bool = False, + search_images: bool = False, + use_content_understanding: bool = False, + content_understanding_endpoint: Union[str, None] = None, +): + sentence_text_splitter = SentenceTextSplitter() + + doc_int_parser: Optional[DocumentAnalysisParser] = None + # check if Azure Document Intelligence credentials are provided + if document_intelligence_service is not None: + documentintelligence_creds: Union[AsyncTokenCredential, AzureKeyCredential] = ( + azure_credential if document_intelligence_key is None else AzureKeyCredential(document_intelligence_key) + ) + doc_int_parser = DocumentAnalysisParser( + endpoint=f"https://{document_intelligence_service}.cognitiveservices.azure.com/", + credential=documentintelligence_creds, + use_content_understanding=use_content_understanding, + content_understanding_endpoint=content_understanding_endpoint, + ) + + pdf_parser: Optional[Parser] = None + if local_pdf_parser or document_intelligence_service is None: + pdf_parser = LocalPdfParser() + elif document_intelligence_service is not None: + pdf_parser = doc_int_parser + else: + logger.warning("No PDF parser available") + + html_parser: Optional[Parser] = None + if local_html_parser or document_intelligence_service is None: + html_parser = LocalHTMLParser() + elif document_intelligence_service is not None: + html_parser = doc_int_parser + else: + logger.warning("No HTML parser available") + + # These file formats can always be parsed: + file_processors = { + ".json": FileProcessor(JsonParser(), SimpleTextSplitter()), + ".md": FileProcessor(TextParser(), sentence_text_splitter), + ".txt": FileProcessor(TextParser(), sentence_text_splitter), + ".csv": FileProcessor(CsvParser(), sentence_text_splitter), + } + # These require either a Python package or Document Intelligence + if pdf_parser is not None: + file_processors.update({".pdf": FileProcessor(pdf_parser, sentence_text_splitter)}) + if html_parser is not None: + file_processors.update({".html": FileProcessor(html_parser, sentence_text_splitter)}) + # These file formats require Document Intelligence + if doc_int_parser is not None: + file_processors.update( + { + ".docx": FileProcessor(doc_int_parser, sentence_text_splitter), + ".pptx": FileProcessor(doc_int_parser, sentence_text_splitter), + ".xlsx": FileProcessor(doc_int_parser, sentence_text_splitter), + ".png": FileProcessor(doc_int_parser, sentence_text_splitter), + ".jpg": FileProcessor(doc_int_parser, sentence_text_splitter), + ".jpeg": FileProcessor(doc_int_parser, sentence_text_splitter), + ".tiff": FileProcessor(doc_int_parser, sentence_text_splitter), + ".bmp": FileProcessor(doc_int_parser, sentence_text_splitter), + ".heic": FileProcessor(doc_int_parser, sentence_text_splitter), + } + ) + return file_processors + + +def setup_image_embeddings_service( + azure_credential: AsyncTokenCredential, vision_endpoint: Union[str, None], search_images: bool +) -> Union[ImageEmbeddings, None]: + image_embeddings_service: Optional[ImageEmbeddings] = None + if search_images: + if vision_endpoint is None: + raise ValueError("A computer vision endpoint is required when GPT-4-vision is enabled.") + image_embeddings_service = ImageEmbeddings( + endpoint=vision_endpoint, + token_provider=get_bearer_token_provider(azure_credential, "https://cognitiveservices.azure.com/.default"), + ) + return image_embeddings_service + + +async def main(strategy: Strategy, setup_index: bool = True): + if setup_index: + await strategy.setup() + + await strategy.run() + + +if __name__ == "__main__": + parser = argparse.ArgumentParser( + description="Prepare documents by extracting content from PDFs, splitting content into sections, uploading to blob storage, and indexing in a search index." + ) + parser.add_argument("files", nargs="?", help="Files to be processed") + + parser.add_argument( + "--category", help="Value for the category field in the search index for all sections indexed in this run" + ) + parser.add_argument( + "--skipblobs", action="store_true", help="Skip uploading individual pages to Azure Blob Storage" + ) + parser.add_argument( + "--disablebatchvectors", action="store_true", help="Don't compute embeddings in batch for the sections" + ) + parser.add_argument( + "--remove", + action="store_true", + help="Remove references to this document from blob storage and the search index", + ) + parser.add_argument( + "--removeall", + action="store_true", + help="Remove all blobs from blob storage and documents from the search index", + ) + + # Optional key specification: + parser.add_argument( + "--searchkey", + required=False, + help="Optional. Use this Azure AI Search account key instead of the current user identity to login (use az login to set current user for Azure)", + ) + parser.add_argument( + "--storagekey", + required=False, + help="Optional. Use this Azure Blob Storage account key instead of the current user identity to login (use az login to set current user for Azure)", + ) + parser.add_argument( + "--datalakekey", required=False, help="Optional. Use this key when authenticating to Azure Data Lake Gen2" + ) + parser.add_argument( + "--documentintelligencekey", + required=False, + help="Optional. Use this Azure Document Intelligence account key instead of the current user identity to login (use az login to set current user for Azure)", + ) + parser.add_argument( + "--searchserviceassignedid", + required=False, + help="Search service system assigned Identity (Managed identity) (used for integrated vectorization)", + ) + + parser.add_argument("--verbose", "-v", action="store_true", help="Verbose output") + args = parser.parse_args() + + if args.verbose: + logging.basicConfig(format="%(message)s", datefmt="[%X]", handlers=[RichHandler(rich_tracebacks=True)]) + # We only set the level to INFO for our logger, + # to avoid seeing the noisy INFO level logs from the Azure SDKs + logger.setLevel(logging.DEBUG) + + load_azd_env() + + if os.getenv("AZURE_PUBLIC_NETWORK_ACCESS") == "Disabled": + logger.error("AZURE_PUBLIC_NETWORK_ACCESS is set to Disabled. Exiting.") + exit(0) + + use_int_vectorization = os.getenv("USE_FEATURE_INT_VECTORIZATION", "").lower() == "true" + use_gptvision = os.getenv("USE_GPT4V", "").lower() == "true" + use_acls = os.getenv("AZURE_ENFORCE_ACCESS_CONTROL") is not None + dont_use_vectors = os.getenv("USE_VECTORS", "").lower() == "false" + use_agentic_retrieval = os.getenv("USE_AGENTIC_RETRIEVAL", "").lower() == "true" + use_content_understanding = os.getenv("USE_MEDIA_DESCRIBER_AZURE_CU", "").lower() == "true" + + # Use the current user identity to connect to Azure services. See infra/main.bicep for role assignments. + if tenant_id := os.getenv("AZURE_TENANT_ID"): + logger.info("Connecting to Azure services using the azd credential for tenant %s", tenant_id) + azd_credential = AzureDeveloperCliCredential(tenant_id=tenant_id, process_timeout=60) + else: + logger.info("Connecting to Azure services using the azd credential for home tenant") + azd_credential = AzureDeveloperCliCredential(process_timeout=60) + + if args.removeall: + document_action = DocumentAction.RemoveAll + elif args.remove: + document_action = DocumentAction.Remove + else: + document_action = DocumentAction.Add + + loop = asyncio.new_event_loop() + asyncio.set_event_loop(loop) + + openai_host = os.environ["OPENAI_HOST"] + # Check for incompatibility + # if openai host is not azure + if openai_host != "azure" and use_agentic_retrieval: + raise Exception("Agentic retrieval requires an Azure OpenAI chat completion service") + + search_info = loop.run_until_complete( + setup_search_info( + search_service=os.environ["AZURE_SEARCH_SERVICE"], + index_name=os.environ["AZURE_SEARCH_INDEX"], + use_agentic_retrieval=use_agentic_retrieval, + agent_name=os.getenv("AZURE_SEARCH_AGENT"), + agent_max_output_tokens=int(os.getenv("AZURE_SEARCH_AGENT_MAX_OUTPUT_TOKENS", 10000)), + azure_openai_endpoint=os.environ["AZURE_OPENAI_ENDPOINT"], + azure_openai_searchagent_deployment=os.getenv("AZURE_OPENAI_SEARCHAGENT_DEPLOYMENT"), + azure_openai_searchagent_model=os.getenv("AZURE_OPENAI_SEARCHAGENT_MODEL"), + azure_credential=azd_credential, + search_key=clean_key_if_exists(args.searchkey), + ) + ) + blob_manager = setup_blob_manager( + azure_credential=azd_credential, + storage_account=os.environ["AZURE_STORAGE_ACCOUNT"], + storage_container=os.environ["AZURE_STORAGE_CONTAINER"], + storage_resource_group=os.environ["AZURE_STORAGE_RESOURCE_GROUP"], + subscription_id=os.environ["AZURE_SUBSCRIPTION_ID"], + search_images=use_gptvision, + storage_key=clean_key_if_exists(args.storagekey), + ) + list_file_strategy = setup_list_file_strategy( + azure_credential=azd_credential, + local_files=args.files, + datalake_storage_account=os.getenv("AZURE_ADLS_GEN2_STORAGE_ACCOUNT"), + datalake_filesystem=os.getenv("AZURE_ADLS_GEN2_FILESYSTEM"), + datalake_path=os.getenv("AZURE_ADLS_GEN2_FILESYSTEM_PATH"), + datalake_key=clean_key_if_exists(args.datalakekey), + ) + + openai_host = os.environ["OPENAI_HOST"] + openai_key = None + if os.getenv("AZURE_OPENAI_API_KEY_OVERRIDE"): + openai_key = os.getenv("AZURE_OPENAI_API_KEY_OVERRIDE") + elif not openai_host.startswith("azure") and os.getenv("OPENAI_API_KEY"): + openai_key = os.getenv("OPENAI_API_KEY") + + openai_dimensions = 1536 + if os.getenv("AZURE_OPENAI_EMB_DIMENSIONS"): + openai_dimensions = int(os.environ["AZURE_OPENAI_EMB_DIMENSIONS"]) + openai_embeddings_service = setup_embeddings_service( + azure_credential=azd_credential, + openai_host=openai_host, + openai_model_name=os.environ["AZURE_OPENAI_EMB_MODEL_NAME"], + openai_service=os.getenv("AZURE_OPENAI_SERVICE"), + openai_custom_url=os.getenv("AZURE_OPENAI_CUSTOM_URL"), + openai_deployment=os.getenv("AZURE_OPENAI_EMB_DEPLOYMENT"), + # https://learn.microsoft.com/azure/ai-services/openai/api-version-deprecation#latest-ga-api-release + openai_api_version=os.getenv("AZURE_OPENAI_API_VERSION") or "2024-06-01", + openai_dimensions=openai_dimensions, + openai_key=clean_key_if_exists(openai_key), + openai_org=os.getenv("OPENAI_ORGANIZATION"), + disable_vectors=dont_use_vectors, + disable_batch_vectors=args.disablebatchvectors, + ) + + ingestion_strategy: Strategy + if use_int_vectorization: + + if not openai_embeddings_service or not isinstance(openai_embeddings_service, AzureOpenAIEmbeddingService): + raise Exception("Integrated vectorization strategy requires an Azure OpenAI embeddings service") + + ingestion_strategy = IntegratedVectorizerStrategy( + search_info=search_info, + list_file_strategy=list_file_strategy, + blob_manager=blob_manager, + document_action=document_action, + embeddings=openai_embeddings_service, + search_field_name_embedding=os.environ["AZURE_SEARCH_FIELD_NAME_EMBEDDING"], + subscription_id=os.environ["AZURE_SUBSCRIPTION_ID"], + search_service_user_assigned_id=args.searchserviceassignedid, + search_analyzer_name=os.getenv("AZURE_SEARCH_ANALYZER_NAME"), + use_acls=use_acls, + category=args.category, + ) + else: + file_processors = setup_file_processors( + azure_credential=azd_credential, + document_intelligence_service=os.getenv("AZURE_DOCUMENTINTELLIGENCE_SERVICE"), + document_intelligence_key=clean_key_if_exists(args.documentintelligencekey), + local_pdf_parser=os.getenv("USE_LOCAL_PDF_PARSER") == "true", + local_html_parser=os.getenv("USE_LOCAL_HTML_PARSER") == "true", + search_images=use_gptvision, + use_content_understanding=use_content_understanding, + content_understanding_endpoint=os.getenv("AZURE_CONTENTUNDERSTANDING_ENDPOINT"), + ) + image_embeddings_service = setup_image_embeddings_service( + azure_credential=azd_credential, + vision_endpoint=os.getenv("AZURE_VISION_ENDPOINT"), + search_images=use_gptvision, + ) + + ingestion_strategy = FileStrategy( + search_info=search_info, + list_file_strategy=list_file_strategy, + blob_manager=blob_manager, + file_processors=file_processors, + document_action=document_action, + embeddings=openai_embeddings_service, + image_embeddings=image_embeddings_service, + search_analyzer_name=os.getenv("AZURE_SEARCH_ANALYZER_NAME"), + # Default to the previous field names for backward compatibility + search_field_name_embedding=os.getenv("AZURE_SEARCH_FIELD_NAME_EMBEDDING", "embedding"), + use_acls=use_acls, + category=args.category, + use_content_understanding=use_content_understanding, + content_understanding_endpoint=os.getenv("AZURE_CONTENTUNDERSTANDING_ENDPOINT"), + ) + + loop.run_until_complete(main(ingestion_strategy, setup_index=not args.remove and not args.removeall)) + loop.close() diff --git a/app/backend/prepdocslib/__init__.py b/app/backend/prepdocslib/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/app/backend/prepdocslib/blobmanager.py b/app/backend/prepdocslib/blobmanager.py new file mode 100644 index 0000000000..d5c21e0d41 --- /dev/null +++ b/app/backend/prepdocslib/blobmanager.py @@ -0,0 +1,178 @@ +import datetime +import io +import logging +import os +import re +from typing import Optional, Union + +import pymupdf +from azure.core.credentials_async import AsyncTokenCredential +from azure.storage.blob import ( + BlobSasPermissions, + UserDelegationKey, + generate_blob_sas, +) +from azure.storage.blob.aio import BlobServiceClient, ContainerClient +from PIL import Image, ImageDraw, ImageFont +from pypdf import PdfReader + +from .listfilestrategy import File + +logger = logging.getLogger("scripts") + + +class BlobManager: + """ + Class to manage uploading and deleting blobs containing citation information from a blob storage account + """ + + def __init__( + self, + endpoint: str, + container: str, + account: str, + credential: Union[AsyncTokenCredential, str], + resourceGroup: str, + subscriptionId: str, + store_page_images: bool = False, + ): + self.endpoint = endpoint + self.credential = credential + self.account = account + self.container = container + self.store_page_images = store_page_images + self.resourceGroup = resourceGroup + self.subscriptionId = subscriptionId + self.user_delegation_key: Optional[UserDelegationKey] = None + + async def upload_blob(self, file: File) -> Optional[list[str]]: + async with BlobServiceClient( + account_url=self.endpoint, credential=self.credential, max_single_put_size=4 * 1024 * 1024 + ) as service_client, service_client.get_container_client(self.container) as container_client: + if not await container_client.exists(): + await container_client.create_container() + + # Re-open and upload the original file + if file.url is None: + with open(file.content.name, "rb") as reopened_file: + blob_name = BlobManager.blob_name_from_file_name(file.content.name) + logger.info("Uploading blob for whole file -> %s", blob_name) + blob_client = await container_client.upload_blob(blob_name, reopened_file, overwrite=True) + file.url = blob_client.url + + if self.store_page_images: + if os.path.splitext(file.content.name)[1].lower() == ".pdf": + return await self.upload_pdf_blob_images(service_client, container_client, file) + else: + logger.info("File %s is not a PDF, skipping image upload", file.content.name) + + return None + + def get_managedidentity_connectionstring(self): + return f"ResourceId=/subscriptions/{self.subscriptionId}/resourceGroups/{self.resourceGroup}/providers/Microsoft.Storage/storageAccounts/{self.account};" + + async def upload_pdf_blob_images( + self, service_client: BlobServiceClient, container_client: ContainerClient, file: File + ) -> list[str]: + with open(file.content.name, "rb") as reopened_file: + reader = PdfReader(reopened_file) + page_count = len(reader.pages) + doc = pymupdf.open(file.content.name) + sas_uris = [] + start_time = datetime.datetime.now(datetime.timezone.utc) + expiry_time = start_time + datetime.timedelta(days=1) + + font = None + try: + font = ImageFont.truetype("arial.ttf", 20) + except OSError: + try: + font = ImageFont.truetype("/usr/share/fonts/truetype/freefont/FreeMono.ttf", 20) + except OSError: + logger.info("Unable to find arial.ttf or FreeMono.ttf, using default font") + + for i in range(page_count): + blob_name = BlobManager.blob_image_name_from_file_page(file.content.name, i) + logger.info("Converting page %s to image and uploading -> %s", i, blob_name) + + doc = pymupdf.open(file.content.name) + page = doc.load_page(i) + pix = page.get_pixmap() + original_img = Image.frombytes("RGB", [pix.width, pix.height], pix.samples) # type: ignore + + # Create a new image with additional space for text + text_height = 40 # Height of the text area + new_img = Image.new("RGB", (original_img.width, original_img.height + text_height), "white") + + # Paste the original image onto the new image + new_img.paste(original_img, (0, text_height)) + + # Draw the text on the white area + draw = ImageDraw.Draw(new_img) + text = f"SourceFileName:{blob_name}" + + # 10 pixels from the top and left of the image + x = 10 + y = 10 + draw.text((x, y), text, font=font, fill="black") + + output = io.BytesIO() + new_img.save(output, format="PNG") + output.seek(0) + + blob_client = await container_client.upload_blob(blob_name, output, overwrite=True) + if not self.user_delegation_key: + self.user_delegation_key = await service_client.get_user_delegation_key(start_time, expiry_time) + + if blob_client.account_name is not None: + sas_token = generate_blob_sas( + account_name=blob_client.account_name, + container_name=blob_client.container_name, + blob_name=blob_client.blob_name, + user_delegation_key=self.user_delegation_key, + permission=BlobSasPermissions(read=True), + expiry=expiry_time, + start=start_time, + ) + sas_uris.append(f"{blob_client.url}?{sas_token}") + + return sas_uris + + async def remove_blob(self, path: Optional[str] = None): + async with BlobServiceClient( + account_url=self.endpoint, credential=self.credential + ) as service_client, service_client.get_container_client(self.container) as container_client: + if not await container_client.exists(): + return + if path is None: + prefix = None + blobs = container_client.list_blob_names() + else: + prefix = os.path.splitext(os.path.basename(path))[0] + blobs = container_client.list_blob_names(name_starts_with=os.path.splitext(os.path.basename(prefix))[0]) + async for blob_path in blobs: + # This still supports PDFs split into individual pages, but we could remove in future to simplify code + if ( + prefix is not None + and ( + not re.match(rf"{prefix}-\d+\.pdf", blob_path) or not re.match(rf"{prefix}-\d+\.png", blob_path) + ) + ) or (path is not None and blob_path == os.path.basename(path)): + continue + logger.info("Removing blob %s", blob_path) + await container_client.delete_blob(blob_path) + + @classmethod + def sourcepage_from_file_page(cls, filename, page=0) -> str: + if os.path.splitext(filename)[1].lower() == ".pdf": + return f"{os.path.basename(filename)}#page={page+1}" + else: + return os.path.basename(filename) + + @classmethod + def blob_image_name_from_file_page(cls, filename, page=0) -> str: + return os.path.splitext(os.path.basename(filename))[0] + f"-{page+1}" + ".png" + + @classmethod + def blob_name_from_file_name(cls, filename) -> str: + return os.path.basename(filename) diff --git a/app/backend/prepdocslib/csvparser.py b/app/backend/prepdocslib/csvparser.py new file mode 100644 index 0000000000..7bf5e1ad75 --- /dev/null +++ b/app/backend/prepdocslib/csvparser.py @@ -0,0 +1,32 @@ +import csv +from collections.abc import AsyncGenerator +from typing import IO + +from .page import Page +from .parser import Parser + + +class CsvParser(Parser): + """ + Concrete parser that can parse CSV into Page objects. Each row becomes a Page object. + """ + + async def parse(self, content: IO) -> AsyncGenerator[Page, None]: + # Check if content is in bytes (binary file) and decode to string + content_str: str + if isinstance(content, (bytes, bytearray)): + content_str = content.decode("utf-8") + elif hasattr(content, "read"): # Handle BufferedReader + content_str = content.read().decode("utf-8") + + # Create a CSV reader from the text content + reader = csv.reader(content_str.splitlines()) + offset = 0 + + # Skip the header row + next(reader, None) + + for i, row in enumerate(reader): + page_text = ",".join(row) + yield Page(i, offset, page_text) + offset += len(page_text) + 1 # Account for newline character diff --git a/app/backend/prepdocslib/embeddings.py b/app/backend/prepdocslib/embeddings.py new file mode 100644 index 0000000000..df56f39c08 --- /dev/null +++ b/app/backend/prepdocslib/embeddings.py @@ -0,0 +1,263 @@ +import logging +from abc import ABC +from collections.abc import Awaitable +from typing import Callable, Optional, Union +from urllib.parse import urljoin + +import aiohttp +import tiktoken +from azure.core.credentials import AzureKeyCredential +from azure.core.credentials_async import AsyncTokenCredential +from azure.identity.aio import get_bearer_token_provider +from openai import AsyncAzureOpenAI, AsyncOpenAI, RateLimitError +from tenacity import ( + AsyncRetrying, + retry_if_exception_type, + stop_after_attempt, + wait_random_exponential, +) +from typing_extensions import TypedDict + +logger = logging.getLogger("scripts") + + +class EmbeddingBatch: + """ + Represents a batch of text that is going to be embedded + """ + + def __init__(self, texts: list[str], token_length: int): + self.texts = texts + self.token_length = token_length + + +class ExtraArgs(TypedDict, total=False): + dimensions: int + + +class OpenAIEmbeddings(ABC): + """ + Contains common logic across both OpenAI and Azure OpenAI embedding services + Can split source text into batches for more efficient embedding calls + """ + + SUPPORTED_BATCH_AOAI_MODEL = { + "text-embedding-ada-002": {"token_limit": 8100, "max_batch_size": 16}, + "text-embedding-3-small": {"token_limit": 8100, "max_batch_size": 16}, + "text-embedding-3-large": {"token_limit": 8100, "max_batch_size": 16}, + } + SUPPORTED_DIMENSIONS_MODEL = { + "text-embedding-ada-002": False, + "text-embedding-3-small": True, + "text-embedding-3-large": True, + } + + def __init__(self, open_ai_model_name: str, open_ai_dimensions: int, disable_batch: bool = False): + self.open_ai_model_name = open_ai_model_name + self.open_ai_dimensions = open_ai_dimensions + self.disable_batch = disable_batch + + async def create_client(self) -> AsyncOpenAI: + raise NotImplementedError + + def before_retry_sleep(self, retry_state): + logger.info("Rate limited on the OpenAI embeddings API, sleeping before retrying...") + + def calculate_token_length(self, text: str): + encoding = tiktoken.encoding_for_model(self.open_ai_model_name) + return len(encoding.encode(text)) + + def split_text_into_batches(self, texts: list[str]) -> list[EmbeddingBatch]: + batch_info = OpenAIEmbeddings.SUPPORTED_BATCH_AOAI_MODEL.get(self.open_ai_model_name) + if not batch_info: + raise NotImplementedError( + f"Model {self.open_ai_model_name} is not supported with batch embedding operations" + ) + + batch_token_limit = batch_info["token_limit"] + batch_max_size = batch_info["max_batch_size"] + batches: list[EmbeddingBatch] = [] + batch: list[str] = [] + batch_token_length = 0 + for text in texts: + text_token_length = self.calculate_token_length(text) + if batch_token_length + text_token_length >= batch_token_limit and len(batch) > 0: + batches.append(EmbeddingBatch(batch, batch_token_length)) + batch = [] + batch_token_length = 0 + + batch.append(text) + batch_token_length = batch_token_length + text_token_length + if len(batch) == batch_max_size: + batches.append(EmbeddingBatch(batch, batch_token_length)) + batch = [] + batch_token_length = 0 + + if len(batch) > 0: + batches.append(EmbeddingBatch(batch, batch_token_length)) + + return batches + + async def create_embedding_batch(self, texts: list[str], dimensions_args: ExtraArgs) -> list[list[float]]: + batches = self.split_text_into_batches(texts) + embeddings = [] + client = await self.create_client() + for batch in batches: + async for attempt in AsyncRetrying( + retry=retry_if_exception_type(RateLimitError), + wait=wait_random_exponential(min=15, max=60), + stop=stop_after_attempt(15), + before_sleep=self.before_retry_sleep, + ): + with attempt: + emb_response = await client.embeddings.create( + model=self.open_ai_model_name, input=batch.texts, **dimensions_args + ) + embeddings.extend([data.embedding for data in emb_response.data]) + logger.info( + "Computed embeddings in batch. Batch size: %d, Token count: %d", + len(batch.texts), + batch.token_length, + ) + + return embeddings + + async def create_embedding_single(self, text: str, dimensions_args: ExtraArgs) -> list[float]: + client = await self.create_client() + async for attempt in AsyncRetrying( + retry=retry_if_exception_type(RateLimitError), + wait=wait_random_exponential(min=15, max=60), + stop=stop_after_attempt(15), + before_sleep=self.before_retry_sleep, + ): + with attempt: + emb_response = await client.embeddings.create( + model=self.open_ai_model_name, input=text, **dimensions_args + ) + logger.info("Computed embedding for text section. Character count: %d", len(text)) + + return emb_response.data[0].embedding + + async def create_embeddings(self, texts: list[str]) -> list[list[float]]: + + dimensions_args: ExtraArgs = ( + {"dimensions": self.open_ai_dimensions} + if OpenAIEmbeddings.SUPPORTED_DIMENSIONS_MODEL.get(self.open_ai_model_name) + else {} + ) + + if not self.disable_batch and self.open_ai_model_name in OpenAIEmbeddings.SUPPORTED_BATCH_AOAI_MODEL: + return await self.create_embedding_batch(texts, dimensions_args) + + return [await self.create_embedding_single(text, dimensions_args) for text in texts] + + +class AzureOpenAIEmbeddingService(OpenAIEmbeddings): + """ + Class for using Azure OpenAI embeddings + To learn more please visit https://learn.microsoft.com/azure/ai-services/openai/concepts/understand-embeddings + """ + + def __init__( + self, + open_ai_service: Union[str, None], + open_ai_deployment: Union[str, None], + open_ai_model_name: str, + open_ai_dimensions: int, + open_ai_api_version: str, + credential: Union[AsyncTokenCredential, AzureKeyCredential], + open_ai_custom_url: Union[str, None] = None, + disable_batch: bool = False, + ): + super().__init__(open_ai_model_name, open_ai_dimensions, disable_batch) + self.open_ai_service = open_ai_service + if open_ai_service: + self.open_ai_endpoint = f"https://{open_ai_service}.openai.azure.com" + elif open_ai_custom_url: + self.open_ai_endpoint = open_ai_custom_url + else: + raise ValueError("Either open_ai_service or open_ai_custom_url must be provided") + self.open_ai_deployment = open_ai_deployment + self.open_ai_api_version = open_ai_api_version + self.credential = credential + + async def create_client(self) -> AsyncOpenAI: + class AuthArgs(TypedDict, total=False): + api_key: str + azure_ad_token_provider: Callable[[], Union[str, Awaitable[str]]] + + auth_args = AuthArgs() + if isinstance(self.credential, AzureKeyCredential): + auth_args["api_key"] = self.credential.key + elif isinstance(self.credential, AsyncTokenCredential): + auth_args["azure_ad_token_provider"] = get_bearer_token_provider( + self.credential, "https://cognitiveservices.azure.com/.default" + ) + else: + raise TypeError("Invalid credential type") + + return AsyncAzureOpenAI( + azure_endpoint=self.open_ai_endpoint, + azure_deployment=self.open_ai_deployment, + api_version=self.open_ai_api_version, + **auth_args, + ) + + +class OpenAIEmbeddingService(OpenAIEmbeddings): + """ + Class for using OpenAI embeddings + To learn more please visit https://platform.openai.com/docs/guides/embeddings + """ + + def __init__( + self, + open_ai_model_name: str, + open_ai_dimensions: int, + credential: str, + organization: Optional[str] = None, + disable_batch: bool = False, + ): + super().__init__(open_ai_model_name, open_ai_dimensions, disable_batch) + self.credential = credential + self.organization = organization + + async def create_client(self) -> AsyncOpenAI: + return AsyncOpenAI(api_key=self.credential, organization=self.organization) + + +class ImageEmbeddings: + """ + Class for using image embeddings from Azure AI Vision + To learn more, please visit https://learn.microsoft.com/azure/ai-services/computer-vision/how-to/image-retrieval#call-the-vectorize-image-api + """ + + def __init__(self, endpoint: str, token_provider: Callable[[], Awaitable[str]]): + self.token_provider = token_provider + self.endpoint = endpoint + + async def create_embeddings(self, blob_urls: list[str]) -> list[list[float]]: + endpoint = urljoin(self.endpoint, "computervision/retrieval:vectorizeImage") + headers = {"Content-Type": "application/json"} + params = {"api-version": "2024-02-01", "model-version": "2023-04-15"} + headers["Authorization"] = "Bearer " + await self.token_provider() + + embeddings: list[list[float]] = [] + async with aiohttp.ClientSession(headers=headers) as session: + for blob_url in blob_urls: + async for attempt in AsyncRetrying( + retry=retry_if_exception_type(Exception), + wait=wait_random_exponential(min=15, max=60), + stop=stop_after_attempt(15), + before_sleep=self.before_retry_sleep, + ): + with attempt: + body = {"url": blob_url} + async with session.post(url=endpoint, params=params, json=body) as resp: + resp_json = await resp.json() + embeddings.append(resp_json["vector"]) + + return embeddings + + def before_retry_sleep(self, retry_state): + logger.info("Rate limited on the Vision embeddings API, sleeping before retrying...") diff --git a/app/backend/prepdocslib/fileprocessor.py b/app/backend/prepdocslib/fileprocessor.py new file mode 100644 index 0000000000..3b58130db8 --- /dev/null +++ b/app/backend/prepdocslib/fileprocessor.py @@ -0,0 +1,10 @@ +from dataclasses import dataclass + +from .parser import Parser +from .textsplitter import TextSplitter + + +@dataclass(frozen=True) +class FileProcessor: + parser: Parser + splitter: TextSplitter diff --git a/app/backend/prepdocslib/filestrategy.py b/app/backend/prepdocslib/filestrategy.py new file mode 100644 index 0000000000..37f399cf4b --- /dev/null +++ b/app/backend/prepdocslib/filestrategy.py @@ -0,0 +1,164 @@ +import logging +from typing import Optional + +from azure.core.credentials import AzureKeyCredential + +from .blobmanager import BlobManager +from .embeddings import ImageEmbeddings, OpenAIEmbeddings +from .fileprocessor import FileProcessor +from .listfilestrategy import File, ListFileStrategy +from .mediadescriber import ContentUnderstandingDescriber +from .searchmanager import SearchManager, Section +from .strategy import DocumentAction, SearchInfo, Strategy + +logger = logging.getLogger("scripts") + + +async def parse_file( + file: File, + file_processors: dict[str, FileProcessor], + category: Optional[str] = None, + image_embeddings: Optional[ImageEmbeddings] = None, +) -> list[Section]: + key = file.file_extension().lower() + processor = file_processors.get(key) + if processor is None: + logger.info("Skipping '%s', no parser found.", file.filename()) + return [] + logger.info("Ingesting '%s'", file.filename()) + pages = [page async for page in processor.parser.parse(content=file.content)] + logger.info("Splitting '%s' into sections", file.filename()) + if image_embeddings: + logger.warning("Each page will be split into smaller chunks of text, but images will be of the entire page.") + sections = [ + Section(split_page, content=file, category=category) for split_page in processor.splitter.split_pages(pages) + ] + return sections + + +class FileStrategy(Strategy): + """ + Strategy for ingesting documents into a search service from files stored either locally or in a data lake storage account + """ + + def __init__( + self, + list_file_strategy: ListFileStrategy, + blob_manager: BlobManager, + search_info: SearchInfo, + file_processors: dict[str, FileProcessor], + document_action: DocumentAction = DocumentAction.Add, + embeddings: Optional[OpenAIEmbeddings] = None, + image_embeddings: Optional[ImageEmbeddings] = None, + search_analyzer_name: Optional[str] = None, + search_field_name_embedding: Optional[str] = None, + use_acls: bool = False, + category: Optional[str] = None, + use_content_understanding: bool = False, + content_understanding_endpoint: Optional[str] = None, + ): + self.list_file_strategy = list_file_strategy + self.blob_manager = blob_manager + self.file_processors = file_processors + self.document_action = document_action + self.embeddings = embeddings + self.image_embeddings = image_embeddings + self.search_analyzer_name = search_analyzer_name + self.search_field_name_embedding = search_field_name_embedding + self.search_info = search_info + self.use_acls = use_acls + self.category = category + self.use_content_understanding = use_content_understanding + self.content_understanding_endpoint = content_understanding_endpoint + + def setup_search_manager(self): + self.search_manager = SearchManager( + self.search_info, + self.search_analyzer_name, + self.use_acls, + False, + self.embeddings, + field_name_embedding=self.search_field_name_embedding, + search_images=self.image_embeddings is not None, + ) + + async def setup(self): + self.setup_search_manager() + await self.search_manager.create_index() + + if self.use_content_understanding: + if self.content_understanding_endpoint is None: + raise ValueError("Content Understanding is enabled but no endpoint was provided") + if isinstance(self.search_info.credential, AzureKeyCredential): + raise ValueError( + "AzureKeyCredential is not supported for Content Understanding, use keyless auth instead" + ) + cu_manager = ContentUnderstandingDescriber(self.content_understanding_endpoint, self.search_info.credential) + await cu_manager.create_analyzer() + + async def run(self): + self.setup_search_manager() + if self.document_action == DocumentAction.Add: + files = self.list_file_strategy.list() + async for file in files: + try: + sections = await parse_file(file, self.file_processors, self.category, self.image_embeddings) + if sections: + blob_sas_uris = await self.blob_manager.upload_blob(file) + blob_image_embeddings: Optional[list[list[float]]] = None + if self.image_embeddings and blob_sas_uris: + blob_image_embeddings = await self.image_embeddings.create_embeddings(blob_sas_uris) + await self.search_manager.update_content(sections, blob_image_embeddings, url=file.url) + finally: + if file: + file.close() + elif self.document_action == DocumentAction.Remove: + paths = self.list_file_strategy.list_paths() + async for path in paths: + await self.blob_manager.remove_blob(path) + await self.search_manager.remove_content(path) + elif self.document_action == DocumentAction.RemoveAll: + await self.blob_manager.remove_blob() + await self.search_manager.remove_content() + + +class UploadUserFileStrategy: + """ + Strategy for ingesting a file that has already been uploaded to a ADLS2 storage account + """ + + def __init__( + self, + search_info: SearchInfo, + file_processors: dict[str, FileProcessor], + embeddings: Optional[OpenAIEmbeddings] = None, + image_embeddings: Optional[ImageEmbeddings] = None, + search_field_name_embedding: Optional[str] = None, + ): + self.file_processors = file_processors + self.embeddings = embeddings + self.image_embeddings = image_embeddings + self.search_info = search_info + self.search_manager = SearchManager( + search_info=self.search_info, + search_analyzer_name=None, + use_acls=True, + use_int_vectorization=False, + embeddings=self.embeddings, + field_name_embedding=search_field_name_embedding, + search_images=False, + ) + self.search_field_name_embedding = search_field_name_embedding + + async def add_file(self, file: File): + if self.image_embeddings: + logging.warning("Image embeddings are not currently supported for the user upload feature") + sections = await parse_file(file, self.file_processors) + if sections: + await self.search_manager.update_content(sections, url=file.url) + + async def remove_file(self, filename: str, oid: str): + if filename is None or filename == "": + logging.warning("Filename is required to remove a file") + return + await self.search_manager.remove_content(filename, oid) diff --git a/app/backend/prepdocslib/htmlparser.py b/app/backend/prepdocslib/htmlparser.py new file mode 100644 index 0000000000..719045b393 --- /dev/null +++ b/app/backend/prepdocslib/htmlparser.py @@ -0,0 +1,50 @@ +import logging +import re +from collections.abc import AsyncGenerator +from typing import IO + +from bs4 import BeautifulSoup + +from .page import Page +from .parser import Parser + +logger = logging.getLogger("scripts") + + +def cleanup_data(data: str) -> str: + """Cleans up the given content using regexes + Args: + data: (str): The data to clean up. + Returns: + str: The cleaned up data. + """ + # match two or more newlines and replace them with one new line + output = re.sub(r"\n{2,}", "\n", data) + # match two or more spaces that are not newlines and replace them with one space + output = re.sub(r"[^\S\n]{2,}", " ", output) + # match two or more hyphens and replace them with two hyphens + output = re.sub(r"-{2,}", "--", output) + + return output.strip() + + +class LocalHTMLParser(Parser): + """Parses HTML text into Page objects.""" + + async def parse(self, content: IO) -> AsyncGenerator[Page, None]: + """Parses the given content. + To learn more, please visit https://pypi.org/project/beautifulsoup4/ + Args: + content (IO): The content to parse. + Returns: + Page: The parsed html Page. + """ + logger.info("Extracting text from '%s' using local HTML parser (BeautifulSoup)", content.name) + + data = content.read() + soup = BeautifulSoup(data, "html.parser") + + # Get text only from html file + result = soup.get_text() + + yield Page(0, 0, text=cleanup_data(result)) diff --git a/app/backend/prepdocslib/integratedvectorizerstrategy.py b/app/backend/prepdocslib/integratedvectorizerstrategy.py new file mode 100644 index 0000000000..9e89facc4c --- /dev/null +++ b/app/backend/prepdocslib/integratedvectorizerstrategy.py @@ -0,0 +1,195 @@ +import logging +from typing import Optional + +from azure.search.documents.indexes._generated.models import ( + NativeBlobSoftDeleteDeletionDetectionPolicy, +) +from azure.search.documents.indexes.models import ( + AzureOpenAIEmbeddingSkill, + IndexProjectionMode, + InputFieldMappingEntry, + OutputFieldMappingEntry, + SearchIndexer, + SearchIndexerDataContainer, + SearchIndexerDataSourceConnection, + SearchIndexerDataSourceType, + SearchIndexerIndexProjection, + SearchIndexerIndexProjectionSelector, + SearchIndexerIndexProjectionsParameters, + SearchIndexerSkillset, + SplitSkill, +) + +from .blobmanager import BlobManager +from .embeddings import AzureOpenAIEmbeddingService +from .listfilestrategy import ListFileStrategy +from .searchmanager import SearchManager +from .strategy import DocumentAction, SearchInfo, Strategy + +logger = logging.getLogger("scripts") + + +class IntegratedVectorizerStrategy(Strategy): + """ + Strategy for ingesting and vectorizing documents into a search service from files stored storage account + """ + + def __init__( + self, + list_file_strategy: ListFileStrategy, + blob_manager: BlobManager, + search_info: SearchInfo, + embeddings: AzureOpenAIEmbeddingService, + search_field_name_embedding: str, + subscription_id: str, + search_service_user_assigned_id: str, + document_action: DocumentAction = DocumentAction.Add, + search_analyzer_name: Optional[str] = None, + use_acls: bool = False, + category: Optional[str] = None, + ): + + self.list_file_strategy = list_file_strategy + self.blob_manager = blob_manager + self.document_action = document_action + self.embeddings = embeddings + self.search_field_name_embedding = search_field_name_embedding + self.subscription_id = subscription_id + self.search_user_assigned_identity = search_service_user_assigned_id + self.search_analyzer_name = search_analyzer_name + self.use_acls = use_acls + self.category = category + self.search_info = search_info + prefix = f"{self.search_info.index_name}-{self.search_field_name_embedding}" + self.skillset_name = f"{prefix}-skillset" + self.indexer_name = f"{prefix}-indexer" + self.data_source_name = f"{prefix}-blob" + + async def create_embedding_skill(self, index_name: str) -> SearchIndexerSkillset: + """ + Create a skillset for the indexer to chunk documents and generate embeddings + """ + + split_skill = SplitSkill( + name="split-skill", + description="Split skill to chunk documents", + text_split_mode="pages", + context="/document", + maximum_page_length=2048, + page_overlap_length=20, + inputs=[ + InputFieldMappingEntry(name="text", source="/document/content"), + ], + outputs=[OutputFieldMappingEntry(name="textItems", target_name="pages")], + ) + + embedding_skill = AzureOpenAIEmbeddingSkill( + name="embedding-skill", + description="Skill to generate embeddings via Azure OpenAI", + context="/document/pages/*", + resource_url=f"https://{self.embeddings.open_ai_service}.openai.azure.com", + deployment_name=self.embeddings.open_ai_deployment, + model_name=self.embeddings.open_ai_model_name, + dimensions=self.embeddings.open_ai_dimensions, + inputs=[ + InputFieldMappingEntry(name="text", source="/document/pages/*"), + ], + outputs=[OutputFieldMappingEntry(name="embedding", target_name="vector")], + ) + + index_projection = SearchIndexerIndexProjection( + selectors=[ + SearchIndexerIndexProjectionSelector( + target_index_name=index_name, + parent_key_field_name="parent_id", + source_context="/document/pages/*", + mappings=[ + InputFieldMappingEntry(name="content", source="/document/pages/*"), + InputFieldMappingEntry(name="sourcepage", source="/document/metadata_storage_name"), + InputFieldMappingEntry(name="sourcefile", source="/document/metadata_storage_name"), + InputFieldMappingEntry(name="storageUrl", source="/document/metadata_storage_path"), + InputFieldMappingEntry( + name=self.search_field_name_embedding, source="/document/pages/*/vector" + ), + ], + ), + ], + parameters=SearchIndexerIndexProjectionsParameters( + projection_mode=IndexProjectionMode.SKIP_INDEXING_PARENT_DOCUMENTS + ), + ) + + skillset = SearchIndexerSkillset( + name=self.skillset_name, + description="Skillset to chunk documents and generate embeddings", + skills=[split_skill, embedding_skill], + index_projection=index_projection, + ) + + return skillset + + async def setup(self): + logger.info("Setting up search index using integrated vectorization...") + search_manager = SearchManager( + search_info=self.search_info, + search_analyzer_name=self.search_analyzer_name, + use_acls=self.use_acls, + use_int_vectorization=True, + embeddings=self.embeddings, + field_name_embedding=self.search_field_name_embedding, + search_images=False, + ) + + await search_manager.create_index() + + ds_client = self.search_info.create_search_indexer_client() + ds_container = SearchIndexerDataContainer(name=self.blob_manager.container) + data_source_connection = SearchIndexerDataSourceConnection( + name=self.data_source_name, + type=SearchIndexerDataSourceType.AZURE_BLOB, + connection_string=self.blob_manager.get_managedidentity_connectionstring(), + container=ds_container, + data_deletion_detection_policy=NativeBlobSoftDeleteDeletionDetectionPolicy(), + ) + + await ds_client.create_or_update_data_source_connection(data_source_connection) + + embedding_skillset = await self.create_embedding_skill(self.search_info.index_name) + await ds_client.create_or_update_skillset(embedding_skillset) + await ds_client.close() + + async def run(self): + if self.document_action == DocumentAction.Add: + files = self.list_file_strategy.list() + async for file in files: + try: + await self.blob_manager.upload_blob(file) + finally: + if file: + file.close() + elif self.document_action == DocumentAction.Remove: + paths = self.list_file_strategy.list_paths() + async for path in paths: + await self.blob_manager.remove_blob(path) + elif self.document_action == DocumentAction.RemoveAll: + await self.blob_manager.remove_blob() + + # Create an indexer + indexer = SearchIndexer( + name=self.indexer_name, + description="Indexer to index documents and generate embeddings", + skillset_name=self.skillset_name, + target_index_name=self.search_info.index_name, + data_source_name=self.data_source_name, + ) + + indexer_client = self.search_info.create_search_indexer_client() + indexer_result = await indexer_client.create_or_update_indexer(indexer) + + # Run the indexer + await indexer_client.run_indexer(self.indexer_name) + await indexer_client.close() + + logger.info( + f"Successfully created index, indexer: {indexer_result.name}, and skillset. Please navigate to search service in Azure Portal to view the status of the indexer." + ) diff --git a/app/backend/prepdocslib/jsonparser.py b/app/backend/prepdocslib/jsonparser.py new file mode 100644 index 0000000000..bc17c7ceec --- /dev/null +++ b/app/backend/prepdocslib/jsonparser.py @@ -0,0 +1,24 @@ +import json +from collections.abc import AsyncGenerator +from typing import IO + +from .page import Page +from .parser import Parser + + +class JsonParser(Parser): + """ + Concrete parser that can parse JSON into Page objects. A top-level object becomes a single Page, while a top-level array becomes multiple Page objects. + """ + + async def parse(self, content: IO) -> AsyncGenerator[Page, None]: + offset = 0 + data = json.loads(content.read()) + if isinstance(data, list): + for i, obj in enumerate(data): + offset += 1 # For opening bracket or comma before object + page_text = json.dumps(obj) + yield Page(i, offset, page_text) + offset += len(page_text) + elif isinstance(data, dict): + yield Page(0, 0, json.dumps(data)) diff --git a/app/backend/prepdocslib/listfilestrategy.py b/app/backend/prepdocslib/listfilestrategy.py new file mode 100644 index 0000000000..bdceef0754 --- /dev/null +++ b/app/backend/prepdocslib/listfilestrategy.py @@ -0,0 +1,178 @@ +import base64 +import hashlib +import logging +import os +import re +import tempfile +from abc import ABC +from collections.abc import AsyncGenerator +from glob import glob +from typing import IO, Optional, Union + +from azure.core.credentials_async import AsyncTokenCredential +from azure.storage.filedatalake.aio import ( + DataLakeServiceClient, +) + +logger = logging.getLogger("scripts") + + +class File: + """ + Represents a file stored either locally or in a data lake storage account + This file might contain access control information about which users or groups can access it + """ + + def __init__(self, content: IO, acls: Optional[dict[str, list]] = None, url: Optional[str] = None): + self.content = content + self.acls = acls or {} + self.url = url + + def filename(self): + return os.path.basename(self.content.name) + + def file_extension(self): + return os.path.splitext(self.content.name)[1] + + def filename_to_id(self): + filename_ascii = re.sub("[^0-9a-zA-Z_-]", "_", self.filename()) + filename_hash = base64.b16encode(self.filename().encode("utf-8")).decode("ascii") + acls_hash = "" + if self.acls: + acls_hash = base64.b16encode(str(self.acls).encode("utf-8")).decode("ascii") + return f"file-{filename_ascii}-{filename_hash}{acls_hash}" + + def close(self): + if self.content: + self.content.close() + + +class ListFileStrategy(ABC): + """ + Abstract strategy for listing files that are located somewhere. For example, on a local computer or remotely in a storage account + """ + + async def list(self) -> AsyncGenerator[File, None]: + if False: # pragma: no cover - this is necessary for mypy to type check + yield + + async def list_paths(self) -> AsyncGenerator[str, None]: + if False: # pragma: no cover - this is necessary for mypy to type check + yield + + +class LocalListFileStrategy(ListFileStrategy): + """ + Concrete strategy for listing files that are located in a local filesystem + """ + + def __init__(self, path_pattern: str): + self.path_pattern = path_pattern + + async def list_paths(self) -> AsyncGenerator[str, None]: + async for p in self._list_paths(self.path_pattern): + yield p + + async def _list_paths(self, path_pattern: str) -> AsyncGenerator[str, None]: + for path in glob(path_pattern): + if os.path.isdir(path): + async for p in self._list_paths(f"{path}/*"): + yield p + else: + # Only list files, not directories + yield path + + async def list(self) -> AsyncGenerator[File, None]: + async for path in self.list_paths(): + if not self.check_md5(path): + yield File(content=open(path, mode="rb")) + + def check_md5(self, path: str) -> bool: + # if filename ends in .md5 skip + if path.endswith(".md5"): + return True + + # if there is a file called .md5 in this directory, see if its updated + stored_hash = None + with open(path, "rb") as file: + existing_hash = hashlib.md5(file.read()).hexdigest() + hash_path = f"{path}.md5" + if os.path.exists(hash_path): + with open(hash_path, encoding="utf-8") as md5_f: + stored_hash = md5_f.read() + + if stored_hash and stored_hash.strip() == existing_hash.strip(): + logger.info("Skipping %s, no changes detected.", path) + return True + + # Write the hash + with open(hash_path, "w", encoding="utf-8") as md5_f: + md5_f.write(existing_hash) + + return False + + +class ADLSGen2ListFileStrategy(ListFileStrategy): + """ + Concrete strategy for listing files that are located in a data lake storage account + """ + + def __init__( + self, + data_lake_storage_account: str, + data_lake_filesystem: str, + data_lake_path: str, + credential: Union[AsyncTokenCredential, str], + ): + self.data_lake_storage_account = data_lake_storage_account + self.data_lake_filesystem = data_lake_filesystem + self.data_lake_path = data_lake_path + self.credential = credential + + async def list_paths(self) -> AsyncGenerator[str, None]: + async with DataLakeServiceClient( + account_url=f"https://{self.data_lake_storage_account}.dfs.core.windows.net", credential=self.credential + ) as service_client, service_client.get_file_system_client(self.data_lake_filesystem) as filesystem_client: + async for path in filesystem_client.get_paths(path=self.data_lake_path, recursive=True): + if path.is_directory: + continue + + yield path.name + + async def list(self) -> AsyncGenerator[File, None]: + async with DataLakeServiceClient( + account_url=f"https://{self.data_lake_storage_account}.dfs.core.windows.net", credential=self.credential + ) as service_client, service_client.get_file_system_client(self.data_lake_filesystem) as filesystem_client: + async for path in self.list_paths(): + temp_file_path = os.path.join(tempfile.gettempdir(), os.path.basename(path)) + try: + async with filesystem_client.get_file_client(path) as file_client: + with open(temp_file_path, "wb") as temp_file: + downloader = await file_client.download_file() + await downloader.readinto(temp_file) + # Parse out user ids and group ids + acls: dict[str, list[str]] = {"oids": [], "groups": []} + # https://learn.microsoft.com/python/api/azure-storage-file-datalake/azure.storage.filedatalake.datalakefileclient?view=azure-python#azure-storage-filedatalake-datalakefileclient-get-access-control + # Request ACLs as GUIDs + access_control = await file_client.get_access_control(upn=False) + acl_list = access_control["acl"] + # https://learn.microsoft.com/azure/storage/blobs/data-lake-storage-access-control + # ACL Format: user::rwx,group::r-x,other::r--,user:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx:r-- + acl_list = acl_list.split(",") + for acl in acl_list: + acl_parts: list = acl.split(":") + if len(acl_parts) != 3: + continue + if len(acl_parts[1]) == 0: + continue + if acl_parts[0] == "user" and "r" in acl_parts[2]: + acls["oids"].append(acl_parts[1]) + if acl_parts[0] == "group" and "r" in acl_parts[2]: + acls["groups"].append(acl_parts[1]) + yield File(content=open(temp_file_path, "rb"), acls=acls, url=file_client.url) + except Exception as data_lake_exception: + logger.error(f"\tGot an error while reading {path} -> {data_lake_exception} --> skipping file") + try: + os.remove(temp_file_path) + except Exception as file_delete_exception: + logger.error(f"\tGot an error while deleting {temp_file_path} -> {file_delete_exception}") diff --git a/app/backend/prepdocslib/mediadescriber.py b/app/backend/prepdocslib/mediadescriber.py new file mode 100644 index 0000000000..5aae79232e --- /dev/null +++ b/app/backend/prepdocslib/mediadescriber.py @@ -0,0 +1,107 @@ +import logging +from abc import ABC + +import aiohttp +from azure.core.credentials_async import AsyncTokenCredential +from azure.identity.aio import get_bearer_token_provider +from rich.progress import Progress +from tenacity import retry, retry_if_exception_type, stop_after_attempt, wait_fixed + +logger = logging.getLogger("scripts") + + +class MediaDescriber(ABC): + + async def describe_image(self, image_bytes) -> str: + raise NotImplementedError # pragma: no cover + + +class ContentUnderstandingDescriber: + CU_API_VERSION = "2024-12-01-preview" + + analyzer_schema = { + "analyzerId": "image_analyzer", + "name": "Image understanding", + "description": "Extract detailed structured information from images extracted from documents.", + "baseAnalyzerId": "prebuilt-image", + "scenario": "image", + "config": {"returnDetails": False}, + "fieldSchema": { + "name": "ImageInformation", + "descriptions": "Description of image.", + "fields": { + "Description": { + "type": "string", + "description": "Description of the image. If the image has a title, start with the title. Include a 2-sentence summary. If the image is a chart, diagram, or table, include the underlying data in an HTML table tag, with accurate numbers. If the image is a chart, describe any axis or legends. The only allowed HTML tags are the table/thead/tr/td/tbody tags.", + }, + }, + }, + } + + def __init__(self, endpoint: str, credential: AsyncTokenCredential): + self.endpoint = endpoint + self.credential = credential + + async def poll_api(self, session, poll_url, headers): + + @retry(stop=stop_after_attempt(60), wait=wait_fixed(2), retry=retry_if_exception_type(ValueError)) + async def poll(): + async with session.get(poll_url, headers=headers) as response: + response.raise_for_status() + response_json = await response.json() + if response_json["status"] == "Failed": + raise Exception("Failed") + if response_json["status"] == "Running": + raise ValueError("Running") + return response_json + + return await poll() + + async def create_analyzer(self): + logger.info("Creating analyzer '%s'...", self.analyzer_schema["analyzerId"]) + + token_provider = get_bearer_token_provider(self.credential, "https://cognitiveservices.azure.com/.default") + token = await token_provider() + headers = {"Authorization": f"Bearer {token}", "Content-Type": "application/json"} + params = {"api-version": self.CU_API_VERSION} + analyzer_id = self.analyzer_schema["analyzerId"] + cu_endpoint = f"{self.endpoint}/contentunderstanding/analyzers/{analyzer_id}" + async with aiohttp.ClientSession() as session: + async with session.put( + url=cu_endpoint, params=params, headers=headers, json=self.analyzer_schema + ) as response: + if response.status == 409: + logger.info("Analyzer '%s' already exists.", analyzer_id) + return + elif response.status != 201: + data = await response.text() + raise Exception("Error creating analyzer", data) + else: + poll_url = response.headers.get("Operation-Location") + + with Progress() as progress: + progress.add_task("Creating analyzer...", total=None, start=False) + await self.poll_api(session, poll_url, headers) + + async def describe_image(self, image_bytes: bytes) -> str: + logger.info("Sending image to Azure Content Understanding service...") + async with aiohttp.ClientSession() as session: + token = await self.credential.get_token("https://cognitiveservices.azure.com/.default") + headers = {"Authorization": "Bearer " + token.token} + params = {"api-version": self.CU_API_VERSION} + analyzer_name = self.analyzer_schema["analyzerId"] + async with session.post( + url=f"{self.endpoint}/contentunderstanding/analyzers/{analyzer_name}:analyze", + params=params, + headers=headers, + data=image_bytes, + ) as response: + response.raise_for_status() + poll_url = response.headers["Operation-Location"] + + with Progress() as progress: + progress.add_task("Processing...", total=None, start=False) + results = await self.poll_api(session, poll_url, headers) + + fields = results["result"]["contents"][0]["fields"] + return fields["Description"]["valueString"] diff --git a/app/backend/prepdocslib/page.py b/app/backend/prepdocslib/page.py new file mode 100644 index 0000000000..857235c571 --- /dev/null +++ b/app/backend/prepdocslib/page.py @@ -0,0 +1,28 @@ +class Page: + """ + A single page from a document + + Attributes: + page_num (int): Page number (0-indexed) + offset (int): If the text of the entire Document was concatenated into a single string, the index of the first character on the page. For example, if page 1 had the text "hello" and page 2 had the text "world", the offset of page 2 is 5 ("hellow") + text (str): The text of the page + """ + + def __init__(self, page_num: int, offset: int, text: str): + self.page_num = page_num + self.offset = offset + self.text = text + + +class SplitPage: + """ + A section of a page that has been split into a smaller chunk. + + Attributes: + page_num (int): Page number (0-indexed) + text (str): The text of the section + """ + + def __init__(self, page_num: int, text: str): + self.page_num = page_num + self.text = text diff --git a/app/backend/prepdocslib/parser.py b/app/backend/prepdocslib/parser.py new file mode 100644 index 0000000000..1552c00fdc --- /dev/null +++ b/app/backend/prepdocslib/parser.py @@ -0,0 +1,15 @@ +from abc import ABC +from collections.abc import AsyncGenerator +from typing import IO + +from .page import Page + + +class Parser(ABC): + """ + Abstract parser that parses content into Page objects + """ + + async def parse(self, content: IO) -> AsyncGenerator[Page, None]: + if False: + yield # pragma: no cover - this is necessary for mypy to type check diff --git a/app/backend/prepdocslib/pdfparser.py b/app/backend/prepdocslib/pdfparser.py new file mode 100644 index 0000000000..c96980d21c --- /dev/null +++ b/app/backend/prepdocslib/pdfparser.py @@ -0,0 +1,251 @@ +import html +import io +import logging +from collections.abc import AsyncGenerator +from enum import Enum +from typing import IO, Union + +import pymupdf +from azure.ai.documentintelligence.aio import DocumentIntelligenceClient +from azure.ai.documentintelligence.models import ( + AnalyzeDocumentRequest, + AnalyzeResult, + DocumentFigure, + DocumentTable, +) +from azure.core.credentials import AzureKeyCredential +from azure.core.credentials_async import AsyncTokenCredential +from azure.core.exceptions import HttpResponseError +from PIL import Image +from pypdf import PdfReader + +from .mediadescriber import ContentUnderstandingDescriber +from .page import Page +from .parser import Parser + +logger = logging.getLogger("scripts") + + +class LocalPdfParser(Parser): + """ + Concrete parser backed by PyPDF that can parse PDFs into pages + To learn more, please visit https://pypi.org/project/pypdf/ + """ + + async def parse(self, content: IO) -> AsyncGenerator[Page, None]: + logger.info("Extracting text from '%s' using local PDF parser (pypdf)", content.name) + + reader = PdfReader(content) + pages = reader.pages + offset = 0 + for page_num, p in enumerate(pages): + page_text = p.extract_text() + yield Page(page_num=page_num, offset=offset, text=page_text) + offset += len(page_text) + + +class DocumentAnalysisParser(Parser): + """ + Concrete parser backed by Azure AI Document Intelligence that can parse many document formats into pages + To learn more, please visit https://learn.microsoft.com/azure/ai-services/document-intelligence/overview + """ + + def __init__( + self, + endpoint: str, + credential: Union[AsyncTokenCredential, AzureKeyCredential], + model_id="prebuilt-layout", + use_content_understanding=True, + content_understanding_endpoint: Union[str, None] = None, + ): + self.model_id = model_id + self.endpoint = endpoint + self.credential = credential + self.use_content_understanding = use_content_understanding + self.content_understanding_endpoint = content_understanding_endpoint + + async def parse(self, content: IO) -> AsyncGenerator[Page, None]: + logger.info("Extracting text from '%s' using Azure Document Intelligence", content.name) + + async with DocumentIntelligenceClient( + endpoint=self.endpoint, credential=self.credential + ) as document_intelligence_client: + file_analyzed = False + if self.use_content_understanding: + if self.content_understanding_endpoint is None: + raise ValueError("Content Understanding is enabled but no endpoint was provided") + if isinstance(self.credential, AzureKeyCredential): + raise ValueError( + "AzureKeyCredential is not supported for Content Understanding, use keyless auth instead" + ) + cu_describer = ContentUnderstandingDescriber(self.content_understanding_endpoint, self.credential) + content_bytes = content.read() + try: + poller = await document_intelligence_client.begin_analyze_document( + model_id="prebuilt-layout", + analyze_request=AnalyzeDocumentRequest(bytes_source=content_bytes), + output=["figures"], + features=["ocrHighResolution"], + output_content_format="markdown", + ) + doc_for_pymupdf = pymupdf.open(stream=io.BytesIO(content_bytes)) + file_analyzed = True + except HttpResponseError as e: + content.seek(0) + if e.error and e.error.code == "InvalidArgument": + logger.error( + "This document type does not support media description. Proceeding with standard analysis." + ) + else: + logger.error( + "Unexpected error analyzing document for media description: %s. Proceeding with standard analysis.", + e, + ) + + if file_analyzed is False: + poller = await document_intelligence_client.begin_analyze_document( + model_id=self.model_id, analyze_request=content, content_type="application/octet-stream" + ) + analyze_result: AnalyzeResult = await poller.result() + + offset = 0 + for page in analyze_result.pages: + tables_on_page = [ + table + for table in (analyze_result.tables or []) + if table.bounding_regions and table.bounding_regions[0].page_number == page.page_number + ] + figures_on_page = [] + if self.use_content_understanding: + figures_on_page = [ + figure + for figure in (analyze_result.figures or []) + if figure.bounding_regions and figure.bounding_regions[0].page_number == page.page_number + ] + + class ObjectType(Enum): + NONE = -1 + TABLE = 0 + FIGURE = 1 + + page_offset = page.spans[0].offset + page_length = page.spans[0].length + mask_chars: list[tuple[ObjectType, Union[int, None]]] = [(ObjectType.NONE, None)] * page_length + # mark all positions of the table spans in the page + for table_idx, table in enumerate(tables_on_page): + for span in table.spans: + # replace all table spans with "table_id" in table_chars array + for i in range(span.length): + idx = span.offset - page_offset + i + if idx >= 0 and idx < page_length: + mask_chars[idx] = (ObjectType.TABLE, table_idx) + # mark all positions of the figure spans in the page + for figure_idx, figure in enumerate(figures_on_page): + for span in figure.spans: + # replace all figure spans with "figure_id" in figure_chars array + for i in range(span.length): + idx = span.offset - page_offset + i + if idx >= 0 and idx < page_length: + mask_chars[idx] = (ObjectType.FIGURE, figure_idx) + + # build page text by replacing characters in table spans with table html + page_text = "" + added_objects = set() # set of object types todo mypy + for idx, mask_char in enumerate(mask_chars): + object_type, object_idx = mask_char + if object_type == ObjectType.NONE: + page_text += analyze_result.content[page_offset + idx] + elif object_type == ObjectType.TABLE: + if object_idx is None: + raise ValueError("Expected object_idx to be set") + if mask_char not in added_objects: + page_text += DocumentAnalysisParser.table_to_html(tables_on_page[object_idx]) + added_objects.add(mask_char) + elif object_type == ObjectType.FIGURE: + if cu_describer is None: + raise ValueError("cu_describer should not be None, unable to describe figure") + if object_idx is None: + raise ValueError("Expected object_idx to be set") + if mask_char not in added_objects: + figure_html = await DocumentAnalysisParser.figure_to_html( + doc_for_pymupdf, figures_on_page[object_idx], cu_describer + ) + page_text += figure_html + added_objects.add(mask_char) + # We remove these comments since they are not needed and skew the page numbers + page_text = page_text.replace("", "") + # We remove excess newlines at the beginning and end of the page + page_text = page_text.strip() + yield Page(page_num=page.page_number - 1, offset=offset, text=page_text) + offset += len(page_text) + + @staticmethod + async def figure_to_html( + doc: pymupdf.Document, figure: DocumentFigure, cu_describer: ContentUnderstandingDescriber + ) -> str: + figure_title = (figure.caption and figure.caption.content) or "" + logger.info("Describing figure %s with title '%s'", figure.id, figure_title) + if not figure.bounding_regions: + return f"
{figure_description}
+ {planning && (
+
+
+
+ )}
+
+ );
+};
diff --git a/app/frontend/src/components/AnalysisPanel/AnalysisPanel.module.css b/app/frontend/src/components/AnalysisPanel/AnalysisPanel.module.css
index 909ac03d44..9dae51a8e8 100644
--- a/app/frontend/src/components/AnalysisPanel/AnalysisPanel.module.css
+++ b/app/frontend/src/components/AnalysisPanel/AnalysisPanel.module.css
@@ -1,6 +1,150 @@
.thoughtProcess {
font-family: source-code-pro, Menlo, Monaco, Consolas, "Courier New", monospace;
word-wrap: break-word;
- padding-top: 12px;
- padding-bottom: 12px;
+ padding-top: 0.75em;
+ padding-bottom: 0.75em;
+}
+
+.tList {
+ padding: 1.25em 1.25em 0 1.25em;
+ display: inline-block;
+ background: #e9e9e9;
+}
+
+.tListItem {
+ list-style: none;
+ margin: auto;
+ margin-left: 1.25em;
+ min-height: 3.125em;
+ border-left: 0.0625em solid #123bb6;
+ padding: 0 0 1.875em 1.875em;
+ position: relative;
+}
+
+.tListItem:last-child {
+ border-left: 0;
+}
+
+.tListItem::before {
+ position: absolute;
+ left: -18px;
+ top: -5px;
+ content: " ";
+ border: 8px solid #d1dbfa;
+ border-radius: 500%;
+ background: #123bb6;
+ height: 20px;
+ width: 20px;
+}
+
+.tStep {
+ color: #123bb6;
+ position: relative;
+ font-size: 0.875em;
+ margin-bottom: 0.5em;
+}
+
+.tQuery {
+ color: #010207;
+ position: relative;
+ font-size: 0.875em;
+ margin-bottom: 0.5em;
+}
+
+.tCodeBlock {
+ max-height: 18.75em;
+}
+
+.tProp {
+ background-color: #d7d7d7;
+ color: #333232;
+ font-size: 0.75em;
+ padding: 0.1875em 0.625em;
+ border-radius: 0.625em;
+ margin-bottom: 0.5em;
+}
+
+.citationImg {
+ height: 28.125rem;
+ max-width: 100%;
+ object-fit: contain;
+}
+
+.header {
+ color: #123bb6;
+ position: relative;
+ font-size: 0.875em;
+ margin-bottom: 0.5em;
+}
+
+.reasoningEffort {
+ font-size: 14px;
+ margin-bottom: 8px;
+}
+
+.tokenUsageGraph {
+ margin: 16px 0;
+}
+
+.tokenBar {
+ height: 20px;
+ margin-bottom: 4px;
+ padding-left: 4px;
+ font-size: 12px;
+ display: flex;
+ align-items: center;
+ white-space: nowrap;
+ overflow: hidden;
+}
+
+.tokenBar {
+ height: 20px;
+ margin-bottom: 4px;
+ padding-left: 4px;
+ font-size: 12px;
+ display: flex;
+ flex-wrap: wrap;
+ align-items: center;
+ background-color: #d7d7d7;
+ white-space: nowrap;
+ overflow: visible;
+ min-width: max-content;
+}
+
+/* Adjust tokenLabel to allow bar-specific text color overrides */
+.tokenLabel {
+ padding-right: 4px;
+}
+
+.primaryBarContainer {
+ width: fit-content;
+ display: flex;
+ gap: 4px;
+}
+
+.promptBar {
+ background-color: #a82424;
+ color: #ffffff; /* White text for contrast */
+}
+
+.reasoningBar {
+ background-color: #265e29;
+ color: #ffffff;
+}
+
+.outputBar {
+ background-color: #12579b;
+ color: #ffffff;
+}
+
+.totalBar {
+ background-color: #424242;
+ color: #ffffff;
+}
+
+.subqueriesTable,
+.subqueriesTable th,
+.subqueriesTable td,
+.subqueriesTable tr {
+ background: #fff;
}
diff --git a/app/frontend/src/components/AnalysisPanel/AnalysisPanel.tsx b/app/frontend/src/components/AnalysisPanel/AnalysisPanel.tsx
index cd01a08e2f..2cee00c761 100644
--- a/app/frontend/src/components/AnalysisPanel/AnalysisPanel.tsx
+++ b/app/frontend/src/components/AnalysisPanel/AnalysisPanel.tsx
@@ -1,11 +1,16 @@
-import { Pivot, PivotItem } from "@fluentui/react";
-import DOMPurify from "dompurify";
-
+import { Stack, Pivot, PivotItem } from "@fluentui/react";
+import { useTranslation } from "react-i18next";
import styles from "./AnalysisPanel.module.css";
import { SupportingContent } from "../SupportingContent";
-import { AskResponse } from "../../api";
+import { ChatAppResponse } from "../../api";
import { AnalysisPanelTabs } from "./AnalysisPanelTabs";
+import { ThoughtProcess } from "./ThoughtProcess";
+import { MarkdownViewer } from "../MarkdownViewer";
+import { useMsal } from "@azure/msal-react";
+import { getHeaders } from "../../api";
+import { useLogin, getToken } from "../../authConfig";
+import { useState, useEffect } from "react";
interface Props {
className: string;
@@ -13,17 +18,58 @@ interface Props {
onActiveTabChanged: (tab: AnalysisPanelTabs) => void;
activeCitation: string | undefined;
citationHeight: string;
- answer: AskResponse;
+ answer: ChatAppResponse;
}
const pivotItemDisabledStyle = { disabled: true, style: { color: "grey" } };
export const AnalysisPanel = ({ answer, activeTab, activeCitation, citationHeight, className, onActiveTabChanged }: Props) => {
- const isDisabledThoughtProcessTab: boolean = !answer.thoughts;
- const isDisabledSupportingContentTab: boolean = !answer.data_points.length;
+ const isDisabledThoughtProcessTab: boolean = !answer.context.thoughts;
+ const isDisabledSupportingContentTab: boolean = !answer.context.data_points;
const isDisabledCitationTab: boolean = !activeCitation;
+ const [citation, setCitation] = useState("");
+
+ const client = useLogin ? useMsal().instance : undefined;
+ const { t } = useTranslation();
+
+ const fetchCitation = async () => {
+ const token = client ? await getToken(client) : undefined;
+ if (activeCitation) {
+ // Get hash from the URL as it may contain #page=N
+ // which helps browser PDF renderer jump to correct page N
+ const originalHash = activeCitation.indexOf("#") ? activeCitation.split("#")[1] : "";
+ const response = await fetch(activeCitation, {
+ method: "GET",
+ headers: await getHeaders(token)
+ });
+ const citationContent = await response.blob();
+ let citationObjectUrl = URL.createObjectURL(citationContent);
+ // Add hash back to the new blob URL
+ if (originalHash) {
+ citationObjectUrl += "#" + originalHash;
+ }
+ setCitation(citationObjectUrl);
+ }
+ };
+ useEffect(() => {
+ fetchCitation();
+ }, []);
+
+ const renderFileViewer = () => {
+ if (!activeCitation) {
+ return null;
+ }
- const sanitizedThoughts = DOMPurify.sanitize(answer.thoughts!);
+ const fileExtension = activeCitation.split(".").pop()?.toLowerCase();
+ switch (fileExtension) {
+ case "png":
+ return Subqueries
+ {queries.length > 0 && (
+ | Subquery | +Total Result Count | +Elapsed MS | +
|---|---|---|
| {q.query.search} | +{q.count} | +{q.elapsed_ms} | +
-
+ {thoughts.map((t, ind) => {
+ return (
+
-
+ {t.title}+
+ {t.props && + (Object.keys(t.props).filter(k => k !== "token_usage" && k !== "query_plan") || []).map((k: any) => ( + + {k}: {JSON.stringify(t.props?.[k])} + + ))} + + {t.props?.token_usage &&} + {t.props?.query_plan && } + {Array.isArray(t.description) ? ( + + {JSON.stringify(t.description, null, 2)} + + ) : ( +{t.description}+ )} +
+ );
+ })}
+
+
+ );
+};
diff --git a/app/frontend/src/components/Answer/Answer.module.css b/app/frontend/src/components/Answer/Answer.module.css
index f2c3bf6950..8722a67b02 100644
--- a/app/frontend/src/components/Answer/Answer.module.css
+++ b/app/frontend/src/components/Answer/Answer.module.css
@@ -1,22 +1,30 @@
.answerContainer {
- padding: 20px;
+ padding: 1.25em;
background: rgb(249, 249, 249);
- border-radius: 8px;
- box-shadow: 0px 2px 4px rgba(0, 0, 0, 0.14), 0px 0px 2px rgba(0, 0, 0, 0.12);
- outline: transparent solid 1px;
+ border-radius: 0.5em;
+ box-shadow:
+ 0em 0.125em 0.25em rgba(0, 0, 0, 0.14),
+ 0em 0em 0.125em rgba(0, 0, 0, 0.12);
+ outline: transparent solid 0.0625em;
}
.answerLogo {
- font-size: 28px;
+ font-size: 1.75em;
}
.answerText {
- font-size: 16px;
+ font-size: 1rem;
font-weight: 400;
- line-height: 22px;
- padding-top: 16px;
- padding-bottom: 16px;
- white-space: pre-line;
+ line-height: 1.375em;
+ padding-top: 1em;
+ padding-bottom: 1em;
+}
+
+.answerText h1,
+h2,
+h2 {
+ font-size: 1rem;
+ font-weight: bold;
}
.answerText table {
@@ -26,25 +34,25 @@
.answerText td,
.answerText th {
border: 1px solid;
- padding: 5px;
+ padding: 0.3125em;
}
.selected {
- outline: 2px solid rgba(115, 118, 225, 1);
+ outline: 0.125em solid rgba(115, 118, 225, 1);
}
.citationLearnMore {
- margin-right: 5px;
+ margin-right: 0.3125em;
font-weight: 600;
- line-height: 24px;
+ line-height: 1.5em;
}
.citation {
font-weight: 500;
- line-height: 24px;
+ line-height: 1.5em;
text-align: center;
- border-radius: 4px;
- padding: 0px 8px;
+ border-radius: 0.25em;
+ padding: 0em 0.5em;
background: #d1dbfa;
color: #123bb6;
text-decoration: none;
@@ -56,21 +64,21 @@
}
.followupQuestionsList {
- margin-top: 10px;
+ margin-top: 0.625em;
}
.followupQuestionLearnMore {
- margin-right: 5px;
+ margin-right: 0.3125em;
font-weight: 600;
- line-height: 24px;
+ line-height: 1.5em;
}
.followupQuestion {
font-weight: 600;
- line-height: 24px;
+ line-height: 1.5em;
text-align: center;
- border-radius: 4px;
- padding: 0px 8px;
+ border-radius: 0.25em;
+ padding: 0em 0.5em;
background: #e8ebfa;
color: black;
font-style: italic;
@@ -92,18 +100,18 @@ sup {
display: inline-flex;
align-items: center;
justify-content: center;
- font-size: 10px;
+ font-size: 0.625em;
font-weight: 600;
vertical-align: top;
top: -1;
- margin: 0px 2px;
- min-width: 14px;
- height: 14px;
- border-radius: 3px;
+ margin: 0em 0.125em;
+ min-width: 0.875em;
+ height: 0.875em;
+ border-radius: 0.1875em;
background: #d1dbfa;
color: #123bb6;
text-decoration-color: transparent;
- outline: transparent solid 1px;
+ outline: transparent solid 0.0625em;
cursor: pointer;
}
diff --git a/app/frontend/src/components/Answer/Answer.tsx b/app/frontend/src/components/Answer/Answer.tsx
index 78c281dc0d..75b0a03504 100644
--- a/app/frontend/src/components/Answer/Answer.tsx
+++ b/app/frontend/src/components/Answer/Answer.tsx
@@ -1,35 +1,65 @@
-import { useMemo } from "react";
+import { useMemo, useState } from "react";
import { Stack, IconButton } from "@fluentui/react";
+import { useTranslation } from "react-i18next";
import DOMPurify from "dompurify";
+import ReactMarkdown from "react-markdown";
+import remarkGfm from "remark-gfm";
+import rehypeRaw from "rehype-raw";
import styles from "./Answer.module.css";
-
-import { AskResponse, getCitationFilePath } from "../../api";
+import { ChatAppResponse, getCitationFilePath, SpeechConfig } from "../../api";
import { parseAnswerToHtml } from "./AnswerParser";
import { AnswerIcon } from "./AnswerIcon";
+import { SpeechOutputBrowser } from "./SpeechOutputBrowser";
+import { SpeechOutputAzure } from "./SpeechOutputAzure";
interface Props {
- answer: AskResponse;
+ answer: ChatAppResponse;
+ index: number;
+ speechConfig: SpeechConfig;
isSelected?: boolean;
+ isStreaming: boolean;
onCitationClicked: (filePath: string) => void;
onThoughtProcessClicked: () => void;
onSupportingContentClicked: () => void;
onFollowupQuestionClicked?: (question: string) => void;
showFollowupQuestions?: boolean;
+ showSpeechOutputBrowser?: boolean;
+ showSpeechOutputAzure?: boolean;
}
export const Answer = ({
answer,
+ index,
+ speechConfig,
isSelected,
+ isStreaming,
onCitationClicked,
onThoughtProcessClicked,
onSupportingContentClicked,
onFollowupQuestionClicked,
- showFollowupQuestions
+ showFollowupQuestions,
+ showSpeechOutputAzure,
+ showSpeechOutputBrowser
}: Props) => {
- const parsedAnswer = useMemo(() => parseAnswerToHtml(answer.answer, onCitationClicked), [answer]);
-
+ const followupQuestions = answer.context?.followup_questions;
+ const parsedAnswer = useMemo(() => parseAnswerToHtml(answer, isStreaming, onCitationClicked), [answer]);
+ const { t } = useTranslation();
const sanitizedAnswerHtml = DOMPurify.sanitize(parsedAnswer.answerHtml);
+ const [copied, setCopied] = useState(false);
+
+ const handleCopy = () => {
+ // Single replace to remove all HTML tags to remove the citations
+ const textToCopy = sanitizedAnswerHtml.replace(/]*>\d+<\/sup><\/a>|<[^>]+>/g, "");
+
+ navigator.clipboard
+ .writeText(textToCopy)
+ .then(() => {
+ setCopied(true);
+ setTimeout(() => setCopied(false), 2000);
+ })
+ .catch(err => console.error("Failed to copy text: ", err));
+ };
return (
Token Usage
+
+
+
+
+ Prompt: {prompt_tokens}
+
+ {reasoningEffort != null && reasoningEffort !== "" && (
+
+ Reasoning: {reasoning_tokens}
+
+ )}
+
+ Output: {completion_tokens - reasoning_tokens}
+
+
+ Total: {total_tokens}
+
+
+ onThoughtProcessClicked()}
- disabled={!answer.thoughts}
+ disabled={!answer.context.thoughts?.length}
/>
onSupportingContentClicked()}
- disabled={!answer.data_points.length}
+ disabled={!answer.context.data_points}
/>
+ {showSpeechOutputAzure && (
+
+
- Generating answer + {t("generatingAnswer")}
- {"Clear chat"}
+ (isUseGPT4V);
+ const [vectorFieldOption, setVectorFieldOption] = useState(gpt4vInputs || GPT4VInput.TextAndImages);
+
+ const onuseGPT4V = (_ev?: React.FormEvent, checked?: boolean) => {
+ updateUseGPT4V(!!checked);
+ setUseGPT4V(!!checked);
+ };
+
+ const onSetGPT4VInput = (_ev: React.FormEvent, option?: IDropdownOption | undefined) => {
+ if (option) {
+ const data = option.key as GPT4VInput;
+ updateGPT4VInputs(data || GPT4VInput.TextAndImages);
+ data && setVectorFieldOption(data);
+ }
+ };
+
+ useEffect(() => {
+ useGPT4V && updateGPT4VInputs(GPT4VInput.TextAndImages);
+ }, [useGPT4V]);
+
+ const useGPT4VId = useId("useGPT4V");
+ const useGPT4VFieldId = useId("useGPT4VField");
+ const gpt4VInputId = useId("gpt4VInput");
+ const gpt4VInputFieldId = useId("gpt4VInputField");
+ const { t } = useTranslation();
+
+ return (
+
+ (
+ (
+
+ );
+};
diff --git a/app/frontend/src/components/GPT4VSettings/index.ts b/app/frontend/src/components/GPT4VSettings/index.ts
new file mode 100644
index 0000000000..90bbd75d48
--- /dev/null
+++ b/app/frontend/src/components/GPT4VSettings/index.ts
@@ -0,0 +1 @@
+export * from "./GPT4VSettings";
diff --git a/app/frontend/src/components/HelpCallout/HelpCallout.tsx b/app/frontend/src/components/HelpCallout/HelpCallout.tsx
new file mode 100644
index 0000000000..595a6f0fca
--- /dev/null
+++ b/app/frontend/src/components/HelpCallout/HelpCallout.tsx
@@ -0,0 +1,52 @@
+import { ITextFieldProps, DefaultButton, IconButton, IButtonStyles, Callout, IStackTokens, Stack, IStackStyles, initializeIcons } from "@fluentui/react";
+import { useBoolean, useId } from "@fluentui/react-hooks";
+import { useTranslation } from "react-i18next";
+
+const stackTokens: IStackTokens = {
+ childrenGap: 4,
+ maxWidth: 300
+};
+
+const labelCalloutStackStyles: Partial = { root: { padding: 20 } };
+const iconButtonStyles: Partial = { root: { marginBottom: -3 } };
+const iconProps = { iconName: "Info" };
+
+interface IHelpCalloutProps {
+ label: string | undefined;
+ labelId: string;
+ fieldId: string | undefined;
+ helpText: string;
+}
+
+export const HelpCallout = (props: IHelpCalloutProps): JSX.Element => {
+ const [isCalloutVisible, { toggle: toggleIsCalloutVisible }] = useBoolean(false);
+ const descriptionId: string = useId("description");
+ const iconButtonId: string = useId("iconButton");
+ const { t } = useTranslation();
+
+ return (
+ <>
+
+
+
+ {isCalloutVisible && (
+
+
+ {props.helpText}
+ {t("labels.closeButton")}
+
+
+ )}
+
+ );
+};
diff --git a/app/frontend/src/components/HelpCallout/index.ts b/app/frontend/src/components/HelpCallout/index.ts
new file mode 100644
index 0000000000..9716f47677
--- /dev/null
+++ b/app/frontend/src/components/HelpCallout/index.ts
@@ -0,0 +1 @@
+export * from "./HelpCallout";
diff --git a/app/frontend/src/components/HistoryButton/HistoryButton.module.css b/app/frontend/src/components/HistoryButton/HistoryButton.module.css
new file mode 100644
index 0000000000..8d998d2d84
--- /dev/null
+++ b/app/frontend/src/components/HistoryButton/HistoryButton.module.css
@@ -0,0 +1,7 @@
+.container {
+ display: flex;
+ align-items: center;
+ gap: 0.375em;
+ cursor: pointer;
+ padding: 0.5rem;
+}
diff --git a/app/frontend/src/components/HistoryButton/HistoryButton.tsx b/app/frontend/src/components/HistoryButton/HistoryButton.tsx
new file mode 100644
index 0000000000..d52a55daf1
--- /dev/null
+++ b/app/frontend/src/components/HistoryButton/HistoryButton.tsx
@@ -0,0 +1,22 @@
+import { History24Regular } from "@fluentui/react-icons";
+import { Button } from "@fluentui/react-components";
+import { useTranslation } from "react-i18next";
+
+import styles from "./HistoryButton.module.css";
+
+interface Props {
+ className?: string;
+ onClick: () => void;
+ disabled?: boolean;
+}
+
+export const HistoryButton = ({ className, disabled, onClick }: Props) => {
+ const { t } = useTranslation();
+ return (
+ ([]);
+ const [isLoading, setIsLoading] = useState(false);
+ const [hasMoreHistory, setHasMoreHistory] = useState(false);
+
+ const client = useLogin ? useMsal().instance : undefined;
+
+ useEffect(() => {
+ if (!isOpen) return;
+ if (notify) {
+ setHistory([]);
+ historyManager.resetContinuationToken();
+ setHasMoreHistory(true);
+ }
+ }, [isOpen, notify]);
+
+ const loadMoreHistory = async () => {
+ setIsLoading(() => true);
+ const token = client ? await getToken(client) : undefined;
+ const items = await historyManager.getNextItems(HISTORY_COUNT_PER_LOAD, token);
+ if (items.length === 0) {
+ setHasMoreHistory(false);
+ }
+ setHistory(prevHistory => [...prevHistory, ...items]);
+ setIsLoading(() => false);
+ };
+
+ const handleSelect = async (id: string) => {
+ const token = client ? await getToken(client) : undefined;
+ const item = await historyManager.getItem(id, token);
+ if (item) {
+ onChatSelected(item);
+ }
+ };
+
+ const handleDelete = async (id: string) => {
+ const token = client ? await getToken(client) : undefined;
+ await historyManager.deleteItem(id, token);
+ setHistory(prevHistory => prevHistory.filter(item => item.id !== id));
+ };
+
+ const groupedHistory = useMemo(() => groupHistory(history), [history]);
+
+ const { t } = useTranslation();
+
+ return (
+ onClose()}
+ onDismissed={() => {
+ setHistory([]);
+ setHasMoreHistory(true);
+ historyManager.resetContinuationToken();
+ }}
+ >
+
+ );
+};
+
+function groupHistory(history: HistoryData[]) {
+ const today = new Date();
+ today.setHours(0, 0, 0, 0);
+ const yesterday = new Date(today);
+ yesterday.setDate(yesterday.getDate() - 1);
+ const lastWeek = new Date(today);
+ lastWeek.setDate(lastWeek.getDate() - 7);
+ const lastMonth = new Date(today);
+ lastMonth.setDate(lastMonth.getDate() - 30);
+
+ return history.reduce(
+ (groups, item) => {
+ const itemDate = new Date(item.timestamp);
+ let group;
+
+ if (itemDate >= today) {
+ group = "history.today";
+ } else if (itemDate >= yesterday) {
+ group = "history.yesterday";
+ } else if (itemDate >= lastWeek) {
+ group = "history.last7days";
+ } else if (itemDate >= lastMonth) {
+ group = "history.last30days";
+ } else {
+ group = itemDate.toLocaleDateString(undefined, { year: "numeric", month: "long" });
+ }
+
+ if (!groups[group]) {
+ groups[group] = [];
+ }
+ groups[group].push(item);
+ return groups;
+ },
+ {} as Record
+ );
+}
+
+const InfiniteLoadingButton = ({ func }: { func: () => void }) => {
+ const buttonRef = useRef(null);
+
+ useEffect(() => {
+ const observer = new IntersectionObserver(
+ entries => {
+ entries.forEach(entry => {
+ if (entry.isIntersecting) {
+ if (buttonRef.current) {
+ func();
+ }
+ }
+ });
+ },
+ {
+ root: null,
+ threshold: 0
+ }
+ );
+
+ if (buttonRef.current) {
+ observer.observe(buttonRef.current);
+ }
+
+ return () => {
+ if (buttonRef.current) {
+ observer.unobserve(buttonRef.current);
+ }
+ };
+ }, []);
+
+ return ;
+};
diff --git a/app/frontend/src/components/HistoryPanel/index.tsx b/app/frontend/src/components/HistoryPanel/index.tsx
new file mode 100644
index 0000000000..e36ac59fe5
--- /dev/null
+++ b/app/frontend/src/components/HistoryPanel/index.tsx
@@ -0,0 +1 @@
+export * from "./HistoryPanel";
diff --git a/app/frontend/src/components/HistoryProviders/CosmosDB.ts b/app/frontend/src/components/HistoryProviders/CosmosDB.ts
new file mode 100644
index 0000000000..5da9df5493
--- /dev/null
+++ b/app/frontend/src/components/HistoryProviders/CosmosDB.ts
@@ -0,0 +1,51 @@
+import { IHistoryProvider, Answers, HistoryProviderOptions, HistoryMetaData } from "./IProvider";
+import { deleteChatHistoryApi, getChatHistoryApi, getChatHistoryListApi, postChatHistoryApi } from "../../api";
+
+export class CosmosDBProvider implements IHistoryProvider {
+ getProviderName = () => HistoryProviderOptions.CosmosDB;
+
+ private continuationToken: string | undefined;
+ private isItemEnd: boolean = false;
+
+ resetContinuationToken() {
+ this.continuationToken = undefined;
+ this.isItemEnd = false;
+ }
+
+ async getNextItems(count: number, idToken?: string): Promise {
+ if (this.isItemEnd) {
+ return [];
+ }
+
+ try {
+ const response = await getChatHistoryListApi(count, this.continuationToken, idToken || "");
+ this.continuationToken = response.continuation_token;
+ if (!this.continuationToken) {
+ this.isItemEnd = true;
+ }
+ return response.sessions.map(session => ({
+ id: session.id,
+ title: session.title,
+ timestamp: session.timestamp
+ }));
+ } catch (e) {
+ console.error(e);
+ return [];
+ }
+ }
+
+ async addItem(id: string, answers: Answers, idToken?: string): Promise {
+ await postChatHistoryApi({ id, answers }, idToken || "");
+ return;
+ }
+
+ async getItem(id: string, idToken?: string): Promise {
+ const response = await getChatHistoryApi(id, idToken || "");
+ return response.answers || null;
+ }
+
+ async deleteItem(id: string, idToken?: string): Promise {
+ await deleteChatHistoryApi(id, idToken || "");
+ return;
+ }
+}
diff --git a/app/frontend/src/components/HistoryProviders/HistoryManager.ts b/app/frontend/src/components/HistoryProviders/HistoryManager.ts
new file mode 100644
index 0000000000..e796d04933
--- /dev/null
+++ b/app/frontend/src/components/HistoryProviders/HistoryManager.ts
@@ -0,0 +1,21 @@
+import { useMemo } from "react";
+import { IHistoryProvider, HistoryProviderOptions } from "../HistoryProviders/IProvider";
+import { NoneProvider } from "../HistoryProviders/None";
+import { IndexedDBProvider } from "../HistoryProviders/IndexedDB";
+import { CosmosDBProvider } from "../HistoryProviders/CosmosDB";
+
+export const useHistoryManager = (provider: HistoryProviderOptions): IHistoryProvider => {
+ const providerInstance = useMemo(() => {
+ switch (provider) {
+ case HistoryProviderOptions.IndexedDB:
+ return new IndexedDBProvider("chat-database", "chat-history");
+ case HistoryProviderOptions.CosmosDB:
+ return new CosmosDBProvider();
+ case HistoryProviderOptions.None:
+ default:
+ return new NoneProvider();
+ }
+ }, [provider]);
+
+ return providerInstance;
+};
diff --git a/app/frontend/src/components/HistoryProviders/IProvider.ts b/app/frontend/src/components/HistoryProviders/IProvider.ts
new file mode 100644
index 0000000000..026443d681
--- /dev/null
+++ b/app/frontend/src/components/HistoryProviders/IProvider.ts
@@ -0,0 +1,19 @@
+import { ChatAppResponse } from "../../api";
+
+export type HistoryMetaData = { id: string; title: string; timestamp: number };
+export type Answers = [user: string, response: ChatAppResponse][];
+
+export const enum HistoryProviderOptions {
+ None = "none",
+ IndexedDB = "indexedDB",
+ CosmosDB = "cosmosDB"
+}
+
+export interface IHistoryProvider {
+ getProviderName(): HistoryProviderOptions;
+ resetContinuationToken(): void;
+ getNextItems(count: number, idToken?: string): Promise;
+ addItem(id: string, answers: Answers, idToken?: string): Promise;
+ getItem(id: string, idToken?: string): Promise;
+ deleteItem(id: string, idToken?: string): Promise;
+}
diff --git a/app/frontend/src/components/HistoryProviders/IndexedDB.ts b/app/frontend/src/components/HistoryProviders/IndexedDB.ts
new file mode 100644
index 0000000000..ca8fa19fe8
--- /dev/null
+++ b/app/frontend/src/components/HistoryProviders/IndexedDB.ts
@@ -0,0 +1,104 @@
+import { IDBPDatabase, openDB } from "idb";
+import { IHistoryProvider, Answers, HistoryProviderOptions, HistoryMetaData } from "./IProvider";
+
+export class IndexedDBProvider implements IHistoryProvider {
+ getProviderName = () => HistoryProviderOptions.IndexedDB;
+
+ private dbName: string;
+ private storeName: string;
+ private dbPromise: Promise | null = null;
+ private cursorKey: IDBValidKey | undefined;
+ private isCusorEnd: boolean = false;
+
+ constructor(dbName: string, storeName: string) {
+ this.dbName = dbName;
+ this.storeName = storeName;
+ this.cursorKey = undefined;
+ this.isCusorEnd = false;
+ }
+
+ private async init() {
+ const storeName = this.storeName;
+ if (!this.dbPromise) {
+ this.dbPromise = openDB(this.dbName, 1, {
+ upgrade(db) {
+ if (!db.objectStoreNames.contains(storeName)) {
+ const store = db.createObjectStore(storeName, { keyPath: "id" });
+ store.createIndex("timestamp", "timestamp");
+ }
+ }
+ });
+ }
+ return this.dbPromise;
+ }
+
+ resetContinuationToken() {
+ this.cursorKey = undefined;
+ this.isCusorEnd = false;
+ }
+
+ async getNextItems(count: number): Promise {
+ const db = await this.init();
+ const tx = db.transaction(this.storeName, "readonly");
+ const store = tx.objectStore(this.storeName);
+ const index = store.index("timestamp");
+
+ // return empty array if cursor is already at the end
+ if (this.isCusorEnd) {
+ return [];
+ }
+
+ // set cursor to the last key
+ let cursor = this.cursorKey ? await index.openCursor(IDBKeyRange.upperBound(this.cursorKey), "prev") : await index.openCursor(null, "prev");
+
+ // return empty array means no more history or no data. set isCursorEnd to true and return empty array
+ if (!cursor) {
+ this.isCusorEnd = true;
+ return [];
+ }
+
+ const loadedItems: { id: string; title: string; timestamp: number; answers: Answers }[] = [];
+ for (let i = 0; i < count && cursor; i++) {
+ loadedItems.push(cursor.value);
+ cursor = await cursor.continue();
+ }
+
+ // set isCursorEnd to true if cursor is null
+ if (!cursor) {
+ this.isCusorEnd = true;
+ }
+
+ // update cursorKey
+ this.cursorKey = cursor?.key;
+
+ return loadedItems;
+ }
+
+ async addItem(id: string, answers: Answers): Promise {
+ const timestamp = new Date().getTime();
+ const db = await this.init(); // 自動的に初期化
+ const tx = db.transaction(this.storeName, "readwrite");
+ const current = await tx.objectStore(this.storeName).get(id);
+ if (current) {
+ await tx.objectStore(this.storeName).put({ ...current, id, timestamp, answers });
+ } else {
+ const title = answers[0][0].length > 50 ? answers[0][0].substring(0, 50) + "..." : answers[0][0];
+ await tx.objectStore(this.storeName).add({ id, title, timestamp, answers });
+ }
+ await tx.done;
+ return;
+ }
+
+ async getItem(id: string): Promise {
+ const db = await this.init();
+ const tx = db.transaction(this.storeName, "readonly");
+ const item = await tx.objectStore(this.storeName).get(id);
+ return item ? item.answers : null;
+ }
+
+ async deleteItem(id: string): Promise {
+ const db = await this.init();
+ await db.delete(this.storeName, id);
+ return;
+ }
+}
diff --git a/app/frontend/src/components/HistoryProviders/None.ts b/app/frontend/src/components/HistoryProviders/None.ts
new file mode 100644
index 0000000000..a662d54f72
--- /dev/null
+++ b/app/frontend/src/components/HistoryProviders/None.ts
@@ -0,0 +1,20 @@
+import { IHistoryProvider, Answers, HistoryProviderOptions, HistoryMetaData } from "./IProvider";
+
+export class NoneProvider implements IHistoryProvider {
+ getProviderName = () => HistoryProviderOptions.None;
+ resetContinuationToken(): void {
+ return;
+ }
+ async getNextItems(count: number): Promise {
+ return [];
+ }
+ async addItem(id: string, answers: Answers): Promise {
+ return;
+ }
+ async getItem(id: string): Promise {
+ return null;
+ }
+ async deleteItem(id: string): Promise {
+ return;
+ }
+}
diff --git a/app/frontend/src/components/HistoryProviders/index.ts b/app/frontend/src/components/HistoryProviders/index.ts
new file mode 100644
index 0000000000..a9d1757802
--- /dev/null
+++ b/app/frontend/src/components/HistoryProviders/index.ts
@@ -0,0 +1,4 @@
+export * from "./HistoryManager";
+export * from "./IndexedDB";
+export * from "./IProvider";
+export * from "./None";
diff --git a/app/frontend/src/components/LoginButton/LoginButton.module.css b/app/frontend/src/components/LoginButton/LoginButton.module.css
new file mode 100644
index 0000000000..1a368294fe
--- /dev/null
+++ b/app/frontend/src/components/LoginButton/LoginButton.module.css
@@ -0,0 +1,6 @@
+.loginButton {
+ border-radius: 0.3125em;
+ font-weight: 100;
+ margin: 0;
+ padding: 0.5rem 1rem;
+}
diff --git a/app/frontend/src/components/LoginButton/LoginButton.tsx b/app/frontend/src/components/LoginButton/LoginButton.tsx
new file mode 100644
index 0000000000..3d8c64323a
--- /dev/null
+++ b/app/frontend/src/components/LoginButton/LoginButton.tsx
@@ -0,0 +1,65 @@
+import { DefaultButton } from "@fluentui/react";
+import { useMsal } from "@azure/msal-react";
+import { useTranslation } from "react-i18next";
+
+import styles from "./LoginButton.module.css";
+import { getRedirectUri, loginRequest, appServicesLogout, getUsername, checkLoggedIn } from "../../authConfig";
+import { useState, useEffect, useContext } from "react";
+import { LoginContext } from "../../loginContext";
+
+export const LoginButton = () => {
+ const { instance } = useMsal();
+ const { loggedIn, setLoggedIn } = useContext(LoginContext);
+ const activeAccount = instance.getActiveAccount();
+ const [username, setUsername] = useState("");
+ const { t } = useTranslation();
+
+ useEffect(() => {
+ const fetchUsername = async () => {
+ setUsername((await getUsername(instance)) ?? "");
+ };
+
+ fetchUsername();
+ }, []);
+
+ const handleLoginPopup = () => {
+ /**
+ * When using popup and silent APIs, we recommend setting the redirectUri to a blank page or a page
+ * that does not implement MSAL. Keep in mind that all redirect routes must be registered with the application
+ * For more information, please follow this link: https://github.com/AzureAD/microsoft-authentication-library-for-js/blob/dev/lib/msal-browser/docs/login-user.md#redirecturi-considerations
+ */
+ instance
+ .loginPopup({
+ ...loginRequest,
+ redirectUri: getRedirectUri()
+ })
+ .catch(error => console.log(error))
+ .then(async () => {
+ setLoggedIn(await checkLoggedIn(instance));
+ setUsername((await getUsername(instance)) ?? "");
+ });
+ };
+ const handleLogoutPopup = () => {
+ if (activeAccount) {
+ instance
+ .logoutPopup({
+ mainWindowRedirectUri: "/", // redirects the top level app after logout
+ account: instance.getActiveAccount()
+ })
+ .catch(error => console.log(error))
+ .then(async () => {
+ setLoggedIn(await checkLoggedIn(instance));
+ setUsername((await getUsername(instance)) ?? "");
+ });
+ } else {
+ appServicesLogout();
+ }
+ };
+ return (
+ = ({ src }) => {
+ const [content, setContent] = useState("");
+ const [isLoading, setIsLoading] = useState(true);
+ const [error, setError] = useState(null);
+ const { t } = useTranslation();
+
+ /**
+ * Anchor links result in HTTP 404 errors as the URL they point to does not exist.
+ * This function removes them from the markdown.
+ */
+ const removeAnchorLinks = (markdown: string) => {
+ const ancorLinksRegex = /\[.*?\]\(#.*?\)/g;
+ return markdown.replace(ancorLinksRegex, "");
+ };
+
+ useEffect(() => {
+ const fetchMarkdown = async () => {
+ try {
+ const response = await fetch(src);
+
+ if (!response.ok) {
+ throw new Error("Failed loading markdown file.");
+ }
+
+ let markdownText = await response.text();
+ markdownText = removeAnchorLinks(markdownText);
+ setContent(markdownText);
+ } catch (error: any) {
+ setError(error);
+ } finally {
+ setIsLoading(false);
+ }
+ };
+
+ fetchMarkdown();
+ }, [src]);
+
+ return (
+ ("");
+ const { loggedIn } = useContext(LoginContext);
+ const { t } = useTranslation();
+ const [isComposing, setIsComposing] = useState(false);
+
+ useEffect(() => {
+ initQuestion && setQuestion(initQuestion);
+ }, [initQuestion]);
const sendQuestion = () => {
if (disabled || !question.trim()) {
@@ -27,12 +41,21 @@ export const QuestionInput = ({ onSend, disabled, placeholder, clearOnSend }: Pr
};
const onEnterPress = (ev: React.KeyboardEvent) => {
+ if (isComposing) return;
+
if (ev.key === "Enter" && !ev.shiftKey) {
ev.preventDefault();
sendQuestion();
}
};
+ const handleCompositionStart = () => {
+ setIsComposing(true);
+ };
+ const handleCompositionEnd = () => {
+ setIsComposing(false);
+ };
+
const onQuestionChange = (_ev: React.FormEvent, newValue?: string) => {
if (!newValue) {
setQuestion("");
@@ -41,12 +64,18 @@ export const QuestionInput = ({ onSend, disabled, placeholder, clearOnSend }: Pr
}
};
- const sendQuestionDisabled = disabled || !question.trim();
+ const disableRequiredAccessControl = requireLogin && !loggedIn;
+ const sendQuestionDisabled = disabled || !question.trim() || disableRequiredAccessControl;
+
+ if (disableRequiredAccessControl) {
+ placeholder = "Please login to continue...";
+ }
return (
);
};
diff --git a/app/frontend/src/components/QuestionInput/SpeechInput.tsx b/app/frontend/src/components/QuestionInput/SpeechInput.tsx
new file mode 100644
index 0000000000..882fd180b6
--- /dev/null
+++ b/app/frontend/src/components/QuestionInput/SpeechInput.tsx
@@ -0,0 +1,117 @@
+import { SetStateAction, useState } from "react";
+import { Button, Tooltip } from "@fluentui/react-components";
+import { Mic28Filled } from "@fluentui/react-icons";
+import { useTranslation } from "react-i18next";
+import styles from "./QuestionInput.module.css";
+import { supportedLngs } from "../../i18n/config";
+
+interface Props {
+ updateQuestion: (question: string) => void;
+}
+
+const useCustomSpeechRecognition = () => {
+ const { i18n } = useTranslation();
+ const currentLng = i18n.language;
+ let lngCode = supportedLngs[currentLng]?.locale;
+ if (!lngCode) {
+ lngCode = "en-US";
+ }
+
+ const SpeechRecognition = (window as any).SpeechRecognition || (window as any).webkitSpeechRecognition;
+ let speechRecognition: {
+ continuous: boolean;
+ lang: string;
+ interimResults: boolean;
+ maxAlternatives: number;
+ start: () => void;
+ onresult: (event: { results: { transcript: SetStateAction }[][] }) => void;
+ onend: () => void;
+ onerror: (event: { error: string }) => void;
+ stop: () => void;
+ } | null = null;
+
+ try {
+ speechRecognition = new SpeechRecognition();
+ if (speechRecognition != null) {
+ speechRecognition.lang = lngCode;
+ speechRecognition.interimResults = true;
+ }
+ } catch (err) {
+ console.error("SpeechRecognition not supported");
+ speechRecognition = null;
+ }
+
+ return speechRecognition;
+};
+
+export const SpeechInput = ({ updateQuestion }: Props) => {
+ let speechRecognition = useCustomSpeechRecognition();
+ const { t } = useTranslation();
+ const [isRecording, setIsRecording] = useState(false);
+ const startRecording = () => {
+ if (speechRecognition == null) {
+ console.error("SpeechRecognition not supported");
+ return;
+ }
+
+ speechRecognition.onresult = (event: { results: { transcript: SetStateAction }[][] }) => {
+ let input = "";
+ for (const result of event.results) {
+ input += result[0].transcript;
+ }
+ updateQuestion(input);
+ };
+ speechRecognition.onend = () => {
+ // NOTE: In some browsers (e.g. Chrome), the recording will stop automatically after a few seconds of silence.
+ setIsRecording(false);
+ };
+ speechRecognition.onerror = (event: { error: string }) => {
+ if (speechRecognition) {
+ speechRecognition.stop();
+ if (event.error == "no-speech") {
+ alert("No speech was detected. Please check your system audio settings and try again.");
+ } else if (event.error == "language-not-supported") {
+ alert(
+ `Speech recognition error detected: ${event.error}. The speech recognition input functionality does not yet work on all browsers, like Edge in Mac OS X with ARM chips. Try another browser/OS.`
+ );
+ } else {
+ alert(`Speech recognition error detected: ${event.error}.`);
+ }
+ }
+ };
+
+ setIsRecording(true);
+ speechRecognition.start();
+ };
+
+ const stopRecording = () => {
+ if (speechRecognition == null) {
+ console.error("SpeechRecognition not supported");
+ return;
+ }
+ speechRecognition.stop();
+ setIsRecording(false);
+ };
+
+ if (speechRecognition == null) {
+ return <>;
+ }
+ return (
+ <>
+ {!isRecording && (
+
+ } disabled={disabled} onClick={onClick}>
+ {t("clearChat")}
+
);
};
diff --git a/app/frontend/src/components/Example/Example.module.css b/app/frontend/src/components/Example/Example.module.css
index 796f2172f5..1b1471c35d 100644
--- a/app/frontend/src/components/Example/Example.module.css
+++ b/app/frontend/src/components/Example/Example.module.css
@@ -1,9 +1,10 @@
.examplesNavList {
list-style: none;
- padding-left: 0;
+ padding-left: 1rem;
+ padding-right: 1rem;
display: flex;
flex-wrap: wrap;
- gap: 10px;
+ gap: 0.625rem;
flex: 1;
justify-content: center;
}
@@ -11,29 +12,57 @@
.example {
word-break: break-word;
background: #dbdbdb;
- border-radius: 8px;
+ border-radius: 0.5rem;
display: flex;
flex-direction: column;
- padding: 20px;
- margin-bottom: 5px;
+ margin-bottom: 0.3125rem;
cursor: pointer;
}
.example:hover {
- box-shadow: 0px 8px 16px rgba(0, 0, 0, 0.14), 0px 0px 2px rgba(0, 0, 0, 0.12);
- outline: 2px solid rgba(115, 118, 225, 1);
+ box-shadow:
+ 0rem 0.5rem 1rem rgba(0, 0, 0, 0.14),
+ 0rem 0rem 0.125rem rgba(0, 0, 0, 0.12);
+ outline: 0.125rem solid rgba(115, 118, 225, 1);
}
.exampleText {
margin: 0;
- font-size: 22px;
- width: 280px;
- height: 100px;
+ font-size: 1.25rem;
+ width: 25rem;
+ padding: 0.5rem;
+ min-height: 4.5rem;
}
-@media only screen and (max-height: 780px) {
+.examplesNavList li {
+ display: none;
+}
+
+.examplesNavList li:nth-of-type(1),
+.examplesNavList li:nth-of-type(2) {
+ display: block; /* Default to showing two list items */
+}
+
+@media only screen and (min-width: 992px) {
+ .examplesNavList {
+ flex-direction: row; /* Switch to row layout for wider screens */
+ padding-left: 0;
+ padding-right: 0;
+ }
+
+ .example {
+ margin-bottom: 0.3125rem;
+ padding: 1.25rem;
+ }
+
+ .examplesNavList li:nth-of-type(2),
+ .examplesNavList li:nth-of-type(3) {
+ display: block; /* Show an additional list item for medium heights */
+ }
.exampleText {
- font-size: 20px;
- height: 80px;
+ font-size: 1.375rem;
+ width: 17.5rem;
+ padding: 0;
+ min-height: 6.25rem;
}
}
diff --git a/app/frontend/src/components/Example/ExampleList.tsx b/app/frontend/src/components/Example/ExampleList.tsx
index bd6e1a6edd..dab4ec97ec 100644
--- a/app/frontend/src/components/Example/ExampleList.tsx
+++ b/app/frontend/src/components/Example/ExampleList.tsx
@@ -1,31 +1,24 @@
import { Example } from "./Example";
+import { useTranslation } from "react-i18next";
import styles from "./Example.module.css";
-export type ExampleModel = {
- text: string;
- value: string;
-};
-
-const EXAMPLES: ExampleModel[] = [
- {
- text: "What is included in my Northwind Health Plus plan that is not in standard?",
- value: "What is included in my Northwind Health Plus plan that is not in standard?"
- },
- { text: "What happens in a performance review?", value: "What happens in a performance review?" },
- { text: "What does a Product Manager do?", value: "What does a Product Manager do?" }
-];
-
interface Props {
onExampleClicked: (value: string) => void;
+ useGPT4V?: boolean;
}
-export const ExampleList = ({ onExampleClicked }: Props) => {
+export const ExampleList = ({ onExampleClicked, useGPT4V }: Props) => {
+ const { t } = useTranslation();
+
+ const DEFAULT_EXAMPLES: string[] = [t("defaultExamples.1"), t("defaultExamples.2"), t("defaultExamples.3")];
+ const GPT4V_EXAMPLES: string[] = [t("gpt4vExamples.1"), t("gpt4vExamples.2"), t("gpt4vExamples.3")];
+
return (
-
- {EXAMPLES.map((x, i) => (
+ {(useGPT4V ? GPT4V_EXAMPLES : DEFAULT_EXAMPLES).map((question, i) => (
-
-
+
))}
+ } disabled={disabled} onClick={onClick}>
+ {t("history.openChatHistory")}
+
+
+ );
+};
diff --git a/app/frontend/src/components/HistoryButton/index.tsx b/app/frontend/src/components/HistoryButton/index.tsx
new file mode 100644
index 0000000000..7d6ff3cfcc
--- /dev/null
+++ b/app/frontend/src/components/HistoryButton/index.tsx
@@ -0,0 +1 @@
+export * from "./HistoryButton";
diff --git a/app/frontend/src/components/HistoryItem/HistoryItem.module.css b/app/frontend/src/components/HistoryItem/HistoryItem.module.css
new file mode 100644
index 0000000000..74245957e9
--- /dev/null
+++ b/app/frontend/src/components/HistoryItem/HistoryItem.module.css
@@ -0,0 +1,120 @@
+.historyItem {
+ display: flex;
+ align-items: center;
+ justify-content: space-between;
+ padding: 4px 8px;
+ border-radius: 6px;
+ transition: background-color 0.2s;
+}
+
+.historyItem:hover {
+ background-color: #f3f4f6;
+}
+
+.historyItemButton {
+ flex-grow: 1;
+ text-align: left;
+ padding: 0;
+ margin-right: 4px;
+ background: none;
+ border: none;
+ cursor: pointer;
+}
+
+.historyItemTitle {
+ font-size: 1rem;
+}
+
+.deleteIcon {
+ width: 20px;
+ height: 20px;
+}
+
+.deleteButton {
+ opacity: 0;
+ transition: opacity 0.2s;
+ background: none;
+ border: none;
+ cursor: pointer;
+ padding: 4px;
+ border-radius: 9999px;
+ color: #6b7280;
+}
+
+.historyItem:hover .deleteButton,
+.deleteButton:focus {
+ opacity: 1;
+}
+
+.deleteButton:hover {
+ color: #111827;
+}
+
+.modalOverlay {
+ position: fixed;
+ top: 0;
+ left: 0;
+ right: 0;
+ bottom: 0;
+ background-color: rgba(0, 0, 0, 0.5);
+ display: flex;
+ align-items: center;
+ justify-content: center;
+ z-index: 50;
+}
+
+.modalContent {
+ background-color: white;
+ padding: 24px;
+ border-radius: 8px;
+ box-shadow:
+ 0 4px 6px -1px rgba(0, 0, 0, 0.1),
+ 0 2px 4px -1px rgba(0, 0, 0, 0.06);
+ max-width: 400px;
+ width: 100%;
+}
+
+.modalTitle {
+ font-size: 20px;
+ font-weight: 600;
+ margin-top: 0px;
+ margin-bottom: 16px;
+}
+
+.modalDescription {
+ margin-top: 0px;
+ margin-bottom: 16px;
+}
+
+.modalActions {
+ display: flex;
+ justify-content: flex-end;
+ gap: 16px;
+}
+
+.modalCancelButton,
+.modalConfirmButton {
+ padding: 8px 16px;
+ border-radius: 4px;
+ font-size: 14px;
+ font-weight: 500;
+ cursor: pointer;
+}
+
+.modalCancelButton {
+ background-color: #f3f4f6;
+ color: #374151;
+}
+
+.modalConfirmButton {
+ background-color: #ef4444;
+ color: white;
+}
+
+.modalCancelButton:hover {
+ background-color: #e5e7eb;
+}
+
+.modalConfirmButton:hover {
+ background-color: #dc2626;
+}
diff --git a/app/frontend/src/components/HistoryItem/HistoryItem.tsx b/app/frontend/src/components/HistoryItem/HistoryItem.tsx
new file mode 100644
index 0000000000..5aca674579
--- /dev/null
+++ b/app/frontend/src/components/HistoryItem/HistoryItem.tsx
@@ -0,0 +1,59 @@
+import { useState, useCallback } from "react";
+import { useTranslation } from "react-i18next";
+import styles from "./HistoryItem.module.css";
+import { DefaultButton } from "@fluentui/react";
+import { Delete24Regular } from "@fluentui/react-icons";
+
+export interface HistoryData {
+ id: string;
+ title: string;
+ timestamp: number;
+}
+
+interface HistoryItemProps {
+ item: HistoryData;
+ onSelect: (id: string) => void;
+ onDelete: (id: string) => void;
+}
+
+export function HistoryItem({ item, onSelect, onDelete }: HistoryItemProps) {
+ const [isModalOpen, setIsModalOpen] = useState(false);
+
+ const handleDelete = useCallback(() => {
+ setIsModalOpen(false);
+ onDelete(item.id);
+ }, [item.id, onDelete]);
+
+ return (
+
+
+
+ setIsModalOpen(false)} onConfirm={handleDelete} />
+
+ );
+}
+
+function DeleteHistoryModal({ isOpen, onClose, onConfirm }: { isOpen: boolean; onClose: () => void; onConfirm: () => void }) {
+ if (!isOpen) return null;
+ const { t } = useTranslation();
+ return (
+
+
+ );
+}
diff --git a/app/frontend/src/components/HistoryItem/index.tsx b/app/frontend/src/components/HistoryItem/index.tsx
new file mode 100644
index 0000000000..314bd98059
--- /dev/null
+++ b/app/frontend/src/components/HistoryItem/index.tsx
@@ -0,0 +1 @@
+export * from "./HistoryItem";
diff --git a/app/frontend/src/components/HistoryPanel/HistoryPanel.module.css b/app/frontend/src/components/HistoryPanel/HistoryPanel.module.css
new file mode 100644
index 0000000000..05b8d07848
--- /dev/null
+++ b/app/frontend/src/components/HistoryPanel/HistoryPanel.module.css
@@ -0,0 +1,14 @@
+.group {
+ margin-top: 1rem;
+}
+.groupLabel {
+ font-size: 0.9rem;
+ font-weight: bold;
+ margin-top: 0.5rem;
+ margin-bottom: 0.2rem;
+}
+
+.footer {
+ display: flex;
+ justify-content: space-between;
+}
diff --git a/app/frontend/src/components/HistoryPanel/HistoryPanel.tsx b/app/frontend/src/components/HistoryPanel/HistoryPanel.tsx
new file mode 100644
index 0000000000..acaf3b7870
--- /dev/null
+++ b/app/frontend/src/components/HistoryPanel/HistoryPanel.tsx
@@ -0,0 +1,172 @@
+import { useMsal } from "@azure/msal-react";
+import { getToken, useLogin } from "../../authConfig";
+import { Panel, PanelType, Spinner } from "@fluentui/react";
+import { useEffect, useMemo, useRef, useState } from "react";
+import { HistoryData, HistoryItem } from "../HistoryItem";
+import { Answers, HistoryProviderOptions } from "../HistoryProviders/IProvider";
+import { useHistoryManager, HistoryMetaData } from "../HistoryProviders";
+import { useTranslation } from "react-i18next";
+import styles from "./HistoryPanel.module.css";
+
+const HISTORY_COUNT_PER_LOAD = 20;
+
+export const HistoryPanel = ({
+ provider,
+ isOpen,
+ notify,
+ onClose,
+ onChatSelected
+}: {
+ provider: HistoryProviderOptions;
+ isOpen: boolean;
+ notify: boolean;
+ onClose: () => void;
+ onChatSelected: (answers: Answers) => void;
+}) => {
+ const historyManager = useHistoryManager(provider);
+ const [history, setHistory] = useState
+
+ {t("history.deleteModalTitle")}
+{t("history.deleteModalDescription")}
+
+
+ {t("history.cancelLabel")}
+
+
+ {t("history.deleteLabel")}
+
+
+
+ {Object.entries(groupedHistory).map(([group, items]) => (
+
+
+
+ ))}
+ {isLoading && {t(group)}
+ {items.map(item => ( +{t("history.noHistory")}
} + {hasMoreHistory && !isLoading &&
+ {isLoading ? (
+
+ );
+};
diff --git a/app/frontend/src/components/MarkdownViewer/index.tsx b/app/frontend/src/components/MarkdownViewer/index.tsx
new file mode 100644
index 0000000000..1dd1eac95b
--- /dev/null
+++ b/app/frontend/src/components/MarkdownViewer/index.tsx
@@ -0,0 +1 @@
+export * from "./MarkdownViewer";
diff --git a/app/frontend/src/components/QuestionInput/QuestionInput.module.css b/app/frontend/src/components/QuestionInput/QuestionInput.module.css
index 4025562e09..0a34dea4f8 100644
--- a/app/frontend/src/components/QuestionInput/QuestionInput.module.css
+++ b/app/frontend/src/components/QuestionInput/QuestionInput.module.css
@@ -1,15 +1,16 @@
.questionInputContainer {
- border-radius: 8px;
- box-shadow: 0px 8px 16px rgba(0, 0, 0, 0.14), 0px 0px 2px rgba(0, 0, 0, 0.12);
- height: 90px;
+ border-radius: 0.5rem;
+ box-shadow:
+ 0px 0.5rem 1rem rgba(0, 0, 0, 0.14),
+ 0px 0px 0.125rem rgba(0, 0, 0, 0.12);
width: 100%;
- padding: 15px;
+ padding: 0.8rem;
background: white;
}
.questionInputTextArea {
width: 100%;
- line-height: 40px;
+ line-height: 2.5rem;
}
.questionInputButtonsContainer {
@@ -18,10 +19,8 @@
justify-content: flex-end;
}
-.questionInputSendButton {
- cursor: pointer;
-}
-
-.questionInputSendButtonDisabled {
- opacity: 0.4;
+@media (min-width: 992px) {
+ .questionInputContainer {
+ height: 5.625rem;
+ }
}
diff --git a/app/frontend/src/components/QuestionInput/QuestionInput.tsx b/app/frontend/src/components/QuestionInput/QuestionInput.tsx
index 8c507739d8..c16ddc1be7 100644
--- a/app/frontend/src/components/QuestionInput/QuestionInput.tsx
+++ b/app/frontend/src/components/QuestionInput/QuestionInput.tsx
@@ -1,18 +1,32 @@
-import { useState } from "react";
+import { useState, useEffect, useContext } from "react";
import { Stack, TextField } from "@fluentui/react";
+import { Button, Tooltip } from "@fluentui/react-components";
import { Send28Filled } from "@fluentui/react-icons";
+import { useTranslation } from "react-i18next";
import styles from "./QuestionInput.module.css";
+import { SpeechInput } from "./SpeechInput";
+import { LoginContext } from "../../loginContext";
+import { requireLogin } from "../../authConfig";
interface Props {
onSend: (question: string) => void;
disabled: boolean;
+ initQuestion?: string;
placeholder?: string;
clearOnSend?: boolean;
+ showSpeechInput?: boolean;
}
-export const QuestionInput = ({ onSend, disabled, placeholder, clearOnSend }: Props) => {
+export const QuestionInput = ({ onSend, disabled, placeholder, clearOnSend, initQuestion, showSpeechInput }: Props) => {
const [question, setQuestion] = useState
+
+ ) : error ? (
+
+
+ {error.message}
+
+ Download the file
+
+
+
+ ) : (
+
+
+ )}
+
-
+ } disabled={sendQuestionDisabled} onClick={sendQuestion} />
+
+ {showSpeechInput &&
-
+
+
+ } onClick={startRecording} />
+
+
+ )}
+ {isRecording && (
+
+
+ } disabled={!isRecording} onClick={stopRecording} />
+
+
+ )}
+
+ );
+};
diff --git a/app/frontend/src/components/Settings/Settings.module.css b/app/frontend/src/components/Settings/Settings.module.css
new file mode 100644
index 0000000000..f33b1f94df
--- /dev/null
+++ b/app/frontend/src/components/Settings/Settings.module.css
@@ -0,0 +1,55 @@
+.settingsSeparator {
+ margin-top: 0.75rem;
+}
+
+.option-slider {
+ width: 100%;
+ max-width: 400px;
+ margin: 20px auto;
+ position: relative;
+ user-select: none;
+}
+
+.slider-track {
+ width: 100%;
+ height: 40px;
+ background: #e0e0e0;
+ border-radius: 20px;
+ position: relative;
+}
+
+.slider-thumb {
+ position: absolute;
+ top: 0;
+ width: 33.33%;
+ height: 40px;
+ background: #0078d4;
+ border-radius: 20px;
+ color: #fff;
+ text-align: center;
+ line-height: 40px;
+ transition: left 0.3s ease;
+ cursor: pointer;
+}
+
+.slider-options {
+ display: flex;
+ justify-content: space-between;
+ position: absolute;
+ top: 0;
+ width: 100%;
+ height: 40px;
+}
+
+.slider-option {
+ width: 33.33%;
+ text-align: center;
+ line-height: 40px;
+ color: #333;
+ cursor: pointer;
+}
+
+.slider-option.active {
+ font-weight: bold;
+ color: #fff;
+}
diff --git a/app/frontend/src/components/Settings/Settings.tsx b/app/frontend/src/components/Settings/Settings.tsx
new file mode 100644
index 0000000000..a1c4f46632
--- /dev/null
+++ b/app/frontend/src/components/Settings/Settings.tsx
@@ -0,0 +1,422 @@
+import { useId } from "@fluentui/react-hooks";
+import { useTranslation } from "react-i18next";
+import { TextField, ITextFieldProps, Checkbox, ICheckboxProps, Dropdown, IDropdownProps, IDropdownOption } from "@fluentui/react";
+import { HelpCallout } from "../HelpCallout";
+import { GPT4VSettings } from "../GPT4VSettings";
+import { VectorSettings } from "../VectorSettings";
+import { RetrievalMode, VectorFields, GPT4VInput } from "../../api";
+import styles from "./Settings.module.css";
+
+// Add type for onRenderLabel
+type RenderLabelType = ITextFieldProps | IDropdownProps | ICheckboxProps;
+
+export interface SettingsProps {
+ promptTemplate: string;
+ temperature: number;
+ retrieveCount: number;
+ maxSubqueryCount: number;
+ resultsMergeStrategy: string;
+ seed: number | null;
+ minimumSearchScore: number;
+ minimumRerankerScore: number;
+ useSemanticRanker: boolean;
+ useSemanticCaptions: boolean;
+ useQueryRewriting: boolean;
+ reasoningEffort: string;
+ excludeCategory: string;
+ includeCategory: string;
+ retrievalMode: RetrievalMode;
+ useGPT4V: boolean;
+ gpt4vInput: GPT4VInput;
+ vectorFields: VectorFields;
+ showSemanticRankerOption: boolean;
+ showQueryRewritingOption: boolean;
+ showReasoningEffortOption: boolean;
+ showGPT4VOptions: boolean;
+ showVectorOption: boolean;
+ useOidSecurityFilter: boolean;
+ useGroupsSecurityFilter: boolean;
+ useLogin: boolean;
+ loggedIn: boolean;
+ requireAccessControl: boolean;
+ className?: string;
+ onChange: (field: string, value: any) => void;
+ streamingEnabled?: boolean; // Only used in chat
+ shouldStream?: boolean; // Only used in Chat
+ useSuggestFollowupQuestions?: boolean; // Only used in Chat
+ promptTemplatePrefix?: string;
+ promptTemplateSuffix?: string;
+ showSuggestFollowupQuestions?: boolean;
+ showAgenticRetrievalOption: boolean;
+ useAgenticRetrieval: boolean;
+}
+
+export const Settings = ({
+ promptTemplate,
+ temperature,
+ retrieveCount,
+ maxSubqueryCount,
+ resultsMergeStrategy,
+ seed,
+ minimumSearchScore,
+ minimumRerankerScore,
+ useSemanticRanker,
+ useSemanticCaptions,
+ useQueryRewriting,
+ reasoningEffort,
+ excludeCategory,
+ includeCategory,
+ retrievalMode,
+ useGPT4V,
+ gpt4vInput,
+ vectorFields,
+ showSemanticRankerOption,
+ showQueryRewritingOption,
+ showReasoningEffortOption,
+ showGPT4VOptions,
+ showVectorOption,
+ useOidSecurityFilter,
+ useGroupsSecurityFilter,
+ useLogin,
+ loggedIn,
+ requireAccessControl,
+ className,
+ onChange,
+ streamingEnabled,
+ shouldStream,
+ useSuggestFollowupQuestions,
+ promptTemplatePrefix,
+ promptTemplateSuffix,
+ showSuggestFollowupQuestions,
+ showAgenticRetrievalOption,
+ useAgenticRetrieval
+}: SettingsProps) => {
+ const { t } = useTranslation();
+
+ // Form field IDs
+ const promptTemplateId = useId("promptTemplate");
+ const promptTemplateFieldId = useId("promptTemplateField");
+ const temperatureId = useId("temperature");
+ const temperatureFieldId = useId("temperatureField");
+ const seedId = useId("seed");
+ const seedFieldId = useId("seedField");
+ const agenticRetrievalId = useId("agenticRetrieval");
+ const agenticRetrievalFieldId = useId("agenticRetrievalField");
+ const searchScoreId = useId("searchScore");
+ const searchScoreFieldId = useId("searchScoreField");
+ const rerankerScoreId = useId("rerankerScore");
+ const rerankerScoreFieldId = useId("rerankerScoreField");
+ const retrieveCountId = useId("retrieveCount");
+ const retrieveCountFieldId = useId("retrieveCountField");
+ const maxSubqueryCountId = useId("maxSubqueryCount");
+ const maxSubqueryCountFieldId = useId("maxSubqueryCountField");
+ const resultsMergeStrategyFieldId = useId("resultsMergeStrategy");
+ const includeCategoryId = useId("includeCategory");
+ const includeCategoryFieldId = useId("includeCategoryField");
+ const excludeCategoryId = useId("excludeCategory");
+ const excludeCategoryFieldId = useId("excludeCategoryField");
+ const semanticRankerId = useId("semanticRanker");
+ const semanticRankerFieldId = useId("semanticRankerField");
+ const queryRewritingFieldId = useId("queryRewritingField");
+ const reasoningEffortFieldId = useId("reasoningEffortField");
+ const semanticCaptionsId = useId("semanticCaptions");
+ const semanticCaptionsFieldId = useId("semanticCaptionsField");
+ const useOidSecurityFilterId = useId("useOidSecurityFilter");
+ const useOidSecurityFilterFieldId = useId("useOidSecurityFilterField");
+ const useGroupsSecurityFilterId = useId("useGroupsSecurityFilter");
+ const useGroupsSecurityFilterFieldId = useId("useGroupsSecurityFilterField");
+ const shouldStreamId = useId("shouldStream");
+ const shouldStreamFieldId = useId("shouldStreamField");
+ const suggestFollowupQuestionsId = useId("suggestFollowupQuestions");
+ const suggestFollowupQuestionsFieldId = useId("suggestFollowupQuestionsField");
+
+ const renderLabel = (props: RenderLabelType | undefined, labelId: string, fieldId: string, helpText: string) => (
+
+ onChange("promptTemplate", val || "")}
+ aria-labelledby={promptTemplateId}
+ onRenderLabel={props => renderLabel(props, promptTemplateId, promptTemplateFieldId, t("helpTexts.promptTemplate"))}
+ />
+
+ onChange("temperature", parseFloat(val || "0"))}
+ aria-labelledby={temperatureId}
+ onRenderLabel={props => renderLabel(props, temperatureId, temperatureFieldId, t("helpTexts.temperature"))}
+ />
+
+ onChange("seed", val ? parseInt(val) : null)}
+ aria-labelledby={seedId}
+ onRenderLabel={props => renderLabel(props, seedId, seedFieldId, t("helpTexts.seed"))}
+ />
+
+ {showAgenticRetrievalOption && (
+ onChange("useAgenticRetrieval", !!checked)}
+ aria-labelledby={agenticRetrievalId}
+ onRenderLabel={props => renderLabel(props, agenticRetrievalId, agenticRetrievalFieldId, t("helpTexts.suggestFollowupQuestions"))}
+ />
+ )}
+ {!useAgenticRetrieval && !useGPT4V && (
+ onChange("minimumSearchScore", parseFloat(val || "0"))}
+ aria-labelledby={searchScoreId}
+ onRenderLabel={props => renderLabel(props, searchScoreId, searchScoreFieldId, t("helpTexts.searchScore"))}
+ />
+ )}
+
+ {showSemanticRankerOption && (
+ onChange("minimumRerankerScore", parseFloat(val || "0"))}
+ aria-labelledby={rerankerScoreId}
+ onRenderLabel={props => renderLabel(props, rerankerScoreId, rerankerScoreFieldId, t("helpTexts.rerankerScore"))}
+ />
+ )}
+
+ {showAgenticRetrievalOption && useAgenticRetrieval && (
+ onChange("maxSubqueryCount", parseInt(val || "10"))}
+ aria-labelledby={maxSubqueryCountId}
+ onRenderLabel={props => renderLabel(props, maxSubqueryCountId, maxSubqueryCountFieldId, t("helpTexts.maxSubqueryCount"))}
+ />
+ )}
+
+ {showAgenticRetrievalOption && useAgenticRetrieval && (
+ , option?: IDropdownOption) =>
+ onChange("resultsMergeStrategy", option?.key)
+ }
+ aria-labelledby={includeCategoryId}
+ options={[
+ { key: "interleaved", text: t("labels.resultsMergeStrategyOptions.interleaved") },
+ { key: "descending", text: t("labels.resultsMergeStrategyOptions.descending") }
+ ]}
+ onRenderLabel={props => renderLabel(props, includeCategoryId, includeCategoryFieldId, t("helpTexts.resultsMergeStrategy"))}
+ />
+ )}
+
+ onChange("retrieveCount", parseInt(val || "3"))}
+ aria-labelledby={retrieveCountId}
+ onRenderLabel={props => renderLabel(props, retrieveCountId, retrieveCountFieldId, t("helpTexts.retrieveNumber"))}
+ />
+
+ , option?: IDropdownOption) => onChange("includeCategory", option?.key || "")}
+ aria-labelledby={includeCategoryId}
+ options={[
+ { key: "", text: t("labels.includeCategoryOptions.all") }
+ // { key: "example", text: "Example Category" } // Add more categories as needed
+ ]}
+ onRenderLabel={props => renderLabel(props, includeCategoryId, includeCategoryFieldId, t("helpTexts.includeCategory"))}
+ />
+
+ onChange("excludeCategory", val || "")}
+ aria-labelledby={excludeCategoryId}
+ onRenderLabel={props => renderLabel(props, excludeCategoryId, excludeCategoryFieldId, t("helpTexts.excludeCategory"))}
+ />
+
+ {showSemanticRankerOption && !useAgenticRetrieval && (
+ <>
+ onChange("useSemanticRanker", !!checked)}
+ aria-labelledby={semanticRankerId}
+ onRenderLabel={props => renderLabel(props, semanticRankerId, semanticRankerFieldId, t("helpTexts.useSemanticReranker"))}
+ />
+
+ onChange("useSemanticCaptions", !!checked)}
+ disabled={!useSemanticRanker}
+ aria-labelledby={semanticCaptionsId}
+ onRenderLabel={props => renderLabel(props, semanticCaptionsId, semanticCaptionsFieldId, t("helpTexts.useSemanticCaptions"))}
+ />
+
+ )}
+
+ {showQueryRewritingOption && !useAgenticRetrieval && (
+ <>
+ onChange("useQueryRewriting", !!checked)}
+ aria-labelledby={queryRewritingFieldId}
+ onRenderLabel={props => renderLabel(props, queryRewritingFieldId, queryRewritingFieldId, t("helpTexts.useQueryRewriting"))}
+ />
+
+ )}
+
+ {showReasoningEffortOption && (
+ , option?: IDropdownOption) =>
+ onChange("reasoningEffort", option?.key || "")
+ }
+ aria-labelledby={reasoningEffortFieldId}
+ options={[
+ { key: "low", text: t("labels.reasoningEffortOptions.low") },
+ { key: "medium", text: t("labels.reasoningEffortOptions.medium") },
+ { key: "high", text: t("labels.reasoningEffortOptions.high") }
+ ]}
+ onRenderLabel={props => renderLabel(props, queryRewritingFieldId, queryRewritingFieldId, t("helpTexts.reasoningEffort"))}
+ />
+ )}
+
+ {useLogin && (
+ <>
+ onChange("useOidSecurityFilter", !!checked)}
+ aria-labelledby={useOidSecurityFilterId}
+ onRenderLabel={props => renderLabel(props, useOidSecurityFilterId, useOidSecurityFilterFieldId, t("helpTexts.useOidSecurityFilter"))}
+ />
+ onChange("useGroupsSecurityFilter", !!checked)}
+ aria-labelledby={useGroupsSecurityFilterId}
+ onRenderLabel={props =>
+ renderLabel(props, useGroupsSecurityFilterId, useGroupsSecurityFilterFieldId, t("helpTexts.useGroupsSecurityFilter"))
+ }
+ />
+
+ )}
+
+ {showGPT4VOptions && !useAgenticRetrieval && (
+ onChange("useGPT4V", val)}
+ updateGPT4VInputs={val => onChange("gpt4vInput", val)}
+ />
+ )}
+
+ {showVectorOption && !useAgenticRetrieval && (
+ onChange("vectorFields", val)}
+ updateRetrievalMode={val => onChange("retrievalMode", val)}
+ />
+ )}
+
+ {/* Streaming checkbox for Chat */}
+ {shouldStream !== undefined && (
+ onChange("shouldStream", !!checked)}
+ aria-labelledby={shouldStreamId}
+ onRenderLabel={props => renderLabel(props, shouldStreamId, shouldStreamFieldId, t("helpTexts.streamChat"))}
+ />
+ )}
+
+ {/* Followup questions checkbox for Chat */}
+ {showSuggestFollowupQuestions && (
+ onChange("useSuggestFollowupQuestions", !!checked)}
+ aria-labelledby={suggestFollowupQuestionsId}
+ onRenderLabel={props =>
+ renderLabel(props, suggestFollowupQuestionsId, suggestFollowupQuestionsFieldId, t("helpTexts.suggestFollowupQuestions"))
+ }
+ />
+ )}
+
+ );
+};
diff --git a/app/frontend/src/components/SettingsButton/SettingsButton.module.css b/app/frontend/src/components/SettingsButton/SettingsButton.module.css
index fb533bfd84..8d998d2d84 100644
--- a/app/frontend/src/components/SettingsButton/SettingsButton.module.css
+++ b/app/frontend/src/components/SettingsButton/SettingsButton.module.css
@@ -1,6 +1,7 @@
.container {
display: flex;
align-items: center;
- gap: 6px;
+ gap: 0.375em;
cursor: pointer;
+ padding: 0.5rem;
}
diff --git a/app/frontend/src/components/SettingsButton/SettingsButton.tsx b/app/frontend/src/components/SettingsButton/SettingsButton.tsx
index 208247e9cd..8445320f45 100644
--- a/app/frontend/src/components/SettingsButton/SettingsButton.tsx
+++ b/app/frontend/src/components/SettingsButton/SettingsButton.tsx
@@ -1,5 +1,6 @@
-import { Text } from "@fluentui/react";
import { Settings24Regular } from "@fluentui/react-icons";
+import { Button } from "@fluentui/react-components";
+import { useTranslation } from "react-i18next";
import styles from "./SettingsButton.module.css";
@@ -9,10 +10,12 @@ interface Props {
}
export const SettingsButton = ({ className, onClick }: Props) => {
+ const { t } = useTranslation();
return (
-
- {"Developer settings"}
+
+ } onClick={onClick}>
+ {t("developerSettings")}
+
);
};
diff --git a/app/frontend/src/components/SupportingContent/SupportingContent.module.css b/app/frontend/src/components/SupportingContent/SupportingContent.module.css
index 3ce11a6a88..54bf03ff09 100644
--- a/app/frontend/src/components/SupportingContent/SupportingContent.module.css
+++ b/app/frontend/src/components/SupportingContent/SupportingContent.module.css
@@ -1,21 +1,22 @@
.supportingContentNavList {
list-style: none;
- padding-left: 5px;
+ padding-left: 0.3125em;
display: flex;
flex-direction: column;
- gap: 10px;
+ gap: 0.625em;
}
.supportingContentItem {
word-break: break-word;
background: rgb(249, 249, 249);
- border-radius: 8px;
- box-shadow: rgb(0 0 0 / 5%) 0px 0px 0px 1px, rgb(0 0 0 / 10%) 0px 2px 3px 0px;
- outline: transparent solid 1px;
-
+ border-radius: 0.5em;
+ box-shadow:
+ rgb(0 0 0 / 5%) 0px 0px 0px 0.0625em,
+ rgb(0 0 0 / 10%) 0px 0.125em 0.1875em 0px;
+ outline: transparent solid 0.0625em;
display: flex;
flex-direction: column;
- padding: 20px;
+ padding: 1.25em;
}
.supportingContentItemHeader {
@@ -26,3 +27,7 @@
margin-bottom: 0;
font-weight: 300;
}
+
+.supportingContentItemImage {
+ object-fit: contain;
+}
diff --git a/app/frontend/src/components/SupportingContent/SupportingContent.tsx b/app/frontend/src/components/SupportingContent/SupportingContent.tsx
index da6f981aa8..94df0ecca2 100644
--- a/app/frontend/src/components/SupportingContent/SupportingContent.tsx
+++ b/app/frontend/src/components/SupportingContent/SupportingContent.tsx
@@ -3,19 +3,27 @@ import { parseSupportingContentItem } from "./SupportingContentParser";
import styles from "./SupportingContent.module.css";
interface Props {
- supportingContent: string[];
+ supportingContent: string[] | { text: string[]; images?: string[] };
}
export const SupportingContent = ({ supportingContent }: Props) => {
+ const textItems = Array.isArray(supportingContent) ? supportingContent : supportingContent.text;
+ const imageItems = !Array.isArray(supportingContent) ? supportingContent?.images : [];
return (
-
- {supportingContent.map((x, i) => {
- const parsed = parseSupportingContentItem(x);
-
+ {textItems.map((c, ind) => {
+ const parsed = parseSupportingContentItem(c);
return (
-
- +
-
{parsed.title}
-{parsed.content}
+ +
+ );
+ })}
+ {imageItems?.map((img, ind) => {
+ return (
+ -
+
+
+ item.name}>
+
+ {({ renderHeaderCell }) => {renderHeaderCell()} }
+
+ >
+ {({ item, rowId }) => key={rowId}>{({ renderCell }) => {renderCell(item)} }}
+
+
+
+ );
+};
diff --git a/app/frontend/src/components/TokenClaimsDisplay/index.tsx b/app/frontend/src/components/TokenClaimsDisplay/index.tsx
new file mode 100644
index 0000000000..baec9b59a1
--- /dev/null
+++ b/app/frontend/src/components/TokenClaimsDisplay/index.tsx
@@ -0,0 +1 @@
+export * from "./TokenClaimsDisplay";
diff --git a/app/frontend/src/components/UploadFile/UploadFile.module.css b/app/frontend/src/components/UploadFile/UploadFile.module.css
new file mode 100644
index 0000000000..28bdaa117b
--- /dev/null
+++ b/app/frontend/src/components/UploadFile/UploadFile.module.css
@@ -0,0 +1,26 @@
+.container {
+ display: flex;
+ align-items: center;
+ gap: 0.375em;
+ cursor: pointer;
+ padding: 0.5rem;
+}
+
+.callout {
+ padding: 1em;
+ min-height: 6.25em;
+}
+
+.list {
+ display: flex;
+ justify-content: center;
+ align-items: center;
+}
+
+.item {
+ width: 15.625em;
+ white-space: nowrap;
+ overflow: hidden;
+ text-overflow: ellipsis;
+ margin-bottom: 0.75em;
+}
diff --git a/app/frontend/src/components/UploadFile/UploadFile.tsx b/app/frontend/src/components/UploadFile/UploadFile.tsx
new file mode 100644
index 0000000000..fe5a5c0535
--- /dev/null
+++ b/app/frontend/src/components/UploadFile/UploadFile.tsx
@@ -0,0 +1,166 @@
+import React, { useState, ChangeEvent } from "react";
+import { Callout, Label, Text } from "@fluentui/react";
+import { Button } from "@fluentui/react-components";
+import { Add24Regular, Delete24Regular } from "@fluentui/react-icons";
+import { useMsal } from "@azure/msal-react";
+import { useTranslation } from "react-i18next";
+
+import { SimpleAPIResponse, uploadFileApi, deleteUploadedFileApi, listUploadedFilesApi } from "../../api";
+import { useLogin, getToken } from "../../authConfig";
+import styles from "./UploadFile.module.css";
+
+interface Props {
+ className?: string;
+ disabled?: boolean;
+}
+
+export const UploadFile: React.FC
+
+ );
+};
diff --git a/app/frontend/src/components/UploadFile/index.tsx b/app/frontend/src/components/UploadFile/index.tsx
new file mode 100644
index 0000000000..c91349002f
--- /dev/null
+++ b/app/frontend/src/components/UploadFile/index.tsx
@@ -0,0 +1 @@
+export * from "./UploadFile";
diff --git a/app/frontend/src/components/UserChatMessage/UserChatMessage.module.css b/app/frontend/src/components/UserChatMessage/UserChatMessage.module.css
index c80bea6f7d..70e73d67a0 100644
--- a/app/frontend/src/components/UserChatMessage/UserChatMessage.module.css
+++ b/app/frontend/src/components/UserChatMessage/UserChatMessage.module.css
@@ -1,15 +1,17 @@
.container {
display: flex;
justify-content: flex-end;
- margin-bottom: 20px;
+ margin-bottom: 1.25em;
max-width: 80%;
margin-left: auto;
}
.message {
- padding: 20px;
+ padding: 1.25em;
background: #e8ebfa;
- border-radius: 8px;
- box-shadow: 0px 2px 4px rgba(0, 0, 0, 0.14), 0px 0px 2px rgba(0, 0, 0, 0.12);
- outline: transparent solid 1px;
+ border-radius: 0.5em;
+ box-shadow:
+ 0px 0.125em 0.25em rgba(0, 0, 0, 0.14),
+ 0px 0px 0.125em rgba(0, 0, 0, 0.12);
+ outline: transparent solid 0.0625em;
}
diff --git a/app/frontend/src/components/VectorSettings/VectorSettings.module.css b/app/frontend/src/components/VectorSettings/VectorSettings.module.css
new file mode 100644
index 0000000000..4cf11f362a
--- /dev/null
+++ b/app/frontend/src/components/VectorSettings/VectorSettings.module.css
@@ -0,0 +1,3 @@
+.container {
+ margin-top: 0.625em;
+}
diff --git a/app/frontend/src/components/VectorSettings/VectorSettings.tsx b/app/frontend/src/components/VectorSettings/VectorSettings.tsx
new file mode 100644
index 0000000000..2010eeb50c
--- /dev/null
+++ b/app/frontend/src/components/VectorSettings/VectorSettings.tsx
@@ -0,0 +1,114 @@
+import { useEffect, useState } from "react";
+import { Stack, IDropdownOption, Dropdown, IDropdownProps } from "@fluentui/react";
+import { useId } from "@fluentui/react-hooks";
+import { useTranslation } from "react-i18next";
+
+import styles from "./VectorSettings.module.css";
+import { HelpCallout } from "../../components/HelpCallout";
+import { RetrievalMode, VectorFields } from "../../api";
+
+interface Props {
+ showImageOptions?: boolean;
+ defaultRetrievalMode: RetrievalMode;
+ defaultVectorFields?: VectorFields;
+ updateRetrievalMode: (retrievalMode: RetrievalMode) => void;
+ updateVectorFields: (vectorFields: VectorFields) => void;
+}
+
+export const VectorSettings = ({ updateRetrievalMode, updateVectorFields, showImageOptions, defaultRetrievalMode, defaultVectorFields }: Props) => {
+ const [retrievalMode, setRetrievalMode] = useState
+ } disabled={disabled} onClick={handleButtonClick}>
+ {t("upload.manageFileUploads")}
+
+
+ {isCalloutVisible && (
+ setIsCalloutVisible(false)}
+ setInitialFocus
+ >
+
+
+ {/* Show a loading message while files are being uploaded */}
+ {isUploading && {t("upload.uploadingFiles")} }
+ {!isUploading && uploadedFileError && {uploadedFileError} }
+ {!isUploading && uploadedFile && {uploadedFile.message} }
+
+ {/* Display the list of already uploaded */}
+ {t("upload.loading")} }
+ {!isLoading && uploadedFiles.length === 0 && {t("upload.noFilesUploaded")} }
+ {uploadedFiles.map((filename, index) => {
+ return (
+
+ )}
+
+ {t("upload.uploadedFilesLabel")}
+ + {isLoading &&
+
+ );
+ })}
+ {filename}
+ {/* Button to remove a file from the list */}
+ }
+ onClick={() => handleRemoveFile(filename)}
+ disabled={deletionStatus[filename] === "pending" || deletionStatus[filename] === "success"}
+ >
+ {!deletionStatus[filename] && t("upload.deleteFile")}
+ {deletionStatus[filename] == "pending" && t("upload.deletingFile")}
+ {deletionStatus[filename] == "error" && t("upload.errorDeleting")}
+ {deletionStatus[filename] == "success" && t("upload.fileDeleted")}
+
+
+ ({
+ key: code,
+ text: details.name,
+ selected: code === i18n.language,
+ data: code
+ }))}
+ onChange={handleLanguageChange}
+ ariaLabel={t("labels.languagePicker")}
+ />
+
+ );
+};
diff --git a/app/frontend/src/i18n/config.ts b/app/frontend/src/i18n/config.ts
new file mode 100644
index 0000000000..c17b074225
--- /dev/null
+++ b/app/frontend/src/i18n/config.ts
@@ -0,0 +1,81 @@
+import i18next from "i18next";
+import LanguageDetector from "i18next-browser-languagedetector";
+import HttpApi from "i18next-http-backend";
+import { initReactI18next } from "react-i18next";
+
+import daTranslation from "../locales/da/translation.json";
+import enTranslation from "../locales/en/translation.json";
+import esTranslation from "../locales/es/translation.json";
+import frTranslation from "../locales/fr/translation.json";
+import jaTranslation from "../locales/ja/translation.json";
+import nlTranslation from "../locales/nl/translation.json";
+import ptBRTranslation from "../locales/ptBR/translation.json";
+import trTranslation from "../locales/tr/translation.json";
+import itTranslation from "../locales/it/translation.json";
+
+export const supportedLngs: { [key: string]: { name: string; locale: string } } = {
+ da: {
+ name: "Dansk",
+ locale: "da-DK"
+ },
+ en: {
+ name: "English",
+ locale: "en-US"
+ },
+ es: {
+ name: "Español",
+ locale: "es-ES"
+ },
+ fr: {
+ name: "Français",
+ locale: "fr-FR"
+ },
+ ja: {
+ name: "日本語",
+ locale: "ja-JP"
+ },
+ nl: {
+ name: "Nederlands",
+ locale: "nl-NL"
+ },
+ ptBR: {
+ name: "Português Brasileiro",
+ locale: "pt-BR"
+ },
+ tr: {
+ name: "Türkçe",
+ locale: "tr-TR"
+ },
+ it: {
+ name: "Italiano",
+ locale: "it-IT"
+ }
+};
+
+i18next
+ .use(HttpApi)
+ .use(LanguageDetector)
+ .use(initReactI18next)
+ // init i18next
+ // for all options read: https://www.i18next.com/overview/configuration-options
+ .init({
+ resources: {
+ da: { translation: daTranslation },
+ en: { translation: enTranslation },
+ es: { translation: esTranslation },
+ fr: { translation: frTranslation },
+ ja: { translation: jaTranslation },
+ nl: { translation: nlTranslation },
+ ptBR: { translation: ptBRTranslation },
+ tr: { translation: trTranslation },
+ it: { translation: itTranslation }
+ },
+ fallbackLng: "en",
+ supportedLngs: Object.keys(supportedLngs),
+ debug: import.meta.env.DEV,
+ interpolation: {
+ escapeValue: false // not needed for react as it escapes by default
+ }
+ });
+
+export default i18next;
diff --git a/app/frontend/src/i18n/index.tsx b/app/frontend/src/i18n/index.tsx
new file mode 100644
index 0000000000..0a84ff2b10
--- /dev/null
+++ b/app/frontend/src/i18n/index.tsx
@@ -0,0 +1 @@
+export * from "./LanguagePicker";
diff --git a/app/frontend/src/index.css b/app/frontend/src/index.css
index 2efb29b41f..f3414a3c95 100644
--- a/app/frontend/src/index.css
+++ b/app/frontend/src/index.css
@@ -2,22 +2,55 @@
box-sizing: border-box;
}
-html,
-body {
- height: 100%;
- margin: 0;
- padding: 0;
-}
-
html {
background: #f2f2f2;
-
- font-family: "Segoe UI", -apple-system, BlinkMacSystemFont, "Roboto", "Oxygen", "Ubuntu", "Cantarell", "Fira Sans", "Droid Sans", "Helvetica Neue",
+ font-size: 12px; /* Root font size for 320px width screens */
+ font-family:
+ "Segoe UI",
+ -apple-system,
+ BlinkMacSystemFont,
+ "Roboto",
+ "Oxygen",
+ "Ubuntu",
+ "Cantarell",
+ "Fira Sans",
+ "Droid Sans",
+ "Helvetica Neue",
sans-serif;
-webkit-font-smoothing: antialiased;
-moz-osx-font-smoothing: grayscale;
}
+body {
+ height: 100%;
+ margin: 0;
+ padding: 0;
+}
+
#root {
height: 100%;
}
+
+@media (min-width: 480px) {
+ html {
+ font-size: 13px; /* Slightly larger font size for larger phones */
+ }
+}
+
+@media (min-width: 600px) {
+ html {
+ font-size: 14px; /* Adjust for small tablets and large phones in landscape */
+ }
+}
+
+@media (min-width: 768px) {
+ html {
+ font-size: 15px; /* Suitable for medium tablets */
+ }
+}
+
+@media (min-width: 992px) {
+ html {
+ font-size: 16px; /* Standard size for desktops */
+ }
+}
diff --git a/app/frontend/src/index.tsx b/app/frontend/src/index.tsx
index 58724ece5b..a8821c8c45 100644
--- a/app/frontend/src/index.tsx
+++ b/app/frontend/src/index.tsx
@@ -1,19 +1,22 @@
import React from "react";
import ReactDOM from "react-dom/client";
import { createHashRouter, RouterProvider } from "react-router-dom";
+import { I18nextProvider } from "react-i18next";
+import { HelmetProvider } from "react-helmet-async";
import { initializeIcons } from "@fluentui/react";
import "./index.css";
-import Layout from "./pages/layout/Layout";
import Chat from "./pages/chat/Chat";
+import LayoutWrapper from "./layoutWrapper";
+import i18next from "./i18n/config";
initializeIcons();
const router = createHashRouter([
{
path: "/",
- element:
+ {/* Setting the page title using react-helmet-async */}
+
+ {t("pageTitle")}
+
+ setIsConfigPanelOpen(false)}
+ closeButtonAriaLabel={t("labels.closeButton")}
+ onRenderFooterContent={() => setIsConfigPanelOpen(false)}>{t("labels.closeButton")} }
+ isFooterAtBottom={true}
+ >
+
+
+ );
+}
+
+Component.displayName = "Ask";
diff --git a/app/frontend/src/pages/chat/Chat.module.css b/app/frontend/src/pages/chat/Chat.module.css
index 91c9818a38..453485241f 100644
--- a/app/frontend/src/pages/chat/Chat.module.css
+++ b/app/frontend/src/pages/chat/Chat.module.css
@@ -2,7 +2,7 @@
flex: 1;
display: flex;
flex-direction: column;
- margin-top: 20px;
+ margin-top: 1rem;
}
.chatRoot {
@@ -16,6 +16,7 @@
flex-direction: column;
align-items: center;
width: 100%;
+ max-height: calc(100vh - 10rem);
}
.chatEmptyState {
@@ -24,92 +25,122 @@
flex-direction: column;
justify-content: center;
align-items: center;
- max-height: 1024px;
- padding-top: 60px;
+ max-height: 64rem;
+ padding-top: 1rem;
}
.chatEmptyStateTitle {
- font-size: 4rem;
+ font-size: 2.75rem;
font-weight: 600;
margin-top: 0;
- margin-bottom: 30px;
+ margin-bottom: 1.875rem;
}
.chatEmptyStateSubtitle {
font-weight: 600;
- margin-bottom: 10px;
-}
-
-@media only screen and (max-height: 780px) {
- .chatEmptyState {
- padding-top: 0;
- }
-
- .chatEmptyStateTitle {
- font-size: 3rem;
- margin-bottom: 0px;
- }
+ margin-bottom: 0.625rem;
}
.chatMessageStream {
flex-grow: 1;
- max-height: 1024px;
- max-width: 1028px;
+ max-height: 64rem;
width: 100%;
overflow-y: auto;
- padding-left: 24px;
- padding-right: 24px;
+ padding: 0 2rem;
display: flex;
flex-direction: column;
+ resize: horizontal;
+ min-width: 50vw;
}
.chatMessageGpt {
- margin-bottom: 20px;
- max-width: 80%;
+ margin-bottom: 1.25rem;
+ max-width: 100%;
display: flex;
- min-width: 500px;
}
.chatMessageGptMinWidth {
- max-width: 500px;
- margin-bottom: 20px;
+ max-width: 31.25rem;
+ margin-bottom: 1.25rem;
}
.chatInput {
position: sticky;
bottom: 0;
- flex: 0 0 100px;
- padding-top: 12px;
- padding-bottom: 24px;
- padding-left: 24px;
- padding-right: 24px;
+ flex: 0 0 6.25rem;
+ padding: 1rem;
width: 100%;
- max-width: 1028px;
+ max-width: 64.25rem;
background: #f2f2f2;
}
.chatAnalysisPanel {
flex: 1;
overflow-y: auto;
- max-height: 89vh;
- margin-left: 20px;
- margin-right: 20px;
+ max-height: 85vh;
+ margin-left: 1.25rem;
+ margin-right: 1.25rem;
}
.chatSettingsSeparator {
- margin-top: 15px;
+ display: flex;
+ flex-direction: column;
+ margin-top: 0.9375rem;
}
.loadingLogo {
- font-size: 28px;
+ font-size: 1.75rem;
+}
+
+.commandsSplitContainer {
+ display: flex;
+ justify-content: space-between;
}
.commandsContainer {
display: flex;
+ padding-left: 1rem;
+ padding-right: 1rem;
align-self: flex-end;
}
.commandButton {
- margin-right: 20px;
- margin-bottom: 20px;
+ margin-bottom: 1.25rem;
+}
+
+@media (min-width: 992px) {
+ .container {
+ margin-top: 1.25rem;
+ }
+
+ .chatEmptyState {
+ padding-top: 3.75rem;
+ }
+
+ .chatEmptyStateTitle {
+ font-size: 4rem;
+ }
+
+ .chatInput {
+ padding: 0.75rem 1.5rem 1.5rem;
+ }
+
+ .commandsSplitContainer {
+ padding-left: 1rem;
+ }
+
+ .commandsContainer {
+ padding-left: 0rem;
+ padding-right: 0rem;
+ }
+
+ .commandButton {
+ margin-right: 1.25rem;
+ margin-bottom: 1.25rem;
+ }
+
+ .chatMessageGpt {
+ max-width: 80%;
+ min-width: 31.25rem;
+ }
}
diff --git a/app/frontend/src/pages/chat/Chat.tsx b/app/frontend/src/pages/chat/Chat.tsx
index 72c3d39fd3..579ad05fea 100644
--- a/app/frontend/src/pages/chat/Chat.tsx
+++ b/app/frontend/src/pages/chat/Chat.tsx
@@ -1,39 +1,199 @@
-import { useRef, useState, useEffect } from "react";
-import { Checkbox, Panel, DefaultButton, TextField, SpinButton, Dropdown, IDropdownOption } from "@fluentui/react";
-import { SparkleFilled } from "@fluentui/react-icons";
+import { useRef, useState, useEffect, useContext } from "react";
+import { useTranslation } from "react-i18next";
+import { Helmet } from "react-helmet-async";
+import { Panel, DefaultButton } from "@fluentui/react";
+import readNDJSONStream from "ndjson-readablestream";
+import appLogo from "../../assets/applogo.svg";
import styles from "./Chat.module.css";
-import { chatApi, RetrievalMode, Approaches, AskResponse, ChatRequest, ChatTurn } from "../../api";
+import {
+ chatApi,
+ configApi,
+ RetrievalMode,
+ ChatAppResponse,
+ ChatAppResponseOrError,
+ ChatAppRequest,
+ ResponseMessage,
+ VectorFields,
+ GPT4VInput,
+ SpeechConfig
+} from "../../api";
import { Answer, AnswerError, AnswerLoading } from "../../components/Answer";
import { QuestionInput } from "../../components/QuestionInput";
import { ExampleList } from "../../components/Example";
import { UserChatMessage } from "../../components/UserChatMessage";
import { AnalysisPanel, AnalysisPanelTabs } from "../../components/AnalysisPanel";
+import { HistoryPanel } from "../../components/HistoryPanel";
+import { HistoryProviderOptions, useHistoryManager } from "../../components/HistoryProviders";
+import { HistoryButton } from "../../components/HistoryButton";
import { SettingsButton } from "../../components/SettingsButton";
import { ClearChatButton } from "../../components/ClearChatButton";
+import { UploadFile } from "../../components/UploadFile";
+import { useLogin, getToken, requireAccessControl } from "../../authConfig";
+import { useMsal } from "@azure/msal-react";
+import { TokenClaimsDisplay } from "../../components/TokenClaimsDisplay";
+import { LoginContext } from "../../loginContext";
+import { LanguagePicker } from "../../i18n/LanguagePicker";
+import { Settings } from "../../components/Settings/Settings";
const Chat = () => {
const [isConfigPanelOpen, setIsConfigPanelOpen] = useState(false);
+ const [isHistoryPanelOpen, setIsHistoryPanelOpen] = useState(false);
const [promptTemplate, setPromptTemplate] = useState
+
+
+ {showUserUpload && setIsConfigPanelOpen(!isConfigPanelOpen)} />
+
+ {t("askTitle")}
+
+ makeApiRequest(question)}
+ showSpeechInput={showSpeechInput}
+ />
+
+
+ {isLoading && onToggleTab(x)}
+ citationHeight="600px"
+ answer={answer}
+ activeTab={activeAnalysisPanelTab}
+ />
+ )}
+
+
+
+ {showLanguagePicker && i18n.changeLanguage(newLang)} />}
+
+ )}
+ {!isLoading && answer && !error && (
+
+ onShowCitation(x)}
+ onThoughtProcessClicked={() => onToggleTab(AnalysisPanelTabs.ThoughtProcessTab)}
+ onSupportingContentClicked={() => onToggleTab(AnalysisPanelTabs.SupportingContentTab)}
+ showSpeechOutputAzure={showSpeechOutputAzure}
+ showSpeechOutputBrowser={showSpeechOutputBrowser}
+ />
+
+ )}
+ {error ? (
+
+ makeApiRequest(lastQuestionRef.current)} />
+
+ ) : null}
+ {activeAnalysisPanelTab && answer && (
+
-
- setIsConfigPanelOpen(!isConfigPanelOpen)} />
+ {/* Setting the page title using react-helmet-async */}
+
+ {t("pageTitle")}
+
+
+
-
+ {((useLogin && showChatHistoryCosmos) || showChatHistoryBrowser) && (
+ setIsHistoryPanelOpen(!isHistoryPanelOpen)} />
+ )}
+
+
+ setIsConfigPanelOpen(!isConfigPanelOpen)} />
+
+
{!lastQuestionRef.current ? (
-
- 
+
+ i18n.changeLanguage(newLang)} />}
+
+