feat: generic example questions & reliable local ingestion script

Author: jacob-gaylord

.gitignore

@@ -148,5 +148,6 @@ node_modules
 static/
 
 data/**/*.md5
+data
 
 .DS_Store

app/backend/approaches/prompts/ask_answer_question.prompty

@@ -4,8 +4,9 @@ description: Answer a single question (with no chat history) using solely text s
 model:
     api: chat
 sample:
-    user_query: What does a product manager do?
+    user_query: Which metrics should I track for Premium Capacity in Fabric?
     text_sources:
+        - "7 - Monitoring.md#L210-L245: Premium capacity KPIs include CPU utilization, memory consumption, query duration, dataset refresh times, and concurrency slots."
         - "role_library.pdf#page=29:  The Manager of Product Management will collaborate with internal teams, such as engineering, sales, marketing, and finance, as well as external partners, suppliers, and customers to ensure successful product execution. Responsibilities: · Lead the product management team and provide guidance on product strategy, design, development, and launch. · Develop and implement product life-cycle management processes. · Monitor and analyze industry trends to identify opportunities for new products. · Develop product marketing plans and go-to-market strategies. · Research customer needs and develop customer-centric product roadmaps. · Collaborate with internal teams to ensure product execution and successful launch. · Develop pricing strategies and cost models. · Oversee product portfolio and performance metrics. · Manage product development budget. · Analyze product performance and customer feedback to identify areas for improvement. Qualifications: · Bachelor's degree in business, engineering, or a related field. · At least 5 years of experience in product management. · Proven track record of successful product launches."
         - "role_library.pdf#page=23: Company: Contoso Electronics Location: Anywhere Job Type: Full-Time Salary: Competitive, commensurate with experience Job Summary: The Senior Manager of Product Management will be responsible for leading the product management team at Contoso Electronics. This role includes developing strategies, plans and objectives for the product management team and managing the day-to-day operations. The Senior Manager of Product Management will be responsible for the successful launch of new products and the optimization of existing products. Responsibilities: · Develop and implement product management strategies, plans and objectives to maximize team performance. · Analyze competitive landscape and market trends to develop product strategies. · Lead the product management team in the development of product plans, roadmaps and launch plans. · Monitor the performance of product management team, analyze results and implement corrective action as needed. · Manage the product lifecycle, including product development, launch, and end of life. · Ensure product features and benefits meet customer requirements. · Establish and maintain relationships with key customers, partners, and vendors."
         - "role_library.pdf#page=28:  · 7+ years of experience in research and development in the electronics sector. · Proven track record of successfully designing, testing, and optimizing products. · Experience leading a team of researchers and engineers. · Excellent problem-solving and analytical skills. · Ability to work in a fast-paced environment and meet tight deadlines.· Knowledge of industry trends, technologies, and regulations. · Excellent communication and presentation skills. Manager of Product Management Job Title: Manager of Product Management, Contoso Electronics Job Summary: The Manager of Product Management is responsible for overseeing the product management team, driving product development and marketing strategy for Contoso Electronics. This individual will be accountable for the successful launch of new products and the implementation of product life-cycle management processes. The Manager of Product Management will collaborate with internal teams, such as engineering, sales, marketing, and finance, as well as external partners, suppliers, and customers to ensure successful product execution."
@@ -14,11 +15,12 @@ system:
 {% if override_prompt %}
 {{ override_prompt }}
 {% else %}
-You are an intelligent assistant helping Contoso Inc employees with their healthcare plan questions and employee handbook questions.
+You are 3Cloud's Solution-Objects Assistant. You help consultants, engineers, and sales teams locate, explain, and compare 3Cloud Solution Objects (architecture decks, runbooks, pricing guides, security practices, etc.) and related internal resources (Employee Handbook, Benefits, Career paths).
 Use 'you' to refer to the individual asking the questions even if they ask with 'I'.
+Provide succinct but complete answers.
 Answer the following question using only the data provided in the sources below.
 Each source has a name followed by colon and the actual information, always include the source name for each fact you use in the response.
-If you cannot answer using the sources below, say you don't know. Use below example to answer
+If you cannot answer using the sources below, say you don't know.
 {{ injected_prompt }}
 {% endif %}
 
@@ -34,6 +36,15 @@ info4.pdf: In-network institutions include Overlake, Swedish and others in the r
 assistant:
 In-network deductibles are $500 for employee and $1000 for family [info1.txt] and Overlake is in-network for the employee plan [info2.pdf][info4.pdf].
 
+user:
+Which metrics should I track for Premium Capacity in Fabric?
+
+Sources:
+7 - Monitoring.md#L210-L245: Premium capacity KPIs include CPU utilization, memory consumption, query duration, dataset refresh times, and concurrency slots.
+
+assistant:
+Track CPU utilization, memory consumption, query duration, dataset refresh times, and concurrency slots to effectively monitor Premium Capacity performance [7 - Monitoring.md#L210-L245].
+
 user:
 {{ user_query }}
 Sources:

app/backend/approaches/prompts/ask_answer_question_vision.prompty

@@ -8,7 +8,8 @@ system:
 {% if override_prompt %}
 {{ override_prompt }}
 {% else %}
-You are an intelligent assistant helping analyze the Annual Financial Report of Contoso Ltd., The documents contain text, graphs, tables and images.
+You are 3Cloud's Solution-Objects Assistant. You help consultants, engineers, and sales teams locate, explain, and compare 3Cloud Solution Objects (architecture decks, runbooks, pricing guides, security practices, etc.) and related internal resources (Employee Handbook, Benefits, Career paths).
+Provide succinct but complete answers.
 Each image source has the file name in the top left corner of the image with coordinates (10,10) pixels and is in the format SourceFileName:<file_name>.
 Each text source starts in a new line and has the file name followed by colon and the actual information.
 Always include the source name from the image or text for each fact you use in the response in the format: [filename].

app/backend/approaches/prompts/chat_answer_question.prompty

@@ -20,7 +20,7 @@ system:
 {% if override_prompt %}
 {{ override_prompt }}
 {% else %}
-Assistant helps the company employees with their healthcare plan questions, and questions about the employee handbook. Be brief in your answers.
+You are 3Cloud's Solution-Objects Assistant. You help consultants, engineers, and sales teams locate, explain, and compare 3Cloud Solution Objects (architecture decks, runbooks, pricing guides, security practices, etc.) and related internal resources (Employee Handbook, Benefits, Career paths). Provide succinct but complete answers.
 Answer ONLY with the facts listed in the list of sources below. If there isn't enough information below, say you don't know. Do not generate answers that don't use the sources below. If asking a clarifying question to the user would help, ask the question.
 If the question is not in English, answer in the language used in the question.
 Each source has a name followed by colon and the actual information, always include the source name for each fact you use in the response. Use square brackets to reference the source, for example [info1.txt]. Don't combine sources, list each source separately, for example [info1.txt][info2.pdf].
@@ -30,9 +30,9 @@ Each source has a name followed by colon and the actual information, always incl
 {% if include_follow_up_questions %}
 Generate 3 very brief follow-up questions that the user would likely ask next.
 Enclose the follow-up questions in double angle brackets. Example:
-<<Are there exclusions for prescriptions?>>
-<<Which pharmacies can be ordered from?>>
-<<What is the limit for over-the-counter medication?>>
+<<Show me the reference architecture diagram?>>
+<<Which roles must own the On-Prem Data Gateway?>>
+<<How often should we run capacity monitoring reports?>>
 Do not repeat questions that have already been asked.
 Make sure the last question ends with ">>".
 {% endif %}

app/backend/approaches/prompts/chat_answer_question_vision.prompty

@@ -8,13 +8,13 @@ system:
 {% if override_prompt %}
 {{ override_prompt }}
 {% else %}
-You are an intelligent assistant helping analyze the Annual Financial Report of Contoso Ltd., The documents contain text, graphs, tables and images.
+You are 3Cloud's Solution-Objects Assistant. You help consultants, engineers, and sales teams locate, explain, and compare 3Cloud Solution Objects (architecture decks, runbooks, pricing guides, security practices, etc.) and related internal resources (Employee Handbook, Benefits, Career paths). Provide succinct but complete answers.
 Each image source has the file name in the top left corner of the image with coordinates (10,10) pixels and is in the format SourceFileName:<file_name>
 Each text source starts in a new line and has the file name followed by colon and the actual information
 Always include the source name from the image or text for each fact you use in the response in the format: [filename]
 Answer the following question using only the data provided in the sources below.
 If asking a clarifying question to the user would help, ask the question.
-Be brief in your answers.
+
 The text and image source can be the same file name, don't use the image title when citing the image source, only use the file name as mentioned
 If you cannot answer using the sources below, say you don't know. Return just the answer without any input texts.
 {{injected_prompt}}
@@ -23,9 +23,9 @@ If you cannot answer using the sources below, say you don't know. Return just th
 {% if include_follow_up_questions %}
 Generate 3 very brief follow-up questions that the user would likely ask next.
 Enclose the follow-up questions in double angle brackets. Example:
-<<Are there exclusions for prescriptions?>>
-<<Which pharmacies can be ordered from?>>
-<<What is the limit for over-the-counter medication?>>
+<<Show me the reference architecture diagram?>>
+<<Which roles must own the On-Prem Data Gateway?>>
+<<How often should we run capacity monitoring reports?>>
 Do not repeat questions that have already been asked.
 Make sure the last question ends with ">>".
 {% endif %}

app/frontend/src/api/api.ts

@@ -79,7 +79,11 @@ export async function getSpeechApi(text: string): Promise<string | null> {
 }
 
 export function getCitationFilePath(citation: string): string {
-    return `${BACKEND_URI}/content/${citation}`;
+    // Encode the actual file path so spaces and other unsafe characters are handled,
+    // but keep any page fragment (e.g. "#page=3") intact so the PDF opens on the correct page.
+    const [pathPart, fragment] = citation.split("#");
+    const encodedPath = encodeURIComponent(pathPart);
+    return `${BACKEND_URI}/content/${encodedPath}${fragment ? `#${fragment}` : ""}`;
 }
 
 export async function uploadFileApi(request: FormData, idToken: string): Promise<SimpleAPIResponse> {

app/frontend/src/components/Answer/AnswerParser.tsx

@@ -8,12 +8,13 @@ type HtmlParsedAnswer = {
 
 // Function to validate citation format and check if dataPoint starts with possible citation
 function isCitationValid(contextDataPoints: any, citationCandidate: string): boolean {
-    const regex = /.+\.\w{1,}(?:#\S*)?$/;
-    if (!regex.test(citationCandidate)) {
+    const trimmed = citationCandidate.trim();
+    const regex = /.+\.[A-Za-z0-9]{1,}(?:#\S*)?$/;
+    if (!regex.test(trimmed)) {
         return false;
     }
 
-    // Check if contextDataPoints is an object with a text property that is an array
+    // Determine the array of data points
     let dataPointsArray: string[];
     if (Array.isArray(contextDataPoints)) {
         dataPointsArray = contextDataPoints;
@@ -23,11 +24,7 @@ function isCitationValid(contextDataPoints: any, citationCandidate: string): boo
         return false;
     }
 
-    const isValidCitation = dataPointsArray.some(dataPoint => {
-        return dataPoint.startsWith(citationCandidate);
-    });
-
-    return isValidCitation;
+    return dataPointsArray.some(dp => dp.startsWith(trimmed));
 }
 
 export function parseAnswerToHtml(answer: ChatAppResponse, isStreaming: boolean, onCitationClicked: (citationFilePath: string) => void): HtmlParsedAnswer {

app/frontend/src/components/Settings/Settings.tsx

@@ -269,8 +269,9 @@ export const Settings = ({
                 onChange={(_ev?: React.FormEvent<HTMLElement | HTMLInputElement>, option?: IDropdownOption) => onChange("includeCategory", option?.key || "")}
                 aria-labelledby={includeCategoryId}
                 options={[
-                    { key: "", text: t("labels.includeCategoryOptions.all") }
-                    // { key: "example", text: "Example Category" } // Add more categories as needed
+                    { key: "", text: t("labels.includeCategoryOptions.all") },
+                    { key: "Solution Object", text: t("labels.includeCategoryOptions.solutionObjects") },
+                    { key: "Employee Info", text: t("labels.includeCategoryOptions.employeeInfo") }
                 ]}
                 onRenderLabel={props => renderLabel(props, includeCategoryId, includeCategoryFieldId, t("helpTexts.includeCategory"))}
             />

app/frontend/src/locales/en/translation.json

@@ -37,17 +37,17 @@
     "chatEmptyStateTitle": "Chat with your data",
     "chatEmptyStateSubtitle": "Ask anything or try an example",
     "defaultExamples": {
-        "1": "What is included in my Northwind Health Plus plan that is not in standard?",
-        "2": "What happens in a performance review?",
-        "3": "What does a Product Manager do?",
-        "placeholder": "Type a new question (e.g. does my plan cover annual eye exams?)"
+        "1": "What deployment options are available for this solution?",
+        "2": "What security controls should I consider for a BI service?",
+        "3": "Which career path levels relate to data engineering?",
+        "placeholder": "Example: Which document covers on-premises gateway setup?"
     },
     "askTitle": "Ask your data",
     "gpt4vExamples": {
-        "1": "Compare the impact of interest rates and GDP in financial markets.",
-        "2": "What is the expected trend for the S&P 500 index over the next five years? Compare it to the past S&P 500 performance",
-        "3": "Can you identify any correlation between oil prices and stock market trends?",
-        "placeholder": "Example: Does my plan cover annual eye exams?"
+        "1": "Summarise a typical architecture diagram for cloud analytics.",
+        "2": "Compare common cost models such as pay-as-you-go and reserved capacity.",
+        "3": "What governance responsibilities apply to a data platform implementation?",
+        "placeholder": "Example: What insights can a capacity planning heatmap provide?"
     },
     "generatingAnswer": "Generating answer",
     "citationWithColon": "Citation:",
@@ -91,7 +91,9 @@
         },
         "includeCategory": "Include category",
         "includeCategoryOptions": {
-            "all": "All"
+            "all": "All",
+            "solutionObjects": "Solution Objects",
+            "employeeInfo": "3Cloud"
         },
         "excludeCategory": "Exclude category",
         "useSemanticRanker": "Use semantic ranker for retrieval",