Merge pull request #1583 from MicrosoftDocs/release-ignite-multimodal-intelligence-preview

Release ignite multimodal intelligence preview -> main -- 11/19 10:00 AM PST

Author: Albertyang0

articles/ai-services/content-understanding/audio/overview.md

@@ -0,0 +1,86 @@
+---
+title: Azure AI Content Understanding audio overview
+titleSuffix: Azure AI services
+description: Learn about Azure AI Content Understanding audio solutions
+author: laujan
+ms.author: lajanuar
+manager: nitinme
+ms.service: azure
+ms.topic: overview
+ms.date: 11/19/2024
+ms.custom: ignite-2024-understanding-release
+---
+
+
+# Content Understanding audio solutions (preview)
+
+> [!IMPORTANT]
+>
+> * Azure AI Content Understanding is available in preview. Public preview releases provide early access to features that are in active development.
+> * Features, approaches, and processes may change or have constrained capabilities, prior to General Availability (GA).
+> * For more information, *see* [**Supplemental Terms of Use for Microsoft Azure Previews**](https://azure.microsoft.com/support/legal/preview-supplemental-terms).
+
+Content Understanding audio analyzers enable transcription and diarization of conversational audio, extracting structured fields such as summaries, sentiments, and key topics. Customize an audio analyzer template to your business needs using [Azure AI Foundry](https://ai.azure.com/) to start generating results.
+
+Here are common scenarios for using Content Understanding with conversational audio data:
+
+* Gain customer insights through summarization and sentiment analysis.
+* Assess and verify call quality and compliance in call centers.
+* Create automated summaries and metadata for podcast publishing.
+
+## Audio analyzer capabilities
+
+:::image type="content" source="../media/audio/overview/workflow-diagram.png" lightbox="../media/audio/overview/workflow-diagram.png" alt-text="Illustration of Content Understanding audio workflow.":::
+
+Content Understanding serves as a cornerstone for Media Asset Management solutions, enabling the following capabilities for audio files:
+  
+### Content extraction
+
+* **Transcription**. Converts conversational audio into searchable and analyzable text-based transcripts in WebVTT format. Customizable fields can be generated from transcription data. Sentence-level and word-level timestamps are available upon request.
+
+* **`Diarization`**. Distinguishes between speakers in a conversation, attributing parts of the transcript to specific speakers.
+
+* **Speaker role detection**. Identifies agent and customer roles within contact center call data.
+
+* **Language detection**. Automatically detects the language in the audio or uses specified language/locale hints.
+
+### Field extraction
+
+Field extraction allows you to extract structured data from audio files, such as summaries, sentiments, and mentioned entities from call logs. You can begin by customizing a suggested analyzer template or creating one from scratch.
+
+## Key Benefits
+Content Understanding offers advanced audio capabilities, including:
+
+* **Customizable data extraction**. Tailor the output to your specific needs by modifying the field schema, allowing for precise data generation and extraction.
+
+* **Generative models**. Utilize generative AI models to specify in natural language the content you want to extract, and the service generates the desired output.
+
+* **Integrated pre-processing**. Benefit from built-in preprocessing steps like transcription, diarization, and role detection, providing rich context for generative models.
+
+* **Scenario adaptability**. Adapt the service to your requirements by generating custom fields and extract relevant data.
+
+## Content Understanding audio analyzer templates
+
+Content Understanding offers customizable audio analyzer templates:
+
+* **Post-call analytics**. Analyze call recordings to generate conversation transcripts, call summaries, sentiment assessments, and more.
+
+* **Conversation summarization**. Generate transcriptions, summaries, and sentiment assessments from conversation audio recordings.
+
+Start with a template or create a custom analyzer to meet your specific business needs.
+
+## Input requirements
+For a detailed list of supported audio formats, refer to our [Service limits and codecs](../service-limits.md) page.
+
+## Supported languages and regions
+
+For a complete list of supported regions, languages, and locales, see our [Language and region support](../language-region-support.md)) page.
+
+## Data privacy and security
+
+Developers using Content Understanding should review Microsoft's policies on customer data. For more information, visit our [Data, protection, and privacy](https://www.microsoft.com/trust-center/privacy) page.
+
+## Next steps
+
+* Try processing your audio content using Content Understanding in [Azure AI Foundry](https://ai.azure.com/).
+* Learn more about audio [**analyzer templates**](../quickstart/use-ai-foundry.md).

articles/ai-services/content-understanding/document/overview.md

@@ -0,0 +1,74 @@
+---
+title: Azure AI Content Understanding document overview
+titleSuffix: Azure AI services
+description: Learn about Azure AI Content Understanding document solutions.
+author: laujan
+ms.author: lajanuar
+manager: nitinme
+ms.service: azure
+ms.topic: overview
+ms.date: 11/19/2024
+ms.custom: ignite-2024-understanding-release
+---
+
+# Content Understanding document solutions (preview)
+
+> [!IMPORTANT]
+>
+> * Azure AI Content Understanding is available in preview. Public preview releases provide early access to features that are in active development.
+> * Features, approaches, and processes may change or have constrained capabilities, prior to General Availability (GA).
+> * For more information, *see* [**Supplemental Terms of Use for Microsoft Azure Previews**](https://azure.microsoft.com/support/legal/preview-supplemental-terms).
+
+Content Understanding is a cloud-based [Azure AI Service](../../what-are-ai-services.md) designed to efficiently extract content and structured fields from documents and forms. It provides a comprehensive suite of APIs and an intuitive UX experience for optimal efficiency.
+
+Content Understanding enables organization to streamline data collection and processing, enhance operational efficiency, optimize data-driven decision making, and empower innovation. With customizable analyzers, Content Understanding allows for easy extraction of content or fields from documents and forms, tailored to specific business needs.
+
+## Business use cases
+
+Document analyzers can process complex documents in various formats and templates:
+
+* **Contract lifecycle management**: Extract key fields, clauses, and obligations from various contract types.
+* **Loan and mortgage applications**: Automate processing to enable quicker handling by banks, lenders, and government entities.
+* **Financial services**: Analyze complex documents like financial reports and asset management reports.
+* **Expense management**: Parse receipts and invoices from various retailers to validate expenses across different formats and templates.
+
+
+## Document analyzer capabilities
+
+:::image type="content" source="../media/document/extraction-overview.png" alt-text="Screenshot of document extraction flow.":::
+
+Content extraction enables the extraction of both printed and handwritten text from forms and documents, delivering business-ready content that is immediately actionable, usable, or adaptable for further development within your organization.
+
+### Add-on capabilities
+
+Enhance your document extraction with optional add-on features, which can incur added costs. These features can be enabled or disabled based on your needs. Currently supported add-ons include:
+
+* **Layout**: Extracts layout information such as paragraphs, sections, tables, and more.
+* **Barcode**: Identifies and decodes all barcodes in the documents.
+* **Formula**: Recognizes all identified mathematical equations from the documents.
+
+
+### Field extraction
+
+Field extraction enables the extraction of structured data from various forms and documents tailored to your specific needs. For instance, you can extract customer names, billing addresses, and line items from invoices; or parties, renewal date, and payment clause from contracts. You can start field extraction right after defining the schema or enhance it by labeling more sample documents to improve extraction quality.
+
+## Key Benefits
+
+* **Accuracy and reliability:** Ensure precise data extraction, reducing errors and boosting efficiency.
+* **Scalability:** Seamlessly scale out document processing to meet business demands.
+* **Customizable:** Adapt document analyzer to fit specific workflows.
+* **Grounding source:** Localize extracted data for human review workflows.
+* **Confidence scores:** Enhance automation with estimated confidence scores to maximize efficiency and minimize costs.
+
+## Input requirements
+For detailed information on supported input document formats, refer to our [Service quotas and limits](../service-limits.md) page.
+
+## Supported languages and regions
+For a detailed list of supported languages and regions, visit our [Language and region support](../language-region-support.md) page.
+
+## Data privacy and security
+Developers using Content Understanding should review Microsoft's policies on customer data. For more information, visit our [Data, protection, and privacy](https://www.microsoft.com/trust-center/privacy) page.
+
+## Next step
+* Try processing your document content using Content Understanding in [Azure AI Foundry](https://ai.azure.com/).
+* Learn more about document [**analyzer templates**](../quickstart/use-ai-foundry.md).

articles/ai-services/content-understanding/faq.yml

@@ -0,0 +1,69 @@
+### YamlMime:FAQ
+metadata:
+  title: Azure AI Content Understanding FAQ
+  description: Get answers to frequently asked questions about the Document Intelligence service.
+  author: laujan
+  manager: nitinme
+  ms.service: azure
+  ms.topic: faq
+  ms.date: 11/19/2024
+  ms.author: lajanuar
+title: Frequently asked questions
+summary: |
+
+ Find answers to commonly asked questions about Azure AI Content Understanding
+sections:
+  - name: Overview
+    questions:
+      - question: |
+         What is Content Understanding
+        answer: |
+          Content Understanding is a new Azure AI Service designed to generate structured insights from unstructured content using artificial intelligence. It provides consistent experience to extract content or a structured schema from audio, video, images, documents, or text inputs.
+      - question: |
+         How does Content Understanding work?
+        answer: |
+          Content Understanding utilizes Generative AI models to analyze and interpret various forms of unstructured content. It integrates data from different modalities (for example, text, images, audio) to generate a cohesive and structured output. The service uses machine learning models trained on diverse datasets and generative AI models to ensure high accuracy and relevance in the insights provided.
+      - question: |
+         What types of unstructured content can Content Understanding process?
+        answer: |
+          Content Understanding can process a wide range of unstructured content, including but not limited to:
+          * Audio recordings
+          * Video content
+          * Documents
+          * Text content
+          * Images
+      - question: |
+         What are the key benefits of using Content Understanding?
+        answer: |
+          The key benefits of using Content Understanding include:
+          * Confidence scores: Ensure the accuracy of extracted values while minimizing the cost of human review.
+          * Defined schema: Define a schema to ensure the extracted values align with intended use.
+          * Quality improvements over time: The service provides capabilities to improve the quality of the schema extracted.
+          * Improved decision-making: Structured insights help organizations make informed decisions quickly and effectively.
+          * Increased efficiency: Automating the analysis of unstructured content saves time and reduces the manual effort required.
+          * Scalability: The service can handle large volumes of data, making it suitable for organizations of all sizes.
+      - question: |
+         How can businesses use Content Understanding?
+        answer: |
+          Businesses can use Content Understanding in various ways, such as:
+          * Automation: Automate processing of content to extract a defined schema. Call center, documents, and other similar scenarios.
+          * Content cataloging: managing a large corpus of digital assets.
+          * Customer sentiment analysis: Understanding customer feedback from reviews, social media, and support interactions.
+          * Market research: Analyzing trends and patterns from diverse data sources to inform business strategies.
+          * Operational insights: Gain insights from internal documents, emails, and other unstructured data to improve operations.
+      - question: |
+         Is Content Understanding easy to integrate with existing systems?
+        answer: |
+          Yes, Content Understanding easily integrates with existing systems and workflows. The service offers a set of easy-to-use APIs that can be integrated into any application.
+      - question: |
+         What security measures are in place to protect data processed by Content Understanding?
+        answer: |
+         Azure AI Services, including Content Understanding, adheres to strict security and compliance standards to ensure data protection. These measures include data encryption, secure access controls, and compliance with industry regulations such as GDPR and HIPAA. The service also adheres to Microsoft’s responsible use of AI.
+      - question: |
+         What base models does Azure AI Content Understanding use?
+        answer: |
+          Content Understanding uses various models and capabilities from Azure OpenAI, Azure AI Speech, Vision, and Language to support single- modality and multi-modal scenarios. The service determines the selection of base models appropriate for each scenario.
+      - question: |
+         What are the pricing tier options for Content Understanding
+        answer: |
+          Content Understanding only supports Standard S0 pricing tier. See more details on the pricing page.

articles/ai-services/content-understanding/glossary.md

@@ -0,0 +1,28 @@
+---
+title: Content Understanding Glossary
+titleSuffix: Azure AI services
+description: Quick reference, detailed description on Content Understanding Terms and Definition
+author: laujan
+manager: nitinme
+ms.service: azure
+ms.topic: conceptual
+ms.date: 11/19/2024
+ms.author: lajanuar
+---
+
+# Content understanding terminologies
+
+| Term | Description |
+|:---------|:----------|
+| **File** | Any type of data, including text, documents, images, videos, and audio. |
+| **File type** | The MIME type of a file, such as text/plain, application/pdf, image/jpeg, audio/wav, and video/mp4. Generic categories like *document* refer to all corresponding MIME types supported by the service. |
+| **Analyzer** | A component that processes and extracts content and structured fields from files. Content Understanding offers a few analyzer templates for common scenarios. |
+| **Analyzer template** | A predefined configuration and field schema for an analyzer. It simplifies creating analyzers by allowing modifications to a template instead of starting from scratch. This feature is available only in AI Foundry, not via REST API/SDKs. |
+| **Analyzer result** | The output generated by an analyzer after processing input data. It typically includes extracted content in Markdown, extracted fields, and optional modality-specific details. |
+| **Add-ons** | Added features that enhance content extraction results, such as layout elements, barcodes, and figures in documents. |
+| **Fields** | List of structured key-value pairs derived from the content, as defined by the field schema. [Learn more about supported field value types.](service-limits.md) |
+| **Field schema** | A formal description of the fields to extract from the input. It specifies the name, description, value type, generation method, and more for each field. |
+| **Generation method** | The process of determining the extracted value of a specified field. Content Understanding supports: <br/> &bullet; **Extract**: Directly extract values from the input content, such as dates from receipts or item details from invoices. <br/> &bullet; **Classify**: Classify content into predefined categories, such as call sentiment or chart type. <br/> &bullet; **Generate**: Generate values from input data, such as summarizing an audio conversation or generating scene descriptions from videos. |
+| **Span** | A reference indicating the location of an element (for example, field, word) within the extracted Markdown content. A character offset and length represent a span. Different programming languages use various character encodings, which can affect the exact offset and length values for Unicode text. To avoid confusion, spans are only returned if the desired encoding is explicitly specified in the request. Some elements can map to multiple spans if they aren't contiguous in the markdown (for example, page). |
+| **Grounding source** | The specific regions in content where a value was generated. It has different representations depending on the file type: <br>&bullet; **Image** - A polygon in the image, often an axis-aligned rectangle (bounding box). <br>&bullet; **PDF/TIFF** - A polygon on a specific page, often a quadrilateral. <br>&bullet; **Audio** - A start and end time range. <br>&bullet; **Video** - A start and end time range with an optional polygon in each frame, often a bounding box.|
+| **Confidence score** | The level of certainty that the extracted data is accurate. |