add docs

Author: sr-remsha

docs/legal-and-compliance.md

@@ -0,0 +1,72 @@
+# Compliance and Legal Q&A
+
+> **This document is intended for self-hosted enterprise editions of DIAL.**
+
+## Data Usage and Processing
+
+##### How is data transmitted to the DIAL Platform utilized? Is such data incorporated into model training datasets?
+
+The DIAL Platform retains conversational data and prompts in system logs exclusively for governance and compliance purposes. When interfacing with third-party cloud-based Large Language Models (including but not limited to Azure OpenAI, Google Vertex, AWS Bedrock), the platform utilizes your organization's designated cloud AI subscriptions. The processing, retention, and utilization of such data by these third-party providers is governed by the applicable service agreements executed between your organization and the respective cloud service provider.
+
+##### Where are conversation logs maintained and what security protocols are implemented?
+
+The DIAL Platform stores all log data within object storage solutions (Azure Blob Storage, Google Cloud Storage, Amazon S3) provisioned within your organization's cloud infrastructure. Access controls and security configurations for such storage repositories remain under the exclusive administration of your organization's IT department.
+
+## Technical Limitations and Safety
+
+##### What are the known technical limitations of the DIAL Platform, such as potential for hallucinations?
+
+The DIAL Platform functions solely as an orchestration layer for Large Language Models and does not incorporate generative AI logic internally. Any limitations such as hallucinations or factual inaccuracies are inherent to the specific LLMs or applications being orchestrated rather than the platform itself.
+
+##### What safety testing protocols have been implemented for the DIAL Platform regarding potentially harmful or malicious queries?
+
+As an orchestration layer, the DIAL Platform does not independently evaluate content or implement content-based restrictions. Responsible AI metrics, content filtering, and safety measures are functionalities of the specific LLMs or applications being orchestrated, not the platform itself.
+
+## Regulatory Compliance
+
+##### How does the DIAL Platform ensure compliance with data privacy regulations (e.g., GDPR) regarding the collection, processing, and storage of personal data?
+
+While specific compliance implementation remains the responsibility of the deploying organization, the DIAL Platform implements several compliance-oriented features:
+
+1.	Configurable option to disable collection of conversation logs
+2.	Configurable data retention policies on cloud data storage level that can automatically purge conversation logs and sensitive files after a predetermined interval (e.g., 30 days)
+3.	An Interceptor framework that examines communications bidirectionally (both user prompts and LLM responses)
+4.	Capability to implement custom validation logic for detecting and handling sensitive information or personally identifiable information (PII)
+5.	Options to modify or block requests containing prohibited content
+
+These capabilities help to minimize and control lifetime of PII within DIAL, helping to build it into organization’s ecosystem, compliant with data privacy regulations.
+
+##### How will the DIAL Platform comply with the EU AI Act concerning general-purpose AI systems?
+
+While specific compliance implementation remains the responsibility of the deploying organization, the DIAL Platform provides technical capabilities that facilitate compliance with the EU AI Act, including:
+1.	Comprehensive conversation logging for audit and investigative purposes
+2.	Transparent AI-generated content labeling mechanisms
+3.	User feedback collection functionality for quality assessment and improvement
+Collected conversations and feedback help deploying organization to perform analysis and make corrections to how DIAL is used and add necessary guardrails to ensure compliance.
+
+## Security and Data Protection
+
+##### What security measures protect sensitive or confidential information processed through the DIAL Platform?
+
+The DIAL Platform incorporates a multi-layered security architecture including:
+1.	In-transit and at-rest protection of sensitive information, using TLS protocol for all exposed connections and built-in encryption mechanisms of cloud storage services.
+2.	Attack surface minimization through reverse proxy implementation and API management
+3.	Network isolation of cloud resources via firewalls and private connectivity
+4.	Mandatory token-based authentication utilizing JWT standards
+5.	Integration with OpenID Connect-compliant Identity Providers
+6.	Centralized role-based access control (RBAC) with granular permission management
+7.	Group-based access controls for precise resource authorization
+
+## Intellectual Property and Liability
+
+##### Who retains intellectual property rights to content generated via the DIAL Platform, and how are copyright matters addressed?
+
+The DIAL Platform functions exclusively as an intermediary orchestration layer and does not independently generate or modify AI content. Consequently, intellectual property rights and copyright ownership for any generated content are governed by the terms and conditions established in the service agreements between your organization and the respective LLM provider (Azure OpenAI, Google Vertex, AWS Bedrock, etc.).
+
+##### How does the system address concerns regarding potential bias in AI-generated content, particularly when processing sensitive or regulated data? Who bears liability for inaccurate or inappropriate AI-generated content?
+
+The DIAL Platform operates solely as an orchestration layer for third-party LLM models and does not independently generate content. Liability considerations regarding content accuracy, appropriateness, and potential bias are determined by the contractual arrangements between your organization and the respective LLM service provider as specified in the applicable service agreements.
+
+##### What degree of human oversight is required for monitoring AI-generated content, and what policies govern the review or override of DIAL Platform outputs?
+
+The requisite level of human oversight is contingent upon the criticality, sensitivity, and regulatory context of the content being generated. Policies governing content review, approval workflows, and override mechanisms are typically established by the implementing organization. Clients are advised to develop and implement comprehensive governance frameworks aligned with their specific operational requirements, risk tolerance, and applicable regulatory obligations.

docs/platform/0.architecture-and-concepts/0.vision.md

@@ -1,18 +1,39 @@
-# Our Vision
+# What is DIAL
 
-DIAL is a secure, enterprise-grade and open-source platform. It has an API-first, cloud and model-agnostic design that makes it suitable for a wide variety of use cases.
-Our primary focus is to avoid reliance on particular cloud or LLM vendors, support scalability and security, avoid increasing tech complexity or licensing risks.
-We prioritize developing use case-agnostic generic features that facilitate developing of GenAI applications.
+DIAL is an acronym for Deterministic Integrator of Applications and Language Models. It is a secure, scalable and customizable enterprise-grade AI platform designed to streamline Software Development Lifecycle (SDLC) for GenAI applications.
 
-> To contribute to AI DIAL development, refer to [CONTRIBUTING](https://github.com/epam/ai-dial/blob/main/CONTRIBUTING.md) instruction on GitHub.
+Watch [introduction videos](/docs/video%20demos/0.dial-product-overview.md) to learn more.
 
 ## Main Principles
 
-* **Open-Source**: AI DIAL is an open-source solution available under the permissive and free Apache License, Version 2.0.
+* **Open-Source**: DIAL is an open-source solution available under the permissive and free [Apache License, Version 2.0](https://github.com/epam/ai-dial/blob/main/LICENSE).
 * **Low TCO**: We aim to minimize technical complexity and licensing risks by utilizing community hardware, affordable storage solutions, and technologies.
-* **Vendor-Agnostic**: AI DIAL can operate on any cloud platform and on-prem, is model and domain agnostic enabling the highest level of flexibility.
-* **API-First**: All AI DIAL features are available via API, which empowers the development of custom extensions.
-* **Scalable**: AI DIAL does not require a centralized DB and relies on Redis and cloud storages making it an ideal platform for building web-scale businesses.
-* **Modular**: Start with a basic setup and scale up to a full ecosystem as needed. Go rapidly from prototyping to fully operational production applications.
+* **Vendor-Agnostic**: DIAL can operate on any cloud platform and on-prem, is model and domain agnostic enabling the highest level of flexibility.
+* **API-First**: All DIAL features are available via [DIAL API](https://dialx.ai/dial_api), which empowers the development of custom extensions.
+* **Scalable**: DIAL does not require a centralized DB and relies on Redis and cloud storages/file system making it an ideal platform for building web-scale businesses.
+* **Modular**: Start with a basic setup and scale up to a full ecosystem as needed. Go rapidly from prototyping to fully operational production applications. Refer to [Architecture](/docs/platform/0.architecture-and-concepts/2.architecture.md) for more details.
 * **Small Tech Footprint**: The system requires minimum resources to operate and can run equally smoothly on clouds and personal laptops.
-* **Secure**: Sensitive information is stored according to the industry best practices. We support integration with your SSO vendors using OpenID and SAML protocols.
+* **Secure**: Sensitive information is stored according to the industry best practices. Scans for dependencies, vulnerabilities, and licenses are required steps in the delivery process.
+
+## Our Vision
+
+DIAL is designed to be cloud-agnostic, providing horizontal features that simplify interactions with various models and LLM-centric applications, as well as the development of LLM applications. We strive to avoid dependencies on specific cloud or LLM vendors, compromises on scalability or security, excessive expansion of our tech footprint, or potential licensing risks. We typically design features as APIs, which custom extensions can leverage.
+
+The solution is domain-agnostic. We prioritize use case-agnostic generic features (such as rendering charts in the frontend based on model responses) over features that are overly specific to a single domain (like supporting medical anthologies). However, specific implementations can be built as extensions or derived works. For more information, refer to our [extension framework](https://dialx.ai/extension-framework) and [API reference](https://dialx.ai/dial_api).
+
+DIAL is a product that is designed to **enhance** or **extend**, rather than replace or compete with:
+ 
+- Model providers such as [Azure](https://azure.microsoft.com/), [AWS](https://aws.amazon.com/), and [GCP](https://cloud.google.com/)
+- Model vendors including [OpenAI](https://openai.com/), [Claude](https://www.anthropic.com/claude), [Google DeepMind](https://deepmind.google/), [Meta](https://ai.meta.com/), and [Mistral](https://mistral.ai/)
+- AI studios like [Azure AI Studio](https://azure.microsoft.com/en-us/products/ai-studio), [Google AI Studio](https://ai.google.dev/aistudio), and [Meta AI Studio](https://ai.meta.com/ai-studio/)
+- Data platforms such as [Databricks](https://www.databricks.com/) and [Dataiku](https://www.dataiku.com/)
+- Application development frameworks including [LangChain](https://www.langchain.com/), [LlamaIndex](https://www.llamaindex.ai/) and [SemanticKernel](https://github.com/microsoft/semantic-kernel)
+- Low-code application designers like [Flowise](https://flowiseai.com/)
+ 
+We aim to complement these services, streamlining the development, deployment, and operation of GenAI applications, as well as facilitating the [interoperability](/docs/platform/0.architecture-and-concepts/2.architecture.md) of various applications to achieve specific business objectives. For this purpose we provide:
+ 
+- [Unified API](https://dialx.ai/dial_api) to access multimodal models, assistants, applications, and embedding models.
+- Essential services such as resilient file/chat/prompt/model storage with user-based [access control](/docs/platform/3.core/2.access-control-intro.md).
+- Important operational features, including [sharing](/docs/platform/7.collaboration-intro.md), organization-wide [publications](/docs/platform/7.collaboration-intro.md#publication), review processes, and flexible [rate limits](/docs/platform/3.core/2.access-control-intro.md).
+> To contribute to DIAL development, refer to [CONTRIBUTING](https://github.com/epam/ai-dial/blob/main/CONTRIBUTING.md) instruction on GitHub.
+

docs/platform/0.architecture-and-concepts/1.concepts.md

@@ -1,29 +1,36 @@
 # Main Concepts and Definitions
 
-## Resources
+## Unified API
 
-In AI DIAL, we call *resources* applications, conversations, prompts and files. When added to the system, they are stored in a dedicated folder within a blob store account bucket assigned to your user account. You can handle resources according to the similar pattern (CRUD operations, publications, sharing and other).
+[DIAL Unified API](https://dialx.ai/dial_api), based on OpenAI API, gives access to all language and embedding models and applications. It helps create a unification layer where all models and applications are interchangeable, delivering a cohesive conversational experience and future-proof development of GenAI applications.
 
-> * In [DIAL API](https://dialx.ai/dial_api) you can find dedicated sections for working with resources: [applications](https://dialx.ai/dial_api#tag/Applications), [conversations](https://dialx.ai/dial_api#tag/Conversations), [prompts](https://dialx.ai/dial_api#tag/Prompts) and [files](https://dialx.ai/dial_api#tag/Files).
-> * Refer to [Collaboration](/docs/platform/7.collaboration-intro.md) to learn about collaboration features such as sharing and publication that can be applied to resources.
-> * Refer to [Applications](/docs/tutorials/0.user-guide.md#applications) to learn how to develop, add and manage apps in DIAL.
+Compatibility with OpenAI, makes it simple to add new adapters for language models or develop them with DIAL SDK.
+
+## Application Server
 
-## Applications
+DIAL acts as application server offering tools  to develop, deploy, host and manage different types of GenAI applications while handling scalability, availability, security and resource optimization tasks.
 
-In AI DIAL, an “application” is any programming logic that conforms to the Unified API of AI DIAL or registered custom endpoints and packaged as a ready-to-deploy solution.
-AI DIAL Chat users can engage with applications via standard or fully customized interfaces. Applications can interact via the Unified API or registered custom endpoints, enabling creation of complex scenarios and utilizing all AI DIAL features.
-You can expand the platform by introducing custom application types to build the applications you need, using the AI DIAL SDK to streamline the development process.
+## Agents
 
-> Refer to [Applications](/docs/tutorials/0.user-guide.md#applications) to learn how to develop, add and manage apps in DIAL.
+Agents in DIAL, are all [available language models](/docs/platform/2.supported-models.md) and [DIAL-native applications](#dial-native-applications). DIAL can serve as an agentic platform, where you can use any agent as building block to create powerful multi-agent and multi-modal applications.
 
-## Application Server
+DIAL includes powerful pre-built agents such as [DIAL ChatHub](/docs/video%20demos/2.Applications/3.dial-chathub.md), [DIAL RAG](/docs/video%20demos/2.Applications/1.dial-rag.md), [DIAL Web RAG](/docs/video%20demos/2.Applications/2.dial-web-rag.md) which you can reuse in your agents as building blocks.
+
+You can create agents using [DIAL SDK](https://github.com/epam/ai-dial-sdk/blob/development/README.md) and [DIAL API](https://dialx.ai/dial_api), or no-code and low-code [application wizards](/docs/tutorials/0.user-guide.md#application-builder).
+
+When ready, you can list your agents in [DIAL Marketplace](/docs/platform/4.chat/1.marketplace.md) and enable others to use them or even contribute to the development.
 
-AI DIAL acts as application server offering tools for users with different tech skillsets to develop, deploy, and manage different types of GenAI applications while handling scalability, availability, security and resource optimization tasks.
+## DIAL-Native Applications
 
-##### Main Principles
+DIAL-native apps have API that adheres to [Unified API of DIAL](https://dialx.ai/dial_api). You can use your apps as agents in DIAL seamlessly if they meet this requirement.
 
-* **Development Acceleration**: Built-in Unified API and DIAL SDK streamline the development of Gen AI apps.
-* **Integration Enablement**: Enable seamless integration with existing enterprise systems and data sources.  
-* **Management Simplification**: Manage all your apps via the DIAL API or the DIAL Control Panel.
-* **Performance Optimization**: Optimize resources, manage availability, scalability, and simplify the deployment of your apps.
-* **Create Business-Driven Apps**: AI DIAL can be extended beyond its standard capabilities to meet specific business requirements. Add new application types to build fully custom business-driven applications.
+Yo can create DIAL-native apps using [DIAL SDK](https://github.com/epam/ai-dial-sdk/blob/development/README.md) and also no-code and low-code [application wizards](/docs/tutorials/0.user-guide.md#application-builder)
+
+
+## Resources
+
+In DIAL, resources refer to applications, conversations, prompts, and files. When added to the system, these resources are stored in a dedicated folder within a BLOB storage account bucket assigned to your user account. Resources can be managed using a consistent pattern, including CRUD operations (Create, Read, Update, Delete), publishing, sharing, and other actions.
+
+> * In [DIAL API](https://dialx.ai/dial_api) you can find dedicated sections for working with resources: [applications](https://dialx.ai/dial_api#tag/Applications), [conversations](https://dialx.ai/dial_api#tag/Conversations), [prompts](https://dialx.ai/dial_api#tag/Prompts) and [files](https://dialx.ai/dial_api#tag/Files).
+> * Refer to [Collaboration](/docs/platform/7.collaboration-intro.md) to learn about collaboration features such as sharing and publication that can be applied to resources.
+> * Refer to [Applications](/docs/tutorials/0.user-guide.md#applications) to learn how to develop, add and manage apps in DIAL.