Upd/docs (#249)

* upd main page

* upd quickstart

* fix quickstart doctests

* upd concepts

* add page about text embeddings

* add page on automl theory

* update dialogue systems page

Author: voorhs

docs/source/concepts.rst

@@ -1,30 +1,123 @@
+============
 Key Concepts
 ============
 
-.. _key-search-space:
+This page introduces the fundamental concepts that underpin AutoIntent's design and functionality. Understanding these concepts will help you effectively use the framework and make informed decisions about your text classification projects.
+
+.. _concepts-pipeline:
+
+Three-Stage Pipeline Architecture
+=================================
+
+AutoIntent organizes text classification into a modular three-stage pipeline, providing clear separation of concerns and flexibility in optimization:
+
+**🔤 Embedding Stage**
+   Transforms raw text into dense vector representations using pre-trained transformer models. This stage handles the computationally intensive text encoding and can be optimized independently from downstream classification tasks.
+
+**📊 Scoring Stage**
+   Processes embeddings to predict class probabilities. This stage supports diverse approaches from classical machine learning (KNN, logistic regression) to deep learning models (BERT fine-tuning, CNNs). All models operate on pre-computed embeddings for efficiency.
+
+**⚖️ Decision Stage**
+   Converts predicted probabilities into final classifications by applying thresholds and decision rules. This stage is crucial for multi-label classification and out-of-scope detection scenarios.
+
+This modular design enables efficient experimentation, allows reusing expensive embedding computations across different models, and supports deployment on CPU-only systems.
+
+.. _concepts-automl:
+
+AutoML Optimization Strategy
+============================
+
+AutoIntent employs a hierarchical optimization approach that balances exploration with computational efficiency:
+
+**🔧 Module-Level Optimization**
+   Components are optimized sequentially: embedding → scoring → decision. Each stage builds upon the best model from the previous stage, creating a cohesive pipeline while preventing combinatorial explosion.
+
+**🤖 Model-Level Optimization**
+   Within each module, both model architectures and hyperparameters are jointly optimized using Optuna's Tree-structured Parzen Estimators and random sampling.
+
+**🗺️ Search Space Configuration**
+   Optimization behavior is controlled through dictionary-like search spaces that define:
+   
+   - Available model types and their hyperparameter ranges
+   - Optimization budget and resource constraints  
+   - Cross-validation and evaluation strategies
+
+.. _concepts-embedding-centric:
+
+Embedding-Centric Design
+========================
+
+AutoIntent's architecture centers around transformer-based text embeddings, providing several key advantages:
+
+**⚡ Pre-computed Embeddings**
+   Text is encoded once and reused across all scoring models, dramatically reducing computational overhead during hyperparameter optimization and enabling efficient experimentation.
+
+**🤗 Model Repository Integration**
+   Seamless access to thousands of pre-trained models from Hugging Face Hub, with intelligent selection strategies based on retrieval metrics or downstream task performance.
+
+**🚀 Deployment Flexibility**
+   Separation of embedding generation from classification enables deploying lightweight classifiers on resource-constrained systems while leveraging powerful transformer representations.
+
+.. _concepts-multiclass-multilabel:
+
+Classification Paradigms
+========================
+
+AutoIntent supports various classification scenarios through its flexible decision module:
+
+**🏷️ Multi-Class Classification**
+   Traditional single-label classification where each input belongs to exactly one class. Uses argmax or threshold-based decisions on predicted probabilities.
+
+**🔖 Multi-Label Classification** 
+   Each input can belong to multiple classes simultaneously. Employs adaptive thresholding strategies that can be sample-specific or learned globally across the dataset.
+
+
+.. _concepts-oos:
+
+Out-of-Scope Detection
+======================
+
+A critical capability for production text classification systems, especially in conversational AI:
+
+**📏 Confidence Thresholding**
+   Uses predicted probability scores to identify inputs that don't belong to any known class. Threshold values can be tuned automatically to balance precision and recall.
+
+**🔗 Integration with Multi-Label**
+   OOS detection works seamlessly with multi-label scenarios, enabling detection of completely unknown inputs vs. partial matches to known classes.
+
+.. _concepts-presets:
+
+Optimization Presets
+====================
+
+AutoIntent provides predefined optimization strategies that balance quality, speed, and resource consumption:
 
-Optimization Search Space
--------------------------
+**⚡ Zero-Shot Presets**
+   Leverage class descriptions and large language models for classification without training data. Ideal for rapid prototyping and cold-start scenarios.
 
-The automatic selection of a classifier occurs through the iteration of hyperparameters within a certain *search space*. Conceptually, this search space is a dictionary where the keys are the names of the hyperparameters, and the values are lists. The hyperparameters act as the coordinate "axes" of the search space, and the values in the lists act as points on this axis.
+**📈 Classic Presets**
+   Focus on traditional ML approaches (KNN, linear models, tree-based methods) operating on transformer embeddings. Offer excellent balance of performance and efficiency.
 
-.. _key-stages:
+**🧠 Neural Network Presets**
+   Include deep learning approaches like CNN, RNN, and transformer fine-tuning. Provide highest potential performance at increased computational cost.
 
-Classification Stages
----------------------
+**🪜 Computational Tiers**
+   Each preset family offers light, medium, and heavy variants that trade optimization time for potential performance improvements.
 
-Intent classification can be divided into two stages: scoring and decision. Scoring involves predicting the probabilities of the presence of each intent in a given utterance. Prediction involves forming the final decision based on the provided probabilities.
+.. _concepts-modularity:
 
-.. _key-oos:
+Modular Architecture
+====================
 
-Out-of-domain utterances
-------------------------
+AutoIntent's design emphasizes modularity and extensibility:
 
-If we want to detect out-of-domain examples, it is necessary to set a probability threshold during the decision stage, at which the presence of some known intent can be asserted.
+**🧩 Plugin Architecture**
+   Each component (embedding models, scoring methods, decision strategies) implements a common interface, enabling easy addition of new approaches without modifying core framework code.
 
-.. _key-nodes-modules:
+**⚙️ Configuration-Driven**
+   All aspects of optimization can be controlled through declarative configuration files, supporting reproducible experiments and easy sharing of optimization strategies.
 
-Nodes and Modules
------------------
+**🔧 Extensibility**
+   Framework can be extended with custom embedding models, scoring algorithms, and decision strategies while maintaining compatibility with the AutoML optimization pipeline.
 
-The scoring or decision model, along with its hyperparameters that need to be iterated, is called an *optimization module*. A set of modules related to one optimization stage (scoring or decision) is called an *optimization node*.
+This modular design ensures that AutoIntent can evolve with advances in NLP research while maintaining stability and backward compatibility for existing users.

docs/source/conf.py

@@ -50,6 +50,7 @@
     "sphinx.ext.intersphinx",
     "sphinx_multiversion",
     "sphinx.ext.napoleon",
+    "sphinx_toolbox.collapse"
 ]
 
 templates_path = ["_templates"]

docs/source/index.rst

@@ -1,13 +1,15 @@
 AutoIntent documentation
 ========================
 
-**AutoIntent** is an open source tool for automatic configuration of a text classification pipeline for intent prediction.
+**AutoIntent** is an open source tool for automatic configuration of text classification pipelines, with specialized support for intent prediction.
 
 .. note::
 
    This project is under active development.
 
-The task of intent detection is one of the main subtasks in creating task-oriented dialogue systems, along with scriptwriting and slot filling. AutoIntent project offers users the following:
+The task of intent detection is one of the main subtasks in creating task-oriented dialogue systems, along with scriptwriting and slot filling. While AutoIntent is particularly well-suited for intent detection, it can be applied to any text classification problem, including sentiment analysis, topic classification, document categorization, and other NLP tasks.
+
+AutoIntent project offers users the following:
 
 - A convenient library of methods for intent classification that can be used in a sklearn-like "fit-predict" format.
 - An AutoML approach to creating classifiers, where the only thing needed is to upload a set of labeled data.
@@ -36,33 +38,34 @@ Example of building an intent classifier in a couple of lines of code:
    for match in glob("vector_db*"):
       shutil.rmtree(match)
 
-Documentation Contents
-----------------------
-
-:doc:`Quickstart <quickstart>`
-..............................
-
-It is recommended to begin with the :doc:`quickstart` page. It contains overview of our capabilities and basic instructions for working with our library.
+Documentation Guide
+-------------------
 
-:doc:`Key Concepts <concepts>`
-..............................
+Getting Started
+...............
 
-Key terms and concepts we use throughout our documentation.
+:doc:`🚀 Quickstart <quickstart>`
+   Jump right in! Install AutoIntent and build your first text classifier in minutes. Perfect for users who want to get up and running quickly with practical examples.
 
-:doc:`User Guides<user_guides>`
-................................
+:doc:`📚 Key Concepts <concepts>`
+   Essential terminology and concepts used throughout AutoIntent. Understanding these will help you navigate the documentation and make the most of the library's features.
 
-A series of notebooks that demonstrate in detail and comprehensively the capabilities of our library and how to use it.
+In-Depth Learning
+.................
 
-:doc:`API Reference <autoapi/autointent/index>`
-...............................................
+:doc:`📖 User Guides <user_guides>`
+   Comprehensive tutorials and examples that walk you through AutoIntent's capabilities step-by-step. These hands-on guides cover everything from basic usage to advanced techniques.
 
-Pay special attention to the sections :doc:`autoapi/autointent/modules/index` and :doc:`autoapi/autointent/metrics/index`.
+:doc:`🎓 Learn AutoIntent <learn/index>`
+   Dive deeper into the theory behind AutoIntent. Learn about dialogue systems, AutoML principles, and the science that powers intelligent text classification.
 
-:doc:`Learn AutoIntent<learn/index>`
-....................................
+Reference
+.........
 
-Some theoretical background on dialogue systems and auto ML.
+:doc:`🔧 API Reference <autoapi/autointent/index>`
+   Complete technical documentation for all classes, methods, and functions. Essential reference for developers integrating AutoIntent into their applications.
+   
+   Key sections: :doc:`Modules <autoapi/autointent/modules/index>` | :doc:`Metrics <autoapi/autointent/metrics/index>`
 
 
 .. toctree::