Skip to content

on_boarding.md

Latest commit

 

History

History
130 lines (84 loc) · 11.4 KB

on_boarding.md

File metadata and controls

130 lines (84 loc) · 11.4 KB
Raw
graph LR
    Data_Ingestion_Validation["Data Ingestion & Validation"]
    Data_Preprocessing["Data Preprocessing"]
    Time_Series_Transformation["Time Series Transformation"]
    Feature_Engineering["Feature Engineering"]
    Anomaly_Detection_Algorithms["Anomaly Detection Algorithms"]
    Post_processing_Calibration["Post-processing & Calibration"]
    Pipeline_Orchestration["Pipeline Orchestration"]
    AutoML_Engine["AutoML Engine"]
    Data_Ingestion_Validation -- "provides raw data to" --> Data_Preprocessing
    Data_Preprocessing -- "provides clean, structured data to" --> Time_Series_Transformation
    Data_Preprocessing -- "provides clean, structured data to" --> Feature_Engineering
    Time_Series_Transformation -- "enhances data for" --> Feature_Engineering
    Feature_Engineering -- "supplies processed features to" --> Anomaly_Detection_Algorithms
    Anomaly_Detection_Algorithms -- "outputs raw detections to" --> Post_processing_Calibration
    Pipeline_Orchestration -- "orchestrates" --> Data_Preprocessing
    Pipeline_Orchestration -- "orchestrates" --> Time_Series_Transformation
    Pipeline_Orchestration -- "orchestrates" --> Feature_Engineering
    Pipeline_Orchestration -- "orchestrates" --> Anomaly_Detection_Algorithms
    Pipeline_Orchestration -- "orchestrates" --> Post_processing_Calibration
    AutoML_Engine -- "optimizes" --> Pipeline_Orchestration
    click Data_Preprocessing href "https://github.com/CodeBoarding/GeneratedOnBoardings/blob/main/tods/Data_Preprocessing.md" "Details"
    click Time_Series_Transformation href "https://github.com/CodeBoarding/GeneratedOnBoardings/blob/main/tods/Time_Series_Transformation.md" "Details"
    click Feature_Engineering href "https://github.com/CodeBoarding/GeneratedOnBoardings/blob/main/tods/Feature_Engineering.md" "Details"
    click Anomaly_Detection_Algorithms href "https://github.com/CodeBoarding/GeneratedOnBoardings/blob/main/tods/Anomaly_Detection_Algorithms.md" "Details"
    click Post_processing_Calibration href "https://github.com/CodeBoarding/GeneratedOnBoardings/blob/main/tods/Post_processing_Calibration.md" "Details"
    click Pipeline_Orchestration href "https://github.com/CodeBoarding/GeneratedOnBoardings/blob/main/tods/Pipeline_Orchestration.md" "Details"
    click AutoML_Engine href "https://github.com/CodeBoarding/GeneratedOnBoardings/blob/main/tods/AutoML_Engine.md" "Details"
Loading

CodeBoardingDemoContact

Details

The TODS (Time-series Outlier Detection System) project is structured around a modular pipeline designed for comprehensive anomaly detection in time-series data. The process begins with Data Ingestion & Validation, ensuring raw datasets are properly loaded and validated. This is followed by Data Preprocessing, which cleans and structures the data, and Time Series Transformation, which applies specialized techniques to enhance time-series characteristics. Feature Engineering then extracts meaningful features, which are fed into Anomaly Detection Algorithms for outlier identification. The raw detections are refined by Post-processing & Calibration. The entire workflow is managed by Pipeline Orchestration, which dynamically builds, fits, and evaluates pipelines. An AutoML Engine further optimizes these pipelines by searching for optimal configurations and hyperparameters.

Data Ingestion & Validation

Responsible for acquiring raw datasets, including downloading from external sources, loading local files, and performing initial validation to ensure data integrity and readiness for processing.

Related Classes/Methods:

Data Preprocessing [Expand]

Transforms raw or semi-processed data into a clean, structured format suitable for subsequent feature engineering and model training. This involves handling missing values, parsing data types, and basic data transformations.

Related Classes/Methods:

Time Series Transformation [Expand]

Applies specialized transformations to time-series data to enhance its characteristics for anomaly detection. This includes techniques like smoothing, scaling, and segmentation to reveal underlying patterns.

Related Classes/Methods:

Feature Engineering [Expand]

Extracts meaningful and discriminative features from the processed time-series data. These features are crucial for anomaly detection algorithms to learn and identify anomalous patterns effectively.

Related Classes/Methods:

Anomaly Detection Algorithms [Expand]

Implements a diverse set of anomaly detection models, ranging from traditional statistical methods to advanced deep learning and ensemble techniques. This component is responsible for identifying outliers based on the engineered features.

Related Classes/Methods:

Post-processing & Calibration [Expand]

Refines the raw outputs from anomaly detection algorithms. This component applies rule-based filtering or other calibration mechanisms to improve the final anomaly scores or labels, enhancing the practical utility of the detections.

Related Classes/Methods:

Pipeline Orchestration [Expand]

Manages the end-to-end execution of anomaly detection pipelines. This includes dynamically building, fitting, and evaluating sequences of data processing, feature engineering, and detection primitives based on user-defined configurations.

Related Classes/Methods:

AutoML Engine [Expand]

Automates the search for optimal pipeline configurations and hyperparameters. It leverages various search strategies (e.g., brute-force, Nevergrad) to find the best-performing anomaly detection solution for a given dataset, optimizing for specified metrics.

Related Classes/Methods:

FAQ