NVIDIA-NeMo
diff --git a/‎docs/_extensions/myst_codeblock_substitutions.py‎
Lines changed: 1 addition & 1 deletion b/‎docs/_extensions/myst_codeblock_substitutions.py‎
Lines changed: 1 addition & 1 deletion
diff --git a/‎docs/about/concepts/deduplication.md‎
Lines changed: 171 additions & 0 deletions b/‎docs/about/concepts/deduplication.md‎
Lines changed: 171 additions & 0 deletions
diff --git a/‎docs/about/concepts/index.md‎
Lines changed: 24 additions & 8 deletions b/‎docs/about/concepts/index.md‎
Lines changed: 24 additions & 8 deletions
@@ -12,7 +12,7 @@
 ```
 
 Or in inline code:
-Use `nvcr.io/nvidia/nemo-curator:{{ current_release }}` for the Docker image.
+Use `nvcr.io/nvidia/nemo-curator:{{ container_version }}` for the Docker image.
 
 The substitutions will be replaced with their values from myst_substitutions in conf.py.
 """
 
@@ -0,0 +1,171 @@
+---
+description: "Comprehensive overview of deduplication techniques across text, image, and video modalities including exact, fuzzy, and semantic approaches"
+categories: ["concepts-architecture"]
+tags: ["deduplication", "exact-dedup", "fuzzy-dedup", "semantic-dedup", "multimodal", "gpu-accelerated"]
+personas: ["data-scientist-focused", "mle-focused"]
+difficulty: "intermediate"
+content_type: "concept"
+modality: "multimodal"
+---
+
+(about-concepts-deduplication)=
+
+# Deduplication Concepts
+
+This guide covers deduplication techniques available across all modalities in NeMo Curator, from exact hash-based matching to semantic similarity detection using embeddings.
+
+## Overview
+
+Deduplication is a critical step in data curation that removes duplicate and near-duplicate content to improve model training efficiency. NeMo Curator provides sophisticated deduplication capabilities that work across text, image, and video modalities.
+
+Removing duplicates offers several benefits:
+
+- **Improved Training Efficiency**: Prevents overrepresentation of repeated content
+- **Reduced Dataset Size**: Significantly reduces storage and processing requirements
+- **Better Model Performance**: Eliminates redundant examples that can bias training
+
+## Deduplication Approaches
+
+NeMo Curator implements three main deduplication strategies, each with different strengths and use cases:
+
+### Exact Deduplication
+
+- **Method**: Hash-based matching (MD5)
+- **Best For**: Identical copies and character-for-character matches
+- **Speed**: Very fast
+- **GPU Required**: Yes (for distributed processing)
+
+Exact deduplication identifies documents or media files that are completely identical by computing cryptographic hashes of their content.
+
+**Modalities Supported**: Text, Image, Video
+
+### Fuzzy Deduplication
+
+- **Method**: MinHash and Locality-Sensitive Hashing (LSH)
+- **Best For**: Near-duplicates with minor changes (reformatting, small edits)
+- **Speed**: Fast
+- **GPU Required**: Yes
+
+Fuzzy deduplication uses statistical fingerprinting to identify content that is nearly identical but may have small variations like formatting changes or minor edits.
+
+**Modalities Supported**: Text
+
+### Semantic Deduplication
+
+- **Method**: Embedding-based similarity using neural networks
+- **Best For**: Content with similar meaning but different expression
+- **Speed**: Moderate (requires embedding generation)
+- **GPU Required**: Yes
+
+Semantic deduplication leverages deep learning embeddings to identify content that conveys similar meaning despite using different words, visual elements, or presentation.
+
+**Modalities Supported**: Text, Image, Video
+
+## Multimodal Applications
+
+### Text Deduplication
+
+Text deduplication is the most mature implementation, offering all three approaches:
+
+- **Exact**: Remove identical documents using MD5 hashing
+- **Fuzzy**: Remove near-duplicates using MinHash and LSH similarity
+- **Semantic**: Remove semantically similar content using embeddings
+
+Text deduplication can handle web-scale datasets and is commonly used for:
+
+- Web crawl data (Common Crawl)
+- Academic papers (ArXiv)
+- Code repositories
+- General text corpora
+
+### Video Deduplication
+
+Video deduplication uses the semantic deduplication workflow with video embeddings:
+
+- **Semantic Clustering**: Uses the general K-means clustering workflow on video embeddings
+- **Pairwise Similarity**: Computes within-cluster similarity using the semantic deduplication pipeline
+- **Representative Selection**: Leverages the semantic workflow to identify and remove redundant content
+
+Video deduplication is particularly effective for:
+
+- Educational content with similar presentations
+- News clips covering the same events
+- Entertainment content with repeated segments
+
+### Image Deduplication
+
+Image deduplication capabilities focus on removing duplicate images from datasets:
+
+- **Duplicate Removal**: Filters out images identified as duplicates from previous deduplication stages
+- **Integration Support**: Works with image processing pipelines through `ImageBatch` tasks
+
+## Architecture and Performance
+
+### Distributed Processing
+
+All deduplication workflows leverage distributed computing frameworks:
+
+- **Ray Backend**: Provides scalable distributed processing
+- **GPU Acceleration**: Essential for embedding generation and similarity computation
+- **Memory Optimization**: Streaming processing for large datasets
+
+### Scalability Characteristics
+
+```{list-table} Deduplication Scalability
+:header-rows: 1
+:widths: 20 25 25 30
+
+* - Method
+  - Dataset Size
+  - Memory Requirements
+  - Processing Time
+* - Exact
+  - Unlimited
+  - Low (hash storage)
+  - Linear with data size
+* - Fuzzy
+  - Petabyte-scale
+  - Moderate (LSH tables)
+  - Sub-linear with LSH
+* - Semantic
+  - Terabyte-scale
+  - High (embeddings)
+  - Depends on model inference
+```
+
+## Implementation Patterns
+
+### Workflow-Based Processing
+
+NeMo Curator provides high-level workflows that encapsulate the complete deduplication process:
+
+```python
+# Text exact deduplication
+from nemo_curator.stages.deduplication.exact.workflow import ExactDeduplicationWorkflow
+
+# Text fuzzy deduplication  
+from nemo_curator.stages.deduplication.fuzzy.workflow import FuzzyDeduplicationWorkflow
+
+# Text semantic deduplication
+from nemo_curator.stages.deduplication.semantic.workflow import SemanticDeduplicationWorkflow
+```
+
+### Stage-Based Processing
+
+For fine-grained control, individual stages can be composed into custom pipelines:
+
+```python
+# Video semantic deduplication stages
+from nemo_curator.stages.deduplication.semantic.kmeans import KMeansStage
+from nemo_curator.stages.deduplication.semantic.pairwise import PairwiseStage
+from nemo_curator.stages.deduplication.semantic.identify_duplicates import IdentifyDuplicatesStage
+```
+
+## Integration with Pipeline Architecture
+
+Deduplication integrates seamlessly with NeMo Curator's pipeline-based architecture:
+
+1. **Input Compatibility**: Works with `DocumentBatch` tasks from any data loading stage
+2. **Output Integration**: Produces standardized outputs for downstream processing
+3. **Chaining Support**: Can be combined with filtering and cleaning stages
+4. **Executor Support**: Compatible with all distributed execution backends
@@ -11,7 +11,7 @@ modality: "universal"
 (about-concepts)=
 # Concepts
 
-Learn about the core components and concepts introduced by NeMo Curator. The following concepts are organized by each major modality.
+Learn about the core components and concepts introduced by NeMo Curator.
 
 ## Modality Concepts
 
@@ -20,18 +20,18 @@ Learn about working with specific modalities using NeMo Curator.
 ::::{grid} 1 1 1 2
 :gutter: 1 1 1 2
 
-:::{grid-item-card} {octicon}`image;1.5em;sd-mr-1` Image Curation Concepts
-:link: about-concepts-image
+:::{grid-item-card} {octicon}`typography;1.5em;sd-mr-1` Text Curation Concepts
+:link: about-concepts-text
 :link-type: ref
 
-Explore key concepts for image data curation, including scalable loading, processing (embedding, classification, filtering, deduplication), and dataset export.
+Learn about text data curation, covering data loading and processing (filtering, classification, deduplication).
 :::
 
-:::{grid-item-card} {octicon}`typography;1.5em;sd-mr-1` Text Curation Concepts
-:link: about-concepts-text
+:::{grid-item-card} {octicon}`image;1.5em;sd-mr-1` Image Curation Concepts
+:link: about-concepts-image
 :link-type: ref
 
-Learn about text data curation, covering data loading, processing (filtering, deduplication, classification), and synthetic data generation.
+Explore key concepts for image data curation, including scalable loading, processing (embedding, classification, filtering), and dataset export.
 :::
 
 :::{grid-item-card} {octicon}`video;1.5em;sd-mr-1` Video Curation Concepts
@@ -49,12 +49,28 @@ Learn about speech data curation, ASR inference, quality assessment, and audio-t
 :::
 ::::
 
+## Universal Concepts
+
+Core concepts that apply across all modalities in NeMo Curator.
+
+::::{grid} 1 1 1 1
+:gutter: 1 1 1 1
+
+:::{grid-item-card} {octicon}`duplicate;1.5em;sd-mr-1` Deduplication Concepts
+:link: about-concepts-deduplication
+:link-type: ref
+
+Comprehensive overview of deduplication techniques across text, image, and video modalities including exact, fuzzy, and semantic approaches.
+:::
+::::
+
 ```{toctree}
 :hidden:
 :maxdepth: 2
 
-Image Concepts <image/index.md>
 Text Concepts <text/index.md>
+Image Concepts <image/index.md>
 Video Concepts <video/index.md>
 Audio Concepts <audio/index.md>
+Deduplication Concepts <deduplication.md>
 ```