Create examples showing ndd usage

[rostan-t]

Category: Documentation

Description:

Currently, most examples in the documentation only show pipeline mode. This PR adds dynamic mode examples.

• The front page now uses tabs to show a both a pipeline mode and dynamic snippets. A sphinx plugin allows it to render properly both on GitHub and in the HTML documentation.
• The getting started page now has variants for both pipeline mode and dynamic mode.
• Some examples and tutorials now have both pipeline mode and dynamic mode variants. Those are stored as subpages:
  

Because of the third point, a lot of pages have been renamed (generally to pipeline_mode.ipynb) with little or no changes (when they contained links with relative paths, those needed to be changed).

Additional information:

Affected modules and functionalities:

Pages affected by this PR:

• Front page
• Getting started tutorial
• Data loading:
  • LMDB Reader
  • MXNet RecordIO Reader
  • TFRecord Reader
  • Webdataset Reader
  • COCO Reader
  • NumPy Reader
• Image processing:
  • Augmentation Gallery (already existed, now moved)
  • Brightness and contrast
  • CLAHE
  • Color Space Conversion
  • Image Decoder
  • HSV
  • Resize Operator
  • Warp Affine
• Audio Processing:
  • Audio Decoder
  • Audio Spectogram
• Video Processing
  • Simple video pipeline
  • Video pipeline with labels
  • Video pipeline with timestamps and frame numbers
  • Video with image processing operators
  • Optical flow

Key points relevant for the review:

The PR contains little code changes apart from documentation pages so it's best reviewed directly as the rendered HTML doc.

• Does the new organization make sense?
• Are links still working correctly?
• Do the dynamic mode examples capture everything that the corresponding pipeline mode examples capture?
• Do the examples cover every aspect of dynamic mode we want to cover? Note that integration with PyTorch and control features are separate tasks.

Tests:

• Existing tests apply
• New tests added
  • Python tests
  • GTests
  • Benchmark
  • Other
• N/A

Checklist

Documentation

• Existing documentation applies
• Documentation updated
  • Docstring
  • Doxygen
  • RST
  • Jupyter
  • Other
• N/A

DALI team only

Requirements

• Implements new requirements
• Affects existing requirements
• N/A

REQ IDs: N/A

JIRA TASK: DALI-4515

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[rostan-t]

!build

[dali-automaton]

CI MESSAGE: [41293348]: BUILD FAILED

[greptile-apps]

Greptile Summary

Overview

This PR successfully adds dynamic mode (ndd) examples alongside existing pipeline mode examples throughout the DALI documentation. The implementation is well-structured and maintains consistency across 22 pairs of example notebooks.

Key Changes

• New Sphinx Extension (dali_tabs.py): Converts .. container:: dali-tabs blocks into .. tab-set:: directives, enabling proper rendering on both GitHub and HTML documentation
• Example Reorganization: Converted single notebooks into paired examples (pipeline_mode.ipynb + dynamic_mode.ipynb) for:
  • Data loading (LMDB, MXNet RecordIO, TFRecord, Webdataset, COCO, NumPy readers)
  • Image processing (Augmentation Gallery, Brightness/Contrast, CLAHE, Color Space, Decoder, HSV, Resize, Warp)
  • Audio processing (Decoder, Spectrogram)
  • Video processing (Simple, with labels, with timestamps, with image ops, optical flow)
• Front page (README.rst): Now demonstrates both modes with practical code examples
• Documentation Structure: Getting started page refactored to present mode options upfront with toctree for each variant
• File Organization: Created nested index.py files for each example category with proper doc_entry references

Content Verification

Both dynamic and pipeline mode examples verified to:

• Use correct imports (ndd.* vs fn.*, @pipeline_def)
• Implement functionally equivalent operations
• Cover the same use cases and parameters
• Show structural differences appropriate to each mode (eager vs declarative)

Technical Quality

• Regex patterns in dali_tabs.py correctly handle RST indentation (intentionally adds 3 spaces to already-indented content)
• OpReference transformation from "dynamic." to "experimental." properly handled in doc_index.py line 88-89
• .gitignore correctly updated for new nested index.rst generation pattern
• All 22 dynamic_mode.ipynb files paired with 22 pipeline_mode.ipynb files

Confidence Score: 5/5

• This PR is safe to merge with no identified issues. All code changes are well-structured, properly tested through documentation examples, and maintain consistency across 22 paired notebooks.
• Maximum confidence score due to: (1) Comprehensive and consistent documentation updates with verified content parity between modes; (2) Clean new Sphinx extension with correct regex patterns and indentation logic; (3) Proper integration of extension into sphinx configuration; (4) Systematic organization of examples with consistent index.py structure; (5) No breaking changes to existing functionality; (6) Good separation of concerns (dali_tabs plugin vs documentation content).
• No files require special attention. All changes are documentation and build configuration related with no potential runtime impacts.

Important Files Changed

Filename	Overview
docs/_extensions/dali_tabs.py	New Sphinx extension for converting dali-tabs container blocks to tab-set directives. Properly handles RST indentation (adds 3 spaces to content that already has 3+ space indent from container). Regex patterns correctly match container blocks and tab headers. Integration with Sphinx via include-read hook is correct.
docs/conf.py	Sphinx configuration correctly registers new dali_tabs extension (line 162). No breaking changes to existing configuration. Extension is properly added to extensions list.
README.rst	Successfully demonstrates both pipeline and dynamic modes using the new dali-tabs container directive. Shows proper code differences: pipeline mode uses @pipeline_def, fn.* prefix, and DALIGenericIterator; dynamic mode uses ndd.* prefix, direct iteration with reader.next_epoch(), and torch integration. Both examples are functionally equivalent and compile-ready.
.gitignore	Updated patterns to ignore generated index.rst files in nested directories (using **/ glob pattern). Changes reflect new documentation structure where each example category has auto-generated index files at multiple levels.
docs/examples/getting_started/index.rst	Properly structured documentation page with clear explanation of pipeline vs dynamic modes. Uses toctree to reference both pipeline_mode and dynamic_mode notebooks. Content correctly distinguishes use cases for each mode.
docs/examples/general/data_loading/lmdb_reader/index.py	Example index.py showing new pattern for paired examples. Correctly references both pipeline_mode.ipynb and dynamic_mode.ipynb. OpReferences properly differentiate between fn.readers.caffe (pipeline) and dynamic.readers.Caffe (which gets transformed to experimental.readers.Caffe by doc_index.py). Structure is consistent with other paired examples.

Sequence Diagram

sequenceDiagram
    participant User as Documentation User
    participant Sphinx as Sphinx Builder
    participant dali_tabs as dali_tabs Extension
    participant RST as RST Parser
    participant HTML as HTML Output

    User->>Sphinx: Build documentation
    Sphinx->>Sphinx: Load dali_tabs extension
    Sphinx->>Sphinx: Process .rst files
    
    Note over RST: Processes README.rst with<br/>container:: dali-tabs blocks
    
    RST->>dali_tabs: include-read hook triggered
    dali_tabs->>dali_tabs: Match container:: dali-tabs
    dali_tabs->>dali_tabs: Extract tab headers<br/>(**Pipeline mode:**, etc)
    dali_tabs->>dali_tabs: Transform to tab-set directive
    dali_tabs->>RST: Return transformed RST
    
    RST->>HTML: Render tab-set with sync groups
    HTML->>User: Display tabs synced across page
    
    Note over User: Same tab selection<br/>persists across all<br/>code examples on page

Loading

[rostan-t]

!build

[dali-automaton]

CI MESSAGE: [41295926]: BUILD FAILED

[greptile-apps]

Greptile Overview

Greptile Summary

This PR adds comprehensive dynamic mode examples to DALI documentation, creating parity with existing pipeline mode examples. A new Sphinx extension enables synchronized tabs for displaying both modes side-by-side.

Major Changes:

• Added dali_tabs.py Sphinx extension to transform container blocks into synchronized tab-sets
• Split getting started tutorial into separate pipeline_mode.ipynb and dynamic_mode.ipynb variants
• Created dynamic_mode.ipynb examples for 20+ topics across data loading, image processing, audio processing, and video processing
• Reorganized documentation structure: many examples moved from flat files into subdirectories with index.py files
• Enhanced doc_index.py to map dynamic.* operator references to experimental.dynamic.* and handle Sphinx title syntax
• Updated README.rst front page with tabbed code examples

Documentation Quality:

• Dynamic mode examples are comprehensive and mirror pipeline mode functionality
• Examples include proper error handling, context managers (EvalContext, EvalMode), and batch/tensor operations
• Code demonstrates key dynamic mode patterns: direct operator calls, reader iteration, GPU/CPU data movement
• Path references updated correctly for relocated notebooks (e.g., ../../data/images for deeper nesting)

Confidence Score: 5/5

• This documentation PR is safe to merge with no code execution risks
• Score reflects that this is a documentation-only change with well-structured code examples, proper organization, and correct path updates for all relocated files. The new Sphinx extension is clean and follows proper patterns.
• No files require special attention - all changes are documentation enhancements

Important Files Changed

File Analysis

Filename	Score	Overview
docs/_extensions/dali_tabs.py	5/5	New Sphinx extension that transforms container:: dali-tabs blocks into proper tab-set RST for rendering pipeline/dynamic mode examples
README.rst	5/5	Front page updated with tabbed examples showing both pipeline mode and dynamic mode usage patterns
docs/conf.py	5/5	Configuration updated to register dali_tabs extension and pass references to dynamic mode documentation
docs/doc_index.py	5/5	Enhanced to support dynamic. operator prefix mapping to experimental.dynamic and handle sphinx title syntax in paths
docs/examples/getting_started/index.rst	5/5	New landing page explaining the difference between pipeline and dynamic modes with links to separate tutorials
docs/examples/getting_started/dynamic_mode.ipynb	5/5	Comprehensive dynamic mode tutorial showing readers, decoding, augmentations, GPU acceleration, and performance comparison

Sequence Diagram

sequenceDiagram
    participant User
    participant Sphinx
    participant dali_tabs
    participant doc_index
    participant Notebooks
    
    User->>Sphinx: Build documentation
    Sphinx->>doc_index: Process examples/index.py
    doc_index->>doc_index: Generate index.rst files
    doc_index->>doc_index: Map operator references<br/>(dynamic.* → experimental.dynamic.*)
    doc_index->>doc_index: Handle title syntax<br/>("Title <filename>")
    
    Sphinx->>dali_tabs: include-read event
    dali_tabs->>dali_tabs: Find "container:: dali-tabs"
    dali_tabs->>dali_tabs: Parse **Tab Name:** sections
    dali_tabs->>dali_tabs: Transform to tab-set RST<br/>with sync-group
    dali_tabs->>Sphinx: Return transformed RST
    
    Sphinx->>Notebooks: Render pipeline_mode.ipynb
    Notebooks->>Sphinx: Display pipeline examples
    
    Sphinx->>Notebooks: Render dynamic_mode.ipynb
    Notebooks->>Sphinx: Display dynamic examples
    
    Sphinx->>User: HTML documentation with<br/>synchronized tabs

Loading

[rostan-t]

!build

[greptile-apps]

Greptile Overview

Greptile Summary

This PR adds comprehensive dynamic mode examples to DALI documentation, complementing the existing pipeline mode examples. The implementation is well-structured and provides excellent coverage of dynamic mode functionality.

Key Changes:

• Added dali_tabs.py Sphinx extension that transforms container:: dali-tabs blocks into synchronized tab-sets for dual-mode display
• Updated front page (README.rst) with tabbed examples showing both pipeline and dynamic modes
• Reorganized documentation structure: converted single notebooks into directories with pipeline_mode.ipynb and dynamic_mode.ipynb variants
• Added 22 new dynamic mode notebooks covering data loading, image processing, audio processing, and video processing
• Updated relative paths in refactored notebooks to account for new directory structure

Documentation Coverage:
The PR adds dynamic mode variants for all major functionality areas: file readers (LMDB, RecordIO, TFRecord, Webdataset, COCO, NumPy), image processing operations (decoders, augmentations, color transforms, resize, warp), audio processing (decoder, spectrogram), and video processing (simple pipelines, labels, timestamps, optical flow).

Technical Quality:

• Dynamic mode examples correctly use ndd. (nvidia.dali.experimental.dynamic) prefix instead of fn.
• Proper device handling: dynamic mode uses device="gpu" while pipeline mode uses device="mixed"
• Relative paths properly updated (e.g., ../data/images to ../../data/images after directory restructure)
• The dali_tabs.py extension is cleanly implemented with appropriate regex patterns and error handling
• Tab synchronization via sync-group: dali-mode ensures consistent user experience across pages

Minor Observations:

• The PR correctly deleted the old monolithic getting_started.ipynb in favor of the new split structure
• Index files use the new doc_entry() structure to properly reference both pipeline and dynamic mode operators
• The doc_index.py update correctly prefixes dynamic operators with "experimental." for internal references

Confidence Score: 5/5

• This PR is safe to merge with very high confidence - it's a well-executed documentation enhancement with no functional code changes
• Score reflects the documentation-only nature of changes (no production code modified), excellent implementation quality of the new Sphinx extension, proper handling of relative paths in refactored notebooks, correct usage of dynamic mode vs pipeline mode syntax, and comprehensive coverage of dynamic mode functionality across all example categories
• No files require special attention - all changes are documentation enhancements with proper structure and correct syntax

Important Files Changed

File Analysis

Filename	Score	Overview
docs/_extensions/dali_tabs.py	5/5	New Sphinx extension that transforms container:: dali-tabs blocks into tab-set directives for dual-mode documentation display. Clean implementation with proper regex patterns and error handling.
README.rst	5/5	Updated front page with tabbed code examples showing both pipeline and dynamic modes. Uses container:: dali-tabs syntax that renders properly in both GitHub and HTML docs.
docs/conf.py	5/5	Added dali_tabs to extensions list and updated doc generation to include dynamic mode operator tables and autodoc.
docs/doc_index.py	5/5	Updated to handle dynamic mode operator references by prefixing "dynamic." with "experimental."
docs/examples/getting_started/index.rst	5/5	New landing page explaining pipeline vs dynamic mode with toctree to both tutorial variants.
docs/examples/getting_started/dynamic_mode.ipynb	5/5	Comprehensive dynamic mode tutorial covering readers, decoding, augmentations, GPU acceleration, and performance testing. Well-structured with clear explanations.
docs/examples/getting_started/pipeline_mode.ipynb	5/5	Refactored from original getting_started.ipynb with updated relative paths. Covers pipeline mode fundamentals comprehensively.
docs/examples/image_processing/decoder/dynamic_mode.ipynb	5/5	Dynamic mode examples for image decoders with proper device="gpu" usage (not "mixed"). Demonstrates decoder variants comprehensively.
docs/examples/image_processing/decoder/pipeline_mode.ipynb	5/5	Renamed from original with updated relative paths (../../data/images). Uses device="mixed" correctly for pipeline mode.
docs/examples/image_processing/decoder/index.py	5/5	New index structure with separate entries for pipeline and dynamic modes, each with appropriate operator references.

Sequence Diagram

sequenceDiagram
    participant User as User/GitHub
    participant Sphinx as Sphinx Build
    participant DaliTabs as dali_tabs.py Extension
    participant DocIndex as doc_index.py
    participant IndexPy as index.py Files
    
    User->>Sphinx: Build Documentation
    Sphinx->>Sphinx: Load conf.py configuration
    Sphinx->>Sphinx: Register dali_tabs extension
    
    Sphinx->>DocIndex: Execute doc_index.py
    DocIndex->>IndexPy: Process examples/index.py
    loop For each example category
        IndexPy->>DocIndex: Return Doc with entries
        DocIndex->>DocIndex: Generate index.rst files
        DocIndex->>DocIndex: Collect operator references
    end
    DocIndex-->>Sphinx: Return operator->example mapping
    
    Sphinx->>Sphinx: Process RST files
    loop For each RST file with dali-tabs
        Sphinx->>DaliTabs: include-read event
        DaliTabs->>DaliTabs: Find "container:: dali-tabs" blocks
        DaliTabs->>DaliTabs: Parse **Pipeline mode:** and **Dynamic mode:** sections
        DaliTabs->>DaliTabs: Transform to tab-set with sync-group
        DaliTabs-->>Sphinx: Return transformed RST
    end
    
    Sphinx->>Sphinx: Render HTML with sphinx_design tabs
    Sphinx-->>User: Generated Documentation

Loading

[dali-automaton]

CI MESSAGE: [41342923]: BUILD FAILED

[rostan-t]

!build

[greptile-apps]

Greptile Overview

Greptile Summary

This PR adds comprehensive dynamic mode examples to DALI documentation, creating parallel examples alongside existing pipeline mode tutorials. The changes are well-structured and primarily additive.

Major Changes:

• Added 22 new dynamic mode example notebooks (dynamic_mode.ipynb) across data loading, image processing, audio processing, and video processing categories
• Created corresponding 22 pipeline mode notebooks (pipeline_mode.ipynb) by reorganizing existing examples
• Implemented dali_tabs.py Sphinx extension to support tabbed code displays in documentation
• Updated README.rst front page to show both pipeline and dynamic mode examples using the new tab system
• Reorganized examples into subdirectories with index.py files for better structure (30 index files total)

Infrastructure Updates:

• Enhanced doc_index.py to handle sphinx toctree title syntax and automatically prefix experimental. to dynamic operator references
• Updated .gitignore patterns to use **/index.rst for subdirectory-generated files
• Registered dali_tabs extension in docs/conf.py

Documentation Quality:
The dynamic mode examples appropriately demonstrate:

• Reader usage with ndd.readers.* classes
• Iterating with next_epoch(batch_size=...)
• GPU acceleration with .gpu() and .cpu() methods
• Batch vs Tensor handling with ndd.as_batch() and ndd.as_tensor()
• EvalContext for controlling execution parameters

All examples maintain consistency with their pipeline mode counterparts while showcasing dynamic mode's more flexible, functional API.

Confidence Score: 5/5

• This PR is safe to merge with minimal risk - it's primarily additive documentation changes with well-implemented infrastructure
• Score reflects the comprehensive nature of the changes (80 files, all documentation), clean implementation of the Sphinx extension, consistent structure across all 22 example pairs, proper handling of paths and references, and no code logic changes to the DALI library itself. The changes are well-organized with matching pipeline/dynamic mode examples.
• No files require special attention - all changes follow consistent patterns and are well-structured

Important Files Changed

File Analysis

Filename	Score	Overview
docs/_extensions/dali_tabs.py	5/5	New Sphinx extension to transform container:: dali-tabs syntax into tab-set directives for displaying pipeline/dynamic mode code examples
README.rst	5/5	Added dali-tabs container with both pipeline mode and dynamic mode examples in the front page code snippet
docs/conf.py	5/5	Registered dali_tabs extension in extensions list (line 162), no other significant changes
docs/doc_index.py	5/5	Added name_to_path() method to handle sphinx toctree title syntax, and automatic prefixing of 'experimental.' to dynamic. operator references
docs/examples/getting_started/index.rst	5/5	New index file that provides overview of pipeline vs dynamic mode and links to separate tutorials for each
docs/examples/getting_started/pipeline_mode.ipynb	5/5	Pipeline mode tutorial (moved from getting_started.ipynb with minor path adjustments)
docs/examples/getting_started/dynamic_mode.ipynb	5/5	New dynamic mode tutorial covering readers, decoding, augmentations, GPU acceleration, and performance comparison
.gitignore	5/5	Updated patterns from specific index.rst paths to **/index.rst to ignore generated index files in subdirectories
docs/index.rst	5/5	Updated to reference new getting_started/index path instead of old getting_started.ipynb

Sequence Diagram

sequenceDiagram
    participant User
    participant Sphinx
    participant dali_tabs
    participant sphinx_design
    participant Browser

    User->>Sphinx: Build documentation
    Sphinx->>Sphinx: Parse README.rst
    Sphinx->>Sphinx: Encounter container:: dali-tabs
    Sphinx->>dali_tabs: include-read event
    dali_tabs->>dali_tabs: Match CONTAINER_PATTERN
    dali_tabs->>dali_tabs: Split by TAB_HEADER_PATTERN
    dali_tabs->>dali_tabs: Transform to tab-set syntax
    dali_tabs-->>Sphinx: Return transformed RST
    Sphinx->>sphinx_design: Process tab-set directive
    sphinx_design-->>Sphinx: Generate HTML tabs
    Sphinx->>Sphinx: Build HTML output
    Sphinx-->>User: Documentation ready
    User->>Browser: View documentation
    Browser->>Browser: Display Pipeline/Dynamic mode tabs

Loading

[dali-automaton]

CI MESSAGE: [41344167]: BUILD STARTED

[dali-automaton]

CI MESSAGE: [41344167]: BUILD FAILED

[greptile-apps]

Greptile Overview

Greptile Summary

This PR adds comprehensive documentation for DALI's dynamic mode (ndd) alongside existing pipeline mode examples. The changes are primarily additive, creating parallel examples for 22 topics covering data loading, image/audio/video processing.

Key Changes:

• New Sphinx Extension: dali_tabs.py enables GitHub-compatible tabbed code examples that render properly in both RST and HTML
• Front Page Update: README.rst now shows both pipeline and dynamic modes side-by-side using the new tab system
• Getting Started: Split into separate pipeline and dynamic mode tutorials with clear mode selection guidance
• 22 New Examples: Each example topic now has both pipeline_mode.ipynb and dynamic_mode.ipynb variants
• Documentation Infrastructure: Added dynamic mode API autodoc generation in conf.py

Example Coverage:

• Data loading: LMDB, MXNet RecordIO, TFRecord, Webdataset, COCO, NumPy readers
• Image processing: Decoders, augmentation gallery, brightness/contrast, CLAHE, color space, HSV, resize, warp affine
• Audio: Audio decoder, spectrogram
• Video: Simple reader, labels, timestamps, image processing ops, optical flow

Strengths:

• Parallel structure makes examples easy to compare between modes
• Dynamic mode examples properly use nvidia.dali.experimental.dynamic as ndd
• Examples cover GPU acceleration, readers, and common operations
• Documentation organization is clear with index.py-based navigation

Observations:

• Many pages renamed to pipeline_mode.ipynb with minimal content changes (just relative link updates)
• The tab synchronization (sync-group: dali-mode) allows consistent mode selection across pages
• All 22 dynamic mode examples follow consistent patterns

Confidence Score: 5/5

• This PR is safe to merge with minimal risk - it's purely additive documentation
• Score of 5 reflects that this is a documentation-only PR with no production code changes. The new Sphinx extension is well-structured, examples are comprehensive and follow consistent patterns, and the parallel structure between pipeline and dynamic modes is well-organized. All changes are additive without breaking existing documentation.
• No files require special attention - the PR is well-structured throughout

Important Files Changed

File Analysis

Filename	Score	Overview
docs/_extensions/dali_tabs.py	5/5	New Sphinx extension for tab rendering, well-structured regex-based RST transformation
README.rst	5/5	Front page updated with tabbed examples showing both pipeline and dynamic modes
docs/conf.py	5/5	Added dali_tabs extension to Sphinx configuration, generates dynamic mode autodoc
docs/examples/getting_started/dynamic_mode.ipynb	5/5	Comprehensive dynamic mode tutorial covering basics, augmentations, and GPU acceleration
docs/examples/getting_started/pipeline_mode.ipynb	5/5	Comprehensive pipeline mode tutorial covering basics, augmentations, and GPU acceleration
docs/examples/image_processing/decoder/dynamic_mode.ipynb	5/5	Image decoder examples for dynamic mode showing CPU, GPU, and different decoding variants

Sequence Diagram

sequenceDiagram
    participant User
    participant Sphinx
    participant dali_tabs
    participant doc_index
    participant Examples
    
    User->>Sphinx: Build documentation
    Sphinx->>doc_index: Generate index.rst files
    doc_index->>doc_index: Parse index.py files
    doc_index->>doc_index: Create operator references
    doc_index-->>Sphinx: Return example mappings
    
    Sphinx->>dali_tabs: Process RST includes
    dali_tabs->>dali_tabs: Find container:: dali-tabs
    dali_tabs->>dali_tabs: Transform to tab-set
    dali_tabs-->>Sphinx: Return transformed RST
    
    Sphinx->>Examples: Render notebooks
    Examples->>Examples: Display pipeline_mode.ipynb
    Examples->>Examples: Display dynamic_mode.ipynb
    Examples-->>User: Show tabbed examples
    
    User->>User: Select tab (Pipeline/Dynamic)
    User->>Examples: View selected mode

Loading