Merge pull request #1889 from roboflow/docs-fusion

Docs fusion

Author: grzegorz-roboflow

inference/core/workflows/core_steps/fusion/buffer/v1.py

@@ -16,11 +16,59 @@
 )
 
 LONG_DESCRIPTION = """
-Returns an array of the last `length` values passed to it. The newest
-elements are added to the beginning of the array.
+Maintain a sliding window buffer of the last N values by storing recent inputs in a FIFO (First-In-First-Out) queue, with newest elements added to the beginning and oldest elements automatically removed when the buffer exceeds the specified length, enabling temporal data collection, frame history tracking, batch processing preparation, and sliding window analysis workflows.
 
-Useful for keeping a sliding window of images or detections for
-later processing, visualization, or comparison.
+## How This Block Works
+
+This block maintains a rolling buffer that stores the most recent values passed to it, creating a sliding window of data over time. The block:
+
+1. Receives input data of any type (images, detections, values, etc.) and configuration parameters (buffer length and padding option)
+2. Maintains an internal buffer that persists across workflow executions:
+   - Buffer is initialized as an empty list when the block is first created
+   - Buffer state persists for the lifetime of the workflow execution
+   - Each buffer block instance maintains its own separate buffer
+3. Adds new data to the buffer:
+   - Inserts the newest value at the beginning (index 0) of the buffer array
+   - Most recent values appear first in the buffer
+   - Older values are shifted to later positions in the array
+4. Manages buffer size:
+   - When buffer length exceeds the specified `length` parameter, removes the oldest elements
+   - Keeps only the most recent `length` values
+   - Automatically maintains the sliding window size
+5. Applies optional padding:
+   - If `pad` is True: Fills the buffer with `None` values until it reaches exactly `length` elements
+   - Ensures consistent buffer size even when fewer than `length` values have been received
+   - If `pad` is False: Buffer size grows from 0 to `length` as values are added, then stays at `length`
+6. Returns the buffered array:
+   - Outputs a list containing the buffered values in order (newest first)
+   - List length equals `length` (if padding enabled) or current buffer size (if padding disabled)
+   - Values are ordered from most recent (index 0) to oldest (last index)
+
+The buffer implements a sliding window pattern where new data enters at the front and old data exits at the back when capacity is reached. This creates a temporal history of recent values, useful for operations that need to look back at previous frames, detections, or measurements. The buffer works with any data type, making it flexible for images, detections, numeric values, or other workflow outputs.
+
+## Common Use Cases
+
+- **Frame History Tracking**: Maintain a history of recent video frames for temporal analysis (e.g., track frame sequences, maintain recent image history, collect frames for comparison), enabling temporal frame analysis workflows
+- **Detection History**: Buffer recent detections for trend analysis or comparison (e.g., track detection changes over time, compare current vs previous detections, analyze detection patterns), enabling detection history workflows
+- **Batch Processing Preparation**: Collect multiple values before processing them together (e.g., batch process recent images, aggregate multiple detections, prepare data for batch operations), enabling batch processing workflows
+- **Sliding Window Analysis**: Perform analysis on a rolling window of data (e.g., analyze trends over recent frames, calculate moving averages, detect changes in sequences), enabling sliding window analysis workflows
+- **Visualization Sequences**: Maintain recent data for animation or sequence visualization (e.g., create frame sequences, visualize temporal changes, display recent history), enabling temporal visualization workflows
+- **Temporal Comparison**: Compare current values with recent historical values (e.g., compare current frame with previous frames, detect changes over time, analyze temporal patterns), enabling temporal comparison workflows
+
+## Connecting to Other Blocks
+
+This block receives data of any type and produces a buffered output array:
+
+- **After any block** that produces values to buffer (e.g., buffer images from image sources, buffer detections from detection models, buffer values from analytics blocks), enabling data buffering workflows
+- **Before blocks that process arrays** to provide batched or historical data (e.g., process buffered images, analyze detection arrays, work with value sequences), enabling array processing workflows
+- **Before visualization blocks** to display sequences or temporal data (e.g., visualize frame sequences, display detection history, show temporal patterns), enabling temporal visualization workflows
+- **Before analysis blocks** that require historical data (e.g., analyze trends over time, compare current vs historical, process temporal sequences), enabling temporal analysis workflows
+- **Before aggregation blocks** to provide multiple values for aggregation (e.g., aggregate buffered values, process multiple detections, combine recent data), enabling aggregation workflows
+- **In temporal processing pipelines** where maintaining recent history is required (e.g., track changes over time, maintain frame sequences, collect data for temporal analysis), enabling temporal processing workflows
+
+## Requirements
+
+This block works with any data type (images, detections, values, etc.). The buffer maintains state across workflow executions within the same workflow instance. The `length` parameter determines the maximum number of values to keep in the buffer. When `pad` is enabled, the buffer will always return exactly `length` elements (padded with `None` if needed). When `pad` is disabled, the buffer grows from 0 to `length` elements as values are added, then maintains `length` elements by removing oldest values. The buffer persists for the lifetime of the workflow execution and resets when the workflow is restarted.
 """
 
 SHORT_DESCRIPTION = "Returns an array of the last `length` values passed to it."
@@ -45,17 +93,21 @@ class BlockManifest(WorkflowBlockManifest):
     data: Selector(
         kind=[WILDCARD_KIND, LIST_OF_VALUES_KIND, IMAGE_KIND],
     ) = Field(
-        description="Reference to step outputs at depth level n to be concatenated and moved into level n-1.",
-        examples=["$steps.visualization"],
+        description="Input data of any type to add to the buffer. Can be images, detections, values, or any other workflow output. Newest values are added to the beginning of the buffer array. The buffer maintains a sliding window of the most recent values.",
+        examples=[
+            "$steps.visualization",
+            "$steps.object_detection_model.predictions",
+            "$steps.image",
+        ],
     )
     length: int = Field(
-        description="The number of elements to keep in the buffer. Older elements will be removed.",
-        examples=[5],
+        description="Maximum number of elements to keep in the buffer. When the buffer exceeds this length, the oldest elements are automatically removed. Determines the size of the sliding window. Must be greater than 0. Typical values range from 2-10 for frame sequences, or higher for longer histories.",
+        examples=[5, 10, 3],
     )
     pad: bool = Field(
-        description="If True, the end of the buffer will be padded with `None` values so its size is always exactly `length`.",
+        description="Enable padding to maintain consistent buffer size. If True, the buffer is padded with `None` values until it reaches exactly `length` elements, ensuring the output always has `length` items even when fewer values have been received. If False, the buffer grows from 0 to `length` as values are added, then maintains `length` by removing oldest values. Use padding when downstream blocks require a fixed-size array.",
         default=False,
-        examples=[True],
+        examples=[True, False],
     )
 
     @classmethod

inference/core/workflows/core_steps/fusion/detections_classes_replacement/v1.py

@@ -31,12 +31,73 @@
 )
 
 LONG_DESCRIPTION = """
-Combine results of detection model with classification results performed separately for 
-each and every bounding box. 
+Replace class labels of detection bounding boxes with classes predicted by a classification model applied to cropped regions, combining generic detection results with specialized classification predictions to enable two-stage detection workflows, fine-grained classification, and class refinement workflows where generic detections are refined with specific class labels from specialized classifiers.
 
-Bounding boxes without top class predicted by classification model are discarded, 
-for multi-label classification results, most confident label is taken as bounding box
-class.  
+## How This Block Works
+
+This block combines results from a detection model (with bounding boxes and generic classes) with classification predictions (from a specialized classifier applied to cropped regions) to replace generic class labels with specific ones. The block:
+
+1. Receives two inputs with different dimensionality levels:
+   - `object_detection_predictions`: Detection results (dimensionality level 1) containing bounding boxes with generic classes (e.g., "dog", "person", "vehicle")
+   - `classification_predictions`: Classification results (dimensionality level 2) from a classifier applied to cropped regions of each detection (e.g., "Golden Retriever", "Labrador" for dog detections)
+2. Matches classifications to detections:
+   - Uses `PARENT_ID_KEY` (detection_id) in classification predictions to link each classification result to its source detection
+   - Creates a mapping from detection IDs to classification results
+3. Extracts leading class from each classification prediction:
+
+   **For single-label classifications:**
+   - Uses the "top" class (predicted class) from the classification result
+   - Extracts class name, class ID, and confidence from the classification prediction
+
+   **For multi-label classifications:**
+   - Finds the class with the highest confidence score
+   - Uses the most confident label as the replacement class
+   - Extracts class name, class ID, and confidence from the highest-confidence prediction
+
+4. Handles missing classifications:
+   - Detections without corresponding classification predictions are discarded by default
+   - If `fallback_class_name` is provided, detections without classifications use the fallback class instead of being discarded
+   - Fallback class ID is set to the provided value, or `sys.maxsize` if not specified or negative
+5. Filters detections:
+   - Keeps only detections that have classification results (or fallback if specified)
+   - Removes detections that cannot be matched to classification predictions
+6. Replaces class information:
+   - Replaces class names in detections with classification class names
+   - Replaces class IDs in detections with classification class IDs
+   - Replaces confidence scores in detections with classification confidence scores
+   - Updates all detection metadata to reflect the new class information
+7. Generates new detection IDs:
+   - Creates new unique detection IDs for updated detections (prevents ID conflicts)
+   - Ensures detection IDs are unique after class replacement
+8. Returns updated detections:
+   - Outputs detections with replaced classes, maintaining bounding box coordinates and other properties
+   - Output dimensionality matches input detection predictions (dimensionality level 1)
+
+The block enables two-stage detection workflows where a generic detection model locates objects and a specialized classification model provides fine-grained labels. This is useful when you need generic localization (e.g., "dog") combined with specific classification (e.g., "Golden Retriever", "German Shepherd") without losing spatial information.
+
+## Common Use Cases
+
+- **Two-Stage Detection and Classification**: Combine generic detection with specialized classification for fine-grained labeling (e.g., detect "dog" then classify breed, detect "vehicle" then classify type, detect "person" then classify age group), enabling two-stage detection workflows
+- **Class Refinement**: Refine generic class labels with specific classifications from specialized models (e.g., refine "animal" to specific species, refine "vehicle" to specific models, refine "food" to specific dishes), enabling class refinement workflows
+- **Multi-Model Workflows**: Combine detection and classification models to leverage the strengths of both (e.g., use generic detector for localization and specialist classifier for identification, combine coarse and fine-grained models, leverage specialized classifiers with general detectors), enabling multi-model workflows
+- **Hierarchical Classification**: Apply hierarchical classification where detection provides high-level classes and classification provides detailed sub-classes (e.g., detect "mammal" then classify species, detect "plant" then classify variety, detect "structure" then classify type), enabling hierarchical classification workflows
+- **Crop-Based Classification**: Use classification results from cropped regions to enhance detection results (e.g., classify crops to improve detection labels, apply specialized classifiers to detected regions, refine detections with crop classifications), enabling crop-based classification workflows
+- **Fine-Grained Object Recognition**: Enable fine-grained recognition by combining localization and detailed classification (e.g., recognize specific product models, identify specific animal breeds, classify specific vehicle types), enabling fine-grained recognition workflows
+
+## Connecting to Other Blocks
+
+This block receives detection and classification predictions and produces detections with replaced classes:
+
+- **After detection and classification model blocks** to combine generic detection with specialized classification (e.g., object detection + classification to refined detections, detection model + classifier to labeled detections), enabling detection-classification fusion workflows
+- **After crop blocks** that create crops from detections for classification (e.g., crop detections then classify crops, create crops for classification then replace classes), enabling crop-classification workflows
+- **Before visualization blocks** to display detections with refined classes (e.g., visualize refined detections, display detections with specific labels, show classification-enhanced detections), enabling refined detection visualization workflows
+- **Before filtering blocks** to filter detections with refined classes (e.g., filter by specific classes, filter refined detections, apply filters to classified detections), enabling refined detection filtering workflows
+- **Before analytics blocks** to perform analytics on refined detections (e.g., analyze specific classes, perform analytics on classified detections, track refined detection metrics), enabling refined detection analytics workflows
+- **In workflow outputs** to provide refined detections as final output (e.g., two-stage detection outputs, classification-enhanced detection outputs, refined detection results), enabling refined detection output workflows
+
+## Requirements
+
+This block requires object detection predictions (with bounding boxes) and classification predictions from crops of those bounding boxes. The classification predictions must have `PARENT_ID_KEY` (detection_id) to link classifications to their source detections. The block accepts different dimensionality levels: detection predictions at level 1 and classification predictions at level 2 (from crops). For single-label classifications, the "top" class is used. For multi-label classifications, the most confident class is selected. Detections without classification results are discarded unless `fallback_class_name` is provided. The block outputs detections with replaced classes, class IDs, and confidences, with new detection IDs generated. Output dimensionality matches input detection predictions (level 1).
 """
 
 SHORT_DESCRIPTION = "Replace classes of detections with classes predicted by a chained classification model."
@@ -70,26 +131,31 @@ class BlockManifest(WorkflowBlockManifest):
         ]
     ) = Field(
         title="Regions of Interest",
-        description="The output of a detection model describing the bounding boxes that will have classes replaced.",
-        examples=["$steps.my_object_detection_model.predictions"],
+        description="Detection predictions (object detection, instance segmentation, or keypoint detection) containing bounding boxes with generic class labels that will be replaced with classification results. These detections should correspond to the regions that were cropped and classified. Detections must have detection IDs that match the PARENT_ID_KEY in classification predictions. Detections at dimensionality level 1.",
+        examples=[
+            "$steps.object_detection_model.predictions",
+            "$steps.instance_segmentation_model.predictions",
+        ],
     )
     classification_predictions: Selector(kind=[CLASSIFICATION_PREDICTION_KIND]) = Field(
         title="Classification results for crops",
-        description="The output of classification model for crops taken based on RoIs pointed as the other parameter",
-        examples=["$steps.my_classification_model.predictions"],
+        description="Classification predictions from a classifier applied to cropped regions of the detections. Each classification result must have PARENT_ID_KEY (detection_id) linking it to its source detection. Supports both single-label (uses 'top' class) and multi-label (uses most confident class) classifications. Classification results at dimensionality level 2 (one classification per crop/detection).",
+        examples=[
+            "$steps.classification_model.predictions",
+            "$steps.breed_classifier.predictions",
+        ],
     )
     fallback_class_name: Union[Optional[str], Selector(kind=[STRING_KIND])] = Field(
         default=None,
         title="Fallback class name",
-        description="The class name to be used as a fallback if no class is predicted for a bounding box",
-        examples=["unknown"],
+        description="Optional class name to use for detections that don't have corresponding classification predictions. If not provided (default None), detections without classifications are discarded. If provided, detections without classifications use this fallback class name instead of being removed. Useful for preserving detections when classification fails or is unavailable.",
+        examples=[None, "unknown", "unclassified"],
     )
     fallback_class_id: Union[Optional[int], Selector(kind=[INTEGER_KIND])] = Field(
         default=None,
         title="Fallback class id",
-        description="The class id to be used as a fallback if no class is predicted for a bounding box;"
-        f"if not specified or negative, the class id will be set to {sys.maxsize}",
-        examples=[77],
+        description="Optional class ID to use with fallback_class_name for detections without classification predictions. If not specified or negative, the class ID is set to sys.maxsize. Only used when fallback_class_name is provided. Should match the class ID mapping used in your model.",
+        examples=[None, 77, 999],
     )
 
     @classmethod