pr05 Visual Regression Testing #1: Initial Visual Testing Framework

[Vaivaswat2244]

Visual Testing Framework for Processing

Summary

This PR introduces a comprehensive visual regression testing framework for Processing, enabling automated detection of visual changes in rendering output across different platforms. This also opens up the scope for further discussions and modifications to the approach.

Implementation Details

Core Components

1. Sketch Runner with Controlled Execution

• Executes Processing sketches in a controlled test environment
• Captures single frame after setup() and one draw() call
• Automatic window cleanup after screenshot capture
• Configurable render wait times for complex sketches

2. Test Runner (VisualTestRunner.java)

• Captures screenshots from Processing sketches
• Manages baseline images with platform-specific naming
• Compares actual output against saved baselines
• Automatically creates baselines on first run

3. JUnit 5 Integration (VisualTest.java)

• Base test class extending JUnit 5
• Provides assertVisualMatch() helper method
• Supports test tagging and suite organization
• Handles first-run baseline creation gracefully

4. Image Comparison Algorithm (ImageComparator.java)

• Implements a sophisticated pixel-matching algorithm
• Detects and filters out minor rendering artifacts (line shifts, anti-aliasing differences)
• Configurable thresholds for pass/fail criteria
• Generates diff images for failed tests showing exact pixel differences

Project Structure

Key Features

1. Hierarchical Screenshot Organization

• Screenshots organized by test suite: __screenshots__/suite-name/test-name-platform.png

2. Test Suite Organization

• Tests organized by feature area (shapes, rendering, text, etc.)
• JUnit 5 @Tag annotations for flexible test selection
• For now, only two suites have been added, i.e., shapes and rendering. These will obviously be extended as tests grow

3. Intelligent Pixel Matching

• Handles minor anti-aliasing differences
• Detects and ignores line-shift artifacts

4. Gradle Tasks

./gradlew :visual-tests:test                    # Run all tests
./gradlew :visual-tests:visualTests             # Run only visual tests
./gradlew :visual-tests:testBasicShapes         # Run specific suite

Usage Example

@Tag("shapes")
public class BasicShapesTest extends VisualTest {
    
    @Test
    @DisplayName("Red circle renders correctly")
    public void testRedCircle() {
        assertVisualMatch("basic-shapes/red-circle", new ProcessingSketch() {
            @Override
            public void setup(PApplet p) {
                p.noStroke();
            }
            
            @Override
            public void draw(PApplet p) {
                p.background(255);
                p.fill(255, 0, 0);
                p.ellipse(p.width/2, p.height/2, 100, 100);
            }
        });
    }
}

Testing Strategy

First Run:

• Test captures screenshot from sketch
• No baseline exists → saves as baseline
• Test marked as "BASELINE CREATED"

Subsequent Runs:

• Test captures screenshot
• Compares against platform-specific baseline
• Reports PASS/FAIL with mismatch details
• Saves diff image if failed

cc @catilac @Stefterv @mingness @SableRaf