High-Performance Order Processor

A robust .NET 8 Windows service that monitors a directory for incoming JSON order files, processes them with validation and business rules, and persists results to a SQLite database.

🚀 Features

• Event-driven file monitoring using FileSystemWatcher for real-time processing
• High-performance concurrent processing with configurable throttling
• Robust error handling with exponential backoff retry logic
• Idempotency guarantees preventing duplicate order processing
• Comprehensive validation with business rule enforcement
• SQLite persistence with Entity Framework Core
• Structured logging using Serilog for observability
• Windows Service support with easy installation scripts
• Extensive testing with unit and integration test coverage

🏗️ Architecture Overview

The system consists of three main components:

1. OrderProcessor.Service - The main Windows service that monitors files and processes orders
2. OrderProcessor.Generator - A command-line tool for generating test order files
3. OrderProcessor.Tests - Comprehensive unit and integration tests

Key Design Decisions

• Event-driven file monitoring using FileSystemWatcher instead of polling for better performance
• Concurrent processing with semaphore-based throttling to handle high-volume bursts
• Idempotency implemented through database constraints and duplicate checking
• Resilient error handling with exponential backoff retry logic
• Structured logging using Serilog for comprehensive observability
• Entity Framework Core with SQLite for reliable data persistence

📋 Prerequisites

• .NET 8.0 SDK or later
• Windows 10/11 or Windows Server 2019+
• Administrator privileges (for Windows Service installation)

🚀 Quick Start

1. Clone and Build

git clone https://github.com/NoCap01/high-performance-order-processor.git
cd high-performance-order-processor
dotnet build OrderProcessor.sln --configuration Release

2. Run as Console Application (Development)

cd OrderProcessor.Service
dotnet run

The service will:

• Create an IncomingOrders directory if it doesn't exist
• Start monitoring for .json files
• Create a SQLite database (orders.db) in the current directory
• Begin processing any existing files

3. Generate Test Files

Open a new terminal and run:

cd OrderProcessor.Generator
dotnet run -- --count 100 --delay 50 --invalid-percentage 0.1 --corrupted-percentage 0.05

This generates 100 files with 10% invalid orders and 5% corrupted JSON files.

🔧 Windows Service Installation

Install as Windows Service

1. Build in Release mode:

dotnet publish OrderProcessor.Service -c Release -o C:\OrderProcessor

2. Install the service:

sc create "OrderProcessorService" binPath="C:\OrderProcessor\OrderProcessor.Service.exe" start=auto
sc description "OrderProcessorService" "High-Performance Order Processing Service"

3. Start the service:

sc start "OrderProcessorService"

Uninstall Windows Service

sc stop "OrderProcessorService"
sc delete "OrderProcessorService"

🛠️ File Generator Usage

The generator supports various scenarios for comprehensive testing:

Basic Usage

dotnet run -- --count 50

High-Volume Burst Testing

dotnet run -- --count 1000 --burst --output-dir "IncomingOrders"

Custom Configuration

dotnet run -- --count 200 --delay 25 --invalid-percentage 0.15 --corrupted-percentage 0.1 --include-high-value true

Available Options

Option	Description	Default
--count	Number of files to generate	10
--delay	Delay between files (ms)	100
--invalid-percentage	Percentage of invalid orders (0.0-1.0)	0.1
--corrupted-percentage	Percentage of corrupted JSON (0.0-1.0)	0.05
--include-high-value	Include high-value orders (>$1000)	true
--output-dir	Output directory	"IncomingOrders"
--burst	Generate all files at once	false

🗄️ Database Schema

ValidOrders Table

• Id (Primary Key)
• OrderId (Unique, Required)
• CustomerName (Required)
• OrderDate
• TotalAmount (Decimal)
• IsHighValue (Boolean)
• ItemsJson (Serialized order items)
• ProcessedAt (UTC timestamp)
• FileName (Unique, for idempotency)

InvalidOrders Table

• Id (Primary Key)
• FileName (Unique)
• RawContent (Original file content)
• FailureReason (Validation error message)
• ProcessedAt (UTC timestamp)

📏 Business Rules

Validation Rules

• TotalAmount must be >= 0
• CustomerName must not be empty or null
• OrderId must not be empty or null

Business Logic

• Orders with TotalAmount > $1000 are marked as "HighValue"
• All order items are serialized as JSON in the ItemsJson field

🔒 Idempotency Implementation

The system ensures each file is processed exactly once through:

1. Unique filename constraints in both ValidOrders and InvalidOrders tables
2. Pre-processing checks that query the database before processing
3. Database-level uniqueness preventing duplicate insertions

🧪 Testing

Run Unit Tests

dotnet test OrderProcessor.Tests

Test Scenarios Covered

1. Valid Order Processing

   • Standard orders are correctly parsed and saved
   • High-value orders are properly flagged
   • Order items are serialized correctly

2. Invalid Order Handling

   • Negative amounts are rejected
   • Empty customer names are rejected
   • Missing order IDs are rejected

3. Corrupted JSON Handling

   • Malformed JSON is caught and logged
   • Corrupted files are saved to InvalidOrders table

4. Idempotency Testing

   • Duplicate file processing is prevented
   • Database constraints prevent duplicate records

5. High-Volume Testing

   • Generate 1000+ files and verify processing
   • Monitor memory usage and performance
   • Verify no files are lost or duplicated

Manual Testing Scenarios

Test 1: High-Volume Burst

# Generate 1000 files at once
cd OrderProcessor.Generator
dotnet run -- --count 1000 --burst

# Monitor service logs for processing completion

Test 2: Locked File Handling

# Generate a file and immediately open it in a text editor
# Verify the service retries and eventually processes it

Test 3: Mixed File Types

# Generate files with various invalid conditions
dotnet run -- --count 100 --invalid-percentage 0.3 --corrupted-percentage 0.2

Test 4: Service Restart Resilience

# Generate files, stop service mid-processing, restart
# Verify idempotency prevents duplicate processing

📊 Logging

The service uses structured logging with Serilog:

• Console output for development
• Rolling file logs in logs/ directory
• Structured JSON format for easy parsing

Key Log Events

• File detection and queuing
• Processing start/completion
• Validation failures
• Database operations
• Error conditions and retries

Log Levels

• Information: Normal operations
• Warning: Validation failures, file locks
• Error: Processing failures, database errors
• Critical: Service startup/shutdown issues

⚡ Performance Characteristics

• Concurrent processing: Up to Environment.ProcessorCount * 2 files simultaneously
• Memory efficient: Streaming file processing, no large in-memory collections
• Resilient: Exponential backoff retry for locked files
• Scalable: Event-driven architecture handles burst loads effectively

⚙️ Configuration

Connection Strings

Modify appsettings.json to change the database location:

{
  "ConnectionStrings": {
    "DefaultConnection": "Data Source=C:\\Data\\orders.db"
  }
}

Logging Configuration

Adjust log levels and outputs in appsettings.json:

{
  "Serilog": {
    "MinimumLevel": {
      "Default": "Information"
    }
  }
}

🔍 Monitoring and Troubleshooting

Common Issues

1. Files not processing: Check directory permissions and file locks
2. Database errors: Verify SQLite file permissions and disk space
3. High memory usage: Monitor concurrent processing limits
4. Service won't start: Check Windows Event Log for detailed errors

Health Checks

Monitor these indicators:

• Log file growth and error rates
• Database file size growth
• IncomingOrders directory file count
• Windows Service status

🚧 Future Improvements

With more time, I would implement:

1. Health Check Endpoints: HTTP endpoints for monitoring service health and metrics
2. Configuration Hot-Reload: Dynamic configuration updates without service restart
3. Distributed Processing: Support for multiple service instances with coordination
4. Advanced Retry Policies: Configurable retry strategies with dead letter queues
5. Metrics and Telemetry: Integration with Application Insights or Prometheus
6. File Archival: Automatic cleanup/archival of processed files
7. Database Migrations: Proper EF Core migrations for schema evolution
8. Performance Counters: Windows Performance Counters for system monitoring

📁 Sample Files

The SampleOrders/ directory contains example files demonstrating:

• Valid standard orders
• High-value orders (>$1000)
• Invalid orders (negative amounts, empty customer names)
• Corrupted JSON files

🤝 Contributing

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add some amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

📞 Contact

Project Link: https://github.com/NoCap01/high-performance-order-processor