Skip to content

Latest commit

 

History

History
1272 lines (1018 loc) · 39.5 KB

Architecture.md

File metadata and controls

1272 lines (1018 loc) · 39.5 KB

Project AirSim Complete Architecture Overview

System Overview

Project AirSim is a comprehensive simulation platform that transforms Unreal Engine 5 into a robotics and autonomous systems development environment. It combines photo-realistic 3D rendering, advanced physics simulation, and network-based APIs to create a complete ecosystem for testing and developing autonomous vehicles.

┌─────────────────────────────────────────────────────────────────────────────────┐
│                              PROJECT AIRSIM SYSTEM                              │
│  ┌─────────────────────────────────────────────────────────────────────────┐    │
│  │                    UNREAL ENGINE 5 SIMULATION ENVIRONMENT               │    │
│  │  ┌─────────────────────────────────────────────────────────────────┐    │    │
│  │  │                PROJECTAIRSIM PLUGIN (UE INTEGRATION)            │    │    │
│  │  │  ┌─────────────────────────────────────────────────────────┐    │    │    │
│  │  │  │          SIM LIBS (C++ CORE SIMULATION)                 │    │    │    │
│  │  │  │  ┌─────────────┬─────────────┬─────────────┐            │    │    │    │
│  │  │  │  │  VEHICLE    │   PHYSICS   │  SENSORS    │            │    │    │    │
│  │  │  │  │   APIS      │   ENGINES   │   SYSTEM    │            │    │    │    │
│  │  │  │  └─────────────┴─────────────┴─────────────┘            │    │    │    │
│  │  │  └─────────────────────────────────────────────────────────┘    │    │    │
│  │  └─────────────────────────────────────────────────────────────────┘    │    │
│  └─────────────────────────────────────────────────────────────────────────┘    │
└─────────────────────┬───────────────────────────────────────────────────────────┘
                      │
                      ▼
┌─────────────────────────────────────────────────────────────────────────────────┐
│                          CLIENT APPLICATIONS                                    │
│  ┌─────────────┬─────────────┬─────────────┬─────────────┐                      │
│  │   PYTHON    │     ROS     │   MAVLINK   │   CUSTOM    │                      │
│  │     API     │   BRIDGE    │     GCS     │ CONTROLLERS │                      │
│  └─────────────┴─────────────┴─────────────┴─────────────┘                      │
└─────────────────────────────────────────────────────────────────────────────────┘

Core Architecture Components

1. Unreal Engine 5 Foundation

Blocks Project (unreal/Blocks/)

  • Primary UE5 project containing the simulation environment
  • Configured for UE versions 5.2 and 5.7 with ProjectAirSim plugin enabled
  • Contains simulation levels (BlocksMap.umap, GISMap.umap)
  • Manages content assets and Blueprints

Key Configuration:

{
  "EngineAssociation": "5.7",
  "Plugins": ["ProjectAirSim"],
  "TargetPlatforms": ["WindowsNoEditor", "LinuxNoEditor"]
}

2. ProjectAirSim Plugin (Bridge Layer)

Location: unreal/Blocks/Plugins/ProjectAirSim/

Purpose: Native UE5 plugin that integrates C++ simulation libraries with Unreal's runtime.

Architecture:

Plugin Structure
├── ProjectAirSim.uplugin (Plugin manifest)
├── Source/ProjectAirSim/ (C++ source)
│   ├── Public/ (API headers)
│   └── Private/ (Implementation)
├── SimLibs/ (Compiled C++ DLLs)
├── Content/ (UE assets)
└── Binaries/ (Platform binaries)

Key Classes:

  • AProjectAirSimGameMode: Manages simulation lifecycle
  • AUnrealSimLoader: Loads and manages Sim Libs
  • AUnrealRobot: UE Actor representing robots
  • AUnrealSensor: Base class for sensors

3. Sim Libs (Core Simulation Engine)

Location: projectairsim/ (root CMake project)

Purpose: Cross-platform C++ libraries providing the simulation framework.

Component Architecture:

Sim Libs Modules
├── core_sim/           (Simulation loop, scene management)
├── vehicle_apis/       (Robot controllers: SimpleFlight, ArduPilot, PX4)
├── physics/           (Physics engines: FastPhysics, Matlab)
├── mavlinkcom/        (MAVLink protocol implementation)
├── rendering/         (Rendering components)
├── sensors/           (Sensor simulation)
└── simserver/         (Network server, API management)

System Startup and Initialization

1. UE5 Editor Launch Sequence

UE5 Editor Start
     ↓
Load Blocks.uproject
     ↓
Initialize ProjectAirSim Plugin
     ↓
FProjectAirSimModule::StartupModule()
     ↓
Register AProjectAirSimGameMode
     ↓
Ready for Play Mode

2. Simulation Launch Sequence

Press "Play" in UE Editor
     ↓
AProjectAirSimGameMode::StartPlay()
     ↓
AUnrealSimLoader::LaunchSimulation()
     ↓
├── Configure UE Settings (CustomDepth, etc.)
├── Load SimServer (C++ DLL)
├── Load Scene Configuration (JSON)
├── Spawn UE Actors (Robots, Sensors)
├── Start Network Server (Ports 8989/8990)
└── Begin Physics Simulation Loop

3. Client Connection Sequence

Client Application Starts
     ↓
ProjectAirSimClient.connect()
     ↓
TCP Connection to Ports 8989/8990
     ↓
Authentication (optional)
     ↓
Subscribe to Topics
     ↓
Ready for Control Commands

Data Flow Architecture

Control Flow (Client → Simulation)

Client Command → TCP Socket → SimServer → Vehicle API → Physics Engine → UE Actor Update → Visual Feedback

Detailed Path:

  1. Client Layer: Python/ROS/MAVLink client sends command
  2. Network Layer: pynng TCP transport (ports 8989/8990)
  3. Server Layer: SimServer receives and deserializes MessagePack
  4. API Layer: Vehicle API (MAVLink, ArduPilot, etc.) processes command
  5. Physics Layer: Physics engine applies forces/torques
  6. UE Integration: UE Actor position/orientation updated
  7. Rendering: UE renders new state

Sensor Data Flow (Simulation → Client)

UE Sensor Tick → Data Collection → Serialization → TCP Socket → Client Processing → User Application

Detailed Path:

  1. UE Sensor: Unreal sensor actor samples environment
  2. Data Processing: Sensor data processed by Sim Libs
  3. Serialization: MessagePack serialization
  4. Network: TCP streaming via pynng
  5. Client: Python/ROS client receives data
  6. Application: User code processes sensor data

Communication Architecture

Network Protocol Stack

Application Layer
├── Python API (projectairsim.Client)
├── ROS Bridge (ros2_node.py)
└── MAVLink Protocol (mavlinkcom/)

Transport Layer
├── TCP/IP (Ports 8989/8990)
├── pynng Sockets (NNG library)
└── Message Serialization (MessagePack + JSON)

Simulation Layer
├── SimServer (C++ core)
├── Scene Management
└── API Routing

Message Patterns

Topics (Publish/Subscribe - Port 8989):

  • Sensor Streaming: Camera images, LiDAR, IMU, GPS
  • State Updates: Robot position, velocity, orientation
  • System Events: Scene changes, actor updates

Services (Request/Response - Port 8990):

  • Control Commands: Movement, configuration changes
  • Synchronous Queries: Current state, settings
  • Administrative: Reset, pause, load scene

Component Integration Details

UE5 ↔ Sim Libs Integration

Runtime DLL Loading:

// In UnrealSimLoader.cpp
SimServer = std::make_shared<projectairsim::SimServer>();

// Bind UE callbacks to C++ simulation
SimServer->SetCallbackLoadExternalScene(
    [this]() { LoadUnrealScene(); });

Actor Synchronization:

// UE Actor position updated from physics
void AUnrealRobot::Tick(float DeltaTime) {
    // Get position from Sim Libs physics
    auto pose = VehicleAPI->getPose();

    // Update UE Actor transform
    SetActorLocation(pose.position);
    SetActorRotation(pose.orientation);
}

Physics Integration

Multi-Physics Backend Support:

Physics Engine Options
├── PhysX (UE5 Default)
├── Bullet Physics
├── Custom Physics
└── Runtime Switching

Integration Pattern:

// Physics abstraction in Sim Libs
class PhysicsEngine {
    virtual void update(float dt) = 0;
    virtual void applyForce(Vector3 force) = 0;
};

// UE integration
void AUnrealSimLoader::UpdatePhysics() {
    PhysicsEngine->update(DeltaTime);
    SyncActorTransforms();
}

Development Workflow

1. Build Process

Source Code Changes
     ↓
Build Sim Libs (CMake)
     ↓
build.sh simlibs_debug
     ↓
Build UE Plugin
     ↓
BlocksEditor Win64 DebugGame
     ↓
Launch UE Editor
     ↓
Test in Play Mode

2. Development Environments

Windows Development:

  • Visual Studio 2019/2022 for Sim Libs
  • UE5 Editor for plugin development
  • VS Code multi-root workspace

Linux Development:

  • CMake + Ninja for Sim Libs
  • UE5 Editor for plugin development
  • VS Code with WSL integration

3. Multi-Root Workspace

VS Code Configuration (Blocks.code-workspace):

{
  "folders": [
    {"path": "unreal/Blocks"},
    {"path": "core_sim"},
    {"path": "vehicle_apis"},
    {"path": "physics"},
    {"path": "mavlinkcom"}
  ]
}

API Ecosystem

Python Client API

Architecture:

Python Client Layers
├── ProjectAirSimClient (Network connection)
├── World (Scene management)
├── Drone/Rover (Vehicle control)
└── Sensor Interfaces (Data access)

Usage Pattern:

# Connect and control
client = ProjectAirSimClient()
client.connect()

world = World(client, "scene_config.jsonc")
drone = Drone(client, world, "Drone1")

# Control loop
while True:
    # Send commands
    await drone.move_by_velocity_async(vx, vy, vz)

    # Receive sensor data
    image = drone.get_camera_image("front_camera")
    lidar = drone.get_lidar_data("lidar")

ROS Integration

ROS2 Bridge Architecture:

ROS2 Integration
├── ros2_node.py (ROS2 node implementation)
├── Message Translation (UE ↔ ROS formats)
├── TF Broadcasting (Coordinate transforms)
└── Service Bridges (ROS services ↔ UE)

Topic Mapping:

/airsim/drone/front_camera → sensor_msgs/Image
/airsim/drone/lidar → sensor_msgs/PointCloud2
/airsim/drone/imu → sensor_msgs/Imu
/airsim/drone/cmd_vel → geometry_msgs/Twist

MAVLink Integration

MAVLink Stack:

MAVLink Integration
├── mavlinkcom/ (Protocol implementation)
├── Vehicle APIs (PX4, ArduPilot integration)
├── Ground Control Station support
└── Custom MAVLink dialects

Performance and Scalability

Rendering Optimization

UE5 Features Utilized:

  • Lumen: Dynamic global illumination
  • Nanite: Geometry LOD system
  • Virtual Shadow Maps: Efficient shadowing
  • Custom Depth: Segmentation rendering

Sensor Processing

GPU Acceleration:

  • Compute Shaders: LiDAR point cloud generation
  • Async Tasks: Image compression and formatting
  • Render Targets: Off-screen rendering for cameras

Network Optimization

Efficient Data Streaming:

  • MessagePack: Compact binary serialization
  • Topic Filtering: Selective data subscription
  • Compression: Optional data compression
  • Async Processing: Non-blocking network operations

Deployment and Distribution

Build Targets

Development Builds:

  • BlocksEditor Win64 DebugGame - Editor debugging
  • BlocksEditor Win64 Development - Full UE features

Production Builds:

  • Blocks Win64 Development - Packaged application
  • Blocks Win64 Shipping - Optimized release

Packaging Process

Build Sim Libs → Package Plugin → Cook Content → Package Game → Distribute

Platform Support

Supported Platforms:

  • Windows 11: Full development and runtime
  • Ubuntu 22.04: Full development and runtime
  • Container Support: Docker integration

Advanced Features

Geographic Information Systems

GIS Integration:

  • Cesium Integration: Real-world terrain and imagery
  • Geodetic Conversion: GPS ↔ UE coordinate systems
  • Large World Support: World partitioning for vast areas

Multi-Robot Simulation

Multi-Agent Support:

  • Concurrent Robots: Multiple vehicles in same scene
  • Independent Control: Separate APIs per robot
  • Inter-Agent Communication: Robot-to-robot messaging

Custom Content Pipeline

Asset Integration:

  • FBX Import: 3D model import with physics
  • Material System: Physically-based rendering
  • Blueprint Scripting: Visual logic for custom behaviors

System Requirements and Dependencies

Hardware Requirements

Minimum:

  • CPU: Quad-core 3.0 GHz
  • RAM: 16 GB
  • GPU: GTX 1060 or equivalent
  • Storage: 50 GB SSD

Recommended:

  • CPU: Octa-core 4.0 GHz
  • RAM: 32 GB
  • GPU: RTX 3070 or equivalent
  • Storage: 100 GB NVMe SSD

Software Dependencies

Core Dependencies:

  • Unreal Engine 5.2 and 5.7
  • Visual Studio 2019/2022 (Windows)
  • CMake 3.20+
  • Python 3.7+

Optional Dependencies:

  • ROS2 Humble/Foxy
  • PX4/ArduPilot
  • Cesium for Unreal

Troubleshooting and Debugging

Common Issues

Build Problems:

  • Sim Libs Build: Check CMake configuration
  • UE Plugin: Verify plugin dependencies
  • Network Issues: Check firewall and port availability

Runtime Issues:

  • Connection Failures: Verify ports 8989/8990 available
  • Physics Problems: Check UE physics settings
  • Sensor Issues: Verify sensor configuration in JSON

Debug Tools

UE5 Debugging:

  • Visual Logger: Record and replay simulation
  • Console Commands: Runtime configuration
  • Stat Commands: Performance profiling

Network Debugging:

  • Wireshark: Network traffic analysis
  • Client Logs: Detailed connection logging
  • Server Logs: Sim Libs debug output

Project AirSim Architecture from Unreal Engine Perspective

From the Unreal Engine developer's viewpoint, Project AirSim is a sophisticated plugin ecosystem that transforms UE5 into a robotics simulation platform. The architecture seamlessly integrates C++ simulation libraries with Unreal's visual and physics systems, providing developers with familiar UE tools while enabling complex autonomous systems simulation.

Unreal Engine Integration Architecture

┌─────────────────────────────────────────────────────────────┐
│                Unreal Engine 5.7 Editor                     │
│  ┌─────────────────────────────────────────────────────┐    │
│  │                Blocks Project                       │    │
│  │  ┌─────────────────────────────────────────────┐   │    │
│  │  │        ProjectAirSim Plugin                 │   │    │
│  │  │  ┌─────────────────────────────────────┐    │   │    │
│  │  │  │      Sim Libs Integration         │    │   │    │
│  │  │  │  (C++ DLLs loaded at runtime)    │    │   │    │
│  │  │  └─────────────────────────────────────┘    │   │    │
│  │  └─────────────────────────────────────────────┘   │    │
│  └─────────────────────────────────────────────────────┘    │
└─────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────┐
│              External Client Applications                   │
│  (Python API, ROS, MAVLink GCS, Custom Controllers)         │
└─────────────────────────────────────────────────────────────┘

Core Components in Unreal Engine

1. Blocks Project (Main UE Project)

Location: unreal/Blocks/

Purpose: The primary Unreal Engine project that serves as the simulation environment.

Key Files:

  • Blocks.uproject - Project manifest with plugin dependencies
  • BlocksMap.umap - Main simulation level
  • GISMap.umap - Geographic information system level

Project Configuration:

{
  "EngineAssociation": "5.7",
  "Plugins": ["ProjectAirSim"],
  "TargetPlatforms": ["WindowsNoEditor", "LinuxNoEditor"]
}

2. ProjectAirSim Plugin Architecture

Location: unreal/Blocks/Plugins/ProjectAirSim/

Plugin Structure:

ProjectAirSim Plugin
├── ProjectAirSim.uplugin (Plugin manifest)
├── Source/ProjectAirSim/ (C++ source)
│   ├── Public/ (API headers)
│   └── Private/ (Implementation)
├── SimLibs/ (Compiled C++ DLLs)
├── Content/ (UE assets)
└── Binaries/ (Platform binaries)

Plugin Manifest:

{
  "Modules": [
    {
      "Name": "ProjectAirSim",
      "Type": "Runtime",
      "LoadingPhase": "PostConfigInit"
    }
  ],
  "Plugins": [
    {"Name": "ProceduralMeshComponent", "Enabled": true},
    {"Name": "SunPosition", "Enabled": true},
    {"Name": "PixelStreaming", "Enabled": true}
  ]
}

3. GameMode Integration

Key Class: AProjectAirSimGameMode

Lifecycle Management:

class AProjectAirSimGameMode : public AGameModeBase {
  void StartPlay() override {
    Super::StartPlay();
    UnrealSimLoader.LaunchSimulation(this->GetWorld());
  }

  void EndPlay() override {
    UnrealSimLoader.TeardownSimulation();
  }
};

Responsibilities:

  • Initializes simulation when Play is pressed in UE Editor
  • Manages UnrealSimLoader lifecycle
  • Sets deterministic seed for reproducible simulations

Simulation Loading Architecture

UnrealSimLoader (Core Integration Point)

Class: AUnrealSimLoader

Purpose: Bridges between Unreal Engine and Project AirSim Sim Libs.

Key Methods:

void LaunchSimulation(UWorld* World) {
  // 1. Configure Unreal Engine settings
  SetUnrealEngineSettings();

  // 2. Load simulator with network ports
  SimServer->LoadSimulator(topicsPort, servicesPort, authKey);

  // 3. Load scene configuration
  SimServer->LoadScene();

  // 4. Create Unreal scene actors
  LoadUnrealScene();

  // 5. Start network server
  SimServer->StartSimulator();

  // 6. Begin simulation
  StartUnrealScene();
  SimServer->StartScene();
}

Architecture Flow:

UE Editor Play Button
        ↓
AProjectAirSimGameMode::StartPlay()
        ↓
AUnrealSimLoader::LaunchSimulation()
        ↓
├── Configure UE Settings (CustomDepth, etc.)
├── Load SimServer (C++ DLL)
├── Load Scene Config (JSON)
├── Spawn UE Actors (Robots, Sensors)
├── Start Network Server (Ports 8989/8990)
└── Begin Physics Simulation

Actor Hierarchy in Unreal Engine

Robot Actor System

Unreal Actor Hierarchy
├── AUnrealRobot (Base Robot Class)
│   ├── Skeletal Mesh Component
│   ├── Physics Components
│   └── Sensor Attachments
├── AUnrealRobotLink (Robot Links)
│   ├── Collision Shapes
│   ├── Visual Meshes
│   └── Joint Attachments
└── AUnrealRobotJoint (Robot Joints)
    ├── Constraint Components
    ├── Motor Controllers
    └── State Publishers

Sensor Actor System

Sensor Integration
├── AUnrealSensor (Base Sensor)
│   ├── Tick-based Updates
│   ├── Data Publishing
│   └── Configuration Management
├── Camera Sensors
│   ├── AUnrealCamera
│   ├── AUnrealViewportCamera
│   └── Render Request System
├── Distance Sensors
│   ├── AUnrealLidar (Raycasting + GPU Compute)
│   ├── AUnrealRadar
│   └── AUnrealDistanceSensor
└── Environmental Sensors
    ├── IMU Simulation
    ├── GPS/Geodetic Conversion
    └── Barometer/Altimeter

Rendering and Sensor Pipeline

Camera Rendering Architecture

Camera Pipeline
├── Unreal Camera Actor
│   ├── Scene Capture Component 2D
│   └── Render Target
├── Image Packing (Async Task)
│   ├── GPU → CPU Transfer
│   ├── Format Conversion
│   └── Compression
└── Network Publishing
    ├── MessagePack Serialization
    └── TCP Streaming (Port 8989)

LiDAR Rendering System

LiDAR Pipeline
├── GPULidar Actor
│   ├── Compute Shader (LidarPointCloudCS)
│   ├── Scene View Extension
│   └── Intensity Calculation
├── Point Cloud Generation
│   ├── Ray Marching
│   ├── Distance Calculation
│   └── Intensity Mapping
└── Data Publishing
    ├── Point Cloud Serialization
    └── ROS/MAVLink Bridge

Development Workflow in Unreal Engine

1. Project Setup and Building

Build Targets:

BlocksEditor Win64 Development  - For Editor development
Blocks Win64 Development       - For packaged game builds
BlocksEditor Win64 DebugGame   - For debugging with Sim Libs

Development Cycle:

1. Modify C++ Sim Libs (CMake)
2. Build Sim Libs (build.sh simlibs_debug)
3. Build UE Plugin (BlocksEditor Win64 DebugGame)
4. Launch UE Editor
5. Press Play → Simulation starts
6. Connect client (Python/ROS/MAVLink)
7. Debug and iterate

2. Multi-Root Workspace Development

VS Code Integration:

  • Blocks.code-workspace - Multi-root workspace
  • Simultaneous editing of UE Plugin and Sim Libs
  • Integrated debugging across C++ and Blueprint

Workspace Structure:

Blocks.code-workspace
├── unreal/Blocks/           (UE Plugin & Project)
├── core_sim/               (Simulation Core)
├── vehicle_apis/           (Robot Controllers)
├── physics/                (Physics Engines)
└── mavlinkcom/             (Communication)

3. Content Creation Pipeline

Asset Organization:

Content/
├── BlocksMap.umap          (Main simulation level)
├── GISMap.umap            (Geographic level)
├── Robots/                (Robot Blueprints/Meshes)
├── Environments/          (Scene assets)
└── Sensors/               (Sensor configurations)

Level Design Considerations:

  • Scale: 1 UE unit = 1 cm (for physics accuracy)
  • Lighting: Must support CustomDepth for segmentation
  • Navigation: Recast navmesh for ground robots
  • Streaming: Level streaming for large environments

Plugin Architecture Deep Dive

Module Dependencies

Build.cs Configuration:

public class ProjectAirSim : ModuleRules {
    PublicIncludePaths.AddRange(new string[] {
        "ProjectAirSim/Public"
    });

    PrivateIncludePaths.AddRange(new string[] {
        EngineDirectory + "/Source/Runtime/Renderer/Private"
    });

    // Sim Libs integration
    if (Target.Platform == UnrealTargetPlatform.Win64) {
        PublicIncludePaths.Add(
            PluginDirectory + "/SimLibs/core_sim/include"
        );
        // ... additional Sim Lib paths
    }
}

Runtime Library Loading

Dynamic Library Integration:

Plugin Load Process
├── UE Plugin loads (ProjectAirSim.uplugin)
├── Module initializes (FProjectAirSimModule)
├── Sim Libs DLLs loaded at runtime
├── SimServer instantiated
└── Callbacks bound for scene management

Callback System:

// Bind C++ simulation callbacks to UE functions
SimServer->SetCallbackLoadExternalScene(
    [this]() { LoadUnrealScene(); });

Communication Architecture from UE Perspective

Network Integration

Server-Side Architecture:

Unreal Engine (Server)
├── SimServer (C++ Core)
│   ├── TCP Listener (Ports 8989/8990)
│   ├── Message Deserialization
│   └── Command Processing
├── Unreal Scene
│   ├── Actor Updates
│   ├── Physics Simulation
│   └── Sensor Data Collection
└── Data Publishing
    ├── Topic Broadcasting
    └── Service Responses

Client Connection Flow:

Client Connects → SimServer Accepts → Authentication → Scene Sync → Ready for Commands

Data Flow Architecture

Control Commands (Client → UE)
├── Network Reception (pynng/TCP)
├── Message Deserialization (MessagePack)
├── Command Routing (SimServer)
├── Physics Updates (UE Physics)
└── Visual Feedback (UE Rendering)

Sensor Data (UE → Client)
├── Sensor Sampling (UE Tick)
├── Data Processing (C++ Sim Libs)
├── Serialization (MessagePack)
├── Network Transmission (pynng/TCP)
└── Client Reception

ROS Integration from UE Perspective

ROS Bridge Architecture

ROS Integration Layers
├── UE Plugin (ProjectAirSim)
│   ├── Sensor Data Publishers
│   ├── Control Command Subscribers
│   └── TF Transform Broadcasting
├── ROS2 Node Bridge (Python)
│   ├── rclpy Integration
│   ├── Topic/Message Translation
│   └── Service Proxies
└── ROS Ecosystem
    ├── RViz Visualization
    ├── ROS Control
    └── Navigation Stack

UE-Specific ROS Features

Transform Broadcasting:

  • UE coordinate system to ROS TF
  • Real-time transform updates
  • Geographic coordinate integration

Sensor Integration:

  • UE cameras → ROS image topics
  • UE LiDAR → ROS PointCloud2
  • UE IMU → ROS Imu messages

Development Best Practices

1. Performance Optimization

UE-Specific Considerations:

  • Tick Groups: Use appropriate tick groups for sensors
  • Async Tasks: Offload heavy computations (image packing)
  • Render Threads: Minimize main thread blocking
  • Memory Management: Pool reusable assets

2. Debugging Techniques

UE Editor Debugging:

  • Visualize Sensors: Debug drawing for raycasts/LiDAR
  • Physics Debugging: Show collision shapes and constraints
  • Network Debugging: Log message traffic and timing

Multi-Target Debugging:

  • Attach debugger to UE Editor + Python client simultaneously
  • Cross-reference UE logs with client application logs

3. Content Pipeline

Asset Optimization:

  • LOD System: Configure level-of-detail for performance
  • Texture Streaming: Manage texture memory for large environments
  • Instancing: Use instanced static meshes for repeated assets

Advanced UE Features Integration

1. Rendering Features

Custom Rendering:

  • Scene View Extensions: For specialized sensor rendering (LiDAR intensity)
  • Compute Shaders: GPU-accelerated sensor processing
  • Post-Processing: Custom materials for sensor simulation

2. Physics Integration

UE Physics Extensions:

  • Custom Physics Engines: Runtime switching between physics backends
  • Constraint Systems: Advanced joint and linkage simulation
  • Collision Detection: Custom collision shapes for sensors

3. Multiplayer/Network Features

Distributed Simulation:

  • Pixel Streaming: Web-based visualization
  • Multi-Client Support: Multiple clients connecting simultaneously
  • State Synchronization: Consistent simulation state across clients

Project AirSim Architecture Analysis

Overview

Project AirSim is a simulation platform for drones, robots, and other autonomous systems built on Unreal Engine 5. It consists of C++ simulation libraries, an Unreal Engine plugin, and Python client libraries for API interactions.

High-Level Architecture

┌─────────────────────────────────────────────────────────────┐
│                    User Applications                        │
│  (Python scripts, ROS nodes, MAVLink GCS, etc.)             │
└─────────────────────┬───────────────────────────────────────┘
                      │
                      ▼
┌─────────────────────────────────────────────────────────────┐
│            Project AirSim Client Library                    │
│  (Python API, ROS Bridge, MAVLink Protocol)                 │
└─────────────────────┬───────────────────────────────────────┘
                      │
                      ▼
┌─────────────────────────────────────────────────────────────┐
│              Project AirSim Plugin                          │
│  (Unreal Engine Plugin - Runtime Integration)               │
└─────────────────────┬───────────────────────────────────────┘
                      │
                      ▼
┌─────────────────────────────────────────────────────────────┐
│              Project AirSim Sim Libs                        │
│  (C++ Core Simulation Framework)                            │
└─────────────────────────────────────────────────────────────┘

Detailed Architecture Components

1. Project AirSim Sim Libs (Core Layer)

Location: projectairsim/ (root CMake project)

Purpose: Base infrastructure for defining a generic robot structure and simulation scene tick loop.

Key Components:

  • Core Simulation (core_sim/): Main simulation loop and scene management
  • Physics Integration (physics/): Physics engines and dynamics
  • Vehicle APIs (vehicle_apis/): Robot-specific control interfaces
  • MAVLink Communication (mavlinkcom/): MAVLink protocol implementation
  • Rendering (rendering/): Visual rendering components

Architecture:

Sim Libs
├── Core Simulation Framework
│   ├── Scene Management
│   ├── Robot Definitions
│   └── Simulation Tick Loop
├── Vehicle APIs
│   ├── Multirotor API
│   │   ├── MAVLink API
│   │   ├── ArduCopter API
│   │   ├── SimpleFlight API
│   │   └── Manual Controller API
│   └── Rover API
├── Physics Engines
├── Sensors Framework
└── Communication Layer

2. Project AirSim Plugin (Integration Layer)

Location: projectairsim/unreal/Blocks/Plugins/ProjectAirSim/

Purpose: Host package (currently an Unreal Plugin) that builds on the sim libs to connect external components (controller, physics, rendering) at runtime that are specific to each configured robot-type scenario (ex. quadrotor drones)

3. Project AirSim Client Library (Interface Layer)

Location: projectairsim/client/python/projectairsim/

Purpose: End-user library to enable API calls to interact with the robot and simulation over a network connection

Communication Architecture:

Client Library
├── Connection Management
│   ├── TCP Socket (pynng)
│   ├── Topics (Port 8989) - Pub/Sub
│   └── Services (Port 8990) - Req/Rep
├── API Objects
│   ├── ProjectAirSimClient
│   ├── World
│   └── Drone/Rover
└── Protocol Bridges
    ├── ROS1/ROS2 Bridge
    └── MAVLink Bridge

Drone Interface Architecture

Vehicle Control Hierarchy

Drone Control Interfaces
├── Python API (High-level)
│   ├── Drone Class
│   │   ├── Movement Commands
│   │   │   ├── move_by_velocity_async()
│   │   │   ├── move_to_position_async()
│   │   │   └── rotate_by_yaw_rate_async()
│   │   ├── Sensor Access
│   │   │   ├── get_camera_images()
│   │   │   ├── get_lidar_data()
│   │   │   └── get_imu_data()
│   │   └── State Queries
│   │       ├── get_position()
│   │       ├── get_velocity()
│   │       └── get_orientation()
│   └── World Class
│       ├── Scene Management
│       ├── Weather Control
│       └── Time Management
├── MAVLink Protocol (Low-level)
│   ├── MAVLink API Implementation
│   ├── PX4/ArduPilot Integration
│   └── Ground Control Station Support
└── ROS Integration
    ├── ROS2 Node Bridge
    ├── Topic Publishing
    └── Service Calls

Communication Flow

Control Flow:
User Code → Python API → TCP (pynng) → Unreal Plugin → Sim Libs → Physics Engine

Data Flow:
Sensors → Sim Libs → Unreal Plugin → TCP (pynng) → Python API → User Code

MAVLink Flow:
GCS/ROS → MAVLink Protocol → MAVLink API → Vehicle Controller → Physics

Communication Logic

Network Architecture

Project AirSim uses a sophisticated multi-channel communication system:

1. TCP-based Transport

  • Library: pynng (NNG - nanomsg next generation)
  • Ports:
    • Topics: 8989 (Publish/Subscribe pattern)
    • Services: 8990 (Request/Response pattern)

2. Message Serialization

  • Primary: MessagePack with JSON hybrid (MSGPACK_JSON)
  • Fallback: Pure JSON
  • Security: Optional client authorization with public key tokens

3. Communication Patterns

Topics (Pub/Sub - Port 8989):
├── Sensor Data Streaming
│   ├── Camera Images
│   ├── LiDAR Point Clouds
│   ├── IMU Data
│   ├── GPS Coordinates
│   └── Vehicle State
└── System Events
    ├── Scene Changes
    └── Actor Updates

Services (Req/Rep - Port 8990):
├── Control Commands
│   ├── Movement Instructions
│   ├── Configuration Changes
│   └── Administrative Actions
└── Synchronous Queries
    ├── State Information
    └── Configuration Data

Protocol Stack

Application Layer
├── Python Client API
├── ROS Bridge
└── MAVLink Interface

Transport Layer
├── TCP/IP
├── pynng Sockets
└── Message Serialization

Simulation Layer
├── Unreal Engine Plugin
├── Physics Integration
└── Sensor Simulation

ROS2 Integration

Project AirSim provides comprehensive ROS2 support through a dedicated bridge package.

ROS2 Architecture

ROS2 Integration
├── ROS2 Node Package
│   ├── projectairsim-ros2/
│   │   ├── setup.py
│   │   ├── ros2_node.py
│   │   └── ros1_compatibility.py
│   └── Dependencies
│       └── rclpy
├── Bridge Components
│   ├── Publisher Wrappers
│   ├── Subscriber Wrappers
│   └── Service Proxies
└── Message Translation
    ├── Sensor Data → ROS Messages
    ├── Control Commands ← ROS Messages
    └── TF Transformations

ROS2 Node Implementation

The ROS2 bridge (ros2_node.py) provides:

1. Publisher Implementation

class Publisher:
    - Wraps rclpy.publisher.Publisher
    - Handles topic naming
    - Manages message publishing

2. Subscriber Implementation

class Subscriber:
    - Wraps rclpy.subscription.Subscription
    - Manages topic subscriptions
    - Handles message callbacks

3. Sensor Helper

class SensorHelper:
    - Camera info configuration
    - Message format adaptation
    - ROS2-specific sensor data handling

ROS2 Usage Pattern

ROS2 Node ←→ Project AirSim Bridge ←→ Simulation Server

Topics:
├── /airsim/drone/camera ← Camera images
├── /airsim/drone/imu ← IMU data
├── /airsim/drone/lidar ← LiDAR scans
├── /airsim/drone/odom ← Odometry
└── /airsim/drone/cmd_vel → Velocity commands

Services:
├── /airsim/takeoff → Takeoff service
├── /airsim/land → Landing service
└── /airsim/reset → Reset simulation

Creating Architecture Diagrams

To create visual graphs of this architecture, I recommend using one of these tools:

1. Draw.io (Free, Web-based)

  • Create flowcharts and system diagrams
  • Export as PNG/SVG/PDF
  • Real-time collaboration

2. PlantUML (Text-based)

  • Define diagrams in text
  • Generate from code comments
  • Integrates with documentation

3. Lucidchart or Visio

  • Professional diagramming tools
  • Template libraries
  • Advanced styling options

Suggested Diagram Types

1. System Context Diagram

  • Show Project AirSim in relation to external systems (ROS, MAVLink GCS, etc.)

2. Component Diagram

  • Detail the three main layers and their interactions

3. Sequence Diagrams

  • Show communication flows for specific operations (takeoff, sensor reading, etc.)

4. Deployment Diagram

  • Show how components are distributed across systems

This architecture enables Project AirSim to serve as a versatile platform for autonomous systems development, supporting everything from simple Python scripting to complex ROS-based robotic applications.