sync CoSys AirSim client and stabilize Docker-to-host RPC mission flow. Fixes #290

[ahmedbektic]

Description

This PR resolves the two blockers from issue #290 and completes the full mission run path (frontend -> backend -> DRV/Cosys-AirSim -> report generation).

• Upgraded the local Python AirSim client layer to the newer Cosys version.
• Fixed backend-to-simulator communication behavior for Docker-on-Windows host execution.
• Added compatibility shims and runtime hardening so mission execution continues instead of failing on API/version mismatches.
• Fixed follow-on monitor/task issues found during end-to-end validation (vehicle naming mismatch, monitor instantiation edge cases, stale task state).

Why

• Root cause 1: the old Microsoft-style AirSim Python layer diverged from Cosys APIs, causing missing/changed methods and unstable behavior.
• Root cause 2: backend RPC connection behavior was brittle in container-to-host mode (wrong/default target assumptions, no readiness gate, and transient startup timing).

Additional blockers discovered during validation:

• Vehicle name mismatch (Drone 1 vs Drone1) caused enableApiControl failures.
• unordered_waypoint_monitor failed on list keys (TypeError: unhashable type: 'list').
• battery_monitor and getTripStats compatibility differences caused repeated runtime failures.
• Settings/materials bootstrap gaps caused simulator initialization issues in some runs.

Where

Cosys AirSim sync + compatibility:

• backend/PythonClient/airsim/client.py:8-23
• backend/PythonClient/airsim/client.py:136-139
• backend/PythonClient/airsim/client.py:1394-1460
• backend/PythonClient/airsim/types.py:442-450
• backend/PythonClient/airsim/init.py:1-5
• backend/PythonClient/airsim/utils.py:1-304
• backend/PythonClient/airsim/pfm.py:1-13

RPC path and connection hardening:

• backend/PythonClient/multirotor/client_factory.py:1-32
• backend/PythonClient/multirotor/control/simulation_task_manager.py:471-509
• docker-compose.dev.yaml:23-25

Mission/monitor robustness:

• backend/PythonClient/multirotor/airsim_application.py:51-99
• backend/PythonClient/multirotor/mission/abstract/abstract_mission.py:16-25
• backend/PythonClient/multirotor/monitor/unordered_waypoint_monitor.py:20-46
• backend/PythonClient/multirotor/monitor/battery_monitor.py:8-64
• backend/PythonClient/multirotor/control/simulation_task_manager.py:93-110
• backend/PythonClient/multirotor/control/simulation_task_manager.py:279-312
• backend/PythonClient/multirotor/control/simulation_task_manager.py:163-178

Settings/bootstrap fixes:

• backend/PythonClient/multirotor/control/simulation_task_manager.py:387-394
• backend/PythonClient/multirotor/control/simulation_task_manager.py:585-594
• backend/PythonClient/multirotor/control/simulation_task_manager.py:675-730
• backend/PythonClient/multirotor/control/jsonFileGenerator.py:15-21
• backend/mock_simulator/mock_task_manager.py:18-24

Tooling/docs:

• backend/PythonClient/check_airsim_api_usage.py:1-160
• readme.md:377-409

Security/logging hardening:

• backend/PythonClient/multirotor/util/geo/geo_util.py:192-208

How

1. Replaced the AirSim Python client layer with the newer Cosys-derived code and aligned imports/exports.
2. Added compatibility shims in the client layer:
   a. simIsPaused() alias for call-site compatibility.
   b. getTripStats() fallback handling for differing RPC signatures and unavailable endpoints.
   c. Default TripStats object return path to avoid hard mission crashes when RPC is unsupported.
3. Added check_airsim_api_usage.py static checker to catch missing airsim.* symbols and missing client methods after vendor upgrades.
4. Introduced a single AirSim client factory and switched mission/stream/wind/control paths to use env-driven RPC target resolution (DRV_UNREAL_HOST, DRV_UNREAL_RPC_PORT).
5. Added explicit RPC readiness polling with timeout/backoff before mission execution to avoid startup race failures.
6. Standardized settings generation for Cosys:
   a. Force SettingsVersion: 2.0.
   b. Seed materials.csv in Documents/AirSim when absent.
   c. Preserve JSON debug override behavior and improve related logs.
7. Hardened mission task lifecycle:
   a. Clear mission/monitor lists on each update and in finally blocks to prevent stale carry-over after exceptions.
   b. Normalize and resolve vehicle names before enableApiControl (Drone 1/Drone1 compatibility).
8. Fixed monitor/runtime issues:
   a. Converted waypoint keys to tuples in unordered waypoint monitor.
   b. Updated battery monitor constructor behavior and added env gate (ENABLE_BATTERY_MONITOR) with safer runtime behavior.
9. Removed sensitive elevation API response logging per CodeQL findings.

Notes

Issue reference: #290 (this PR moves and closes acceptance criteria items for that issue).

Acceptance criteria status:

• AirSim folder updated to newer Cosys client files.
• Communication failure cause identified/documented (container-to-host RPC target/readiness + naming mismatches).
• Communication path fixed (env-based target + readiness + resilient client usage).
• Full simulator run path now executes and generates reports.

Operational defaults after this work:

• Use DRV_UNREAL_HOST=host.docker.internal and DRV_UNREAL_RPC_PORT=41451 for backend-in-container + DRV-on-host development.
• ENABLE_BATTERY_MONITOR now defaults to true; set false on simulator builds that do not reliably expose getTripStats.

Addendum

The sim is now facing this new error:

which is beyond the scope of issue #290.

DRV logs demonstrate the root cause of the on-screen "API call was not received, entering hover mode for safety" is a DRV teardown bug which causes this side effect. The blocker to full stability is now in DRV blueprint/game-mode lifecycle (BP_PIPCamera_C EndPlay path).

Example report and DRV log file:
FlyToPoints_Drone 1_log.txt
DRV.log

• Mission logic is completing before failure. Example report shows normal progression and explicit completion:
  • FlyToPoints_Drone 1_log.txt:1-7 (task over at line 7).
• DRV successfully enters AirSim simulation mode:
  • DRV.log:1049.
• Right after mission completion, DRV tears down the sim world and returns to menu:
  • DRV.log:1132-1135.
• During teardown, Unreal logs a handled ensure tied to BP_PIPCamera_C EndPlay (Super::EndPlay() not called):
  • .DRV.log:1143-1144.
• This pattern is repeated in multiple crash logs, indicating a simulator-side teardown/lifecycle defect rather than primary backend RPC connectivity failure.
• The on-screen API call was not received, entering hover mode for safety is consistent with command stream interruption during/after this teardown path.

[CharlieWells13]

awesome work fixing this