Skip to content

Commit 3931b11

Browse files
odincodeshenodincodeshen
committed
OpenAD Kit#2 multiple AWS
1 parent d32b890 commit 3931b11

File tree

1 file changed

+262
-4
lines changed
  • content/learning-paths/automotive/openadkit2_virtualplatform

1 file changed

+262
-4
lines changed

‎content/learning-paths/automotive/openadkit2_virtualplatform/3_aws_setup.md‎

Lines changed: 262 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -6,11 +6,269 @@ weight: 4
66
layout: learningpathall
77
---
88

9-
## System Architecture and Component Design
9+
### System Architecture and Component Design
1010

11-
### Deployment of DDS in AWS Environment
11+
Once we understand Safety Island —- the safety-critical subsystem responsible for executing essential control logic in automotive systems—and the DDS (Data Distribution Service) communication mechanism, we can refactor the original OpenAD Kit architecture, which previously ran on a single instance, by splitting it into two independent execution environments.
1212

13-
### Network Latency and Performance Optimization Strategies
13+
In the previous [learning path](http://learn.arm.com/learning-paths/automotive/openadkit1_container/), OpenAD Kit launched three containers on the same hardware, each responsible for the `Simulation environment`, `Visualization` and `Planning-Control` components.
1414

15-
### Advantages and Challenges of This Solution
15+
In this session, we decouple the simulation and visualization system from the on-vehicle autonomous driving stack, deploying them on two separate Arm-based instances. Communication between the two is facilitated through the ROS 2 software architecture, using DDS as the underlying middleware, which enables real-time data exchange and modular integration across distributed systems.
16+
17+
This architecture brings several practical benefits:
18+
19+
- Improved system stability: Decoupling reduces the risk of resource contention or runtime interference between visualization workloads and safety-critical components.
20+
21+
- Leverages DDS’s QoS and discovery features: Ensures robust, real-time, and scalable communication across nodes without requiring a central broker or manual configuration.
22+
23+
- Better scalability and performance tuning: Each instance can be provisioned with resources optimized for its specific task.
24+
25+
- Supports modular development and CI/CD: Teams can develop, test, and iterate on components independently, enabling better DevOps practices for autonomous systems.
26+
27+
### Networking Setting
28+
29+
First, launch two Arm instances (either cloud instances or on-premise servers).
30+
31+
The specifications of the two Arm instances don’t need to be identical. In my tests, 16 CPUs and 32GB of RAM have already provided good performance.
32+
33+
Once the two machines are up and running, you need to decide where the `Planning-Control` will execute. After making the decision, the other machine will run `Simulation environment` and `Visualization`.
34+
35+
In order for the two instances to communicate, you must configure them to allow network access to each other. For example, in AWS EC2, both instances should belong to the same security group.
36+
37+
In the AWS EC2 Security Groups inbound rules setting, ensure that there is a rule allowing traffic from other members of the same security group (i.e., the security group itself as a source). Outbound traffic is typically allowed by default and usually does not require modification.
38+
This setup ensures that both EC2 instances can discover and communicate with each other over the network as required by ROS 2 and DDS.
39+
40+
Once both of your machines are set up, please note down the IP addresses of both machines, as you will need them for the upcoming configuration.
41+
42+
### New Docker YAML Configur Setting
43+
44+
Ensure that Docker is installed on your development environment, then clone the same repository from the previous learning path onto both machines.
45+
46+
```bash
47+
git clone https://github.com/autowarefoundation/openadkit_demo.autoware.git
48+
49+
cd openadkit_demo.autoware
50+
cp docker/docker-compose.yml docker/docker-compose-2ins.yml
51+
touch docker/cycloneDDS.xml
52+
```
53+
54+
First, you need create xml file called `cycloneDDS.xml`
55+
56+
This CycloneDDS XML configuration file is used to customize the behavior of the CycloneDDS middleware, which is used in ROS2 for inter-process communication (IPC) and network communication over DDS.
57+
Please replace the previously written IP addresses of the two machines with the 192.168.xx.yy and 192.168.aa.bb in the configuration.
58+
59+
```xml
60+
<CycloneDDS>
61+
<Domain>
62+
<General>
63+
<AllowMulticast>false</AllowMulticast>
64+
<Interfaces>
65+
<NetworkInterface
66+
autodetermine="false"
67+
name="ens5"
68+
priority="default"
69+
multicast="false"
70+
/>
71+
</Interfaces>
72+
</General>
73+
<Discovery>
74+
<MaxAutoParticipantIndex>1000</MaxAutoParticipantIndex>
75+
<ParticipantIndex>auto</ParticipantIndex>
76+
<Peers>
77+
<Peer address="192.168.xx.yy"/>
78+
<Peer address="192.168.aa.bb"/>
79+
</Peers>
80+
</Discovery>
81+
<Tracing>
82+
<OutputFile>/root/workspace/cyclonelog.log</OutputFile>
83+
<Verbosity>config</Verbosity>
84+
</Tracing>
85+
</Domain>
86+
</CycloneDDS>
87+
```
88+
89+
{{% notice Note %}}
90+
You can find the more detail about CycloneDDS setting [Configuration](https://cyclonedds.io/docs/cyclonedds/latest/config/config_file_reference.html#cyclonedds-domain-internal-socketreceivebuffersize)
91+
{{% /notice %}}
92+
93+
Next, we need to configure the newly created YML file `docker-compose-2ins.yml`.
94+
95+
- Use the host network instead of the docker network bridge.
96+
This is necessary for allowing the all of three of containers to access the host network interfaces.
97+
98+
```YAML
99+
visualizer:
100+
network_mode: host
101+
```
102+
103+
- Add the newly created XML file as an environment variable in each container, and mount the current folder as `/root/workspace`. Configure the environment variable CYCLONEDDS_URI in the docker-compose.yaml file for all containers to ensure they use this configuration file.
104+
105+
```YAML
106+
volumes:
107+
- .:/root/workspace
108+
environment:
109+
- CYCLONEDDS_URI=/root/workspace/cycloneDDS.xml
110+
```
111+
112+
- Remove the dependency between the `planning-control` and `simulator containers`.
113+
Since these containers will now be launched independently, it is necessary to eliminate the dependency configuration between them.
114+
115+
```YAML
116+
planning-control:
117+
depends_on:
118+
- simulator
119+
```
120+
121+
Here is the complete XML file:
122+
```YAML
123+
services:
124+
simulator:
125+
image: ghcr.io/autowarefoundation/demo-packages:simulator
126+
container_name: simulator
127+
network_mode: host
128+
volumes:
129+
- ./etc/simulation:/autoware/scenario-sim
130+
- .:/root/workspace
131+
- ./log:/root/.ros/log
132+
environment:
133+
- ROS_DOMAIN_ID=88
134+
- RMW_IMPLEMENTATION=rmw_cyclonedds_cpp
135+
- CYCLONEDDS_URI=/root/workspace/cycloneDDS.xml
136+
command: >
137+
ros2 launch scenario_test_runner scenario_test_runner.launch.py
138+
record:=false
139+
scenario:=/autoware/scenario-sim/scenario/yield_maneuver_demo.yaml
140+
sensor_model:=sample_sensor_kit
141+
vehicle_model:=sample_vehicle
142+
initialize_duration:=90
143+
global_timeout:=$TIMEOUT
144+
global_frame_rate:=20
145+
launch_autoware:=false
146+
launch_rviz:=false
147+
148+
planning-control:
149+
image: ghcr.io/autowarefoundation/demo-packages:planning-control
150+
container_name: planning-control
151+
network_mode: host
152+
deploy:
153+
volumes:
154+
- ./etc/simulation:/autoware/scenario-sim
155+
- $CONF_FILE:/opt/autoware/share/autoware_launch/config/planning/scenario_planning/lane_driving/behavior_planning/behavior_path_planner/autoware_behavior_path_static_obstacle_avoidance_module/static_obstacle_avoidance.param.yaml
156+
- $COMMON_FILE:/opt/autoware/share/autoware_launch/config/planning/scenario_planning/common/common.param.yaml
157+
- .:/root/workspace
158+
- ./log:/root/.ros/log
159+
environment:
160+
- ROS_DOMAIN_ID=88
161+
- RMW_IMPLEMENTATION=rmw_cyclonedds_cpp
162+
- CYCLONEDDS_URI=/root/workspace/cycloneDDS.xml
163+
command: >
164+
ros2 launch autoware_launch planning_simulator.launch.xml
165+
map_path:=/autoware/scenario-sim/map
166+
vehicle_model:=sample_vehicle
167+
sensor_model:=sample_sensor_kit
168+
scenario_simulation:=true
169+
rviz:=false
170+
perception/enable_traffic_light:=false
171+
172+
visualizer:
173+
image: ghcr.io/autowarefoundation/demo-packages:visualizer
174+
network_mode: host
175+
container_name: visualizer
176+
volumes:
177+
- ./etc/simulation:/autoware/scenario-sim
178+
- .:/root/workspace
179+
- ./log:/root/.ros/log
180+
ports:
181+
- 6080:6080
182+
- 5999:5999
183+
environment:
184+
- ROS_DOMAIN_ID=88
185+
- VNC_ENABLED=true
186+
- RVIZ_CONFIG=/autoware/scenario-sim/rviz/scenario_simulator.rviz
187+
- NGROK_AUTHTOKEN=${NGROK_AUTHTOKEN}
188+
- NGROK_URL=${NGROK_URL}
189+
- CYCLONEDDS_URI=/root/workspace/cycloneDDS.xml
190+
command: >-
191+
sleep infinity
192+
```
193+
Before moving to the next step, make sure that `docker-compose-2ins.yml` and `cycloneDDS.xml` are already present on both instances.
194+
195+
### Network Latency and Performance Optimization
196+
197+
This distributed network architecture may lead to IP packet fragmentation, which can consume system memory under certain network conditions.
198+
199+
To mitigate issues related to IP fragmentation and socket receive buffer limitations, you can tune the network configuration to support heavy UDP load.
200+
201+
```bash
202+
sudo su
203+
sysctl net.ipv4.ipfrag_time=3
204+
sysctl net.ipv4.ipfrag_high_thresh=134217728
205+
sysctl -w net.core.rmem_max=2147483647
206+
exit
207+
```
208+
209+
To make the configuration permanent across reboot, modify the create the file `/etc/sysctl.d/10-cyclone-max.conf`:
210+
211+
```bash
212+
sudo su
213+
cat << EOF > /etc/sysctl.d/10-cyclone-max.conf
214+
net.core.rmem_max=2147483647
215+
net.ipv4.ipfrag_time=3
216+
net.ipv4.ipfrag_high_thresh=134217728 # (128 MB)
217+
EOF
218+
exit
219+
```
220+
221+
Reference:
222+
- [Autoware dds-setting](https://autowarefoundation.github.io/autoware-documentation/main/installation/additional-settings-for-developers/network-configuration/dds-settings/)
223+
- [ROS2 documentation](https://docs.ros.org/en/humble/How-To-Guides/DDS-tuning.html#cyclone-dds-tuning)
224+
225+
226+
### Execution
227+
228+
Next, you need to configure the execution commands on the different machines.
229+
Copy the corresponding commands into the `openadkit_demo.autoware/docker` directory of two instances.
230+
231+
{{< tabpane code=true >}}
232+
{{< tab header="Planning-Control" language="bash">}}
233+
#!/bin/bash
234+
# Configure the environment variables
235+
export SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
236+
export SCRIPT_DIR="$( pwd )"
237+
export CONF_FILE_PASS=$SCRIPT_DIR/etc/simulation/config/pass_static_obstacle_avoidance.param.yaml
238+
export CONF_FILE_FAIL=$SCRIPT_DIR/etc/simulation/config/fail_static_obstacle_avoidance.param.yaml
239+
export CONF_FILE=$CONF_FILE_FAIL
240+
export COMMON_FILE=$SCRIPT_DIR/etc/simulation/config/common.param.yaml
241+
export NGROK_AUTHTOKEN=$NGROK_AUTHTOKEN
242+
export NGROK_URL=$NGROK_URL
243+
# Start planning-control
244+
echo "Running planning v1.."
245+
CONF_FILE=$CONF_FILE_FAIL docker compose -f "$SCRIPT_DIR/docker-compose-2ins.yml" up planning-control --abort-on-container-exit
246+
{{< /tab >}}
247+
{{< tab header="Visualizer & Simulator" language="bash">}}
248+
#!/bin/bash
249+
# Configure the environment variables
250+
export SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
251+
export SCRIPT_DIR="$( pwd )"
252+
export CONF_FILE_PASS=$SCRIPT_DIR/etc/simulation/config/pass_static_obstacle_avoidance.param.yaml
253+
export CONF_FILE_FAIL=$SCRIPT_DIR/etc/simulation/config/fail_static_obstacle_avoidance.param.yaml
254+
export CONF_FILE=$CONF_FILE_FAIL
255+
export COMMON_FILE=$SCRIPT_DIR/etc/simulation/config/common.param.yaml
256+
export NGROK_AUTHTOKEN=$NGROK_AUTHTOKEN
257+
export NGROK_URL=$NGROK_URL
258+
259+
# Start visualizer and show logs
260+
docker compose -f "$SCRIPT_DIR/docker-compose.yml" up visualizer -d
261+
echo "Waiting 10 seconds for visualizer to start..."
262+
sleep 10
263+
docker compose -f "$SCRIPT_DIR/docker-compose-2ins.yml" logs visualizer
264+
265+
# Start simulator
266+
echo "Running simulator v1.."
267+
TIMEOUT=70 CONF_FILE=$CONF_FILE_FAIL docker compose -f "$SCRIPT_DIR/docker-compose-2ins.yml" up simulator --abort-on-container-exit {{< /tab >}}
268+
{{< /tabpane >}}
269+
270+
271+
Once both machines execute their respective scripts, the visualizer will provide a link that can be accessed through the public IP. When you access the link, you will see that the demo execution will be very similar to the [previous learning path](http://learn.arm.com/learning-paths/automotive/openadkit1_container/4_run_openadkit/).
272+
273+
The only difference is that the containers are distributed across two physical machines, and at the underlying layer of the demo, there will be frequent packet exchanges.
16274

0 commit comments

Comments
 (0)