Enhancement: Improved UPF routing and NAT handling #38

hxngillani · 2025-02-26T22:58:45Z

Enhancement: Dynamic UPF Routing and NAT Handling

Changes Made:

Replaced static aether-ue-nat.service with aether-ue-nat.service.j2 for dynamic NAT configuration..
Updated install.yaml to generate the NAT service dynamically based on UPF configuration.
Improved 20-aether-core.network to dynamically add routes for additional UPFs.
Modified the UPF module to check for existing routes before adding, preventing duplicate route errors.
Ensured a consistent and streamlined route management approach between the core network and UPF installation.

Why This Change?

The original aether-ue-nat.service was static, meaning new UPFs required manual modifications.
Using Jinja2 templating allows the NAT service to automatically adapt to new UPFs.
Previously, additional UPFs required route configuration, which is being done in UPF role in install task which near me should be part of router role.
The new approach automates UPF routing in router role section
Ensures that UPF routes do not fail with RTNETLINK answers: File exists when adding new UPFs.

Testing & Validation:

Successfully deployed Aether 5GC with multiple UPFs without issues.
Verified dynamic generation of NAT service using Jinja2.
Ensured UPF routing was correctly applied without requiring manual intervention.
No duplicate route errors occurred when adding multiple UPFs.

Notes for Reviewers:

Please verify if moving route handling to the router section aligns with project best practices.
Open to feedback on improving the logic for route management.

gab-arrobo · 2025-02-27T01:23:47Z

Hi @hxngillani, thanks for your contribution

Hi @llpeterson, @mbilal92, can you please review this PR? Thanks!

gab-arrobo · 2025-02-27T18:40:11Z

roles/router/templates/systemd/aether-ue-nat.service.j2

+    {% if '1' in core.upf.additional_upfs %} \
+    sudo iptables -t nat -C POSTROUTING -s {{ core.upf.additional_upfs['1'].ue_ip_pool }} -o {{ core.data_iface }} -j MASQUERADE || \
+    sudo iptables -t nat -A POSTROUTING -s {{ core.upf.additional_upfs['1'].ue_ip_pool }} -o {{ core.data_iface }} -j MASQUERADE; \
+    {% endif %} \
+    {% if '2' in core.upf.additional_upfs %} \
+    sudo iptables -t nat -C POSTROUTING -s {{ core.upf.additional_upfs['2'].ue_ip_pool }} -o {{ core.data_iface }} -j MASQUERADE || \
+    sudo iptables -t nat -A POSTROUTING -s {{ core.upf.additional_upfs['2'].ue_ip_pool }} -o {{ core.data_iface }} -j MASQUERADE; \
+    {% endif %} \


What happens if someone wants to deploy more than 2 additional UPFs? Would it be better to replace these if statements by a for loop?

Thank you for review great idea let me do that and test if it works will then amend in PR

i have updated and tested it is working now for multiple UPFs use case.

gab-arrobo · 2025-03-03T22:36:43Z

roles/upf/tasks/install.yml

+
  # ignore_errors: yes


If ignore_errors is not used, I think it is better to just remove it

Suggested change

# ignore_errors: yes

@gab-arrobo I thought ignore_errors was added for some reason, but it was already there should i remove it ?

@gab-arrobo i for now i have removed as it is not being used

mbilal92 · 2025-03-12T01:20:39Z

roles/router/templates/systemd/20-aether-core.network

 [Route]
 Gateway={{ core.upf.default_upf.ip.core }}
 Destination={{ core.upf.default_upf.ue_ip_pool }}
+
+# Additional UPFs - Dynamically Generated Routes


@hxngillani, this approach works well if you know the number of UPFs to deploy upfront. However, if you add more UPFs later, you’d need to rerun the setup. It would be better to make this independent, allowing users to deploy additional UPFs without reconfiguring the system.

yeah i agree my idea was to put routing related configs in route section because when we enable multiple upfs in main.yml i assumed these will be deployed so route will configure but yeah route config while ups is added is also fine.

i will amend with previous route configuration :)

great, I’d recommend creating a router configuration file as part of the additional UPF role.

@mbilal92 done please check

mbilal92 · 2025-03-12T01:22:24Z

@hxngillani left a comment. Let's keep support to add additional UPFs apart from the router.

gab-arrobo · 2025-03-12T02:03:53Z

roles/router/templates/systemd/20-aether-core.network

@@ -10,4 +10,4 @@ Address={{ core.upf.core_subnet }}

 [Route]
 Gateway={{ core.upf.default_upf.ip.core }}
-Destination={{ core.upf.default_upf.ue_ip_pool }}
+Destination={{ core.upf.default_upf.ue_ip_pool }}


You need to add an empty line to the files that you modified and have the red symbol

gab-arrobo · 2025-03-21T18:12:04Z

@mbilal92, any update on the review for this PR? Thanks!

mbilal92 · 2025-03-24T06:12:10Z

@gab-arrobo, looks good to me.
@hxngillani, could you please write down how you tested it?

hxngillani · 2025-03-24T14:27:28Z

@mbilal92 for testing i used below commands to make sure that UE NAT service is UP and running and MASQUERADE rules show expected subnets and also i used UERANSIM to connected UE with additional upf and i able to access internet successfully.

hassan@Aether:~/aether-onramp$ cat /etc/systemd/system/aether-ue-nat.service 
# Copyright 2022-present Open Networking Foundation
# SPDX-License-Identifier: Apache-2.0

[Unit]
Description=Aether UE NAT Setup
After=network.target

[Service]
Type=oneshot
ExecStart=/bin/bash -c "\
    sudo iptables -t nat -C POSTROUTING -s 172.250.0.0/16 -o enp0s9 -j MASQUERADE || \
    sudo iptables -t nat -A POSTROUTING -s 172.250.0.0/16 -o enp0s9 -j MASQUERADE; \
     \
    sudo iptables -t nat -C POSTROUTING -s 172.248.0.0/16 -o enp0s9 -j MASQUERADE || \
    sudo iptables -t nat -A POSTROUTING -s 172.248.0.0/16 -o enp0s9 -j MASQUERADE; \
     \
"

[Install]
WantedBy=sys-subsystem-net-devices-core.devicehassan@Aether:~/aether-onramp$ 
hassan@Aether:~/aether-onramp$ systemctl status aether-ue-nat.service
○ aether-ue-nat.service - Aether UE NAT Setup
     Loaded: loaded (/etc/systemd/system/aether-ue-nat.service; enabled; vendor preset: enabled)
     Active: inactive (dead) since Mon 2025-03-24 14:02:35 UTC; 14min ago
    Process: 46315 ExecStart=/bin/bash -c      sudo iptables -t nat -C POSTROUTING -s 172.250.0.0/16 >
   Main PID: 46315 (code=exited, status=0/SUCCESS)
        CPU: 17ms

Mar 24 14:02:35 Aether sudo[46316]:     root : PWD=/ ; USER=root ; COMMAND=/usr/sbin/iptables -t nat >
Mar 24 14:02:35 Aether sudo[46316]: pam_unix(sudo:session): session opened for user root(uid=0) by (u>
Mar 24 14:02:35 Aether bash[46317]: MASQUERADE  all opt -- in * out enp0s9  172.250.0.0/16  -> 0.0.0.>
Mar 24 14:02:35 Aether sudo[46316]: pam_unix(sudo:session): session closed for user root
Mar 24 14:02:35 Aether sudo[46318]:     root : PWD=/ ; USER=root ; COMMAND=/usr/sbin/iptables -t nat >
Mar 24 14:02:35 Aether sudo[46318]: pam_unix(sudo:session): session opened for user root(uid=0) by (u>
Mar 24 14:02:35 Aether bash[46319]: MASQUERADE  all opt -- in * out enp0s9  172.248.0.0/16  -> 0.0.0.>
Mar 24 14:02:35 Aether sudo[46318]: pam_unix(sudo:session): session closed for user root
Mar 24 14:02:35 Aether systemd[1]: aether-ue-nat.service: Deactivated successfully.
Mar 24 14:02:35 Aether systemd[1]: Finished Aether UE NAT Setup.

hassan@Aether:~/aether-onramp$ sudo iptables -t nat -L -n -v | grep MASQUERADE
    0     0 MASQUERADE  all  --  *      !docker0  172.17.0.0/16        0.0.0.0/0           
    0     0 MASQUERADE  all  --  *      enp0s9  172.250.0.0/16       0.0.0.0/0           
    0     0 MASQUERADE  all  --  *      enp0s9  172.248.0.0/16       0.0.0.0/0           
    0     0 MASQUERADE  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x2000/0x2000
  440 26846 MASQUERADE  all  --  *      *       10.42.0.0/16        !224.0.0.0/4          /* flanneld masq */ random-fully
    0     0 MASQUERADE  all  --  *      *      !10.42.0.0/16         10.42.0.0/16         /* flanneld masq */ random-fully
    0     0 MASQUERADE  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kubernetes service traffic requiring SNAT */ random-fully
    0     0 MASQUERADE  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:flqWnvo8yq4ULQLa */ match-set cali40masq-ipam-pools src ! match-set cali40all-ipam-pools dst random-fully
hassan@Aether:~/aether-onramp$

jrmcclurg · 2025-04-14T17:12:28Z

Hi all, I ran into this issue (improperly configured NAT for additional UPFs) and found the following simple fix:

Add the following at the end of deps/5gc/roles/upf/tasks/install.yml:

- name: configure NAT for upf traffic
  shell: |
    sudo iptables -t nat -C POSTROUTING -s {{ item.value.ue_ip_pool }} -o {{ core.data_iface }} -j MASQUERADE || sudo iptables -t nat -A POSTROUTING -s {{ item.value.ue_ip_pool }} -o {{ core.data_iface }} -j MASQUERADE
  when: inventory_hostname in groups['master_nodes']
  with_dict: "{{ core.upf.additional_upfs}}"
  become: true
  # ignore_errors: yes

Then, any time a new UPF is added, simply do make 5gc-upf-install.

I'm a bit worried that this PR would require a make 5gc-install followed by make 5gc-upf-install when adding a new UPF.

gab-arrobo · 2025-04-14T17:19:05Z

Hi all, I ran into this issue (improperly configured NAT for additional UPFs) and found the following simple fix:

Add the following at the end of deps/5gc/roles/upf/tasks/install.yml:
- name: configure NAT for upf traffic
  shell: |
    sudo iptables -t nat -C POSTROUTING -s {{ item.value.ue_ip_pool }} -o {{ core.data_iface }} -j MASQUERADE
    sudo iptables -t nat -A POSTROUTING -s {{ item.value.ue_ip_pool }} -o {{ core.data_iface }} -j MASQUERADE
  when: inventory_hostname in groups['master_nodes']
  with_dict: "{{ core.upf.additional_upfs}}"
  become: true
  # ignore_errors: yes
Then, any time a new UPF is added, simply do make 5gc-upf-install.

I'm a bit worried that this PR would require a make 5gc-install followed by make 5gc-upf-install when adding a new UPF.

Hi @hxngillani, what are your thoughts about Jed's comment?

Hi @jrmcclurg, feel free to open a PR with your solution.

jrmcclurg · 2025-04-14T17:22:33Z

Thanks @gab-arrobo. @hxngillani please let me know if my approach above works in your setup. I haven't done extensive testing with it, so I may be missing something. If it works for you, I can make it into a PR.

gab-arrobo · 2025-04-14T17:34:46Z

@hxngillani, please rebase your PR. Thanks!

- Replaced static aether-ue-nat.service with dynamic aether-ue-nat.service.j2 - Updated install.yaml to use Jinja2 templating for NAT service - Improved 20-aether-core.network to dynamically add routes for additional UPFs - Modified add UPF module to check for existing routes before adding - Ensured consistent route management between core network and UPF installation Signed-off-by: hxngillani <[email protected]>

Signed-off-by: hxngillani <[email protected]>

hxngillani · 2025-04-16T06:21:15Z

Thanks @gab-arrobo. @hxngillani please let me know if my approach above works in your setup. I haven't done extensive testing with it, so I may be missing something. If it works for you, I can make it into a PR.

@jrmcclurg Thank you for the solution. can you please try to reboot your system and check if routes that you are configuring in install are persistent?

jrmcclurg · 2025-04-17T18:12:40Z

@hxngillani I can confirm that the NAT routes installed by my fix are not persistent after a restart. However, I have also found that many other routes are also not persistent (e.g., those installed by the many iptables commands executed when performing make 5gc-install, make gnbsim-install, etc.).

So I think your PR is targeting an important issue (persistence of routes), but it only addresses it in the context of UPF-related routes. Much more work would be needed to fully address that issue. More importantly, I am concerned about the need to do make 5gc-install every time a new UPF is added/reconfigured -- that is very slow on my machine. I think the functionality you're proposing would be better in make 5gc-upf-install.

jrmcclurg · 2025-04-18T15:35:13Z

Hi all, I have added my fix as a PR.

hxngillani force-pushed the upf-routing-improvements branch from 081a98a to fd8760a Compare February 26, 2025 23:26

gab-arrobo reviewed Feb 27, 2025

View reviewed changes

hxngillani force-pushed the upf-routing-improvements branch from fd8760a to cea71ab Compare February 27, 2025 19:40

gab-arrobo reviewed Mar 3, 2025

View reviewed changes

mbilal92 reviewed Mar 12, 2025

View reviewed changes

hxngillani force-pushed the upf-routing-improvements branch 3 times, most recently from 470ec9e to fda8928 Compare March 12, 2025 01:53

gab-arrobo requested changes Mar 12, 2025

View reviewed changes

hxngillani force-pushed the upf-routing-improvements branch 3 times, most recently from c537d64 to d73e970 Compare March 12, 2025 13:36

hxngillani added 2 commits April 16, 2025 08:04

UE NAT setup to support dynamic UPF scaling

8985a13

Signed-off-by: hxngillani <[email protected]>

hxngillani force-pushed the upf-routing-improvements branch from d73e970 to 8985a13 Compare April 16, 2025 06:05

Enhancement: Improved UPF routing and NAT handling #38

Are you sure you want to change the base?

Enhancement: Improved UPF routing and NAT handling #38

Conversation

hxngillani commented Feb 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes Made:

Why This Change?

Testing & Validation:

Notes for Reviewers:

Uh oh!

gab-arrobo commented Feb 27, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mbilal92 Mar 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mbilal92 commented Mar 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gab-arrobo commented Mar 21, 2025

Uh oh!

mbilal92 commented Mar 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hxngillani commented Mar 24, 2025

Uh oh!

jrmcclurg commented Apr 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gab-arrobo commented Apr 14, 2025

Uh oh!

jrmcclurg commented Apr 14, 2025

Uh oh!

gab-arrobo commented Apr 14, 2025

Uh oh!

hxngillani commented Apr 16, 2025

Uh oh!

jrmcclurg commented Apr 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jrmcclurg commented Apr 18, 2025

Uh oh!

Uh oh!

hxngillani commented Feb 26, 2025 •

edited

Loading

mbilal92 Mar 12, 2025 •

edited

Loading

mbilal92 commented Mar 12, 2025 •

edited

Loading

mbilal92 commented Mar 24, 2025 •

edited

Loading

jrmcclurg commented Apr 14, 2025 •

edited

Loading

jrmcclurg commented Apr 17, 2025 •

edited

Loading