Skip to content

Commit a19e5fb

Browse files
QxBytesQxBytes
authored
feat: Add SNAT bridge to Native, decouple SNAT bridge (#1506)
* Native Endpoint Client Add Endpoints * AddEndpointRules, ConfigureContainerInterfacesAndRoutes * Changed interface names, log statements nw.extIf.Name > eth0 (eth0) eth0.vlanid > eth0.X (eth0.1) %s%s hostIfName > vnet (A1veth0) %s%s-2 contIfName > container (B1veth0) * Renaming, using lib to set ns * Namespace "path" is /var/run/netns/<NS> * Loopback set up, Remove auto kernel subnet route * Cannot set link to up if it's in another NS * Multiple containers on same VNET NS * Delete Endpoint routes on Delete * Minimizing netns usage * Moving NS Exec Code * Further minimized netns.Set usage * Moved helper methods down, drafted tests * Removed DevName from Route Info, more tests * Test existing vnet ns, delete endpoint * NetNS interface for testing * Separated tests by namespace * Endpoints delete if they cannot be moved into NS * Namespace netns tests * Added Native Client to deleteEndpointImpl * Deletion of Endpoints Impl and Tests * Cleaned code (Tests ok) * Moved mock/netns to package (Tests ok) * Fixing Netns (wip) Moved netnsinterface to consumer package (network). Removed "Netns" from "NewNetns" and "NewMockNetns" as it is unambiguous. Changed uintptr to int and casted the int to uintptr when needed later. * Using errors.Wrap for error context (wip) * Removed sentence case (wip) * Removing variable predeclaration * Removed NewNativeEndpointClient Directly instantiating struct because nothing special happens in NewNativeEndpointClient * Removed generics from ExecuteInNS * Removed uintptr from mocknetns, tests compile Forgot to remove uintptr from mocknetns * Fix tests, lint * Fixes from linter Works on VMSS * Replacing references to ethX with vlan veth * Removed unnecessary log * Removed unnecessary mac, fix tests * Mockns method name enum * Unable to use GetNetworkInterfaceByName due to NS If I use GetNetworkInterface, I need to be in the vnet NS, but that means I will need to call ExecuteInNS, which causes tests to fail. * Fixes from linter * Assume if NS exists, vlan veth exists Tests ok * Fixes for Linter * Snat refactor * Fix delete tests * Fix delete tests bug * More snat refactor * Breaking, prepping for Native Snat Delete native endpoint snat route linux to remove errors and in theory, ovs should work fine again. * Go mod tidy for linting Hopefully this fixes the windows lint error * Add fields to native endpoint client for snat * Using New() func to create Native Client Creation of the native endpoint client is too complicated to directly instantiate. * Snat defaults * Insert SNAT entry points * Native Snat error handling * Breaking, decouple ovsctl from snat Proposed Solution implementation Moved ovsctlClient.AddPortOnOVSBridge to ovs_endpoint_snatroute_linux.go. Removed ovsctlclient from NewSnatClient. Removed ovsctlClient from testing file. * Delete unecessary ovssnat files * No lint on vishvananda netns Maybe this will fix the windows linter? * Build linux only for netns package Maybe this fixes the linter error? * Remove nolint to see if linter fails * Breaking, removed bridgeName bridgeName refers to the OVS Switch I believe * If native uses snat bridge, should also get IP * Breaking, Decouple or Wrap snat route * Check to see if snat triggered * Snat behaviors specific to ovs/native * Pass the pointer Add/Delete ok * Renaming to make consts public * Breaking, moving ovs specific parts of snat to ovs * Remove enable infra vnet (Tests ok) Tested: Allow Host to NC only Allow NC to Host only Allow both Wget Ping between containers Warning: Enable snat is still hard coded to true!!! * Move add port to after exists() check * Moved netns interface to caller, generalized tests Tests ok, Native ok * Typos * Reordered if statement, unwrapped arp Tests ok, ping ok, wget ok * Linted, wrapping errors * Go fumpt entire network package * Code markers removed, clean (Tests ok) OVS & Native: - Ping between two containers same VM, no packets on bridge - Ping between two containers diff VM, no packets on bridge - Ping other container not in vnet, no packets on bridge - Ping snat to container, packets on bridge - Ping container to snat, packets on bridge - Tcpdump confirmed on azSnatBr - Deletion of containers deletes appropriate interfaces * Renamed veth, fixed logs * Made deleteEndpoints logic clearer, renamed error * Renamed eth0 to primaryHostIfName, vlanEth to vlanIf * Deleted debug log * Corrected merge (hardware addr) (Tests ok) * Renamed vlan veth to hostExtIf_vlanID, Disabled RA eth0.2 makes disable RA look for a folder eth0 and then another sub folder "2". ("eth0/2") However, it should look for a folder named "eth0.2" literally. To solve this, we change the naming scheme to use an underscore instead. (Tests ok) * Renamed Native to TransparentVlan Confirmed basic functionality on VM with correct mode * Make file updated * Create azure-windows-multitenancy-transparent-vlan.conflist * Unified snat err format * Rename to transparent-vlan * Route table support added to local netlink * Moved SNAT to end of function * Defer deleting vlan interface on failure
1 parent 717b7a0 commit a19e5fb

15 files changed

+619
-299
lines changed

‎Makefile‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -48,6 +48,7 @@ CNM_BUILD_DIR = $(BUILD_DIR)/cnm
4848
CNI_BUILD_DIR = $(BUILD_DIR)/cni
4949
ACNCLI_BUILD_DIR = $(BUILD_DIR)/acncli
5050
CNI_MULTITENANCY_BUILD_DIR = $(BUILD_DIR)/cni-multitenancy
51+
CNI_MULTITENANCY_TRANSPARENT_VLAN_BUILD_DIR = $(BUILD_DIR)/cni-multitenancy-transparent-vlan
5152
CNI_SWIFT_BUILD_DIR = $(BUILD_DIR)/cni-swift
5253
CNI_OVERLAY_BUILD_DIR = $(BUILD_DIR)/cni-overlay
5354
CNI_BAREMETAL_BUILD_DIR = $(BUILD_DIR)/cni-baremetal
@@ -77,6 +78,7 @@ CNM_ARCHIVE_NAME = azure-vnet-cnm-$(GOOS)-$(GOARCH)-$(VERSION).$(ARCHIVE_EXT)
7778
CNI_ARCHIVE_NAME = azure-vnet-cni-$(GOOS)-$(GOARCH)-$(VERSION).$(ARCHIVE_EXT)
7879
ACNCLI_ARCHIVE_NAME = acncli-$(GOOS)-$(GOARCH)-$(VERSION).$(ARCHIVE_EXT)
7980
CNI_MULTITENANCY_ARCHIVE_NAME = azure-vnet-cni-multitenancy-$(GOOS)-$(GOARCH)-$(VERSION).$(ARCHIVE_EXT)
81+
CNI_MULTITENANCY_TRANSPARENT_VLAN_ARCHIVE_NAME = azure-vnet-cni-multitenancy-transparent-vlan-$(GOOS)-$(GOARCH)-$(VERSION).$(ARCHIVE_EXT)
8082
CNI_SWIFT_ARCHIVE_NAME = azure-vnet-cni-swift-$(GOOS)-$(GOARCH)-$(VERSION).$(ARCHIVE_EXT)
8183
CNI_OVERLAY_ARCHIVE_NAME = azure-vnet-cni-overlay-$(GOOS)-$(GOARCH)-$(VERSION).$(ARCHIVE_EXT)
8284
CNI_BAREMETAL_ARCHIVE_NAME = azure-vnet-cni-baremetal-$(GOOS)-$(GOARCH)-$(VERSION).$(ARCHIVE_EXT)
@@ -439,6 +441,15 @@ ifeq ($(GOOS),linux)
439441
endif
440442
cd $(CNI_MULTITENANCY_BUILD_DIR) && $(ARCHIVE_CMD) $(CNI_MULTITENANCY_ARCHIVE_NAME) azure-vnet$(EXE_EXT) azure-vnet-ipam$(EXE_EXT) azure-vnet-telemetry$(EXE_EXT) 10-azure.conflist azure-vnet-telemetry.config
441443

444+
$(MKDIR) $(CNI_MULTITENANCY_TRANSPARENT_VLAN_BUILD_DIR)
445+
cp cni/azure-$(GOOS)-multitenancy-transparent-vlan.conflist $(CNI_MULTITENANCY_TRANSPARENT_VLAN_BUILD_DIR)/10-azure.conflist
446+
cp $(CNI_BUILD_DIR)/azure-vnet$(EXE_EXT) $(CNI_BUILD_DIR)/azure-vnet-ipam$(EXE_EXT) $(CNI_MULTITENANCY_TRANSPARENT_VLAN_BUILD_DIR)
447+
ifeq ($(GOOS),linux)
448+
cp telemetry/azure-vnet-telemetry.config $(CNI_MULTITENANCY_TRANSPARENT_VLAN_BUILD_DIR)/azure-vnet-telemetry.config
449+
cp $(CNI_BUILD_DIR)/azure-vnet-telemetry$(EXE_EXT) $(CNI_MULTITENANCY_TRANSPARENT_VLAN_BUILD_DIR)
450+
endif
451+
cd $(CNI_MULTITENANCY_TRANSPARENT_VLAN_BUILD_DIR) && $(ARCHIVE_CMD) $(CNI_MULTITENANCY_TRANSPARENT_VLAN_ARCHIVE_NAME) azure-vnet$(EXE_EXT) azure-vnet-ipam$(EXE_EXT) azure-vnet-telemetry$(EXE_EXT) 10-azure.conflist azure-vnet-telemetry.config
452+
442453
$(MKDIR) $(CNI_SWIFT_BUILD_DIR)
443454
cp cni/azure-$(GOOS)-swift.conflist $(CNI_SWIFT_BUILD_DIR)/10-azure.conflist
444455
cp telemetry/azure-vnet-telemetry.config $(CNI_SWIFT_BUILD_DIR)/azure-vnet-telemetry.config

‎cni/azure-linux-multitenancy-transparent-vlan.conflist‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
{
2+
"cniVersion":"0.3.0",
3+
"name":"azure",
4+
"plugins":[
5+
{
6+
"type":"azure-vnet",
7+
"mode":"transparent-vlan",
8+
"bridge":"azure0",
9+
"multiTenancy":true,
10+
"infraVnetAddressSpace":"",
11+
"podNamespaceForDualNetwork":[],
12+
"enableExactMatchForPodName": false,
13+
"enableSnatOnHost":true,
14+
"ipam":{
15+
"type":"azure-cns"
16+
},
17+
"dns":{
18+
"nameservers":[]
19+
}
20+
},
21+
{
22+
"type":"portmap",
23+
"capabilities":{
24+
"portMappings":true
25+
},
26+
"snat":true
27+
}
28+
]
29+
}

‎cni/azure-windows-multitenancy-transparent-vlan.conflist‎

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
{
2+
"cniVersion": "0.3.0",
3+
"name": "azure",
4+
"plugins": [
5+
{
6+
"type": "azure-vnet",
7+
"mode": "transparent-vlan",
8+
"bridge": "azure0",
9+
"multiTenancy":true,
10+
"enableSnatOnHost":true,
11+
"enableExactMatchForPodName": true,
12+
"capabilities": {
13+
"portMappings": true
14+
},
15+
"ipam": {
16+
"type": "azure-cns"
17+
},
18+
"dns": {
19+
"Nameservers": [
20+
"10.0.0.10",
21+
"168.63.129.16"
22+
],
23+
"Search": [
24+
"svc.cluster.local"
25+
]
26+
},
27+
"AdditionalArgs": [
28+
{
29+
"Name": "EndpointPolicy",
30+
"Value": {
31+
"Type": "OutBoundNAT",
32+
"ExceptionList": [
33+
"10.240.0.0/16",
34+
"10.0.0.0/8"
35+
]
36+
}
37+
},
38+
{
39+
"Name": "EndpointPolicy",
40+
"Value": {
41+
"Type": "ROUTE",
42+
"DestinationPrefix": "10.0.0.0/8",
43+
"NeedEncap": true
44+
}
45+
}
46+
],
47+
"windowsSettings": {
48+
"hnsTimeoutDurationInSeconds" : 120
49+
}
50+
}
51+
]
52+
}

‎network/endpoint.go‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -93,6 +93,7 @@ type RouteInfo struct {
9393
DevName string
9494
Scope int
9595
Priority int
96+
Table int
9697
}
9798

9899
type apipaClient interface {

‎network/endpoint_linux.go‎

Lines changed: 10 additions & 38 deletions
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,6 @@ import (
1313
"github.com/Azure/azure-container-networking/log"
1414
"github.com/Azure/azure-container-networking/netio"
1515
"github.com/Azure/azure-container-networking/netlink"
16-
"github.com/Azure/azure-container-networking/netns"
1716
"github.com/Azure/azure-container-networking/network/networkutils"
1817
"github.com/Azure/azure-container-networking/ovsctl"
1918
"github.com/Azure/azure-container-networking/platform"
@@ -90,25 +89,12 @@ func (nw *network) newEndpointImpl(_ apipaClient, nl netlink.NetlinkInterface, p
9089
}
9190

9291
if vlanid != 0 {
93-
if nw.Mode == opModeNative {
94-
log.Printf("Native client")
95-
vlanVethName := fmt.Sprintf("%s.%d", nw.extIf.Name, vlanid)
96-
vnetNSName := fmt.Sprintf("az_ns_%d", vlanid)
97-
98-
epClient = &NativeEndpointClient{
99-
primaryHostIfName: nw.extIf.Name,
100-
vlanIfName: vlanVethName,
101-
vnetVethName: hostIfName,
102-
containerVethName: contIfName,
103-
vnetNSName: vnetNSName,
104-
nw: nw,
105-
vlanID: vlanid,
106-
netnsClient: netns.New(),
107-
netlink: nl,
108-
netioshim: &netio.NetIO{},
109-
plClient: plc,
110-
netUtilsClient: networkutils.NewNetworkUtils(nl, plc),
92+
if nw.Mode == opModeTransparentVlan {
93+
log.Printf("Transparent vlan client")
94+
if _, ok := epInfo.Data[SnatBridgeIPKey]; ok {
95+
nw.SnatBridgeIP = epInfo.Data[SnatBridgeIPKey].(string)
11196
}
97+
epClient = NewTransparentVlanEndpointClient(nw, epInfo, hostIfName, contIfName, vlanid, localIP, nl, plc)
11298
} else {
11399
log.Printf("OVS client")
114100
if _, ok := epInfo.Data[SnatBridgeIPKey]; ok {
@@ -261,25 +247,10 @@ func (nw *network) deleteEndpointImpl(nl netlink.NetlinkInterface, plc platform.
261247
// entering the container netns and hence works both for CNI and CNM.
262248
if ep.VlanID != 0 {
263249
epInfo := ep.getInfo()
264-
if nw.Mode == opModeNative {
265-
log.Printf("Native client")
266-
vlanVethName := fmt.Sprintf("%s.%d", nw.extIf.Name, ep.VlanID)
267-
vnetNSName := fmt.Sprintf("az_ns_%d", ep.VlanID)
268-
269-
epClient = &NativeEndpointClient{
270-
primaryHostIfName: nw.extIf.Name,
271-
vlanIfName: vlanVethName,
272-
vnetVethName: ep.HostIfName,
273-
containerVethName: "",
274-
vnetNSName: vnetNSName,
275-
nw: nw,
276-
vlanID: ep.VlanID,
277-
netnsClient: netns.New(),
278-
netlink: nl,
279-
netioshim: &netio.NetIO{},
280-
plClient: plc,
281-
netUtilsClient: networkutils.NewNetworkUtils(nl, plc),
282-
}
250+
if nw.Mode == opModeTransparentVlan {
251+
log.Printf("Transparent vlan client")
252+
epClient = NewTransparentVlanEndpointClient(nw, epInfo, ep.HostIfName, "", ep.VlanID, ep.LocalIP, nl, plc)
253+
283254
} else {
284255
epClient = NewOVSEndpointClient(nw, epInfo, ep.HostIfName, "", ep.VlanID, ep.LocalIP, nl, ovsctl.NewOvsctl(), plc)
285256
}
@@ -330,6 +301,7 @@ func addRoutes(nl netlink.NetlinkInterface, netioshim netio.NetIOInterface, inte
330301
Priority: route.Priority,
331302
Protocol: route.Protocol,
332303
Scope: route.Scope,
304+
Table: route.Table,
333305
}
334306

335307
if err := nl.AddIPRoute(nlRoute); err != nil {

‎network/endpoint_snatroute_linux.go‎

Lines changed: 100 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,100 @@
1+
package network
2+
3+
import (
4+
"fmt"
5+
6+
"github.com/Azure/azure-container-networking/log"
7+
"github.com/Azure/azure-container-networking/netlink"
8+
"github.com/Azure/azure-container-networking/network/networkutils"
9+
"github.com/Azure/azure-container-networking/network/snat"
10+
"github.com/Azure/azure-container-networking/platform"
11+
"github.com/pkg/errors"
12+
)
13+
14+
func GetSnatHostIfName(epInfo *EndpointInfo) string {
15+
return fmt.Sprintf("%s%s", snatVethInterfacePrefix, epInfo.Id[:7])
16+
}
17+
18+
func GetSnatContIfName(epInfo *EndpointInfo) string {
19+
return fmt.Sprintf("%s%s-2", snatVethInterfacePrefix, epInfo.Id[:7])
20+
}
21+
22+
func AddSnatEndpoint(snatClient *snat.Client) error {
23+
if err := snatClient.CreateSnatEndpoint(); err != nil {
24+
return errors.Wrap(err, "failed to add snat endpoint")
25+
}
26+
return nil
27+
}
28+
29+
func AddSnatEndpointRules(snatClient *snat.Client, hostToNC, ncToHost bool, nl netlink.NetlinkInterface, plc platform.ExecClient) error {
30+
// Allow specific Private IPs via Snat Bridge
31+
if err := snatClient.AllowIPAddressesOnSnatBridge(); err != nil {
32+
return errors.Wrap(err, "failed to allow ip addresses on snat bridge")
33+
}
34+
35+
// Block Private IPs via Snat Bridge
36+
if err := snatClient.BlockIPAddressesOnSnatBridge(); err != nil {
37+
return errors.Wrap(err, "failed to block ip addresses on snat bridge")
38+
}
39+
nuc := networkutils.NewNetworkUtils(nl, plc)
40+
if err := nuc.EnableIPForwarding(snat.SnatBridgeName); err != nil {
41+
return errors.Wrap(err, "failed to enable ip forwarding")
42+
}
43+
44+
if hostToNC {
45+
if err := snatClient.AllowInboundFromHostToNC(); err != nil {
46+
return errors.Wrap(err, "failed to allow inbound from host to nc")
47+
}
48+
}
49+
50+
if ncToHost {
51+
if err := snatClient.AllowInboundFromNCToHost(); err != nil {
52+
return errors.Wrap(err, "failed to allow inbound from nc to host")
53+
}
54+
}
55+
return nil
56+
}
57+
58+
func MoveSnatEndpointToContainerNS(snatClient *snat.Client, netnsPath string, nsID uintptr) error {
59+
if err := snatClient.MoveSnatEndpointToContainerNS(netnsPath, nsID); err != nil {
60+
return errors.Wrap(err, "failed to move snat endpoint to container ns")
61+
}
62+
return nil
63+
}
64+
65+
func SetupSnatContainerInterface(snatClient *snat.Client) error {
66+
if err := snatClient.SetupSnatContainerInterface(); err != nil {
67+
return errors.Wrap(err, "failed to setup snat container interface")
68+
}
69+
return nil
70+
}
71+
72+
func ConfigureSnatContainerInterface(snatClient *snat.Client) error {
73+
if err := snatClient.ConfigureSnatContainerInterface(); err != nil {
74+
return errors.Wrap(err, "failed to configure snat container interface")
75+
}
76+
return nil
77+
}
78+
79+
func DeleteSnatEndpoint(snatClient *snat.Client) error {
80+
if err := snatClient.DeleteSnatEndpoint(); err != nil {
81+
return errors.Wrap(err, "failed to delete snat endpoint")
82+
}
83+
return nil
84+
}
85+
86+
func DeleteSnatEndpointRules(snatClient *snat.Client, hostToNC, ncToHost bool) {
87+
if hostToNC {
88+
err := snatClient.DeleteInboundFromHostToNC()
89+
if err != nil {
90+
log.Errorf("failed to delete inbound from host to nc rules")
91+
}
92+
}
93+
94+
if ncToHost {
95+
err := snatClient.DeleteInboundFromNCToHost()
96+
if err != nil {
97+
log.Errorf("failed to delete inbound from nc to host rules")
98+
}
99+
}
100+
}

‎network/network.go‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -15,11 +15,11 @@ import (
1515

1616
const (
1717
// Operational modes.
18-
opModeBridge = "bridge"
19-
opModeTunnel = "tunnel"
20-
opModeTransparent = "transparent"
21-
opModeNative = "native"
22-
opModeDefault = opModeTunnel
18+
opModeBridge = "bridge"
19+
opModeTunnel = "tunnel"
20+
opModeTransparent = "transparent"
21+
opModeTransparentVlan = "transparent-vlan"
22+
opModeDefault = opModeTunnel
2323
)
2424

2525
const (

‎network/network_linux.go‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -88,8 +88,8 @@ func (nm *networkManager) newNetworkImpl(nwInfo *NetworkInfo, extIf *externalInt
8888
return nil, fmt.Errorf("Ipv6 forwarding failed: %w", err)
8989
}
9090
}
91-
case opModeNative:
92-
log.Printf("Native mode")
91+
case opModeTransparentVlan:
92+
log.Printf("Transparent vlan mode")
9393
ifName = extIf.Name
9494
default:
9595
return nil, errNetworkModeInvalid

0 commit comments

Comments
 (0)