Submariner deployment on VPC-native GKE: traffic not forwarded to pods

[etesami]

I’m exploring deploying Submariner on GKE, but I noticed that the GKE quickstart guide
references version 0.8, which is quite outdated.

Specifically, I’m wondering whether Submariner is fully deployable on VPC-native GKE clusters, and if the absence of IP forwarding on GKE nodes could cause issues (ip forwarding disabled on GKE).

In my testing, I deployed Submariner on a single-node VPC-native GKE cluster to connect it to a local cluster. Gateways appear to be connected correctly, and I can reach the gateway node from pods in the remote cluster. However, traffic does not get forwarded to pods that are running on the same gateway node.

GKE version: v1.32.6-gke.1060000
Submariner version: 0.21.0

Cluster setup: 1-node GKE cluster + local k3s cluster

Questions:

• Is missing IP forwarding on GKE nodes expected to cause traffic forwarding issues?
• Are there updated deployment instructions for Submariner on GKE?
• Any guidance on enabling intra-node traffic forwarding on GKE nodes when using Submariner?

Any insights or updated recommendations would be greatly appreciated.

[dfarrell07]

Thanks for the question @etesami. We don't try to keep the version number examples in the docs updated, but do update other details. Yossi is an SME that will look into this.

[yboaron]

Hi @etesami

A. Since you created a single-node cluster, there’s no need for intra-node routing.
Submariner’s handles traffic in the egress direction, and for ingress (after the IPSec packet is decrypted), the CNI is responsible for forwarding the packet to its destination.
To make this work, rp_filter needs to be set to loose mode. Submariner code takes care of that for the supported CNIs,

Which CNI are you using? If you are using GKE CNI , you might need to apply the rp_filter workaround described here: GKE node preparation

B. IIRC, @VaishnaviRajulu successfully deployed Submariner on GKE a few months ago ,
@VaishnaviRajulu can you share how you deployed Submariner on GKE and provide some insights ?

Once you’ve addressed the issue, it would be great if you could submit PR to update GKE deployment section on Submariner documentation.

[VaishnaviRajulu]

Hi @etesami,

I have deployed Submariner on GKE a few months ago, I had given the mentioned the steps in below mentioned issue. Please check.
https://github.com/submariner-io/lighthouse/issues/1789

[etesami]

Thank you, @yboaron and @VaishnaviRajulu

The discussion mentioned above does not address the cluster specification.

I am still not sure if Submariner works with GKE in VPC-native mode. GKE supports two network models: the recommended VPC-native model and the legacy route-based model (referecnes here and here.

I provide more details below to reproduce the issue. Please let me know if I'm missing anything. The following setup creates a GKE cluster in VPC-native mode with two nodes. Despite my first comment, pods on the gateway node are reachable from pods in remote clusters but reachability to pods on nodes other than the gateway is not working.

GKE Setup

PROJECT_ID="replace-with-your-project"
SA_NAME="my-sa-name"  

# network with manual subnets creation
gcloud compute networks create my-vpc \
  --subnet-mode=custom \
  --project=$PROJECT_ID

# subnetwork
gcloud compute networks subnets create my-subnet \
  --network=my-vpc \
  --region=us-central1 \
  --range=10.17.0.0/17 \
  --secondary-range=my-pod-range=10.17.128.0/18,my-service-range=10.17.192.0/18	 \
  --project=$PROJECT_ID

# Create service account
gcloud iam service-accounts create $SA_NAME \
  --project=$PROJECT_ID

# Create service account key
SA_EMAIL="${SA_NAME}@${PROJECT_ID}.iam.gserviceaccount.com"
KEY_FILE="k8s-gke-sa-${SA_NAME}.json"
gcloud iam service-accounts keys create $KEY_FILE \
  --iam-account=$SA_EMAIL \
  --project=$PROJECT_ID

# Create cluster
gcloud container clusters create my-gke-cluster \
  --zone=us-central1-a \
  --num-nodes=2 \
  --enable-ip-alias \
  --network=my-vpc \
  --subnetwork=my-subnet \
  --cluster-secondary-range-name=my-pod-range \
  --services-secondary-range-name=my-service-range \
  --workload-pool=${PROJECT_ID}.svc.id.goog \
  --service-account=${SA_EMAIL}

# IAM policy binding
gcloud projects add-iam-policy-binding ${PROJECT_ID} \
  --member="serviceAccount:${SA_EMAIL}" \
  --role="roles/container.admin"

# Once ready:
gcloud container clusters get-credentials my-gke-cluster --location us-central1-a

# Firewall rules
gcloud compute firewall-rules create "allow-tcp-in" --allow=tcp \
  --direction=IN --source-ranges=10.0.0.0/8 --network=my-vpc
gcloud compute firewall-rules create "allow-udp-in" --allow=udp \
  --direction=IN --source-ranges=10.0.0.0/8 --network=my-vpc
gcloud compute firewall-rules create "allow-icmp-in" --allow=icmp \
  --direction=INGRESS --source-ranges=0.0.0.0/0 --network=my-vpc 

gcloud compute firewall-rules create "allow-tcp-out" --allow=tcp \
  --direction=OUT --destination-ranges=0.0.0.0/0 --network=my-vpc
gcloud compute firewall-rules create "allow-udp-out" --allow=udp \
  --direction=OUT --destination-ranges=0.0.0.0/0 --network=my-vpc

# Using Kubectl get nodes name and set one as gateway
NODE=gke-my-gke-cluster-default-pool-25552bc2-7jmz
kubectl label nodes $NODE "submariner.io/gateway=true"

Submariner Join

Assuming you have a broker ready and deployed somewhere, I use Helm to deploy Submariner:

helm install submariner-operator submariner-latest/submariner-operator \
    --create-namespace \
    --namespace submariner-operator \
    --set ipsec.psk="${PSK}" \
    --set broker.server="${BROKER_URL}" \
    --set broker.token="${TOKEN}" \
    --set broker.namespace="${BROKER_NS}" \
    --set broker.ca="${CA}" \
    --set submariner.serviceDiscovery=false \
    --set submariner.cableDriver=wireguard \
    --set submariner.clusterId="${CLUSTER_ID}" \
    --set submariner.clusterCidr="${CLUSTER_CIDR}" \
    --set submariner.serviceCidr="${SERVICE_CIDR}" \
    --set submariner.natEnabled="true" \
    --set serviceAccounts.lighthouseAgent.create=true \
    --set serviceAccounts.lighthouseCoreDns.create=true \
    --kube-context $CTX

Testing

Output of subctl show all and subctl diagnose all:

subctl show all --context $CTX
 ✓ Detecting broker(s)
 ✓ No brokers found

 ✓ Showing Connections
GATEWAY                        CLUSTER                    REMOTE IP       NAT   CABLE DRIVER   SUBNETS                       STATUS      RTT avg.
skycluster-control-plane       broker         abc.d.abc.abc   yes   wireguard      10.0.32.0/19, 10.0.0.0/19     connected   25.107953ms
example-kube-os-scinet-n9z2k   example-kube-os-scinet-j   abc.a.abc.abc   yes   wireguard      172.16.0.0/16, 10.16.0.0/17   connected   25.282758ms

 ✓ Showing Endpoints
CLUSTER                    ENDPOINT IP    PUBLIC IP        CABLE DRIVER   TYPE
gke                        10.17.0.3      abc.abc.abc.ad   wireguard      local
broker-skycluster          172.18.0.3     abc.d.abc.abc    wireguard      remote
example-kube-os-scinet-j   10.16.128.16   abc.d.abc.abc    wireguard      remote

 ✓ Showing Gateways
NODE                             HA STATUS   SUMMARY
gke-my-gke-cluster-default-poo   active      All connections (2) are established

 ✓ Showing Network details
    Discovered network details via Submariner:
        Network plugin:  generic
        Service CIDRs:   [10.17.192.0/18]
        Cluster CIDRs:   [10.17.128.0/18]

 ✓ Showing versions
COMPONENT                  REPOSITORY           CONFIGURED   RUNNING                     ARCH
submariner-gateway         quay.io/submariner   0.21.0       release-0.21-2f0111a4d643   amd64
submariner-routeagent      quay.io/submariner   0.21.0       release-0.21-2f0111a4d643   amd64
submariner-metrics-proxy   quay.io/submariner   0.21.0       release-0.21-f8d812e114a6   amd64
submariner-operator        quay.io/submariner   0.21.0       release-0.21-134dda265e13   amd64

subctl diagnose all --context $CTX
 ✓ Checking Submariner support for the Kubernetes version
 ✓ Kubernetes version "v1.33.3-gke.1136000" is supported
   
 ✓ Non-Globalnet deployment detected - checking that cluster CIDRs do not overlap
 ✓ Checking DaemonSet "submariner-gateway"
 ✓ Checking DaemonSet "submariner-routeagent"
 ✓ Checking DaemonSet "submariner-metrics-proxy"
 ✓ Checking the status of all Submariner pods
 ✓ Checking that gateway metrics are accessible from non-gateway nodes
 ✓ Skipping this check as it's a single node cluster
   
 ⚠ Checking Submariner support for the CNI network plugin
 ⚠ Submariner could not detect the CNI network plugin and is using ("generic") plugin. It may or may not work.
 ✓ Checking gateway connections
 ✓ Retrieving route agent connections
 ✓ Checking remote connections for route agent "gke-my-gke-cluster-default-pool-25552bc2-7jmz"
 ✓ Checking Submariner support for the kube-proxy mode
 ✓ The kube-proxy mode is supported
 ✓ Checking that firewall configuration allows intra-cluster VXLAN traffic
 ✓ Skipping this check as it's a single node cluster
   
 ⚠ Submariner service discovery feature is not installed

With this setup, you cannot reach pods on nodes other than the gateway.

[etesami]

Adding to the setup above, please note that rp_filter workaround is already applied but there is still no reachability to pods on nodes other than the gateway. For reference, here is the command:

NODES=(
  gke-my-gke-cluster-default-pool-25552bc2-7jmz
  gke-my-gke-cluster-default-pool-1a143183-abcd
)

for NODE in "${NODES[@]}"; do
  kubectl --v=4 run netshoot-hostmount-$(uuidgen) --overrides='{
    "spec": {
      "hostNetwork": true,
      "nodeName": "'$NODE'",
      "containers": [{
        "args": [
          "/bin/bash", "-c",
          "echo 2 > /proc/sys/net/ipv4/conf/all/rp_filter"
        ],
        "stdin": true,
        "stdinOnce": true,
        "terminationMessagePath": "/dev/termination-log",
        "terminationMessagePolicy": "File",
        "tty": true,
        "securityContext": {
          "allowPrivilegeEscalation": true,
          "privileged": true,
          "runAsUser": 0,
          "capabilities": {
            "add": ["ALL"]
          }
        },
        "name": "netshoot-hostmount",
        "image": "nicolaka/netshoot",
        "volumeMounts": [{
          "mountPath": "/host",
          "name": "host-slash",
          "readOnly": true
        }]
      }],
      "restartPolicy": "Never",
      "volumes": [{
        "hostPath": {
          "path": "/",
          "type": ""
        },
        "name": "host-slash"
      }]
    }
  }' --image nicolaka/netshoot -- /bin/bash
done

[etesami]

You may also face an issue when deploying Submariner related to the failure in finding the service IP range, as described here. I also want to confirm that if you follow the steps above but create the cluster using legacy route-based networking, Submariner works as expected. Hence, I suspect this issue is due to the disabled IP forwarding feature in VPC-native GKE mode.

[VaishnaviRajulu]

Hi @etesami

My GKE cluster is created with VPC-native traffic routing. I found this information under details section of my GKE cluster

VPC-native traffic routing | Enabled

So, GKE works with Submariner in VPC-native traffic routing

[yboaron]

@etesami

Adding to the setup above, please note that rp_filter workaround is already applied but there is still no reachability to pods on nodes other than the gateway.

A. Isn’t your GKE cluster a single-node ? Maybe I'm missing something — on which cluster are you seeing the issue with lack of reachability to pods on non-gateway nodes?

B. I also noticed you have 3 clusters in your clusterset [1]. do you see the same connectivity issue when testing between the other two clusters as well?

[1]
gke 10.17.0.3 abc.abc.abc.ad wireguard local
broker-skycluster 172.18.0.3 abc.d.abc.abc wireguard remote
example-kube-os-scinet-j 10.16.128.16 abc.d.abc.abc wireguard remote

[yboaron]

@etesami Any updates on this issue?

[etesami]

I’m sorry, I’ve had other priorities recently. Based on my latest observations, the issue appears to be related to GKE VPC-native mode and the automatically inserted routes in the nodes’ routing tables. This causes intra-cluster traffic to bypass the VXLAN tunnels and instead be routed through the default gateway. Since IP forwarding is disabled in VPC-native mode, traffic from gateway nodes cannot reach other nodes. I’ll aim to provide details for reproducing the environment within the next couple of weeks.