Problems with Networking & multiple SGs in PodSecurityGroups

[armenr]

I'm having one heck of a time understanding how to configure EKS networking. It's really difficult to make sense of the documentation against my requirements. I've been fighting this problem for days.

Context

ALL Nodes & Pods in our infra run in private subnets. This isn't going to change, ever.

VPC CNI is configured as follows:

      kubectl set env daemonset aws-node -n kube-system \
        AWS_VPC_K8S_CNI_EXTERNALSNAT=true \
        ENABLE_POD_ENI=true \
        ENABLE_PREFIX_DELEGATION=true \
        POD_SECURITY_GROUP_ENFORCING_MODE=strict \
        WARM_PREFIX_TARGET=1

There is a "baseline" security group named: core-test-1-eks-node which is attached to all EC2 "worker" nodes as the default and ONLY securityGroup attached to the Node/EC2 itself.

Requirement 1: We want to use PodSecurityGroups alongside this configuration. So, we apply a default PodSecurityGroup manifest in each namespace, and that PodSecurityGroup is given a list, without using any selectors...we want to apply sets of SecurityGroups to whole namespaces.

So, in the "default" PodSecurityGroup manifest for a namespace, we have a list:

• core-test-1-eks-node
• some-specific-security-group-for-more-ports

Example:

• core-test-1-eks-node --> all the standard ingress/egress securityGroupRules you would expect for "ephemeral" traffic and communication with the control plane/masters and NO egress ALL to 0.0.0.0/0
• some-specific-security-group-for-more-ports --> Additional SG that has a rule to permit egress from POD to a specific security group attached to EFS filesystem

Problem(s) / Observed Behavior

Unfortunately, my pods are not able to connect to EFS when "stacking" the PodSecurityGroups this way. The ONLY way I get them to connect is if I edit core-test-1-eks-node and add an egress rule for EFS to the target EFS-attached security group...which tells me that the some-specific-security-group-for-more-ports is not actually being applied/working.

Functionally, I cannot "stack" or "layer" my PodSecurityGroups. I have also tried testing with setting POD_SECURITY_GROUP_ENFORCING_MODE=standard which has not produced any results, either.

Can someone please help me?

[jdn5126]

Closing this discussion as outdated. In the future, please file an issue at https://github.com/aws/amazon-vpc-cni-k8s/issues to get more visibility, as issues are the better place to discuss something like this.