Exception when calling CoreV1Api->create_namespaced_config_map

[chandrapratapsingh08]

@sukumarsengott aws_auth_config.py is failing with the following error:

Exception when calling CoreV1Api->create_namespaced_config_map: (401)
Reason: Unauthorized

Here are the complete lambda failure logs:

---

-k8s-api-1-cluster', 'aws:cloudformation:logical-id': 'ControlPlane', 'alpha.eksctl.io/cluster-name': 'kumo-k8s-api-1', 'aws:cloudformation:stack-id': 'arn:aws:cloudformation:us-east-1:707082674943:stack/eksctl-kumo-k8s-api-1-cluster/db268900-ae82-11ec-b5c9-0a3629edb87f', 'alpha.eksctl.io/eksctl-version': '0.66.0', 'eksctl.cluster.k8s.io/v1alpha1/cluster-name': 'kumo-k8s-api-1'}}}
eks cluster endpoint= https://95A5BA33EC1CA7A25E692CD78839A057.sk1.us-east-1.eks.amazonaws.com
completed creating kube config file.
accessing k8s api through client:
aws auth configMap:{'api_version': 'v1',
'binary_data': None,
'data': {'mapRoles': '- rolearn: '
'arn:aws:iam::707082674943:role/eksctl-kumo-k8s-api-1-cluster-ServiceRole-1XLD1K3X4GENE\n'
' username: system:node:{{EC2PrivateDNSName}}\n'
' groups:\n'
' - system:bootstrappers\n'
' - system:nodes\n'
'- rolearn: '
'arn:aws:iam::707082674943:role/mic-SAMLAdmin\n'
' username: mic-SAMLAdmin\n'
' groups:\n'
' - system:masters\n',
'mapUsers': '- userarn: '
'arn:aws:iam::707082674943:user/chandraprataps@kumolus.com\n'
' username: chandraprataps@kumolus.com\n'
' groups:\n'
' - system:masters\n'},
'kind': 'ConfigMap',
'metadata': {'annotations': None,
'cluster_name': None,
'creation_timestamp': None,
'deletion_grace_period_seconds': None,
'deletion_timestamp': None,
'finalizers': None,
'generate_name': None,
'generation': None,
'initializers': None,
'labels': None,
'managed_fields': None,
'name': 'aws-auth',
'namespace': 'kube-system',
'owner_references': None,
'resource_version': None,
'self_link': None,
'uid': None}}
Exception when calling CoreV1Api->create_namespaced_config_map: (401)
Reason: Unauthorized
HTTP response headers: HTTPHeaderDict({'Audit-Id': '7a1dceb4-116e-4ae3-8edb-d49239593704', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Date': 'Mon, 28 Mar 2022 12:30:15 GMT', 'Content-Length': '129'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
[ERROR] ApiException: (401)
Reason: Unauthorized
HTTP response headers: HTTPHeaderDict({'Audit-Id': '7a1dceb4-116e-4ae3-8edb-d49239593704', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Date': 'Mon, 28 Mar 2022 12:30:15 GMT', 'Content-Length': '129'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}

Traceback (most recent call last):
File "/var/task/lambda_function.py", line 26, in lambda_handler
apply_aws_auth_config_map(aws_session, REGION, account_id, CLUSTER_NAME, CFG_MAP_USER_NAME, CFG_MAP_ROLE_NAME,
File "/var/task/aws_eks_k8s_client.py", line 49, in apply_aws_auth_config_map
apply_configmap(v1, configmapObj)
File "/var/task/aws_auth_config.py", line 68, in apply_configmap
api_response = client_api_instance.create_namespaced_config_map(
File "/var/task/kubernetes/client/api/core_v1_api.py", line 5594, in create_namespaced_config_map
(data) = self.create_namespaced_config_map_with_http_info(namespace, body, **kwargs) # noqa: E501
File "/var/task/kubernetes/client/api/core_v1_api.py", line 5671, in create_namespaced_config_map_with_http_info
return self.api_client.call_api(
File "/var/task/kubernetes/client/api_client.py", line 340, in call_api
return self.__call_api(resource_path, method,
File "/var/task/kubernetes/client/api_client.py", line 172, in __call_api
response_data = self.request(
File "/var/task/kubernetes/client/api_client.py", line 382, in request
return self.rest_client.POST(url,
File "/var/task/kubernetes/client/rest.py", line 272, in POST
return self.request("POST", url,
File "/var/task/kubernetes/client/rest.py", line 231, in request
raise ApiException(http_resp=r)END RequestId: 2765ecc4-6d10-4918-b71e-0596ce25067e
REPORT RequestId: 2765ecc4-6d10-4918-b71e-0596ce25067e Duration: 1517.64 ms Billed Duration: 1518 ms Memory Size: 128 MB Max Memory Used: 127 MB Init Duration: 2560.88 ms

[sukumarsengott]

Hi @chandrapratapsingh08,
Apologies, i could not look into this last week.
Hope you are accessing api with role/user who created the cluster. if not, then request you to access api with user/role that created cluster.
if you are doing this correctly, then can you please let me know if you are using IAM user or IAM role/federated login.
Also, let me know the contents of your requirements.txt.

[chandrapratapsingh007]

Hi Sukumar,

Sorry for the delay in response. I am accessing the API with the same IAM user who created my cluster and trying to add another user with the standalone test.py script. Here is my requirement.txt file:

kubernetes==11.0.0 boto3==1.13.4 botocore==1.16.4

I have generated the session token using the access & secret key of the user created this cluster and then export the access key, secret key and session token received from "aws sts get-session-token" command.

Here is the output of my test.py script execution:

eks-aws-auth-configmap-main/standalone$ python3 test.py
in get_aws_session ... caller identity:::{'UserId': '###############', 'Account': '#############', 'Arn': 'arn:aws:iam::#########:user/###########', 'ResponseMetadata': {'RequestId': '792f11e1-c8f6-471e-b0b3-6d60d3cd8088', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amzn-requestid': '792f11e1-c8f6-471e-b0b3-6d60d3cd8088', 'content-type': 'text/xml', 'content-length': '410', 'date': 'Wed, 27 Apr 2022 08:01:27 GMT'}, 'RetryAttempts': 0}}
sts end point=sts.us-east-1.amazonaws.com
*** eks cluster info::{'ResponseMetadata': {'RequestId': '26da83d7-fe77-4607-a32a-54897f2267d3', 'HTTPStatusCode': 200, 'HTTPHeaders': {'date': 'Wed, 27 Apr 2022 08:01:30 GMT', 'content-type': 'application/json', 'content-length': '2582', 'connection': 'keep-alive', 'x-amzn-requestid': '26da83d7-fe77-4607-a32a-54897f2267d3', 'access-control-allow-origin': '', 'access-control-allow-headers': ',Authorization,Date,X-Amz-Date,X-Amz-Security-Token,X-Amz-Target,content-type,x-amz-content-sha256,x-amz-user-agent,x-amzn-platform-id,x-amzn-trace-id', 'x-amz-apigw-id': 'ROu-KFsNIAMF63g=', 'access-control-allow-methods': 'GET,HEAD,PUT,POST,DELETE,OPTIONS', 'access-control-expose-headers': 'x-amzn-errortype,x-amzn-errormessage,x-amzn-trace-id,x-amzn-requestid,x-amz-apigw-id,date', 'x-amzn-trace-id': 'Root=1-6268f85a-299210195dabc54f4e052694'}, 'RetryAttempts': 0}, 'cluster': {'name': '#######', 'arn': 'arn:aws:eks:us-east-1:#########:cluster/#########', 'createdAt': datetime.datetime(2022, 4, 27, 12, 53, 38, 271000, tzinfo=tzlocal()), 'version': '1.21', 'endpoint': 'https://0660070B658FCA476Bx1E6484707A1C63.gr7.us-east-1.eks.amazonaws.com', 'roleArn': 'arn:aws:iam::##########:role/########', 'resourcesVpcConfig': {'subnetIds': ['##############################'], 'securityGroupIds': ['#####################'], 'clusterSecurityGroupId': '#################', 'vpcId': '###############', 'endpointPublicAccess': True, 'endpointPrivateAccess': False, 'publicAccessCidrs': ['0.0.0.0/0']}, 'logging': {'clusterLogging': [{'types': ['api', 'audit', 'authenticator', 'controllerManager', 'scheduler'], 'enabled': False}]}, 'identity': {'oidc': {'issuer': 'https://oidc.eks.us-east-1.amazonaws.com/id/0660070B658FCA476B1E6484707A1C63'}}, 'status': 'ACTIVE', 'certificateAuthority': {'data': '###################'}, 'platformVersion': 'eks.6', 'tags': {}}}
eks cluster endpoint= https://0660070B658FCA476B1E648470x7A1C63.gr7.us-east-1.eks.amazonaws.com
completed creating kube config file.
get_pods - accessing k8s api through client:
Listing pods with their IPs:
--pod listed---{'found_pod': 'NO'}

[sukumarsengott]

Hi Chandrapratap,
I could reproduce the issue you have mentioned.

When you run standalone program, it has worked because you have run it in the context of the user who has created the cluster and in this case you have exported credentials of user who has created cluster.

When running from Lambda, it uses Lambda's execution role. This role should have access to the cluster. As mentioned in "Pre requisites" section of readme, "manually create/edit EKS aws auth ConfigMap to provide permission for the Lambda execution role or edit and run standalone code".
When you perform above, it will move to next issue of conflict in auth config map (as config map exists). I will analyze api to add feature to enhance updating config map similar to create that is currently used. This sample currently handles only create config map.

Note: In the meantime, for quick fix you can do one of the below.

1. update lambda code to create boto3 session using access keys(read from env variable or from secret manager) of user who created cluster and pass the session (similar to stand alone code)
2. update lambda code to assume the role that created cluster, so that lambda runs with specific role that has created ESK cluster.