Issue with getting /api and /apis inside a pod

[bulatgizyatullin]

I'm running my python code inside a pod, please see below:

from kubernetes import client, config

config.load_incluster_config()
api_client = client.ApiClient()
core_version = api_client.call_api('/api', 'GET', response_type='object')[0]

or

discovery = client.ApisApi(api_client)
api_versions = discovery.get_api_versions()

And getting the next issue:
"apiVersion":"v1","metadata":{},"status":"Failure","message":"forbidden: User "system:anonymous" cannot get path "/api"","reason":"Forbidden","details":{},"code":403

When I try to do the same but with

config.load_kube_config(config_file=KUBE_CONFIG_FILE) 

instead of

config.load_incluster_config()

everything is working fine.

I've attached service account with full admin permission to the pod. I'm able to see token in file /var/run/secrets/kubernetes.io/serviceaccount/token inside the pod.
Kubernetes lib version is kubernetes==33.1.0.

Could you please assist with the issue ?

[RehanK-123]

I would like to help with this issue.

[RehanK-123]

I think config.load_incluster_config() isnt loading the Service Account token properly, please check what account is being used by running kubectl get pod -o json.

[bulatgizyatullin]

"serviceAccount": "kube-controller",
why kubernetes sees it as User "system:anonymous" not clear.
Version of kubernetes is v1.31.3

[RehanK-123]

Could you send the whole output, if you can?

[bulatgizyatullin]

Sure,

{
    "apiVersion": "v1",
    "items": [
        {
            "apiVersion": "v1",
            "kind": "Pod",
            "metadata": {
                "annotations": {
                    "cni.projectcalico.org/containerID": "8e78d4278b01fb233e94eb4e60794390bc5255ccac41f5154907b13c21e8bd69",
                    "cni.projectcalico.org/podIP": "10.10.137.69/32",
                    "cni.projectcalico.org/podIPs": "10.10.137.69/32"
                },
                "creationTimestamp": "2025-09-30T18:35:53Z",
                "generateName": "kube-control-app-6bb47759fc-",
                "labels": {
                    "app": "kube-control",
                    "pod-template-hash": "6bb47759fc"
                },
                "name": "kube-control-app-6bb47759fc-l8jwf",
                "namespace": "kube-control",
                "ownerReferences": [
                    {
                        "apiVersion": "apps/v1",
                        "blockOwnerDeletion": true,
                        "controller": true,
                        "kind": "ReplicaSet",
                        "name": "kube-control-app-6bb47759fc",
                        "uid": "381d6fba-3d69-4dca-8975-bdb6cf533209"
                    }
                ],
                "resourceVersion": "33282337",
                "uid": "5e352045-c9f5-445b-ae78-f6b5491b3b04"
            },
            "spec": {
                "automountServiceAccountToken": true,
                "containers": [
                    {
                        "image": "kubefc:latest",
                        "imagePullPolicy": "Never",
                        "name": "kubefc",
                        "ports": [
                            {
                                "containerPort": 8080,
                                "protocol": "TCP"
                            }
                        ],
                        "resources": {},
                        "terminationMessagePath": "/dev/termination-log",
                        "terminationMessagePolicy": "File",
                        "volumeMounts": [
                            {
                                "mountPath": "/var/run/secrets/kubernetes.io/serviceaccount",
                                "name": "kube-api-access-9c2xg",
                                "readOnly": true
                            }
                        ]
                    }
                ],
                "dnsPolicy": "ClusterFirst",
                "enableServiceLinks": true,
                "nodeName": "master1",
                "preemptionPolicy": "PreemptLowerPriority",
                "priority": 0,
                "restartPolicy": "Always",
                "schedulerName": "default-scheduler",
                "securityContext": {},
                "serviceAccount": "kube-controller",
                "serviceAccountName": "kube-controller",
                "terminationGracePeriodSeconds": 30,
                "tolerations": [
                    {
                        "effect": "NoExecute",
                        "key": "node.kubernetes.io/not-ready",
                        "operator": "Exists",
                        "tolerationSeconds": 300
                    },
                    {
                        "effect": "NoExecute",
                        "key": "node.kubernetes.io/unreachable",
                        "operator": "Exists",
                        "tolerationSeconds": 300
                    }
                ],
                "volumes": [
                    {
                        "name": "kube-api-access-9c2xg",
                        "projected": {
                            "defaultMode": 420,
                            "sources": [
                                {
                                    "serviceAccountToken": {
                                        "expirationSeconds": 3607,
                                        "path": "token"
                                    }
                                },
                                {
                                    "configMap": {
                                        "items": [
                                            {
                                                "key": "ca.crt",
                                                "path": "ca.crt"
                                            }
                                        ],
                                        "name": "kube-root-ca.crt"
                                    }
                                },
                                {
                                    "downwardAPI": {
                                        "items": [
                                            {
                                                "fieldRef": {
                                                    "apiVersion": "v1",
                                                    "fieldPath": "metadata.namespace"
                                                },
                                                "path": "namespace"
                                            }
                                        ]
                                    }
                                }
                            ]
                        }
                    }
                ]
            },
            "status": {
                "conditions": [
                    {
                        "lastProbeTime": null,
                        "lastTransitionTime": "2025-09-30T18:35:55Z",
                        "status": "True",
                        "type": "PodReadyToStartContainers"
                    },
                    {
                        "lastProbeTime": null,
                        "lastTransitionTime": "2025-09-30T18:35:53Z",
                        "status": "True",
                        "type": "Initialized"
                    },
                    {
                        "lastProbeTime": null,
                        "lastTransitionTime": "2025-09-30T18:35:55Z",
                        "status": "True",
                        "type": "Ready"
                    },
                    {
                        "lastProbeTime": null,
                        "lastTransitionTime": "2025-09-30T18:35:55Z",
                        "status": "True",
                        "type": "ContainersReady"
                    },
                    {
                        "lastProbeTime": null,
                        "lastTransitionTime": "2025-09-30T18:35:53Z",
                        "status": "True",
                        "type": "PodScheduled"
                    }
                ],
                "containerStatuses": [
                    {
                        "containerID": "containerd://33f4acae02bbb2046afe827e5e89b5f0e8fe65247c833a680a82bb50586dd049",
                        "image": "docker.io/library/kubefc:latest",
                        "imageID": "sha256:09d85781a4e03e6ffc51ab0432fcd974f6fe7f2e9fc1d5bb07a805f5c676045b",
                        "lastState": {},
                        "name": "kubefc",
                        "ready": true,
                        "restartCount": 0,
                        "started": true,
                        "state": {
                            "running": {
                                "startedAt": "2025-09-30T18:35:54Z"
                            }
                        },
                        "volumeMounts": [
                            {
                                "mountPath": "/var/run/secrets/kubernetes.io/serviceaccount",
                                "name": "kube-api-access-9c2xg",
                                "readOnly": true,
                                "recursiveReadOnly": "Disabled"
                            }
                        ]
                    }
                ],
                "hostIP": "63.108.0.10",
                "hostIPs": [
                    {
                        "ip": "63.108.0.10"
                    }
                ],
                "phase": "Running",
                "podIP": "10.10.137.69",
                "podIPs": [
                    {
                        "ip": "10.10.137.69"
                    }
                ],
                "qosClass": "BestEffort",
                "startTime": "2025-09-30T18:35:53Z"
            }
        }
    ],
    "kind": "List",
    "metadata": {
        "resourceVersion": ""
    }
}

I've also checkend inside the pod next:

TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
curl -k https://kubernetes.default.svc/api -H "Authorization: Bearer $TOKEN"
{
  "kind": "APIVersions",
  "versions": [
    "v1"
  ],
  "serverAddressByClientCIDRs": [
    {
      "clientCIDR": "0.0.0.0/0",
      "serverAddress": "63.108.0.10:6443"
    }
  ]

it returned /api

[bulatgizyatullin]

@RehanK-123 any ideas what is wrong ?

[RehanK-123]

Hi, i think you need to check whether RBAC is binded or not, if it is not bound, then it might treat the user as anonymous, please check using
----- kubectl get clusterrolebinding | grep kube-controller
----- kubectl get rolebinding -n kube-control | grep kube-controller

[bulatgizyatullin]

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"rbac.authorization.k8s.io/v1","kind":"ClusterRoleBinding","metadata":{"annotations":{},"name":"kube-controller-binding"},"roleRef":{"apiGroup":"rbac.authorization.k8s.io","kind":"ClusterRole","name":"kube-controller-role"},"subjects":[{"kind":"ServiceAccount","name":"kube-controller","namespace":"kube-control"}]}
  creationTimestamp: "2025-09-15T19:36:45Z"
  name: kube-controller-binding
  resourceVersion: "32105299"
  uid: 89950d5d-e31c-49c5-b1f7-d8fe4d6f105c
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kube-controller-role
subjects:
- kind: ServiceAccount
  name: kube-controller
  namespace: kube-control

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"rbac.authorization.k8s.io/v1","kind":"ClusterRole","metadata":{"annotations":{},"name":"kube-controller-role"},"rules":[{"apiGroups":["certificates.k8s.io"],"resources":["certificatesigningrequests","certificatesigningrequests/approval"],"verbs":["create","get","list","watch","delete","patch","update","approve"]},{"apiGroups":["certificates.k8s.io"],"resources":["certificatesigningrequests/status"],"verbs":["get","list","watch","update","patch"]},{"apiGroups":["certificates.k8s.io"],"resourceNames":["kubernetes.io/kube-apiserver-client"],"resources":["signers"],"verbs":["approve"]},{"nonResourceURLs":["/api","/apis","/version"],"verbs":["get"]},{"apiGroups":[""],"resources":["serviceaccounts"],"verbs":["create","get","list","watch","delete","patch","update"]},{"apiGroups":["rbac.authorization.k8s.io"],"resources":["roles","rolebindings"],"verbs":["create","get","list","watch","delete","patch","update"]},{"apiGroups":["rbac.authorization.k8s.io"],"resources":["clusterroles","clusterrolebindings"],"verbs":["create","get","list","watch","delete","patch","update"]},{"apiGroups":["authentication.k8s.io"],"resources":["tokenreviews"],"verbs":["*"]}]}
  creationTimestamp: "2025-09-15T19:36:45Z"
  name: kube-controller-role
  resourceVersion: "33283420"
  uid: 8885a388-c818-4802-b4c4-df71b82742c9
rules:
- apiGroups:
  - certificates.k8s.io
  resources:
  - certificatesigningrequests
  - certificatesigningrequests/approval
  verbs:
  - create
  - get
  - list
  - watch
  - delete
  - patch
  - update
  - approve
- apiGroups:
  - certificates.k8s.io
  resources:
  - certificatesigningrequests/status
  verbs:
  - get
  - list
  - watch
  - update
  - patch
- apiGroups:
  - certificates.k8s.io
  resourceNames:
  - kubernetes.io/kube-apiserver-client
  resources:
  - signers
  verbs:
  - approve
- nonResourceURLs:
  - /api
  - /apis
  - /version
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - serviceaccounts
  verbs:
  - create
  - get
  - list
  - watch
  - delete
  - patch
  - update
- apiGroups:
  - rbac.authorization.k8s.io
  resources:
  - roles
  - rolebindings
  verbs:
  - create
  - get
  - list
  - watch
  - delete
  - patch
  - update
- apiGroups:
  - rbac.authorization.k8s.io
  resources:
  - clusterroles
  - clusterrolebindings
  verbs:
  - create
  - get
  - list
  - watch
  - delete
  - patch
  - update
- apiGroups:
  - authentication.k8s.io
  resources:
  - tokenreviews
  verbs:
  - '*'

I modified a little bit clusterrole, now it doesn't have full admin permissions, but here as you can see I added permission to work with urls:

- nonResourceURLs:
  - /api
  - /apis
  - /version
  verbs:
  - get
  - list
  - watch

So when I did

TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
curl -k https://kubernetes.default.svc/api -H "Authorization: Bearer $TOKEN"
{
  "kind": "APIVersions",
  "versions": [
    "v1"
  ],
  "serverAddressByClientCIDRs": [
    {
      "clientCIDR": "0.0.0.0/0",
      "serverAddress": "63.108.0.10:6443"
    }
  ]

everything worked fine. But If I do it via kubernetes lib in python I'm still getting

User "system:anonymous" cannot get path "/api"

I researched a little bit method config.load_incluster_config() and saw what vars are there. There are 4 vars:

SERVICE_HOST_ENV_NAME = "KUBERNETES_SERVICE_HOST"
SERVICE_PORT_ENV_NAME = "KUBERNETES_SERVICE_PORT"
SERVICE_TOKEN_FILENAME = "/var/run/secrets/kubernetes.io/serviceaccount/token"
SERVICE_CERT_FILENAME = "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"

env vars KUBERNETES_SERVICE_HOST and PORT exist inside the pod as well as files token and ca.crt. I don't get any exceptions like Service host/port is not set or something like this.

[RehanK-123]

client.Configuration, please run this, to check what the config settings are before and after running load_incluster_config, i am suspecting that load_incluster_config() isnt loading the configuration settings properly, leading to the issue.

[RehanK-123]

/assign

[bulatgizyatullin]

>>> from kubernetes import client, config
>>>
>>>
>>>
>>> client.Configuration
<class 'kubernetes.client.configuration.Configuration'>
>>>
>>>
>>> config.load_incluster_config()
>>>
>>> client.Configuration
<class 'kubernetes.client.configuration.Configuration'>
>>>

Please let me know if you need some specific properties of client configuration

[RehanK-123]

import json
cfg = client.Configuration.get_default_copy()
print(json.dumps(cfg.to_dict(), indent=2))

use this sorry, I am new at this so bear with me a little :(

[bulatgizyatullin]

There is no to_dict() method, I've used another way:

>>> from kubernetes import config, client
>>> import json
>>>
>>> cfg = client.Configuration.get_default_copy()
>>> print(json.dumps(vars(cfg), indent=2, default=str))
{
  "host": "http://localhost",
  "temp_folder_path": null,
  "api_key": {},
  "api_key_prefix": {},
  "refresh_api_key_hook": null,
  "username": null,
  "password": null,
  "discard_unknown_keys": false,
  "logger": {
    "package_logger": "<Logger client (WARNING)>",
    "urllib3_logger": "<Logger urllib3 (WARNING)>"
  },
  "_Configuration__logger_format": "%(asctime)s %(levelname)s %(message)s",
  "logger_formatter": "<logging.Formatter object at 0x7f2048d1b040>",
  "logger_stream_handler": null,
  "logger_file_handler": null,
  "_Configuration__logger_file": null,
  "_Configuration__debug": false,
  "verify_ssl": true,
  "ssl_ca_cert": null,
  "cert_file": null,
  "key_file": null,
  "assert_hostname": null,
  "tls_server_name": null,
  "connection_pool_maxsize": 10,
  "proxy": null,
  "no_proxy": null,
  "proxy_headers": null,
  "safe_chars_for_path_param": "",
  "retries": null,
  "client_side_validation": true
}
>>>

and after loading incluster config:

>>> config.load_incluster_config()
>>> cfg = client.Configuration.get_default_copy()
>>> print(json.dumps(vars(cfg), indent=2, default=str))
{
  "host": "https://10.96.0.1:443",
  "temp_folder_path": null,
  "api_key": {
    "authorization": "bearer eyJhbGciOiJSUzI1..."
  },
  "api_key_prefix": {},
  "refresh_api_key_hook": "<function InClusterConfigLoader._set_config.<locals>._refresh_api_key at 0x7f204638c670>",
  "username": null,
  "password": null,
  "discard_unknown_keys": false,
  "_Configuration__logger_format": "%(asctime)s %(levelname)s %(message)s",
  "logger_formatter": "<logging.Formatter object at 0x7f2046380d00>",
  "logger_stream_handler": null,
  "_Configuration__logger_file": null,
  "_Configuration__debug": false,
  "verify_ssl": true,
  "ssl_ca_cert": "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt",
  "cert_file": null,
  "key_file": null,
  "assert_hostname": null,
  "tls_server_name": null,
  "connection_pool_maxsize": 10,
  "proxy": null,
  "no_proxy": null,
  "proxy_headers": null,
  "safe_chars_for_path_param": "",
  "retries": null,
  "client_side_validation": true,
  "logger": {
    "package_logger": "<Logger client (WARNING)>",
    "urllib3_logger": "<Logger urllib3 (WARNING)>"
  }
}

[bulatgizyatullin]

I did a bit deeper debug, so what I've found

>>> from kubernetes import config, client
>>>
>>> config.load_incluster_config()

cfg.api_key['authorization'] = "Bearer " + token
cfg.api_key_prefix['authorization'] = "Bearer"

cfg.debug = True

api_client = client.ApiClient(configuration=cfg)
>>>
>>> cfg = client.Configuration.get_default_copy()
>>>
>>> with open('/var/run/secrets/kubernetes.io/serviceaccount/token') as f:
...     token = f.read().strip()
...
>>> cfg.api_key['authorization'] = "Bearer " + token
>>> cfg.api_key_prefix['authorization'] = "Bearer"
>>>
>>> cfg.debug = True
>>>
>>> api_client = client.ApiClient(configuration=cfg)
>>>
>>> try:
...     core_version = api_client.call_api('/api', 'GET', response_type='object')[0]
...     print(core_version)
... except client.exceptions.ApiException as e:
...     print(f"ApiException: {e}")
...
send: b'GET /api HTTP/1.1\r\nHost: 10.96.0.1\r\nAccept-Encoding: identity\r\nUser-Agent: OpenAPI-Generator/30.1.0/python\r\nContent-Type: application/json\r\n\r\n'
reply: 'HTTP/1.1 403 Forbidden\r\n'
header: Audit-Id: a32fa9af-ffee-46ce-a944-7739a46cf760
header: Cache-Control: no-cache, private
header: Content-Type: application/json
header: X-Content-Type-Options: nosniff
header: X-Kubernetes-Pf-Flowschema-Uid: 31d7f2a0-11c7-489d-b80e-a457520f092e
header: X-Kubernetes-Pf-Prioritylevel-Uid: 47ffadef-1dbc-49d5-a87c-7389bd7bb26f
header: Date: Mon, 06 Oct 2025 19:42:01 GMT
header: Content-Length: 188
ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Audit-Id': 'a32fa9af-ffee-46ce-a944-7739a46cf760', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': '31d7f2a0-11c7-489d-b80e-a457520f092e', 'X-Kubernetes-Pf-Prioritylevel-Uid': '47ffadef-1dbc-49d5-a87c-7389bd7bb26f', 'Date': 'Mon, 06 Oct 2025 19:42:01 GMT', 'Content-Length': '188'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"forbidden: User \"system:anonymous\" cannot get path \"/api\"","reason":"Forbidden","details":{},"code":403}

There is no authorization header.

So then I found solution: we need to manually add Authorization header to call_api method:

from kubernetes import client, config

config.load_incluster_config()
api_client = client.ApiClient()

cfg = client.Configuration.get_default_copy()
token = cfg.api_key.get('authorization')

header_params = {"Authorization": token}

core_version = api_client.call_api('/api', 'GET', response_type='object', header_params=header_params)[0]

and then it works :)