Support Private IP

[mik-laj]

@olchas

Hi.
In order to make this script work with Composer environment behind private IP you have to:

1. Make sure that it is possible to use IAP for SSH tunneling in your network (you need to set up a firewall rule, see here). Your account also needs a permission: iap.tunnelInstances.accessViaIAP.

2. Collect credentials using private master endpoint:
   gcloud container clusters get-credentials "${COMPOSER_GKE_CLUSTER_NAME}" --zone "${COMPOSER_GKE_CLUSTER_ZONE}" --internal-ip &>/dev/null

3. Start a process in the background that will open dynamic port forwarding to node of your cluster:
   gcloud compute ssh "${COMPOSER_GKE_NODE_NAME}" --zone "${COMPOSER_GKE_CLUSTER_ZONE}" --quiet --ssh-flag=-N --ssh-flag=-vvv --ssh-flag="-D 127.0.0.1:${SOCKS_PORT}"

Where:

• ${COMPOSER_GKE_NODE_NAME} is any node belonging to ${COMPOSER_GKE_CLUSTER_NAME}
• ${SOCKS_PORT} is any free port on your local machine

This command by default will use the ssh key called google_compute_engine located at ~/.ssh/ and will generate one if it does not exist. If you already have this key, make sure it has no passphrase, otherwise you will have to specify it every time you connect. You can also specify path to other key with --ssh-key-file flag. Your account will need compute.instances.get and compute.instances.setMetadata permissions, as well as Service Account User role on Composer’s service account.

Since you do not have access to kubectl yet, then in order to find a node you will need to collect the instance group from node-pool and then the node from the instance group (compute.instanceGroups.list permission needed):

COMPOSER_GKE_INSTANCE_GROUP_NAME=$(gcloud container node-pools describe default-pool --cluster="${COMPOSER_GKE_CLUSTER_NAME}" --zone "${COMPOSER_GKE_CLUSTER_ZONE}" '--format=value(instanceGroupUrls)' | rev| cut -d '/' -f 1 | rev)`

COMPOSER_GKE_NODE_NAME=$(gcloud compute instance-groups list-instances $COMPOSER_GKE_INSTANCE_GROUP_NAME --zone="${COMPOSER_GKE_CLUSTER_ZONE}" | grep RUNNING | head -1 | cut -d " " -f 1)

At this point you should be able to execute some kubectl commands by specifying https_proxy variable, for example:

https_proxy=socks5://localhost:${SOCKS_PORT} kubectl get pod -A

However, you will not be able to execute kubectl exec commands, as they do not support socks proxies (see this PR: kubernetes/kubernetes#84205).

To make it work, you will need to use HTTP proxy instead, or use some way that will convert http requests to socks requests (see this superuser thread for possible solutions: https://superuser.com/questions/280129/http-proxy-over-ssh-not-socks)

For example, you can use http-proxy-to-socks nodejs tool (https://github.com/oyyd/http-proxy-to-socks).

After installing it, simply run another background process:

hpts -s 127.0.0.1:${SOCKS_PORT} -p ${HTTP_PORT}

Where ${HTTP_PORT} is another free port on your local machine.

Then execute kubectl commands with following https_proxy variable set, for example:

https_proxy=http://localhost:${HTTP_PORT} kubectl exec --namespace="${COMPOSER_GKE_NAMESPACE_NAME}" -t "${COMPOSER_GKE_WORKER_NAME}" --container airflow-worker -- echo "hello"