GCP GKE – run kubectl through bastion host

Re-posting my answer to a Google cloud platform’s Google Kubernetes Engine (GKE) related question in Serverfault. The question is how do you run kubectl commands from a local development laptop to GKE via bastion tunnel. Run ‘kubectl’ commands from my localhost to GKE – but via tunnelling through a bastion host

Adding your IP to the Master authorized network would be easier. You can write a script which gets your laptop external IP and adds it to GKE’s master authorized network list and use the same script to remove the IP once done. But since this doesn’t seem to be your preference, let me give a long answer to how you can accomplish this with a jump host.

First, you will need a port redirector program on the bastion host. This will forward requests hitting bastion port to be redirected to the GKE master IP address. I am assuming here you are using a private cluster – both nodes and master api server are in private network.

On bastion host –

sudo apt-get update && sudo apt-get install redir -y 
redir --laddr=0.0.0.0 --lport=8443 --caddr=172.16.0.32 --cport=443 -l debug

Above command will redirect requests on bastion port of 8443 to GKE master node (172.16.0.3) – feel free to change this based on your setup.

On your laptop –

gcloud compute ssh bastion --zone $ZONE --ssh-flag="-L 8443:localhost:8443"

You have now created an ssh tunnel from your laptop to bastion host. Calls to localhost on port 8443 will be redirected to the GKE api server.

Generate kube config –

gcloud container clusters get-credentials $CLUSTER_NAME [--zone $ZONE | --region $REGION]

Finally, update the server section in ~/.kube/config as follows –

server: https://127.0.0.1:8443

If you run the kubectl commands now, you will still encounter ssl certificate error –

$ kubectl version --short
Client Version: v1.15.11-dispatcher
Unable to connect to the server: x509: certificate is valid for ***, not 127.0.0.1

To verify the setup is working, you can skip tls verifiction with --insecure-skip-tls-verify. NOTE – skipping TLS verification is NOT recommended for security reasons.

$ kubectl version --short --insecure-skip-tls-verify 
Client Version: v1.15.11-dispatcher
Server Version: v1.15.12-gke.2

One way you can address the certificate verification issue is, to create an alias IP address on your laptop which matches the GKE api server private address. In addition to this, update your kube config to match this.

$ sudo ifconfig eth0:0 172.16.0.3 up
$ gcloud compute ssh linux --zone us-east1-c --ssh-flag="-L 172.16.0.3:8443:localhost:8443"

$ kubectl version --short
Client Version: v1.15.11-dispatcher
Server Version: v1.15.12-gke.2

References –

https://linux.die.net/man/1/redir

https://cloud.google.com/kubernetes-engine/docs/how-to/private-clusters