The following link https://cloud.google.com/kubernetes-engine/docs/how-to/private-clusters talks about the setting up of a private GKE cluster in a separate custom VPC. The Terraform code that creates the cluster and VPCs are available from https://github.com/rajtmana/gcp-terraform/blob/master/k8s-cluster/main.tf Cluster creation completed and I wanted to use some kubectl commands from the Google Cloud Shell. I used the following commands
$ gcloud container clusters get-credentials mservice-dev-cluster --region europe-west2
$ gcloud container clusters update mservice-dev-cluster \
> --region europe-west2 \
> --enable-master-authorized-networks \
> --master-authorized-networks "35.241.216.229/32"
Updating mservice-dev-cluster...done.
ERROR: (gcloud.container.clusters.update) Operation [<Operation
clusterConditions: []
detail: u'Patch failed'
$ gcloud container clusters update mservice-dev-cluster \
> --region europe-west2 \
> --enable-master-authorized-networks \
> --master-authorized-networks "172.17.0.2/32"
Updating mservice-dev-cluster...done.
Updated [https://container.googleapis.com/v1/projects/protean-
XXXX/zones/europe-west2/clusters/mservice-dev-cluster].
To inspect the contents of your cluster, go to:
https://console.cloud.google.com/kubernetes/workload_/gcloud/europe-
west2/mservice-dev-cluster?project=protean-XXXX
$ kubectl config current-context
gke_protean-XXXX_europe-west2_mservice-dev-cluster
$ kubectl get services
Unable to connect to the server: dial tcp 172.16.0.2:443: i/o timeout
When I give the public IP of the Cloud Shell, it says that public IP is not allowed with error message given above. If I give the internal IP of Cloud Shell starting with 172, the connection is timing out as well. Any thoughts? Appreciate the help.
Google suggest creating a VM within the same network as the cluster and then accessing that via SSH in the cloud shell and running kubectl commands from there:
https://cloud.google.com/solutions/creating-kubernetes-engine-private-clusters-with-net-proxies
Try to perform the following
gcloud container clusters get-credentials [CLUSTER_NAME]
And confirm that kubectl is using the right credentials:
gcloud auth application-default login
Related
I'm trying to connect to a cluster and I'm getting the following error:
gcloud container clusters get-credentials cluster1 --region europe-west2 --project my-project
Fetching cluster endpoint and auth data.
CRITICAL: ACTION REQUIRED: gke-gcloud-auth-plugin, which is needed for continued use of kubectl, was not found or is not executable.
Install gke-gcloud-auth-plugin for use with kubectl by following https://cloud.google.com/blog/products/containers-kubernetes/kubectl-auth-changes-in-gke
kubeconfig entry generated for dbcell-cluster.
I have installed Google Cloud SDK 400, kubektl 1.22.12, gke-gcloud-auth-plugin 0.3.0, and also setup /~.bashrc with
export USE_GKE_GCLOUD_AUTH_PLUGIN=True
gke-gcloud-auth-plugin --version
Kubernetes v1.24.0-alpha+f42d1572e39979f6f7de03bd163f8ec04bc7950d
but when I try to connect to the cluster always I'm getting the same error, any idea here?
Thanks
The cluster exist in that region, also I verfied the env variable
with
echo $USE_GKE_GCLOUD_AUTH_PLUGIN
True
I installed the gke-gcloud-auth-plugin using gcloud components install... I do not know what more can I check
gcloud components list
I solved the same problem by removing my current kubeconfig context for GCP.
Get your context name running:
kubectl config get-contexts
Delete the context:
kubectl config delete-context CONTEXT_NAME
Reconfigure the credentials
gcloud container clusters get-credentials CLUSTER_NAME --region REGION --project PROJECT
The warning message should be gone by now.
Currently...
I have a GKE/kubernetes/k8s cluster in GCP. I have a bastion host (Compute Engine VM Instance) in GCP. I have allowlisted my bastion host's IP in the GKE cluster's Master authorized networks section. Hence, in order to run kubectl commands to my GKE, I first need to SSH into my bastion host by running the gcloud beta compute ssh command; then I run the gcloud container clusters get-credentials command to authenticate with GKE, then from there I can run kubectl commands like usual.
Later...
I want to be able to run kubectl commands to my GKE cluster directly from my local development CLI. In order to do that, I can add my local development machine IP as an allowlisted entry into my GKE's Master authorized networks, and that should be it. Then i can run the gcloud container clusters get-credentials first and then run kubectl commands like usual.
However...
I am looking for a way to avoid having to allowlist my local development machine IP. Every time i take my laptop somewhere new, i have to update the allowlist my new IP from there before i can run the gcloud container clusters get-credentials command before running kubectl commands.
I wonder...
Is there a way to assign a port number in the bastion-host that can be used to invoke kubectl commands to the remote GKE cluster securely? And then, i can just use the gcloud compute start-iap-tunnel command (which BTW takes care of all permission issues using Cloud IAM) from my local dev CLI to establish a ssh-tunnel to that specific port number in the bastion host. That way, for the GKE cluster, it is receiving kubectl commands from the bastion host (which is already allowlisted in its Master authorized networks). But behind the scene, i am authenticating with the bastion host from my local dev CLI (using my glcoud auth credentails) and invoking kubectl commands from there securely.
Is this possible? Any ideas from anyone?
This would help accessing to your secured GKE cluster from localhost
https://github.com/GoogleCloudPlatform/gke-private-cluster-demo
Once the bastion host is setup with tinyproxy as in the above doc, we can use the below shell functions to quickly enable/disable the bastion host access
enable_secure_kubectl() {
# Aliasing kubectl/helm commands to use local proxy
alias kubectl="HTTPS_PROXY=localhost:8888 kubectl"
alias helm="HTTPS_PROXY=localhost:8888 helm"
# Open SSH tunnel for 1 hour
gcloud compute ssh my-bastion-host -- -o ExitOnForwardFailure=yes -M -S /tmp/sslsock -L8888:127.0.0.1:8888 -f sleep 3600
# Get kubernetes credentials with internal ip for kube-apiserver in kubeconfig
gcloud container clusters get-credentials my-gke-cluster --region us-east1 --project myproject --internal-ip
}
disable_secure_kubectl() {
unalias kubectl
unalias helm
ssh -S /tmp/sslsock -O exit my-bastion-host
}
I'm trying to follow this step by step to upload the airflow in Kubernetes (https://github.com/EamonKeane/airflow-GKE-k8sExecutor-helm) but in this part of the execution I have problems as follows:
Researching on the topic did not find anything that solved so far my problem, does anyone have any suggestions of what to do?
SQL_ALCHEMY_CONN=postgresql+psycopg2://$AIRFLOW_DB_USER:$AIRFLOW_DB_USER_PASSWORD#$KUBERNETES_POSTGRES_CLOUDSQLPROXY_SERVICE:$KUBERNETES_POSTGRES_CLOUDSQLPROXY_PORT/$AIRFLOW_DB_NAME
echo $SQL_ALCHEMY_CONN > /secrets/airflow/sql_alchemy_conn
# Create the fernet key which is needed to decrypt database the database
FERNET_KEY=$(dd if=/dev/urandom bs=32 count=1 2>/dev/null | openssl base64)
echo $FERNET_KEY > /secrets/airflow/fernet-key
kubectl create secret generic airflow \
--from-file=fernet-key=/secrets/airflow/fernet-key \
--from-file=sql_alchemy_conn=/secrets/airflow/sql_alchemy_conn
Unable to connect to the server: error executing access token command
"/google/google-cloud-sdk/bin/gcloud config config-helper
--format=json": err=exit status 1 output= stderr=ERROR: gcloud crashed (BadStatusLine): '' If you would like to report this issue, please run
the following command: gcloud feedback To check gcloud for common
problems, please run the following command: gcloud info
--run-diagnostics
I solved this by creating a new cloud shell tab to connect the cluster:
gcloud container clusters get-credentials testcluster1 --zone = your_zone
Example:
get the name and location of your cluster
gcloud container clusters list
then
gcloud container clusters get-credentials demo --region=us-west1-a
I have a Kubernetes cluster in Azure using AKS and I'd like to 'login' to one of the nodes. The nodes do not have a public IP.
Is there a way to accomplish this?
The procedure is longly decribed in an article of the Azure documentation:
https://learn.microsoft.com/en-us/azure/aks/ssh. It consists of running a pod that you use as a relay to ssh into the nodes, and it works perfectly fine:
You probably have specified the ssh username and public key during the cluster creation. If not, you have to configure your node to accept them as the ssh credentials:
$ az vm user update \
--resource-group MC_myResourceGroup_myAKSCluster_region \
--name node-name \
--username theusername \
--ssh-key-value ~/.ssh/id_rsa.pub
To find your nodes names:
az vm list --resource-group MC_myResourceGroup_myAKSCluster_region -o table
When done, run a pod on your cluster with an ssh client inside, this is the pod you will use to ssh to your nodes:
kubectl run -it --rm my-ssh-pod --image=debian
# install ssh components, as their is none in the Debian image
apt-get update && apt-get install openssh-client -y
On your workstation, get the name of the pod you just created:
$ kubectl get pods
Add your private key into the pod:
$ kubectl cp ~/.ssh/id_rsa pod-name:/id_rsa
Then, in the pod, connect via ssh to one of your node:
ssh -i /id_rsa theusername#10.240.0.4
(to find the nodes IPs, on your workstation):
az vm list-ip-addresses --resource-group MC_myAKSCluster_myAKSCluster_region -o table
This Gist and this page have pretty good explanations of how to do it. Sshing into the nodes and not shelling into the pods/containers.
you can use this instead of SSH. This will create a tiny priv pod and use nsenter to access the noed.
https://github.com/mohatb/kubectl-wls
I have created Kubernetes on Alibaba Cloud and would like to control from client, such as kube master master/nodes kubectl, kubernetes-dashboard, deploy (manifests) from local to cloud, etc without SSH.
I know that we can use kubeconfig, but no idea for it now, please help me more, thanks.
If you created a cluster using kubeadm for example, you will need to enter the instance through SSH and download the kube-apiserver client certificates and CA from /etc/kubernetes/pki.
Once you have them, you can add the configuration to kubeconfig using these commands (based on https://github.com/kelseyhightower/kubernetes-the-hard-way/blob/master/docs/10-configuring-kubectl.md). Make sure you replace the IP_ADDRESS_OF_YOUR_CLUSTER, CLIENT_CERTIFICATE, CLIENT_KEY placeholders (instead of admin you can choose another name for the credentials):
kubectl config set-cluster your cluster \
--certificate-authority=CA_CERTIFICATE \
--embed-certs=true \
--server=https://IP_ADDRESS_OF_YOUR_CLUSTER:6443
kubectl config set-credentials admin \
--client-certificate=CLIENT_CERTIFICATE \
--client-key=CLIENT_KEY
kubectl config set-context your-cluster-context \
--cluster=your-cluster \
--user=admin
If you get authentication errors, then you used the incorrect certificates.
In addition, make sure that you open port 6443 in your cloud firewall, otherwise you will not be able to access.