I tried to configure istio multicluster in different networkg using https://istio.io/latest/docs/setup/install/multicluster/ in AWS EC2 instances.
Tried creating 2 clusters and tried to verify the communication between cluster, but is failing.
In Istio doc, I found this statement "The API Server in each cluster must be accessible to the other clusters in the mesh" under https://istio.io/latest/docs/setup/install/multicluster/before-you-begin/. How to make API Server accessible to other cluster?
In the same link which you had used https://istio.io/latest/docs/setup/install/multicluster/multi-primary_multi-network/ the steps are there to provide access to api server.
Install a remote secret in cluster2 that provides access to cluster1’s API server.
$ istioctl x create-remote-secret \
--context="${CTX_CLUSTER1}" \
--name=cluster1 | \
kubectl apply -f - --context="${CTX_CLUSTER2}"
Install a remote secret in cluster1 that provides access to cluster2’s API server.
$ istioctl x create-remote-secret \
--context="${CTX_CLUSTER2}" \
--name=cluster2 | \
kubectl apply -f - --context="${CTX_CLUSTER1}"
Related
I am running a regional GKE kubernetes cluster in is-central1-b us-central-1-c and us-central1-f. I am running 1.21.14-gke.700. I am adding a confidential node pool to the cluster with this command.
gcloud container node-pools create card-decrpyt-confidential-pool-1 \
--cluster=svcs-dev-1 \
--disk-size=100GB \
--disk-type=pd-standard \
--enable-autorepair \
--enable-autoupgrade \
--enable-gvnic \
--image-type=COS_CONTAINERD \
--machine-type="n2d-standard-2" \
--max-pods-per-node=8 \
--max-surge-upgrade=1 \
--max-unavailable-upgrade=1 \
--min-nodes=4 \
--node-locations=us-central1-b,us-central1-c,us-central1-f \
--node-taints=dedicatednode=card-decrypt:NoSchedule \
--node-version=1.21.14-gke.700 \
--num-nodes=4 \
--region=us-central1 \
--sandbox="type=gvisor" \
--scopes=https://www.googleapis.com/auth/cloud-platform \
--service-account="card-decrpyt-confidential#corp-dev-project.iam.gserviceaccount.com" \
--shielded-integrity-monitoring \
--shielded-secure-boot \
--tags=testingdonotuse \
--workload-metadata=GKE_METADATA \
--enable-confidential-nodes
This creates a node pool but there is one problem... I can still SSH to the instances that the node pool creates. This is unacceptable for my use case as these node pools need to be as secure as possible. I went into my node pool and created a new machine template with ssh turned off using an instance template based off the one created for my node pool.
gcloud compute instance-templates create card-decrypt-instance-template \
--project=corp-dev-project
--machine-type=n2d-standard-2
--network-interface=aliases=gke-svcs-dev-1-pods-10a0a3cd:/28,nic-type=GVNIC,subnet=corp-dev-project-private-subnet,no-address
--metadata=block-project-ssh-keys=true,enable-oslogin=true
--maintenance-policy=TERMINATE --provisioning-model=STANDARD
--service-account=card-decrpyt-confidential#corp-dev-project.iam.gserviceaccount.com
--scopes=https://www.googleapis.com/auth/cloud-platform
--region=us-central1 --min-cpu-platform=AMD\ Milan
--tags=testingdonotuse,gke-svcs-dev-1-10a0a3cd-node
--create-disk=auto-delete=yes,boot=yes,device-name=card-decrpy-instance-template,image=projects/confidential-vm-images/global/images/cos-89-16108-766-5,mode=rw,size=100,type=pd-standard
--shielded-secure-boot
--shielded-vtpm -
-shielded-integrity-monitoring
--labels=component=gke,goog-gke-node=,team=platform --reservation-affinity=any
When I change the instance templates of the nodes in the node pool the new instances come online but they do not attach to the node pool. The cluster is always trying to repair itself and I can't change any settings until I delete all the nodes in the pool. I don't receive any errors.
What do I need to do to disable ssh into the node pool nodes with the original node pool I created or with the new instance template I created. I have tried a bunch of different configurations with a new node pool and the cluster and have not had any luck. I've tried different tags network configs and images. None of these have worked.
Other info:
The cluster was not originally a confidential cluster. The confidential nodes are the first of its kind added to the cluster.
One option you have here is to enable private IP addresses for the nodes in your cluster. The --enable-private-nodes flag will make it so the nodes in your cluster get private IP addresses (rather than the default public, internet-facing IP addresses).
Note that in this case, you would still be able to SSH into these nodes, but only from within your VPC network.
Also note that this means you would not be able to access NodePort type services from outside of your VPC network. Instead, you would need to use a LoadBalancer type service (or provide some other way to route traffic to your service from outside of the cluster, if required).
If you'd like to prevent SSH access even from within your VPC network, your easiest option would likely be to configure a firewall rule to deny SSH traffic to your nodes (TCP/UDP/SCTP port 22). Use network tags (the --tags flag) to target your GKE nodes.
Something along the lines of:
gcloud compute firewall-rules create fw-d-i-ssh-to-gke-nodes \
--network NETWORK_NAME \
--action deny \
--direction ingress \
--rules tcp:22,udp:22,sctp:22 \
--source-ranges 0.0.0.0/0 \
--priority 65534 \
--target-tags my-gke-node-network-tag
Finally, one last option I'll mention for creating a hardened GKE cluster is to use Google's safer-cluster Terraform module. This is an opinionated setup of a GKE cluster that follows many of the principles laid out in Google's cluster hardening guide and the Terraform module takes care of a lot of the nitty-gritty boilerplate here.
I needed the metadata flag when creating the node pool
--metadata=block-project-ssh-keys=TRUE \
This blocked ssh.
However, enable-os-login=false won't work because it is reserved for use by the Kubernetes Engine
I've set up a basic GKE cluster using Autopilot settings. I am able to install Helm charts on it using kubectl with proper kubeconfig pointing to the GKE cluster.
I'd like to do the same without the kubeconfig, by providing the cluster details with relevant parameters.
To do that I'm running a docker container using alpine/helm image and passing the paramtrised command which looks like this:
docker run --rm -v $(pwd):/chart alpine/helm install <my_chart_name> /chart --kube-apiserver <cluster_endpoint> --kube-ca-file /chart/<cluster_certificate_file> --kube-as-user <my_gke_cluster_username> --kube-token <token>
unfortunately it returns :
Error: INSTALLATION FAILED: Kubernetes cluster unreachable: Get "http://<cluster_endpoint>/version": dial tcp <cluster_endpoint>:80: i/o timeout
Is this even doable with GKE?
One challenge will be that GKE leverages a plugin (currently built in to kubectl itself but soon the standlone gke-gcloud-auth-plugin) to obtain an access token for the default gcloud user.
This token expires hourly.
If you can, it would be better to mount the kubeconfig (${HOME}/.kube/config) file into the container as it should (!) then authenticate as if it were kubectl which will not only leverage the access token correctly but will renew it as appropriate.
https://github.com/alpine-docker/helm
docker run \
--interactive --tty --rm \
--volume=${PWD}/.kube:/root/.kube \
--volume=${PWD}/.helm:/root/.helm \
--volume=${PWD}/.config/helm:/root/.config/helm \
--volume=${PWD}/.cache/helm:/root/.cache/helm \
alpine/helm ...
NOTE It appears there are several (.helm, .config and .cache) other local paths that may be required too.
Problem solved! A more experienced colleague has found the solution.
I should have used the address including "http://" protocol specification. That however still kept returning "Kubernetes cluster unreachable: " error, with "unknown" details instead.
I had been using incorect username. Instead the one from kubeconfig file, a new service account should be created and its name used instead in a form system:serviceaccount:<namespace>:<service_account>. However that would not alter the error either.
The service account lacked proper role, following command did the job: kubectl create rolebinding --clusterrole=cluster-admin --serviceaccount=<namespace>:<service_account>. Ofc, cluster-admin might now be the role we want to give away freely.
I am trying to explore vault enterprise but getting permission denied for sidecar when I use the vault enterprise but seems to work fine when I tried to use local vault server.
Here is the repository that contains a working example with the local vault vault-sidecar-injector-app
Vault config
export VAULT_ADDR="https://vault-cluster.vault.c1c633fa-91ef-4e86-b025-4f31b3f14730.aws.hashicorp.cloud:8200"
export VAULT_NAMESPACE="admin"
#install agent
helm upgrade --install vault hashicorp/vault --set "injector.externalVaultAddr=$VAULT_ADDR"
vault auth enable kubernetes
# get certs & host
VAULT_HELM_SECRET_NAME=$(kubectl get secrets --output=json | jq -r '.items[].metadata | select(.name|startswith("vault-token-")).name')
TOKEN_REVIEW_JWT=$(kubectl get secret $VAULT_HELM_SECRET_NAME --output='go-template={{ .data.token }}' | base64 --decode)
KUBE_CA_CERT=$(kubectl config view --raw --minify --flatten --output='jsonpath={.clusters[].cluster.certificate-authority-data}' | base64 --decode)
KUBE_HOST=$(kubectl config view --raw --minify --flatten --output='jsonpath={.clusters[].cluster.server}')
# set Kubernetes config
vault write auth/kubernetes/config \
token_reviewer_jwt="$TOKEN_REVIEW_JWT" \
kubernetes_host="$KUBE_HOST" \
kubernetes_ca_cert="$KUBE_CA_CERT" \
issuer="https://kubernetes.default.svc.cluster.local" \
disable_iss_validation="true" \
disable_local_ca_jwt="true"
vault auth enable approle
# create admin policy
vault policy write admin admin-policy.hcl
vault write auth/approle/role/admin policies="admin"
vault read auth/approle/role/admin/role-id
# generate secret
vault write -f auth/approle/role/admin/secret-id
#Enable KV
vault secrets enable -version=2 kv
I can see the role and policy
Admin policy
Here is the admin policy for the enterprise
path "*" {
capabilities = ["create", "read", "update", "delete", "list", "sudo"]
}
Deploy Script for helm
here is the deploy script, tried hcp-root root policy but no luck
RELEASE_NAME=demo-managed
NAMESPACE=default
ENVIRONMENT=develop
export role_id="f9782a53-823e-2c08-81ae-abc"
export secret_id="1de3b8c5-18c7-60e3-24ca-abc"
export VAULT_ADDR="https://vault-cluster.vault.c1c633fa-91ef-4e86-b025-4f31b3f14730.aws.hashicorp.cloud:8200"
export VAULT_TOKEN=$(vault write -field="token" auth/approle/login role_id="${role_id}" secret_id="${secret_id}")
vault write auth/kubernetes/role/${NAMESPACE}-${RELEASE_NAME} bound_service_account_names=${RELEASE_NAME} bound_service_account_namespaces=${NAMESPACE} policies=hcp-root ttl=1h
helm upgrade --install $RELEASE_NAME ../helm-chart --set environment=$ENVIRONMENT --set nameOverride=$RELEASE_NAME
also tried with root token
RELEASE_NAME=demo-managed
NAMESPACE=default
ENVIRONMENT=develop
vault write auth/kubernetes/role/${NAMESPACE}-${RELEASE_NAME} bound_service_account_names=${RELEASE_NAME} bound_service_account_namespaces=${NAMESPACE} policies=hcp-root ttl=1h
helm upgrade --install $RELEASE_NAME ../helm-chart --set environment=$ENVIRONMENT --set nameOverride=$RELEASE_NAME
Sidecar config
With namespace annotation, as my understanding namespace is required
vault.hashicorp.com/namespace - configures the Vault Enterprise namespace to be used when requesting secrets from Vault.
https://www.vaultproject.io/docs/platform/k8s/injector/annotations
vault.hashicorp.com/namespace : "admin"
Error
| Error making API request.
|
| URL: PUT https://vault-cluster.vault.c1c633fa-91ef-4e86-b025-4f31b3f14730.aws.hashicorp.cloud:8200/v1/admin/auth/kubernetes/login
| Code: 403. Errors:
|
| * permission denied
Without namespace annotation getting below error
| URL: PUT https://vault-cluster.vault.c1c633fa-91ef-4e86-b025-4f31b3f14730.aws.hashicorp.cloud:8200/v1/auth/kubernetes/login
| Code: 400. Errors:
|
| * missing client token
Even enabling debug logs vault.hashicorp.com/log-level : "debug" does not help me with this error, any help or suggestions will be appreciated.
Also tried
https://support.hashicorp.com/hc/en-us/articles/4404389946387-Kubernetes-auth-method-Permission-Denied-error
So seems like I am missing something very specific to the vault enterprise
Finally able to resolve the weird issue with vault, posting as an answer might help someone else.
The only thing that I missed to understand the flow between vault server, sidecar, and Kubernetes.
Kubernetes should be reachable to vault enterprise for Token review API calls. As you can see the when sidecar makes a request to the vault, then the vault enterprise server performs a token review API call.
Use the /config endpoint to configure Vault to talk to Kubernetes. Use kubectl cluster-info to validate the Kubernetes host address and TCP port.
https://www.vaultproject.io/docs/auth/kubernetes
| Error making API request.
|
| URL: PUT https://vault-cluster.vault.c1c633fa-91ef-4e86-b025-4f31b3f14730.aws.hashicorp.cloud:8200/v1/admin/auth/kubernetes/login
| Code: 403. Errors:
|
| * permission denied
backoff=2.99s
This error does not indicate it has a connectivity issue but this also happens when the vault is not able to communicate with the Kubernetes cluster.
Kube Host
vault write auth/kubernetes/config \
token_reviewer_jwt="$TOKEN_REVIEW_JWT" \
kubernetes_host="$KUBE_HOST" \
kubernetes_ca_cert="$KUBE_CA_CERT" \
issuer="https://kubernetes.default.svc.cluster.local"
KUBE_HOST should be reachable for vault enterprise for tokenreview process.
So for the vault to communicate with our cluster, we need a few changes.
minikube start --apiserver-ips=14.55.145.30 --vm-driver=none
Now update the vaul-config.sh file
KUBE_HOST=$(kubectl config view --raw --minify --flatten --output='jsonpath={.clusters[].cluster.server}')
change this to
KUBE_HOST=""https://14.55.145.30:8443/"
No manual steps, for the first time configuration run
./vault-config.sh
and for the rest of the deployment in your CI/CD you can use
./vault.sh
Each release has only been able to access its own secrets.
Furter details can be found start-minikube-in-ec2
TLDR,
Note: Kubernetes cluster should be reachable to vault enterprise for authentication, so vault enterprise would not able to communicate with your local minikube cluster. Better to test it out on EC2
When you have set the Kubernetes auth into the vault you have used the
vault write auth/kubernetes/config \
token_reviewer_jwt="$TOKEN_REVIEW_JWT" \
kubernetes_host="$KUBE_HOST" \
kubernetes_ca_cert="$KUBE_CA_CERT" \
issuer="https://kubernetes.default.svc.cluster.local" \
disable_iss_validation="true" \
disable_local_ca_jwt="true"
you have mentioned
issuer="https://kubernetes.default.svc.cluster.local" \
it's HTTP and looks like running the local K8s server, not sure vault will be able to access this endpoint or not.
Did you try sending auth at this endpoint bypassing the Token & CA cert?
Your vault is able to connect and authenticate over the internet of your local cluster?
also i have faced this weird issue, with vault i used configure using both ways CLI and UI then it was working for me while in a single way it was giving error of permission denied.
I'd like to deploy a single app to multiple servers in one time.
I'm using Kubernetes and K3S to easily deploy containers.
Basically, I have a master server that I run and multiple servers that are localed in my customers facilities.
Master server was initialized with the following command:
k3sup install \
--ip $MASTER_IP \
--user ubuntu \
--cluster --k3s-channel latest \
--k3s-extra-args "--node-label ols.role=master"
Customer's servers were launched with:
k3sup join \
--ip $WORKER01_IP \
--user ubuntu \
--server-ip $MASTER_IP \
--server-user ubuntu \
--k3s-channel latest \
--k3s-extra-args "--node-label ols.role=worker"
When I want to deploy a new web service on each customer's server, I've tried the following code:
helm install node-red k8s-at-home/node-red --set nodeSelector."ols\.role"=worker
Problem: Only one single pod is deployed.
What I'd like is to deploy a single pod on each server and make it independent.
Is there a way to do that ?
Here there are two different things that we need to consider.
If the requirement is just to run more number of replicas of the application a change to the deployment template in the helm chart or through values you can pass number of minimum replicas need to be working in the cluster.
Reference documentation for deployments
Coming to next thing, if the requirements is just to run application across all the nodes existing in the cluster, Daemonsets is the workload which gives the capability to run across all the existing nodes.
Reference documentation for daemonsets
Again if you are using helm to deploy, appropriate templates for either daemonsets or deployments need to be added or modified based on the existing contents of the helm chart.
There are also different workloads k8s supports so based on requirements they can be picked appropriately.
I wish to access Kubernetes APIs from my local machine. I'm trying to get list of pods using kubernetes Rest APIs.
I've created a kubernetes cluster and some pods on Google Cloud.
On my local Windows machine, I've installed gcloud sdk and kubectl component with it.
I connected to my cluster using:
gcloud container clusters get-credentials my-cluster --region us-central1 --project my-project
I can get the list of pods using kubectl get pods
Although, I want to get pods list using kubernetes Rest APIs.
GET https://kubernetes.default/api/v1/namespaces/default/pods
Authorization: Bearer my_access_token
But I think the request is not going through.
In Postman, I get the error:
Error: tunneling socket could not be established, cause=socket hang up
Or in Python using requests library (from my local machine), I get the error
HTTPSConnectionPool(host='kubernetes.default', port=443): Max retries exceeded with url: /api/v1/namespaces/default/pods (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x00000277DCD04D90>: Failed to establish a new connection: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond'))
What am I missing here?
The endpoint https://kubernetes.default only works if you want to access Kubernetes REST API from inside the cluster i.e from another pod. For accessing Kubernetes REST API from outside the kubernetes cluster i.e from your local machine you need to use the API server IP or host which is externally accessible i.e the one which is there in kubeconfig file.
For accessing it from outside the kubernetes cruster i.e from your local machine there are three ways referring from the docs here
Run kubectl in proxy mode (recommended). This method is recommended, since it uses the stored apiserver location and verifies the identity of the API server using a self-signed cert. No man-in-the-middle (MITM) attack is possible using this method.
kubectl proxy --port=8080 &
curl http://localhost:8080/api/v1/namespaces/default/pods
It is possible to avoid using kubectl proxy by passing an authentication token directly to the API server, like this:
Check all possible clusters, as your .KUBECONFIG may have multiple contexts:
kubectl config view -o jsonpath='{"Cluster name\tServer\n"}{range .clusters[*]}{.name}{"\t"}{.cluster.server}{"\n"}{end}'
Select name of cluster you want to interact with from above output:
export CLUSTER_NAME="some_server_name"
Point to the API server referring the cluster name
APISERVER=$(kubectl config view -o jsonpath="{.clusters[?(#.name==\"$CLUSTER_NAME\")].cluster.server}")
Gets the token value
TOKEN=$(kubectl get secrets -o jsonpath="{.items[?(#.metadata.annotations['kubernetes\.io/service-account\.name']=='default')].data.token}"|base64 --decode)
Explore the API with TOKEN
curl -X GET $APISERVER/api/v1/namespaces/default/pods --header "Authorization: Bearer $TOKEN" --insecure
Using client library
To use Python client, run the following command: pip install kubernetes See Python Client Library page for more installation options.
The Python client can use the same kubeconfig file as the kubectl CLI does to locate and authenticate to the API server. See this example:
from kubernetes import client, config
config.load_kube_config()
v1=client.CoreV1Api()
print("Listing pods with their IPs:")
ret = v1.list_pod_for_all_namespaces(watch=False)
for i in ret.items:
print("%s\t%s\t%s" % (i.status.pod_ip, i.metadata.namespace, i.metadata.name))
You can also do it the way you are doing without using kubeconfig file but it's more work and you need to use the kubernetes API Server IP or hostname from the kubeconfig file.
using below kubectl command start a proxy to the Kubernetes API server:
kubectl proxy --port=8080
Get the API versions:
curl http://localhost:8080/api/
The output should look similar to this:
{
"kind": "APIVersions",
"versions": [
"v1"
],
"serverAddressByClientCIDRs": [
{
"clientCIDR": "0.0.0.0/0",
"serverAddress": "10.0.2.15:8443"
}
]
}
Your api server address is not correct for external REST access.
Get the address like this.
kubectl config view
Find your cluster name in the list and get the APi.
Here is the cURL (without the real IP or the token) which worked in my local pc.
curl --location --request GET 'https://nnn.nnn.nnnn.nnn/api/v1/namespaces/develop/pods' \
--header 'Authorization: bearer xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
If you run in POSTMAN, you might have to disable certificate verification.