I need to use a domain for GKE cluster to access ingress into the cluster and applications, similar like azure AKS http add-on which gives a generic-created domain(not a custom domain)
https://learn.microsoft.com/en-us/azure/aks/http-application-routing
Is there any solution on Google cloud as well?
Our GKE creating/deleting process is a part of IaC tooling and we are automating cluster and our app deployment for dev/test/staging. And the generic domain creation and binding managed dns zone to the cluster resources gives us great flexibility. Otherwise we have to create custom domain and managed dns zone which will be static and bring unnecessary complexity to the provisioning tooling.
GCP has not implemented resources like that, however this operation could be automated using one of the available Cloud DNS APIs 1, as for example the ResourceRecordSets 2 to configure A records to the ManagedZone you want to assign the host, scripting this configuration after the Ingress controller creation.
Example, retrieving the IP address allocated to the ingress controller issuing the command like kubectl describe ing <ingress-name> |grep “Address:” |awk ‘{print $2}’ than using the IP information to construct the API body request 3.
There is not generic domain options in gke so I have to purchase a domain and update NS according to created managed dns zone NS and they will be automated sync when I update ingress in gke by external-dns
I can say I solve this problem with this steps,
1- Create a managed zone which has domain name belongs own and be sure it has permission to access domain from dns zones which you create. Mean is giving access the google project which your dns zone exist
Note: when you create the cluster be sure giving scopes for readwrite perm for managed dns zone
gcloud container clusters create “external-dns” \
—num-nodes 1 \
—scopes “https://www.googleapis.com/auth/ndev.clouddns.readwrite
Create a DNS zone which will contain the managed DNS records.
$ gcloud dns managed-zones create “xxx.test-dev” \
—dns-name “xxx.test.dev.” \
—description “Automatically managed zone by kubernetes.io/external-dns test.dev domain name”
2- Please deploy the resources to gke which name is external-dns
https://github.com/kubernetes-sigs/external-dns/blob/master/docs/tutorials/gke.md#deploy-externaldns
And check the logs with
kubectl logs $(kubectl get pods --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}' | grep dns)
Or
kubectl logs $(kubectl get pods --no-headers -o custom-columns=":metadata.name" | grep dns)
And if you see something like everything is going smoothly
time="2021-01-20T11:37:46Z" level=info msg="Add records: xxx.test.dev. A [34.89.xx.xx] 300"
time="2021-01-20T11:37:46Z" level=info msg="Add records: xxx.test.dev. TXT [\"heritage=external-dns,external-dns/owner=my-identifier,external-dns/resource=ingress/default/ingress-test\"] 300"
time="2021-01-20T11:38:47Z" level=info msg="All records are already up to date"
Note created TXT record alongside A record. TXT record signifies that the corresponding A record is managed by ExternalDNS. This makes ExternalDNS safe for running in environments where there are other records managed via other means.
Let’s check that we can resolve this DNS name. We’ll ask the nameservers assigned to your zone first.
$ dig +short #ns-cloud-e1.googledomains.com. xxx.test.dev.
104.155.xx.xx
And you can check the ip of the domain is correct or has a problem
host https://xxx.test.dev/
Host https://xxx.test.dev/ not found: 3(NXDOMAIN)
It can be complained bed domain for a while but then you will get the correct response
host xxx.test.dev
xxx.test.dev has address 35.197.xx.xx
Related
I have a project A in which I have created a service account.
I want to create a GKE in project B.
I followed the steps of service account impersonation listed here https://cloud.google.com/iam/docs/impersonating-service-accounts
in project A,
the default-service-accounts of project B have roles/iam.serviceAccountTokenCreator and roles/iam.serviceAccountUser on the service account I created which is my-service-account
in project B,
my-service-account has Kubernetes admin role
When I try to create, I end up with the error
Error: Error waiting for creating GKE NodePool: All cluster resources were brought up, but: only 0 nodes out of 1 have registered; cluster may be unhealthy.
I am using terraform to create this cluster and the service account being used by terraform has kubernetes admin and service account user role.
This is what it shows in the console
GKE error
Edit:
I tried using Gcloud command line to create GKE
gcloud beta container --project "my-project" clusters create "test-gke-sa" --zone "us-west1-a" --no-enable-basic-auth --cluster-version "1.18.16-gke.502" --release-channel "regular" --machine-type "e2-standard-16" --image-type "COS" --disk-type "pd-standard" --disk-size "100" --metadata disable-legacy-endpoints=true --scopes "https://www.googleapis.com/auth/devstorage.read_only","https://www.googleapis.com/auth/logging.write","https://www.googleapis.com/auth/monitoring","https://www.googleapis.com/auth/servicecontrol","https://www.googleapis.com/auth/service.management.readonly","https://www.googleapis.com/auth/trace.append" --num-nodes "3" --enable-stackdriver-kubernetes --enable-private-nodes --master-ipv4-cidr "192.168.0.16/28" --enable-ip-alias --network "projects/infgprj-sbo-n-hostgs-gl-01/global/networks/my-network" --subnetwork "projects/my-network/regions/us-west1/subnetworks/my-subnetwork" --cluster-secondary-range-name "gke1-pods" --services-secondary-range-name "gke1-services" --default-max-pods-per-node "110" --no-enable-master-authorized-networks --addons HorizontalPodAutoscaling,HttpLoadBalancing,GcePersistentDiskCsiDriver --enable-autoupgrade --enable-autorepair --max-surge-upgrade 1 --max-unavailable-upgrade 0 --enable-shielded-nodes --shielded-secure-boot --node-locations "us-west1-a" --service-account="my-service-account#project-a.iam.gserviceaccount.com"
Got the same errors.
I see that the node-pool is created, but not nodes. (or atleast they are not attached to the node-pool?)
here are some more pics of errors
VM page
GKE page
Solution: Finally, I figured, what was wrong. I had given token creator role to only default service accounts. It started working when I gave the same role to default service agents as well. So basically
role = "roles/iam.serviceAccountTokenCreator",
members = [
"serviceAccount:{project-number}-compute#developer.gserviceaccount.com",
"serviceAccount:service-{project-number}#container-engine-robot.iam.gserviceaccount.com",
"serviceAccount:service-{project-number}#compute-system.iam.gserviceaccount.com",
]
Just to confirm that it's a service account error and not something involving Terraform, I recommend that you:
A. impersonate Project A's service account and confirm that you are who you're trying to be with this command - gcloud auth list (the active account is the one with the star next to it), and then
B. try creating a cluster in Project B with gcloud container clusters create - here are the reference docs but you can also:
go to Console > Kubernetes Engine
click on "Create,"
scroll down to the bottom of the form and click on the "COMMAND LINE" link to launch a modal that generates the syntax of the CLI command you'd want to run
copy, paste, tweak to make it create only one node and what other basic settings you want to change...make sure it's specifying --project=project-B
run the command
That will likely give you a more helpful error message. Or at least a different one, so, hurray?
Usually the above error may be caused by following reasons
1] If Shared VPC, verify IAM permissions are correct.
2] Verify Auto generated Ingress Firewall Rules are created
Usually three firewall rules are created
gke-${cluster_name}-${random_char}-all : Firewall Rule for pod to pod communication
gke-${cluster_name}-${random_char}-master : Rule for Master to talk to Nodes
gke-${cluster_name}-${random_char}-vms : Node to Node communication
random char: Random Character
3] Check firewall rules for denial of egress.
By default GCP creates a firewall rule of allowing all egress. If the you delete the rule or denies all egress, then you must configure a firewall rule that allows egress on the master CIDR block via tcp ports 443, 10250. Private Cluster Firewall Rules Private Cluster Firewall Rules documents how to obtain the master CIDR block.
-If you enable other GKE Add-Ons you may require adding additional egress firewall rules.
4] Check DNS Configuration for communication to Google APIs.
Leverage Kubelet logs to check for any curl failed request. Ex: Unable to resolve host or Connection Timeout during kubelet installation. There may be a chance that dns configuration is incorrect (ex resolve Private Google API's or hitting public google APIs). A dig command or looking at 'etc/resolv.conf' for dns servers should confirm where requests are being routed to.
When I created a given k8 cluster I didn't specify anything specific for service-cluster-ip-range. Now when I create new loadBalancer services k8 is assigning values that walk on existing ip addresses within the network.
Checking the allowed range via kubectl cluster-info dump | grep service-cluster-ip-range gives me:
"--service-cluster-ip-range=10.96.0.0/12"
which (oddly enough) isn't where the assigned values are coming from. New values seem to have started at 10.95.96.235 and incremented from there.
Attempts to preset a valid ip in a service descriptor via spec.loadBalancerIP gives me errors from kubelet:
Failed to allocate IP for "newservice": "10.95.96.233" is not allowed in config
My questions are:
is it possible to change service-cluster-ip-range without rebuilding the entire cluster?
if not, do I have any other options for (pre)setting loadBalancerIP ?
I have
an openstack, it is Queens, it has octavia for lbaas
a small (test) k8s cluster on top of it (3 nodes, 1 master), version 9.1.2
a deployment called hello which serves a simple webpage saying 'hello world', it works when accessed from within the cluster
I want to expose my deployment as a load balanced service with a floating IP.
I did kubectl expose deployment hello --type=LoadBalancer --name=my-service
It says (kubectl describe service my-service)
Error creating load balancer (will retry): failed to ensure load balancer for service default/my-service: error getting floating ip for port 9cc6442b-2b2f-4b6a-8f91-65dbc2ff13d0: Resource not found
If I manually do: openstack floating ip --port 9cc6442b-2b2f-4b6a-8f91-65dbc2ff13d0 356c8ffa-7bc2-43a9-a8d3-29147ae01727
where:
| ID | Floating IP Address | Port | Floating Network |
| 356c8ffa-7bc2-43a9-a8d3-29147ae01727 | 172.27.81.241 | None | eb31cc74-96ba-4394-aef4-0e94bec46d85 |
and /etc/kubernetes/cloud_config has:
[LoadBalancer]
subnet-id=6a6cdc35-8dda-4982-850e-53c6ee5a5085
floating-network-id=eb31cc74-96ba-4394-aef4-0e94bec46d85
use-octavia=True
(so it is looking for floating IPs on the correct network, and that subnet is the k8s internal subnet)
It all works.
So everything except "associate an IP" has worked. Why does this step fail? Where has k8s logged what it did and how it failed? I can only find docs for pod level logging (and my pod is fine, and serving it's test webpage just great).
(I have lots of quota remaining for 'make more floating ips', and several unused ones hanging around)
I was able to find this No Ports Available when trying to associate a floating IP and this Failed to Associate Floating IP. Maybe those will point you into right direction.
I would recommend that you check this page OpenStack community and look for more answers as I'm not an expert in OpenStack.
As for your question
Where has k8s logged what it did and how it failed?
You can use kubectl describe service <service_name>
Show details of a specific resource or group of resources
Print a detailed description of the selected resources, including related resources such as events or controllers. You may select a single object by name, all objects of that type, provide a name prefix, or label selector. For example:
$ kubectl describe TYPE NAME_PREFIX
For mode debug description please check Debug Services.
Newbie setup :
Created First project in GCP
Created cluster with default, 3 nodes. Node version 1.7.6. cluster master version 1.7.6-gke.1.
Deployed aan application in a pod, per example.
Able to access "hello world" and the hostname, using the external-ip and the port.
In GCP / GKE webpage of my cloud console, clicked "discovery and loadbalancing", I was able to see the "kubernetes-dashboard" process in green-tick, but cannot access throught the IP listed. tried 8001,9090, /ui and nothing worked.
not using any cloud shell or gcloud commands on my local laptop. Everything is done on console.
Questions :
How can anyone access the kubernetes-dashboard of the cluster created in console?
docs are unclear, are the dashboard components incorporated in the console itself? Are the docs out of sync with GCP-GKE screens?
tutorial says run "kubectl proxy" and then to open
"http://localhost:8001/ui", but it doesnt work, why?
If you create a cluster with with version 1.9.x or greater, then u can access using tokens.
get secret.
kubectl -n kube-system describe secrets `kubectl -n kube-system get secrets | awk '/clusterrole-aggregation-controller/ {print $1}'` | awk '/token:/ {print $2}'
Copy secret.
kubectl proxy.
Open UI using 127.0.0.1:8001/ui. This will redirect to login page.
there will be two options to login, kubeconfig and token.
Select token and paste the secret copied earlier.
hope this helps
It seems to be an issue with the internal Kubernetes DNS service starting at version 1.7.6 on Google Cloud.
The solution is to access the dashboard at this endpoint instead:
http://localhost:8001/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard
Github Issue links:
https://github.com/kubernetes/dashboard/issues/2368
https://github.com/kubernetes/kubernetes/issues/52729
The address of the dashboard service is only accessible from inside of the cluster. If you ssh into a node in your cluster, you should be able to connect to the dashboard. You can verify this by noticing that the address is within the services CIDR range for your cluster.
The dashboard in running as a pod inside of your cluster with an associated service. If you open the Workloads view you will see the kubernetes-dashboard deployment and can see the pod that was created by the deployment. I'm not sure which docs you are referring to, since you didn't provide a link.
When you run kubectl proxy it creates a secure connection from your local machine into your cluster. It works by connecting to your master and then running through a proxy on the master to the pod/service/host that you are connecting to via an ssh tunnel. It's possible that it isn't working because the ssh tunnels are not running; you should verify that your project has newly created ssh rules allowing access from the cluster endpoint IP address. Otherwise, if you could explain more about how it fails, that would be useful for debugging.
First :
gcloud container clusters get-credentials cluster-1 --zone my-zone --project my-project
Then find your kubernetes dashboard endpoint doing :
kubectl cluster-info
It will be something like https://42.42.42.42/api/v1/namespaces/kube-system/services/kubernetes-dashboard/proxy
Install kube-dashboard
kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v1.10.1/src/deploy/recommended/kubernetes-dashboard.yaml
Run:
$ kubectl proxy
Access:
http://localhost:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/#!/login
I need a way to get service cluster ip range (as CIDR) that works accross all Kubernetes clusters.
I tried the following, which works fine for clusters created with kubeadm as it greps arguments of apiserver pod:
$ kubectl cluster-info dump | grep service-cluster-ip-range
"--service-cluster-ip-range=10.96.0.0/12",
This does not work on all Kubernetes clusters, i.e. gcloud
So the question is, what is the best way to get service ip range programatically?
I don't think there is a way to access such information through K8s Api, there is an open issue to address lack of this functionality: https://github.com/kubernetes/kubernetes/issues/25533 . If you have access to the etcd of the k8s cluster in question, then there is a key with information about service cidr range: /registry/ranges/serviceips . You can get the value, by using etcdctl (assuming, you have the proper permissions): etcdctl --enpoints=<your etcd> --<any authentication flags> get "/registry/ranges/serviceips" --prefix=true.