When I created a given k8 cluster I didn't specify anything specific for service-cluster-ip-range. Now when I create new loadBalancer services k8 is assigning values that walk on existing ip addresses within the network.
Checking the allowed range via kubectl cluster-info dump | grep service-cluster-ip-range gives me:
"--service-cluster-ip-range=10.96.0.0/12"
which (oddly enough) isn't where the assigned values are coming from. New values seem to have started at 10.95.96.235 and incremented from there.
Attempts to preset a valid ip in a service descriptor via spec.loadBalancerIP gives me errors from kubelet:
Failed to allocate IP for "newservice": "10.95.96.233" is not allowed in config
My questions are:
is it possible to change service-cluster-ip-range without rebuilding the entire cluster?
if not, do I have any other options for (pre)setting loadBalancerIP ?
Related
I have set these environment variables inside my pod named main_pod.
$ env
HTTP_PROXY=http://myproxy.com
http_proxy=http://myproxy.com
I also have another dynamic pod in pattern sub_pod-{number} which has a service attached to it called sub_pod-{number}.
So, if I add NO_PROXY=sub_pod-1 environment variable in main_pod, request with URL http://sub_pod-1:5000/health_check will run successfully as it won't be directed through proxy which is fine.
But I want this process to be dynamic. sub_pod_45 might spawn at runtime and sub_pod-1 might get destroyed. Is there any better way to handle this rather than updating NO_PROXY for every pod creation / destruction ?
Is there any resource / network policy / egress rule from which I can tell pod that if domain name belongs to kubernetes service, do not route it through proxy server?
Or can I simply use regex or glob patterns in NO_PROXY env variable like NO_PROXY=sub_pod-* ?
Edited
Result of nslookup
root#tmp-shell:/# nslookup sub_pod-1
Server: 10.43.0.10
Address: 10.43.0.10#53
Name: sub_pod-1.default.svc.cluster.local
Address: 10.43.22.139
When no_proxy=cluster.local,
Proxy bypassed when requested with FQDN
res = requests.get('http://sub_pod-1.default.svc.cluster.local:5000')
Proxy didn't bypass when requested with service name only
res = requests.get('http://sub_pod-1:5000') # I want this to work
I would not want to ask my developers to change the application to use FQDN.
Is there any way cluster can identify if URL resolves to a service present within the network and if it happens do not route the request to proxy ?
Libraries that support the http_proxy environment variable generally also support a matching no_proxy that names things that shouldn't be proxied. The exact syntax seems to vary across languages and libraries but it does seem to be universal that setting no_proxy=example.com causes anything.example.com to not be proxied either.
This is relevant because the Kubernetes DNS system creates its names in a domain based on the cluster name, by default cluster.local. The canonical form of a Service DNS name, for example, is service-name.namespace-name.svc.cluster.local., where service-name and namespace-name are the names of the corresponding Kubernetes objects.
I suspect this means it would work to do two things:
Set an environment variable no_proxy=cluster.local; and
Make sure to use the FQDN form when calling other services, service.namespace.svc.cluster.local.
Pods have similar naming, but are in a pod.cluster.local subdomain. The cluster.local value is configurable at a cluster level and it may be different in your environment.
We have to use two name servers in the pod deployed in the cluster managed by other team.
The coredns for service name resolution of other pods inside cluster
The custom dns server for some external dns queries made by
application inside pod.
We have use the dnsPolicy as ClusterFirst and also specify the nameserver in dnsConfig
as ip of custom dns server and options with ndots = 1.
After deploying the pod we got the correct entries in /etc/resolv.conf with coredns entry as first entry. But when the application tries to resolve some domain, it first query with the absolute name (as option ndots=1 specified in /etc/resolv.conf from the first name server specified in /etc/resolv.conf, after failure it appends the search string that was automatically inserted when we specified the dnsPolicy as ClusterFirst, again query with the first nameserver and then try with second name server.
Why it is not trying the absolute name query with the second name server after failure from first name server, while in case when it append the search string, it query with both the name server in sequential order?
Is there any way we can insert the custom dns entry at top?
Note:- We cannot use the forward functionality of coredns as this dns server is used by other pods/services in the cluster.
I need to use a domain for GKE cluster to access ingress into the cluster and applications, similar like azure AKS http add-on which gives a generic-created domain(not a custom domain)
https://learn.microsoft.com/en-us/azure/aks/http-application-routing
Is there any solution on Google cloud as well?
Our GKE creating/deleting process is a part of IaC tooling and we are automating cluster and our app deployment for dev/test/staging. And the generic domain creation and binding managed dns zone to the cluster resources gives us great flexibility. Otherwise we have to create custom domain and managed dns zone which will be static and bring unnecessary complexity to the provisioning tooling.
GCP has not implemented resources like that, however this operation could be automated using one of the available Cloud DNS APIs 1, as for example the ResourceRecordSets 2 to configure A records to the ManagedZone you want to assign the host, scripting this configuration after the Ingress controller creation.
Example, retrieving the IP address allocated to the ingress controller issuing the command like kubectl describe ing <ingress-name> |grep “Address:” |awk ‘{print $2}’ than using the IP information to construct the API body request 3.
There is not generic domain options in gke so I have to purchase a domain and update NS according to created managed dns zone NS and they will be automated sync when I update ingress in gke by external-dns
I can say I solve this problem with this steps,
1- Create a managed zone which has domain name belongs own and be sure it has permission to access domain from dns zones which you create. Mean is giving access the google project which your dns zone exist
Note: when you create the cluster be sure giving scopes for readwrite perm for managed dns zone
gcloud container clusters create “external-dns” \
—num-nodes 1 \
—scopes “https://www.googleapis.com/auth/ndev.clouddns.readwrite
Create a DNS zone which will contain the managed DNS records.
$ gcloud dns managed-zones create “xxx.test-dev” \
—dns-name “xxx.test.dev.” \
—description “Automatically managed zone by kubernetes.io/external-dns test.dev domain name”
2- Please deploy the resources to gke which name is external-dns
https://github.com/kubernetes-sigs/external-dns/blob/master/docs/tutorials/gke.md#deploy-externaldns
And check the logs with
kubectl logs $(kubectl get pods --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}' | grep dns)
Or
kubectl logs $(kubectl get pods --no-headers -o custom-columns=":metadata.name" | grep dns)
And if you see something like everything is going smoothly
time="2021-01-20T11:37:46Z" level=info msg="Add records: xxx.test.dev. A [34.89.xx.xx] 300"
time="2021-01-20T11:37:46Z" level=info msg="Add records: xxx.test.dev. TXT [\"heritage=external-dns,external-dns/owner=my-identifier,external-dns/resource=ingress/default/ingress-test\"] 300"
time="2021-01-20T11:38:47Z" level=info msg="All records are already up to date"
Note created TXT record alongside A record. TXT record signifies that the corresponding A record is managed by ExternalDNS. This makes ExternalDNS safe for running in environments where there are other records managed via other means.
Let’s check that we can resolve this DNS name. We’ll ask the nameservers assigned to your zone first.
$ dig +short #ns-cloud-e1.googledomains.com. xxx.test.dev.
104.155.xx.xx
And you can check the ip of the domain is correct or has a problem
host https://xxx.test.dev/
Host https://xxx.test.dev/ not found: 3(NXDOMAIN)
It can be complained bed domain for a while but then you will get the correct response
host xxx.test.dev
xxx.test.dev has address 35.197.xx.xx
I have
an openstack, it is Queens, it has octavia for lbaas
a small (test) k8s cluster on top of it (3 nodes, 1 master), version 9.1.2
a deployment called hello which serves a simple webpage saying 'hello world', it works when accessed from within the cluster
I want to expose my deployment as a load balanced service with a floating IP.
I did kubectl expose deployment hello --type=LoadBalancer --name=my-service
It says (kubectl describe service my-service)
Error creating load balancer (will retry): failed to ensure load balancer for service default/my-service: error getting floating ip for port 9cc6442b-2b2f-4b6a-8f91-65dbc2ff13d0: Resource not found
If I manually do: openstack floating ip --port 9cc6442b-2b2f-4b6a-8f91-65dbc2ff13d0 356c8ffa-7bc2-43a9-a8d3-29147ae01727
where:
| ID | Floating IP Address | Port | Floating Network |
| 356c8ffa-7bc2-43a9-a8d3-29147ae01727 | 172.27.81.241 | None | eb31cc74-96ba-4394-aef4-0e94bec46d85 |
and /etc/kubernetes/cloud_config has:
[LoadBalancer]
subnet-id=6a6cdc35-8dda-4982-850e-53c6ee5a5085
floating-network-id=eb31cc74-96ba-4394-aef4-0e94bec46d85
use-octavia=True
(so it is looking for floating IPs on the correct network, and that subnet is the k8s internal subnet)
It all works.
So everything except "associate an IP" has worked. Why does this step fail? Where has k8s logged what it did and how it failed? I can only find docs for pod level logging (and my pod is fine, and serving it's test webpage just great).
(I have lots of quota remaining for 'make more floating ips', and several unused ones hanging around)
I was able to find this No Ports Available when trying to associate a floating IP and this Failed to Associate Floating IP. Maybe those will point you into right direction.
I would recommend that you check this page OpenStack community and look for more answers as I'm not an expert in OpenStack.
As for your question
Where has k8s logged what it did and how it failed?
You can use kubectl describe service <service_name>
Show details of a specific resource or group of resources
Print a detailed description of the selected resources, including related resources such as events or controllers. You may select a single object by name, all objects of that type, provide a name prefix, or label selector. For example:
$ kubectl describe TYPE NAME_PREFIX
For mode debug description please check Debug Services.
I need a way to get service cluster ip range (as CIDR) that works accross all Kubernetes clusters.
I tried the following, which works fine for clusters created with kubeadm as it greps arguments of apiserver pod:
$ kubectl cluster-info dump | grep service-cluster-ip-range
"--service-cluster-ip-range=10.96.0.0/12",
This does not work on all Kubernetes clusters, i.e. gcloud
So the question is, what is the best way to get service ip range programatically?
I don't think there is a way to access such information through K8s Api, there is an open issue to address lack of this functionality: https://github.com/kubernetes/kubernetes/issues/25533 . If you have access to the etcd of the k8s cluster in question, then there is a key with information about service cidr range: /registry/ranges/serviceips . You can get the value, by using etcdctl (assuming, you have the proper permissions): etcdctl --enpoints=<your etcd> --<any authentication flags> get "/registry/ranges/serviceips" --prefix=true.