PROBLEM:
For some reason client pod can only resolve fully qualified fully-qualified DNS names including the cluster service suffix.
That problem is stated in this quesion:
AKS, WIndows Node, dns does not resolve service until fully qualified name is used
To workaround this issue, I'm using useServiceDnsDomain flag. The documentation (https://strimzi.io/docs/operators/master/using.html#type-GenericKafkaListenerConfiguration-schema-reference ) explains it as
Configures whether the Kubernetes service DNS domain should be used or
not. If set to true, the generated addresses with contain the service
DNS domain suffix (by default .cluster.local, can be configured using
environment variable KUBERNETES_SERVICE_DNS_DOMAIN). Defaults to
false.This field can be used only with internal type listener.
Part of my yaml is as follows
apiVersion: kafka.strimzi.io/v1beta1
kind: Kafka
metadata:
name: tt-kafka
namespace: shared
spec:
kafka:
version: 2.5.0
replicas: 3
listeners:
- name: local
port: 9092
type: internal
tls: false
useServiceDnsDomain: true
This didn't do anything, so I also tried adding KUBERNETES_SERVICE_DNS_DOMAIN like below
template:
kafkaContainer:
env:
- name: KUBERNETES_SERVICE_DNS_DOMAIN
value: .cluster.local
strimzi/operator:0.20.0 image is being used.
In my client (.net Confluent.Kafka 1.4.4), I used tt-kafka-kafka-bootstrap.shared.svc.cluster.local as BootstrapServers.
It gives me error as
Error: GroupCoordinator: Failed to resolve
'tt-kafka-kafka-2.tt-kafka-kafka-brokers.shared.svc:9092': No such
host is known.
I'm expecting the broker service to provide full names to the client but from the error it seems useServiceDnsDomain has no effect.
Any help is appreciated. Thanks.
As exmaplained in https://github.com/strimzi/strimzi-kafka-operator/issues/3898, there is a typo in the docs. The right YAML is:
listeners:
- name: plain
port: 9092
type: internal
tls: false
configuration:
useServiceDnsDomain: true
If your domain is different from .cluster.local, you can use the KUBERNETES_SERVICE_DNS_DOMAIN env var to override it. But you have to configure it on the Strimzi Cluster Operator pod. Not on the Kafka pods: https://strimzi.io/docs/operators/latest/full/using.html#ref-operator-cluster-str
Related
I am running Kafka connect worker from Openshift cluster and able to access the connector APIs from the PODs terminal like below (listener in connect-distributed.properties as https://localhost:20000):
sh-4.2$ curl -k -X GET https://localhost:20000
{"version":"5.5.0-ce","commit":"dad78e2df6b714e3","kafka_cluster_id":"XojxTYmbTXSwHguxJ_flWg"}
I have created OpenShift routes with below config:
- kind: Route
apiVersion: v1
metadata:
name: '${APP_SHORT_NAME}-route'
labels:
app: '${APP_SHORT_NAME}'
annotations:
description: Route for application's http service.
spec:
host: '${APP_SHORT_NAME}.${ROUTE_SUFFIX}'
port:
targetPort: 20000-tcp
tls:
termination: reencrypt
destinationCACertificate: '${DESTINATION_CA_CERTIFICATE}'
to:
kind: Service
name: '${APP_NAME}-service'
Port 20000 is exposed from Dockerfile but the route URL is throwing below error instead:
Possible reasons you are seeing this page:
The host doesn't exist. Make sure the hostname was typed correctly and that a route matching this hostname exists.
The host exists, but doesn't have a matching path. Check if the URL path was typed correctly and that the route was created using the desired path.
Route and path matches, but all pods are down. Make sure that the resources exposed by this route (pods, services, deployment configs, etc) have at least one pod running
Same OpenShift route URL works fine with a normal Spring boot service but Kafka connect worker URL is not getting bind to the Route URL create as above. (Note the Kafka connect worker is running fine along with the logs in the OpenShift pods)
Set the listeners back to the default of bind-address 0.0.0.0 so that external connections can be accepted.
If you need a different port, you can use port forwarding in the k8s service, rather than do that at the application config.
If you're not using Strimzi (which doesn't look like you are, based on 5.5.0-ce), you'll also want to add this env-var so a cluster can be formed.
- name: CONNECT_REST_ADVERTISED_HOST_NAME
valueFrom:
fieldRef:
fieldPath: status.podIP
I am trying to set up a multi-cluster architecture. I have a Spring Boot API that I want to run on a second cluster (for isolation purposes). I have set that up using the gateway.networking.k8s.io API. I am using a Gateway that has an SSL certificate and matches an IP address that's registered to my domain in the DNS registry. I am then setting up an HTTPRoute for each service that I am running on the second cluster. That works fine and I can communicate between our clusters and everything works as intended but there is a problem:
There is a timeout of 30s by default and I cannot change it. I want to increase it as the application in the second cluster is a WebSocket and I obviously would like our WebSocket connections to stay open for more than 30s at a time. I can see that in the backend service that's created from our HTTPRoute there is a timeout specified as 30s. I found a command to increase it gcloud compute backend-services update gkemcg1-namespace-store-west-1-8080-o1v5o5p1285j --timeout=86400
When I run that command it would increase the timeout and the webSocket connection will be kept alive. But after a few minutes this change gets overridden (I suspect that it's because it's managed by the yaml file). This is the yaml file for my backend service
kind: HTTPRoute
apiVersion: gateway.networking.k8s.io/v1beta1
metadata:
name: public-store-route
namespace: namespace
labels:
gateway: external-http
spec:
hostnames:
- "my-website.example.org"
parentRefs:
- name: external-http
rules:
- matches:
- path:
type: PathPrefix
value: /west
backendRefs:
- group: net.gke.io
kind: ServiceImport
name: store-west-1
port: 8080
I have tried to add either a timeout, timeoutSec, or timeoutSeconds under every level with no success. I always get the following error:
error: error validating "public-store-route.yaml": error validating data: ValidationError(HTTPRoute.spec.rules[0].backendRefs[0]): unknown field "timeout" in io.k8s.networking.gateway.v1beta1.HTTPRoute.spec.rules.backendRefs; if you choose to ignore these errors, turn validation off with --validate=false
Surely there must be a way to configure this. But I wasn't able to find anything in the documentation referring to a timeout. Am I missing something here?
How do I configure the timeout?
Edit:
I have found this resource: https://cloud.google.com/kubernetes-engine/docs/how-to/configure-gateway-resources
I have been trying to set up a LBPolicy and attatch it it the Gateway, HTTPRoute, Service, or ServiceImport but nothing has made a difference. Am I doing something wrong or is this not working how it is supposed to? This is my yaml:
kind: LBPolicy
apiVersion: networking.gke.io/v1
metadata:
name: store-timeout-policy
namespace: sandstone-test
spec:
default:
timeoutSec: 50
targetRef:
name: public-store-route
group: gateway.networking.k8s.io
kind: HTTPRoute
I'm messing around with kubernetes and I've set up a kluster on my local pc using kind. I have also installed traefik as an ingress controller, and I have already managed to access an api that I have deployed in the kluster and a grafana through some ingress (without doing port forwards or anything like that). But with mongo I can't. While the API and grafana need an IngressRoute, mongo needs an IngressRouteTCP
The IngressRouteTCP that I have defined is this:
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRouteTCP
metadata:
name: mongodb-ingress-tcp
namespace: mongo_namespace
spec:
entryPoints:
- web
routes:
- match: HostSNI(`mongo.localhost`)
services:
- name: mongodb
port: 27017
But I get this error:
I know i can use a port forward, but I want do in this way (with ingress)
A lot of thanks
You need to specify tls parameters
like this
tls:
certResolver: "bar"
domains:
- main: "snitest.com"
sans:
- "*.snitest.com"
To avoid using TLS u need to match all routes HostSNI(`*`)
It is important to note that the Server Name Indication is an extension of the TLS protocol.
Hence, only TLS routers will be able to specify a domain name with that rule.
However, there is one special use case for HostSNI with non-TLS routers: when one wants a non-TLS router that matches all (non-TLS) requests, one should use the specific HostSNI(`*`) syntax.
DOCS
I have two services, say svcA and svcB that may sit in different namespaces or even in different k8s clusters. I want to configure the services so that svcA can refer to svcB using some constant address, then deploy an Istio Service Entry object depending on the environment to route the request. I will use Helm to do the deployment, so using a condition to choose the object to deploy is easy.
If svcB is in a completely different cluster, it is just like any external server and is easy to configure.
But when it is in a different namespace on the same cluster, I just could not get the Service Entry work. Maybe I don't understand all the options it provides.
Istio objects
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: demo-gateway
spec:
selector:
istio: ingressgateway # use Istio default gateway implementation
servers:
- port:
number: 80
name: http
protocol: HTTP
hosts:
- "*"
---
apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
name: svcB-se
spec:
hosts:
- svcB.alias
ports:
- number: 80
name: http
protocol: HTTP2
location: MESH_INTERNAL
resolution: svcB.svcb-ns.svc.cluster.local
Update
After doing some random/crazy test, I found that the alias domain name must ends with well know suffix, like .com, .org, arbitrary suffix, like .svc, .alias, won't work.
If I update the above ServiceEntry object to like this. My application works.
apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
name: svcB-se
spec:
hosts:
- svcB.com
ports:
- number: 80
name: http
protocol: HTTP2
location: MESH_INTERNAL
resolution: svcB.svcb-ns.svc.cluster.local
I searched for a while and checked the Istio documentations, but could not find any reference about domain name suffix restrictions.
Is it implicit knowledge that only domain names like .com and .org are valid? I have left school for a long time.
I have posted community wiki answer to summarize the topic and paste explanation of the problem:
After doing some random/crazy test, I found that the alias domain name must ends with well know suffix, like .com, .org, arbitrary suffix, like .svc, .alias, won't work.
If I update the above ServiceEntry object to like this. My application works.
apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
name: svcB-se
spec:
hosts:
- svcB.com
ports:
- number: 80
name: http
protocol: HTTP2
location: MESH_INTERNAL
resolution: svcB.svcb-ns.svc.cluster.local
I searched for a while and checked the Istio documentations, but could not find any reference about domain name suffix restrictions.
Is it implicit knowledge that only domain names like .com and .org are valid? I have left school for a long time.
Explanation:
You can find ServiceEntry requirements in the offical documentation. You can find description how you can properly set this value:
The hosts associated with the ServiceEntry. Could be a DNS name with wildcard prefix.
The hosts field is used to select matching hosts in VirtualServices and DestinationRules.
For HTTP traffic the HTTP Host/Authority header will be matched against the hosts field.
For HTTPs or TLS traffic containing Server Name Indication (SNI), the SNI value will be matched against the hosts field.
NOTE 1: When resolution is set to type DNS and no endpoints are specified, the host field will be used as the DNS name of the endpoint to route traffic to.
NOTE 2: If the hostname matches with the name of a service from another service registry such as Kubernetes that also supplies its own set of endpoints, the ServiceEntry will be treated as a decorator of the existing Kubernetes service. Properties in the service entry will be added to the Kubernetes service if applicable. Currently, the only the following additional properties will be considered by istiod:
subjectAltNames: In addition to verifying the SANs of the service accounts associated with the pods of the service, the SANs specified here will also be verified.
Based on this issue you don't have to use FQDN in your hosts field, but you need to set proper value to select matching hosts in VirtualServices and DestinationRules.
I've got a kubernetes 1.6.2 cluster, and am creating a service like:
kind: Service
apiVersion: v1
metadata:
name: hello
namespace: myns
annotations:
service.beta.kubernetes.io/aws-load-balancer-internal: 0.0.0.0/0
dns.alpha.kubernetes.io/internal: mydomain.com
spec:
selector:
app: hello-world
ports:
- protocol: "TCP"
port: 80
targetPort: 5000
type: LoadBalancer
I'd expect this to create an internal ELB (which it does) but also set up an A record on the AWS Route53 hosted zone for mydomain.com as per https://github.com/kubernetes/kops/tree/master/dns-controller (which it doesn't). Is there something I need to do to enable A record creation?
In route 53 create a type A record with Alias Yes prior to launching your cluster and Kurbernetes will audio update it with proper Alias Target which gets resolved to the correct IP upon app boot up when you issue
kubectl expose rs .....
This might not work since dns.alpha.kubernetes.io/internal requires that you use NodePort, at least that's what's written in here https://github.com/kubernetes/kops/issues/1082
Update:
There's an open issue about this A-record. CNAME record can be created but A record is not yet working. Forgot the issue number but I think you can find it from kops github issues.