I'm trying to use this confluent helm chart:
https://github.com/confluentinc/cp-helm-charts
I know it is not maintained anymore, but I cannot use the another one, because of firewall issue.
I installed it on our dev Openshift 4 cluster, but I'm having problem when I test my application from local. I cannot access to the cluster from local.
The error message:
2022-12-05 10:46:40.532 WARN 15096 --- [ntainer#0-0-C-1] org.apache.kafka.clients.NetworkClient : [Consumer clientId=consumer-consumer-1, groupId=consumer] Error connecting to node prf-confluent-cp-kafka-0.prf-confluent-cp-kafka-headless.lapc-dev:9092 (id: 2147483647 rack: null)
java.net.UnknownHostException: No such host is known (prf-confluent-cp-kafka-0.prf-confluent-cp-kafka-headless.lapc-dev)
I enabled the the external listeners:
"advertised.listeners": |-
EXTERNAL://${HOST_IP}:$((32090 + ${KAFKA_BROKER_ID}))
"listener.security.protocol.map": |-
PLAINTEXT:PLAINTEXT,EXTERNAL:PLAINTEXT
and the nodeport:
nodeport:
enabled: true
servicePort: 9092
firstListenerPort: 32090
I see the following in the logs (I masked the ip with "x"):
$ kubectl logs prf-confluent-cp-kafka-0 --container cp-kafka-broker | grep liste
advertised.listeners = PLAINTEXT://prf-confluent-cp-kafka-0.prf-confluent-cp-kafka-headless.lapc-dev:9092,EXTERNAL://10.xxx.x.xx:32090
control.plane.listener.name = null
controller.listener.names = null
early.start.listeners = null
inter.broker.listener.name = null
listener.security.protocol.map = PLAINTEXT:PLAINTEXT,EXTERNAL:PLAINTEXT
listeners = PLAINTEXT://0.0.0.0:9092,EXTERNAL://0.0.0.0:32090
remote.log.metadata.manager.listener.name = null
How can I troubleshoot this error?
It looks like I used a wrong port. Now I'm using this and it is working.
nodeport:
enabled: true
servicePort: 19092
firstListenerPort: 32090
I suggest you open your firewall to download and use Strimzi, which RedHat maintains to be compatible with OpenShift.
Kubernetes service names will never be accessible from your local machine. You'll need to either kubectl port-forward, or find if there's a NodePort exposed for the bootstrap address. Look with kubectl get svc (or whatever the equivalent oc command is), and connect to the IP:port of the exposed node.
Related
Helm version: v3.5.2
Kubernetes version: v1.20.4
nifi chart version:latest : 1.0.2 rel
Issue: [cetic/nifi]-issue
I'm trying to connect to nifi UI deployed in kubernetes.
I have set following properties in values yaml
properties:
# use externalSecure for when inbound SSL is provided by nginx-ingress or other external mechanism
sensitiveKey: changeMechangeMe # Must to have minimal 12 length key
algorithm: NIFI_PBKDF2_AES_GCM_256
externalSecure: false
isNode: false
httpsPort: 8443
webProxyHost: 10.0.39.39:30666
clusterPort: 6007
# ui service
service:
type: NodePort
httpsPort: 8443
nodePort: 30666
annotations: {}
# loadBalancerIP:
## Load Balancer sources
## https://kubernetes.io/docs/tasks/access-application-cluster/configure-cloud-provider-firewall/#restrict-access-for-loadbalancer-service
##
# loadBalancerSourceRanges:
# - 10.10.10.0/24
## OIDC authentication requires "sticky" session on the LoadBalancer for JWT to work properly...but AWS doesn't like it on creation
# sessionAffinity: ClientIP
# sessionAffinityConfig:
# clientIP:
# timeoutSeconds: 10800
10.0.39.39 - is the kubernetes masternode internal ip.
When nifi get started i get follwoing
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /home/k8sadmin/.kube/config
WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /home/k8sadmin/.kube/config
NAME: nifi
LAST DEPLOYED: Thu Nov 25 12:38:00 2021
NAMESPACE: jeed-cluster
STATUS: deployed
REVISION: 1
NOTES:
Cluster endpoint IP address will be available at:
kubectl get svc nifi -n jeed-cluster -o jsonpath='{.status.loadBalancer.ingress[*].ip}'
Cluster endpoint domain name is: 10.0.39.39:30666 - please update your DNS or /etc/hosts accordingly!
Once you are done, your NiFi instance will be available at:
https://10.0.39.39:30666/nifi
and when i do a curl
curl https://10.0.39.39:30666 put sample.txt -k
<h1>System Error</h1>
<h2>The request contained an invalid host header [<code>10.0.39.39:30666</
the request [<code>/</code>]. Check for request manipulation or third-part
t.</h2>
<h3>Valid host headers are [<code>empty
<ul><li>127.0.0.1</li>
<li>127.0.0.1:8443</li>
<li>localhost</li>
<li>localhost:8443</li>
<li>[::1]</li>
<li>[::1]:8443</li>
<li>nifi-0.nifi-headless.jeed-cluste
<li>nifi-0.nifi-headless.jeed-cluste
<li>10.42.0.8</li>
<li>10.42.0.8:8443</li>
<li>0.0.0.0</li>
<li>0.0.0.0:8443</li>
</ul>
Tried lot of things but still cannot whitelist master node ip in
proxy hosts
Ingress is not used
edit: it looks like properties set in values.yaml is not set in nifi.properties in side the pod. Is there any reason for this?
Appreciate help!
As NodePort service you can also assign port number from 30000-32767.
You can apply values when you install your chart with:
properties:
webProxyHost: localhost
httpsPort:
This should let nifi whitelist your https://localhost:
I have a host running HAProxy already. It's been up and running since before I did anything with Kubernetes. It works flawlessly as a reverse proxy and SSL terminator for various web things in various Docker containers on various other host machines.
Now I have a Kubernetes cluster up and running across some of those other machines. I've created the NodePort Service that exposes port 30080 on each worker node, as follows:
apiVersion: v1
kind: Service
metadata:
name: snginx
labels:
app: nginx
spec:
type: NodePort
externalTrafficPolicy: Local #Cluster or Local
selector:
app: nginx
ports:
- protocol: TCP
port: 8080
targetPort: 80
nodePort: 30080
From the machine running HAProxy (which is not part of the cluster), I can curl the NodePort successfully ( curl 10.0.0.20:30080 ), and I get "Welcome to nginx!..." However, if I set that NodePort as a backend in HAProxy, I get a 503 "No server is available", and HAProxy traffic log says:
localhost haproxy[18489]: [redactedIP]:49399 [30/Aug/2021:19:24:00.764] http-in~ load/load 66/0/-1/-1/67 503 212 - - SC-- 1/1/0/0/3 0/0 "GET / HTTP/1.1"
The haproxy admin log says:
Aug 30 20:07:13 localhost haproxy[18839]: Server load/load is DOWN, reason: Layer4 connection problem, info: "General socket error (Permission denied)"
However, I've disabled the firewall with
sudo systemctl disable --now firewalld
and verified the status is not running. Also, SELinux was disabled when I installed the cluster. Also, I can ping 10.0.0.20 just fine.
"load" is the hostname I'm using for testing load balancing (i.e. load.mydomain.com).
Also, if I use PAT on my physical router to route directly to that NodePort, from outside the building, it works as expected.
What gives? What's the difference between the proxied request and curl?
Thank you.
SELinux is the difference. That is, SELinux on the HAProxy host (not a cluster node):
"SELinux only allows the web server to make outbound connections to a limited set of ports"
That is, you can't make an outbound http request to any port in the NodePort range (30000-32768) without opening that port on the "client", which is the HAProxy server in this case.
sudo semanage port --add --type http_port_t --proto tcp 30080
I am able to get eclipse mosquitto broker up and running with the MQTT clients able to talk to the broker using Broker's IP address. However, as am running these on kubernetes, the broker IP keeps changing on restart. I would like to enable DNS name service for the broker, so the clients can use broker-name instead of the IP. coreDNS is running default in kubernetes..
Any suggestions on what can be done ?
$ nslookup kubernetes.default
Server: 10.43.0.10
Address: 10.43.0.10:53
** server can't find kubernetes.default: NXDOMAIN
** server can't find kubernetes.default: NXDOMAIN
You can achieve that using headless service. You create it by setting the clusterIP field in a service spec to None. Once you do that the DNS server will return the pod IPs instead of the single service and instead of returning a single DNS A record, the DNS server will return multiple A records for the service each pointing to the IP of an individual pod backing the service at the moment.
With this your client can perform a single DNS A record lookup to fetch the IP of all the pods that are part of the service. Headless service is also often used as service discovery system.
apiVersion: v1
kind: Service
metadata:
name: your-headless-service
spec:
clusterIP: None # <-- This makes the service hadless!
selector:
app: your-mosquito-broker-pod
ports:
- protocol: TCP
port: 80
targetPort: 3000
You are able also to resolve the dns name with regular service as well. The difference is that with headless service you are able to talk to the pod directly instead having service as load-balancer or proxy.
Resolving the service thru dns is easy and you do that with the following pattern:
backend-broker.default.svc.cluster.local
Whereas backend-broker corresponds to the service name, default stands for the namespace the service is defined in, and svc.cluster.local is a configurable cluster domain suffix used in all cluster local service names.
Note that if you client and broker are in the same namespace you can omit the svc.cluster.local suffix and the namespace. You then reffer the servie as:
backend-broker
I high encourage you to read more about Dns in kubernetes.
All,
Thanks for answering the query, especially Thomas for code pointers. With your suggestions, once I create a Service for the POD, I was able to get the DNS working as core-dns was already running .. I was able to use the hostname in MQTT broker also after this.
opts.AddBroker(fmt.Sprintf("tcp://mqtt-broker:1883"))
apiVersion: v1
kind: Service
metadata:
creationTimestamp: "2021-02-01T19:08:46Z"
labels:
app: ipc
name: mqtt-broker
namespace: default
BTW, I wasnt able to get the headless service working, was getting this error, so continued with ClusterIP itself + 1883 exposed port for MQTT. Any suggestions please ?
`services "mqtt-broker" was not valid:`
`spec.clusterIPs[0]: Invalid value: []string{"None"}: may not change once set`
My setup is like, I have a producer service provisioned as part of minikube cluster and it is trying to publish messages to a kafka instance running on the host machine.
I have written a kafka service and endpoints yaml as follows:
kind: Service
apiVersion: v1
metadata:
name: kafka
spec:
ports:
- name: "broker"
protocol: "TCP"
port: 9092
targetPort: 9092
nodePort: 0
---
kind: Endpoints
apiVersion: v1
metadata:
name: kafka
namespace: default
subsets:
- addresses:
- ip: 10.0.2.2
ports:
- name: "broker"
port: 9092
The ip address of the host machine from inside the minikube cluster mentioned in the endpoint is acquired from the following command:
minikube ssh "route -n | grep ^0.0.0.0 | awk '{ print \$2 }'"
The problem I am facing is that the topic is getting created when producer tries to publish message for the first time but no messages are getting written on to that topic.
Digging into the pod logs, I found that producer is trying to connect to kafka instance on localhost or something (not really sure of it).
2020-05-17T19:09:43.021Z [warn] org.apache.kafka.clients.NetworkClient [] -
[Producer clientId=/system/sharding/kafkaProducer-greetings/singleton/singleton/producer]
Connection to node 0 (omkara/127.0.1.1:9092) could not be established. Broker may not be available.
Following which I suspected that probably I need to modify server.properties with the following change:
listeners=PLAINTEXT://localhost:9092
This however resulted in the change in the ip address in the log:
2020-05-17T19:09:43.021Z [warn] org.apache.kafka.clients.NetworkClient [] -
[Producer clientId=/system/sharding/kafkaProducer-greetings/singleton/singleton/producer]
Connection to node 0 (omkara/127.0.0.1:9092) could not be established. Broker may not be available.
I am not sure what ip address must be mentioned here? Or what is an alternate solution? And if it is even possible to connect from inside the kubernetes cluster to the kafka instance installed outside the kubernetes cluster.
Since producer kafka client is on the very same network as the brokers, we need to configure an additional listener like so:
listeners=INTERNAL://0.0.0.0:9093,EXTERNAL://0.0.0.0:9092
listener.security.protocol.map=INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
advertised.listeners=INTERNAL://localhost:9093,EXTERNAL://10.0.2.2:9092
inter.broker.listener.name=INTERNAL
We can verify messages in topic like so:
kafka-console-consumer.sh --bootstrap-server INTERNAL://0.0.0.0:9093 --topic greetings --from-beginning
{"name":"Alice","message":"Namastey"}
You can find a detailed explaination on understanding & provisioning kafka listeners here.
Our company has an external service that needs to connect to an application running in one of our Kubernetes clusters. The external service must connect via an IP address (it cannot use a host name) with a static port (port 80). This is not ideal, but we cannot control the external service.
The only externally exposed service in our cluster runs HAProxy which routes traffic to all of the internal (not publicly exposed) pods/services in the cluster. HAProxy's service is deployed as type LoadBalancer which sits behind an AWS ELB for SSL termination.
The problem comes when I try to deploy HAProxy with a static port. According to the documentation, I should be able to specify "nodePort" in the ports section for service types=NodePort or LoadBalancer:
nodePort:
The port on each node on which this service is exposed when type=NodePort or
LoadBalancer. Usually assigned by the system. If specified, it will be
allocated to the service if unused or else creation of the service will fail.
Default is to auto-allocate a port if the ServiceType of this Service
requires one.
Note that this Service will be visible as both <NodeIP>:spec.ports[*].nodePort and .spec.clusterIP:spec.ports[*].port. (If the --nodeport-addresses flag in kube-proxy is set, would be filtered NodeIP(s).)
So of course, I tried that with the following k8s configuration file:
apiVersion: v1
kind : Service
metadata:
name: haproxy
labels:
app : haproxy
env : {{ .Values.env | lower }}
annotations:
dns.alpha.kubernetes.io/external: "{{ .Values.externalUrl | lower }}"
service.beta.kubernetes.io/aws-load-balancer-ssl-cert : {{ .Values.sslcert | lower }}
service.beta.kubernetes.io/aws-load-balancer-ssl-ports : "443"
service.beta.kubernetes.io/aws-load-balancer-backend-protocol : http
service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout : "500"
service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags: "Environment={{ .Values.env | lower }},Type=k8s"
spec:
type : LoadBalancer
ports:
- name : https
port : 443
targetPort: 80
nodePort : 80
selector:
app : haproxy
env : {{ .Values.env | lower }}
Which produces the following error when deployed with Helm:
Error: UPGRADE FAILED: Service "haproxy" is invalid: spec.ports[0].nodePort: Invalid value: 80:
I simply need to be able to hit the haproxy service by entering the node's ip into my browser, but maybe I'm misplacing the nodePort configuration key. Should it go somewhere else in the configuration file? I've tried moving it to various places under the "ports" section, but that just throws parsing errors.
Thanks in advance! I'm at a complete loss.
I believe if you just don't try to configure the nodePort it will do what you want.
The port and targetPort are important: they specify in your case that port 443 (either on the ELB or the Kubernetes Service endpoint haproxy.default.svc.cluster.local) will get forwarded to port 80 in the pod. In an AWS environment like you describe, the nodePort is mostly a side effect: you'd use ClusterIP Services to communicate between Pods, and LoadBalancer services for things you want to expose outside the immediate cluster, and generally not make direct connections to nodes themselves.
There's a more specific limitation that the nodePort must be between 30000 and 32767; you can't use it to publish arbitrary ports from nodes, and that's why you're getting the error you're getting.