I have my Kafka Server running on other system. I am trying to run the client from my local machine by giving the broker url of the machine where Kafka server is running. But unfortunately i am not able to connect to kafka server.
server.properties files has the below attributes:
group.initial.rebalance.delay.ms=0
listeners=SASL_PLAINTEXT://localhost:9093
advertised.listeners=SASL_PLAINTEXT://localhost:9093
#advertised.listeners=SASL_PLAINTEXT://10.97.123.52:9093
security.inter.broker.protocol=SASL_PLAINTEXT
sasl.mechanism.inter.broker.protocol=PLAIN
sasl.enabled.mechanisms=PLAIN
while running my client from my local machine, i am passing the broker url of the server machine, but unable to connect:( . Can anyone help in this problem?
A bit simplified, but ... the client first connects to the bootstrap server to get the metadata. A based on that metadata it will open another TCP connection to the broker which is leader for the topic/partition the clients wants to speak with.
The first connection is done based on the bootstrap server address which you set. The second connection is opened to the address from the metadata. And the metadata will in your case contain the address from the advertised.listeners field, which is localhost. So the client will try to connect to localhost:9093 and not to your broker. So you need to set the advertised.listeners to use the address under which the broker is visible for the clients. (which is maybe the line which is commented out in your config example?)
You have also the listener field set to listen on localhost only. So it will not be accessible from the external IP address. You have to change it to listen on the external IP address. Most propbably setting it to the following value (i.e. without localhost) should help:
listeners=SASL_PLAINTEXT://:9093
Related
We have following kafka-ssh-tunneling setup.
ssh -N $JUMPHOST -L 2181:w.x.y.z:2181 -L 9092:a.b.c.d:9092 -L 9091:e.e.f.f:9092
broker IP is a.b.c.d , There is local lo0 device alias with same IP address
zookeper IP is w.x.y.z , There is local device alias with same IP address
kafkahost "entry" host is e.e.f.f
Our planned use case is kafkacat -C -b localhost:9091 -t <topic>
Problem:
Connecting to kafka host/ zookepers works fine, however
kafka clients ( e.g. kafkacat ) are accessing brokers by their hostname, ip-a.b.c.d.eu-central-1.compute.internal , not by their IP's.
To counteract, I've added entry to /etc/hosts
a.b.c.d ip-a.b.c.d.eu-central-1.compute.internal
Still doesn't work,
although pinging to that hostname is successful.
Nslookup gives
Non-authoritative answer:
Name: ip-a.b.c.d.eu-central-1.compute.internal
Address: a.b.c.d
** server can't find ip-a.b.c.d.eu-central-1.compute.internal: NXDOMAIN
Question:
Is there a way telling kafka to connect brokers through IP's and not via hostnames?
If not, will starting local dns server might resolve the issue?
What is happening here is:
The broker receives a petition from the client and returns him ip-a.b.c.d.eu-central-1.compute.internal, as this is the host name of the broker and the default value for listeners. (this is the key)
Your client tries to send data to the broker using the metadata it was given. But as it can't resolve ip-a.b.c.d.eu-central-1.compute.internal, it fails without even reaching the cluster, caused by a networking issue out of Kafka's scope.
If you set the value on /etc/hosts, you will fix the address resolution problem; The client will now be able to reach the cluster, solving the previous networking issue.
The following step involves Kafka replying with the 448_in_your_face error code (the exact code name may differ). Your petition fails again, now on cluster-side: your client is asking for a broker called/referenced a.b.c.d, but there is no registered listener with that name, as its identifier is still ip-a.b.c.d.eu-central-1.compute.internal.
The key here is within the advertised.listeners property, located in the server.properties configuration file.
In order your clients to be able to connect, modify that property, directly setting the ip there, or a resolvable dns (using IP for this example):
advertised.listeners=PLAINTEXT://a.b.c.d:9092
Now on client side, just use the IP in order to connect with the broker:
bootstrap.servers = a.b.c.d:9092
When the petition from the client is received, kafka will recognize the content of bootstrap.servers as one of its registered listeners, hence accepting the connection.
Found workaround. Posting here If anyone might face my problem.
Steps are following:
Create dummy aliases for all host you are planning to use, sudo ip add a dev lo $ip
These aliases should NOT have the same broker/zookeper IPs, BUT 127.0.j.k format
Add ip-<>.<>.<>.<>.eu-central-1.compute.internal < -- > 127.0.[].[] mapping to /etc/hosts
Create tunnel via SSH, taking account relation of broker/zookeper's IPs and your local (aliased) IPs
ssh -N $JUMPHOST -L 2181:<localIP>:<remoteIP>:2181 -L 9092:<localIP>:<remoteIP>:9092 ...
then you can consume messages via
kafkacat -C -b 127.0.[].[]:9092 -t <topic>
I'm a little confused here at the moment. I'm working with Kafka v1.0 and it is being hosted in a VM in the cloud. The single VM hosts Zookeeper as well as Kafka.
I can't seem to get a listeners and advertised.listeners configuration for server properties figured out so I can connect remotely with my producer and consumers.
If I set it like this...
listeners=CLIENT://:9090,PLAINTEXT://:9092
advertised.listeners=CLIENT://:9090,PLAINTEXT://:9092
From my external client I get a NoBrokersAvailable error. If I try this...
listeners=CLIENT://0.0.0.0:9090,PLAINTEXT://:9092
advertised.listeners=CLIENT://0.0.0.0:9090,PLAINTEXT://:9092
I get an error that it can't listen on the meta-address 0.0.0.0.
This is currently being hosted with a public interface, but if I try to set this:
listeners=CLIENT://my.public.dns.name:9090,PLAINTEXT://:9092
advertised.listeners=CLIENT://my.public.dns.name:9090,PLAINTEXT://:9092
Then I get an error that it can't bind to the requested address. This is sitting behind some networking infrastructure, as it's obvious ip addr only shows my private IP address.
How would I get around this to setup a listener so an external/public producer/consumer could connect to this broker?
Thank you in advance!
I set up a broker on a windows pc which is publishing messages to raspberry pis (clients). On the same windows machine I'm running a node.js Server which is also a mosquitto client which can publish messages to the broker running on the same windows machine.
I looked up (by simple googling and reading the official documentation) how I can secure the moquittoconnections. But I still ran into some issues:
How can I only allow a mosquitto broker to communicate with clients
running on the same machine? (e.g. for simply publishing sensor
values to a local frontend via websockets - which I'm already doing)
Do local (on the same machine) clients require the username and
password if specified?
Why won't this configuration not require an username and a password
when the client is connecting to the broker?
My mosquitto.conf:
allow_anonymous false
password_file C:\Program Files (x86)\mosquitto
Password file is at the specified location and valid.
mosquitto -c mosquitto.conf is not throwing any error.
Can I still use the username and password when including encryption
mosquitto-tls?
Listeners can be bound to a specific interface e.g. 127.0.0.1 which will only allow connections from the localhost.
listener 1885 127.0.0.1
If you want to restrict the whole broker to only listen for local connections use the bind_adddress config option to change the default listener to only listen on 127.0.0.1 rather than 0.0.0.0 (this address represents ALL network interfaces on the machine)
If password based authentication is configured it applies to ALL listeners so if you set up a localhost only listener you will still need to supply a username and password.
You can use TLS and username and password based authentication unless you use client side certificates and enable the use_identity_as_username and require_certificate
Here is the scenario:
My kafka server and the zookeeper are running and working fine on the remote server as long as I launch the process on the same remote server.
I don't have any connectivity issues between my local machine and the server. I can ssh and access all other applications on my remote server from my local machine.
When trying to connect a client to the remote zookeeper directly from my local machine I get a connection timeout. Debugging shows that a NoRouteToHost exception is being thrown.
The client logfile contains the following lines:
org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=remotehost:2181 sessionTimeout=120000
watcher=org.I0Itec.zkclient.ZkClient#62807a4d
org.apache.zookeeper.ClientCnxn: Opening socket connection to server
remotehost/192.13.12.1:2181. Will not attempt to authenticate using
SASL (unknown error)
Note the "remotehost/192.13.12.1:2181". The URL I'm specifying is remotehost:2181. He appears to resolve the hostname to IP correctly and then... slapping a forward slash and the IP on to the URL which looks weird to me. Is this how he should resolve and pass down the URL to the socket?? Or is this the cause of my problem.
Found the issue. The zookeeper listen port was not open to the firewall.
I have learn basic zookeeper concept and did a sample project, But I only it only local pc or one computer.
I understand the zookeeper but still confused on how the client connect to the zookeeper server if they are not in one computer? for instance, if we start a zookeeper server in my own computer, and we can use connect() like connect 2181 to connect to the zookeeper server, that make sense, since they are all in one computer have have some association in lower layer. But what if the zookeeper server and client they are separated into two computer? how can we handle that?
I'm not sure what language you're using for the client, so this will have to be a generic answer.
The client and server communicate over TCP. This requires that the client simply know the server's host and port. In general, your ZooKeeper servers bind to some private network interface. For instance, your zoo.conf configuration file might contain a line like the following:
clientPort=2181
server.1=123.456.789.1:2888:3888
The first portion of the server.1 section 123.456.789.1 is the host to which the ZooKeeper server will bind. As long as this host is not the loop back interface (i.e. localhost or 127.0.0.1) you should be able to connect to that host from another machine on the client port 2181. So, for instance, in Java I create a new ZkClient that points to that host and port:
ZkClient client = new ZkClient("123.456.789.1:2181");