We have following kafka-ssh-tunneling setup.
ssh -N $JUMPHOST -L 2181:w.x.y.z:2181 -L 9092:a.b.c.d:9092 -L 9091:e.e.f.f:9092
broker IP is a.b.c.d , There is local lo0 device alias with same IP address
zookeper IP is w.x.y.z , There is local device alias with same IP address
kafkahost "entry" host is e.e.f.f
Our planned use case is kafkacat -C -b localhost:9091 -t <topic>
Problem:
Connecting to kafka host/ zookepers works fine, however
kafka clients ( e.g. kafkacat ) are accessing brokers by their hostname, ip-a.b.c.d.eu-central-1.compute.internal , not by their IP's.
To counteract, I've added entry to /etc/hosts
a.b.c.d ip-a.b.c.d.eu-central-1.compute.internal
Still doesn't work,
although pinging to that hostname is successful.
Nslookup gives
Non-authoritative answer:
Name: ip-a.b.c.d.eu-central-1.compute.internal
Address: a.b.c.d
** server can't find ip-a.b.c.d.eu-central-1.compute.internal: NXDOMAIN
Question:
Is there a way telling kafka to connect brokers through IP's and not via hostnames?
If not, will starting local dns server might resolve the issue?
What is happening here is:
The broker receives a petition from the client and returns him ip-a.b.c.d.eu-central-1.compute.internal, as this is the host name of the broker and the default value for listeners. (this is the key)
Your client tries to send data to the broker using the metadata it was given. But as it can't resolve ip-a.b.c.d.eu-central-1.compute.internal, it fails without even reaching the cluster, caused by a networking issue out of Kafka's scope.
If you set the value on /etc/hosts, you will fix the address resolution problem; The client will now be able to reach the cluster, solving the previous networking issue.
The following step involves Kafka replying with the 448_in_your_face error code (the exact code name may differ). Your petition fails again, now on cluster-side: your client is asking for a broker called/referenced a.b.c.d, but there is no registered listener with that name, as its identifier is still ip-a.b.c.d.eu-central-1.compute.internal.
The key here is within the advertised.listeners property, located in the server.properties configuration file.
In order your clients to be able to connect, modify that property, directly setting the ip there, or a resolvable dns (using IP for this example):
advertised.listeners=PLAINTEXT://a.b.c.d:9092
Now on client side, just use the IP in order to connect with the broker:
bootstrap.servers = a.b.c.d:9092
When the petition from the client is received, kafka will recognize the content of bootstrap.servers as one of its registered listeners, hence accepting the connection.
Found workaround. Posting here If anyone might face my problem.
Steps are following:
Create dummy aliases for all host you are planning to use, sudo ip add a dev lo $ip
These aliases should NOT have the same broker/zookeper IPs, BUT 127.0.j.k format
Add ip-<>.<>.<>.<>.eu-central-1.compute.internal < -- > 127.0.[].[] mapping to /etc/hosts
Create tunnel via SSH, taking account relation of broker/zookeper's IPs and your local (aliased) IPs
ssh -N $JUMPHOST -L 2181:<localIP>:<remoteIP>:2181 -L 9092:<localIP>:<remoteIP>:9092 ...
then you can consume messages via
kafkacat -C -b 127.0.[].[]:9092 -t <topic>
Related
I am not able to send/receive messages with public IP in Kafka. I tried by changing IP with private and public. I also tried by changing advertised.host.name to 0.0.0.0.
What am I missing in Kafka ?
I suppose you are running Kafka on one machine and trying to access it from other machine.
It can be debugged as follows:
Try pinging the public IP from your machine. ping public_ip
If ping works, then try doing a telnet to that public IP along with the Kafka bootstrap server port.
For example, telnet 1.2.3.4 9092
If you are able to telnet means, that you are able to connect to the public IP (here, 1.2.3.4) and port from your machine.
If you are not able to connect, check your iptables rules on your Kafka server. You may want to allow the port to be accessible from outside.
Example to allow 9092 port.
iptables -A INPUT -p tcp --dport 9092 -j ACCEPT
You may also want to check if any firewall is blocking access like UFW or firewalld. Try disabling them or allow Kafka port there and check.
If you are using OpenStack or similar software, you may want to check Security group rules there and allow those ports. This can be applicable to AWS also.
Check that your advertised.listeners have the public IP which you are using to connect to. By default this property is found in etc/kafka/server.properties file.
Change it to something like (if you are using PLAINTEXT)
advertised.listeners=PLAINTEXT://<PUBLIC_IP>:<PORT>
For example,
advertised.listeners=PLAINTEXT://1.2.3.4:9092
advertised.host.name seems to be DEPRECATED now (see documentation)
I have my Kafka Server running on other system. I am trying to run the client from my local machine by giving the broker url of the machine where Kafka server is running. But unfortunately i am not able to connect to kafka server.
server.properties files has the below attributes:
group.initial.rebalance.delay.ms=0
listeners=SASL_PLAINTEXT://localhost:9093
advertised.listeners=SASL_PLAINTEXT://localhost:9093
#advertised.listeners=SASL_PLAINTEXT://10.97.123.52:9093
security.inter.broker.protocol=SASL_PLAINTEXT
sasl.mechanism.inter.broker.protocol=PLAIN
sasl.enabled.mechanisms=PLAIN
while running my client from my local machine, i am passing the broker url of the server machine, but unable to connect:( . Can anyone help in this problem?
A bit simplified, but ... the client first connects to the bootstrap server to get the metadata. A based on that metadata it will open another TCP connection to the broker which is leader for the topic/partition the clients wants to speak with.
The first connection is done based on the bootstrap server address which you set. The second connection is opened to the address from the metadata. And the metadata will in your case contain the address from the advertised.listeners field, which is localhost. So the client will try to connect to localhost:9093 and not to your broker. So you need to set the advertised.listeners to use the address under which the broker is visible for the clients. (which is maybe the line which is commented out in your config example?)
You have also the listener field set to listen on localhost only. So it will not be accessible from the external IP address. You have to change it to listen on the external IP address. Most propbably setting it to the following value (i.e. without localhost) should help:
listeners=SASL_PLAINTEXT://:9093
I have setup a simple HTTP java server running locally on port 8000. It simply prints a message "Hello world" when a request comes. When I try to ping it from the browser by running http://localhost:8000/test I get my message printed.
I want to get the same results from another computer that is not local. When I try to use my public IP lets say http:/43.xxx.xxx.xxx:8000/test (even from the same machine) I get an ERR_CONNECTION_REFUSED .
I probably suspect that has something to do with the firewall. Can anyone guide me a little more because I lack the experience?
Thanks in advance
You don't specify what host OS your server/firewall is running so I'll keep this generic...
Without knowing your application, it seems like the server is sending a reset (RST packet) when the first SYN packet shows up indicating that the port (on that interface [your external]) is closed. You can do a quick port scan from here (https://mxtoolbox.com/PortScan.aspx) if you don't have access to a remote machine to test with. Odds are, TCP/8000 will not be open.
If it is, in fact, closed, you'll have to look at the firewall that your host OS is running and find out how to allow TCP/8000 to your host. In a major firewall vendor, your rule would look similar to this:
Source: Any
Destination: Your Public IP Address
Service: TCP/8000
Action: Allow
Logging: Full
That being said, you mentioned this was a PC so look into "iptables" (if you're running *nix) or the Windows Firewall (if you're running Windows) on adding firewall rules (Unfortunately I just joined and can't ask questions/comments, yet).
If you really want to find out what packet is being sent, run a tcpdump on your external interface (let's say eth1) (assuming your remote IP is 1.2.3.4 and your home public IP is 4.5.6.7):
tcpdump -nn -vvv -e -s 0 -X -c 100 -i eth1 host 1.2.3.4 and host
4.5.6.7 and port 8000
Here you're looking for the SYN/SYN-ACK/ACK for a successful TCP negotiation or SYN/RST if there is a firewall rejecting (not dropping) the TCP stream to the port.
Once the port is open on the host OS firewall, take a look at the application to make sure it's configured properly. If this were a standard webserver, you could take a look at the configuration files for the "Allow from" directives to make sure that everyone can access the site. If this is a custom application that you've created, you'll have to check this yourself.
I finally solved my problem. I needed to open a forwarding port in my router that maps my local ip address to the public. My router is TP Link so this what I did:
http://www.tp-link.com/us/faq-72.html
Also in order for this to work every time and not to have to reconfigure this every time I reconected to the router (because I get a new local IP), I have created a static local ip for my server following this guide:
http://www.tp-link.com/us/faq-182.html
Thanks for all the replies.
I set up a broker on a windows pc which is publishing messages to raspberry pis (clients). On the same windows machine I'm running a node.js Server which is also a mosquitto client which can publish messages to the broker running on the same windows machine.
I looked up (by simple googling and reading the official documentation) how I can secure the moquittoconnections. But I still ran into some issues:
How can I only allow a mosquitto broker to communicate with clients
running on the same machine? (e.g. for simply publishing sensor
values to a local frontend via websockets - which I'm already doing)
Do local (on the same machine) clients require the username and
password if specified?
Why won't this configuration not require an username and a password
when the client is connecting to the broker?
My mosquitto.conf:
allow_anonymous false
password_file C:\Program Files (x86)\mosquitto
Password file is at the specified location and valid.
mosquitto -c mosquitto.conf is not throwing any error.
Can I still use the username and password when including encryption
mosquitto-tls?
Listeners can be bound to a specific interface e.g. 127.0.0.1 which will only allow connections from the localhost.
listener 1885 127.0.0.1
If you want to restrict the whole broker to only listen for local connections use the bind_adddress config option to change the default listener to only listen on 127.0.0.1 rather than 0.0.0.0 (this address represents ALL network interfaces on the machine)
If password based authentication is configured it applies to ALL listeners so if you set up a localhost only listener you will still need to supply a username and password.
You can use TLS and username and password based authentication unless you use client side certificates and enable the use_identity_as_username and require_certificate
I can't connect to ContextBroker from another machine, even a machine in the same LAN.
Accessing by ssh without any problem
ssh geezar#192.168.1.115
and then
curl localhost:1026/statistics
the terminal shows the statistics, all right
<orion>
<xmlRequests>3</xmlRequests>
<jsonRequests>1</jsonRequests>
<updates>1</updates>
<versionRequests>1</versionRequests>
<statisticsRequests>2</statisticsRequests>
<uptime_in_secs>84973</uptime_in_secs>
<measuring_interval_in_secs>84973</measuring_interval_in_secs>
</orion>
But when I try without ssh connection...
curl 192.168.1.115:1026/statistics
curl: (7) Failed to connect to 192.168.1.115 port 1026: No route to host
Even, I routed the port 1026 to that machine (192.168.1.115) on the router configuration, and tried to access from my public IP, the result is the same, failed to connect
I think I am missing something, but.. what is it?
The most probable causes of this problem are:
Something in the host (e.g a firewall or security group) is blocking the incoming connection
Something in the client (e.g a firewall) is blocking the outcoming connection
There is some other network issue is causing the connection problem.
EDIT: in GNU/Linux system, iptables is usually used as firewall. It can be disabled typically running iptables -F.