I have kafka installed in a ubuntu server, and node-red is in my personal laptop. I want to send data from node-red to the kafka topic.
I tried using the kafka node in node-red to connect, but I am getting error like "Client is not a constructor". Also I am bit confused between the listeners and advertised listeners configuration. How should I configure the server.properties file for the same, and also which nodes should I include the node-red flow to achieve this, please suggest me some ways.
I expect that if I send some message from node-red, I should be able to see the same in kafka topic.
Related
I am new to Apache Kafka and I'm trying to build a Python app which is able to handle Kafka messages. I've set Kafka up to produce and consume messages locally. Now I also want this to work non-locally, so that I can send messages from everywhere to my Python app.
My idea was to just expose the specific port that Kafka is using by using Localtunnel. I thought this would just mirror the local messages, so that I can consume them via the generated URL. But surprise, it doesn't work.
I just don't receive any messages at all. Do you have an idea why this is? Do I maybe have to configure the listeners in the Kafka server.properties first?
Thanks!
My pipeline is:
Kerberized Kafka --> Logstash (hosted on a different server) --> Splunk.
Can I replace the Logstash component with Kafka Connect?
Could you point me to a resource/guide where I can use kerberized Kafka as a source for my Kafka connect (which is hosted separately)?
From the documentation, what I understood is that if Kafka Connect is hosted on the same cluster as that of Kafka, that's quite possible. But I don't have that option right now, as our Kafka cluster is multi-tenant and hence not approved for additional processes on the cluster.
Kerberos keytabs aren't commonly machine/JVM specific, so yes, Kafka Connect should be able to be configured very similarly to Logstash since both are JVM processes using native Kafka protocol.
You shouldn't run Connect on the brokers anyway
If you can't add Kafka Connect to an existing Kafka cluster, you will have to spin up a separate Kafka Connect (Cluster or standalone).
I've written about it here: enter link description here
I have a scenario where we are forwarding our application logs to Kafka topic using fluentD agents,
as Kafka team introduced Kerberos authentication and fluentD version not supporting this authentication, I cannot directly use forward logs.
Now we have introduced a new Kafka server without authentication and created a topic there, I want forward messages from this topic in the new server to another topic in another server using Kafka connectors,
want to know how I can achieve this?
There's several different tools that enable you to stream messages from a Kafka topic on one cluster to a different cluster, including:
MirrorMaker (open source, part of Apache Kafka)
Confluent's Replicator (commercial tool, 30 day free trial)
uReplicator (open sourced from Uber)
Mirus (open sourced from Salesforce)
Brucke (open source)
Disclaimer: I work for Confluent.
There's a cloud machine Whose public ip is known in which kafka broker is up and running. How do I produce and consume a message to this kafka broker from my laptop.
Open Kafka ports and connect directly via TCP or use Kafka REST proxy
https://github.com/confluentinc/kafka-rest
Click on the address bar.
Type google.com, and enter.
Within the textbox, write kafka producer consumer example, and enter.
There will be plenty of resources showing you how.
I have installed node-red on raspberry pi 3 to collect data from sensors and then store them in kafka but now I have some issue with kafka producer node.
I've setup a kafka server on my laptop that correctly works in console: if I send messages on kafka producer console I correctly receive it on consumer console.
Unfortunately when I try to inject a timestamp in kafka producer in node-red on raspberry, server gives no response.
Debug page of node-red says: "BrokerNotAvailableError: Broker not available"
In producer node ZKQuorum field I've typed the ip of the laptop and set port to 9092, as I seen in example on npm site.
I'm sure the topic is correct.
I'm sure zookeeper is running and kafka server also. Indeed if at same time I try to use kafka with laptop console it works great.
I've also tried to reach kafka producer port with telnet: connections are accepted.
I've already posted same question on node-red community, without success for now.
Any hint about this issue?
UPDATE:
Update. I've tried to implement a python function in node-red to send a simple message to the kafka producer and I have obtained a deeper error log in:
/usr/local/lib/python2.7/dist-packages/kafka/client_async.py", line 808
I opened file and at line 808 there is a function with this comment:
It can be helpful?
You have to configure the accepted listeners field on the kafka server properties to the IP address of your laptop. Try to change zookeeper connect to actual ip, not localhost.
Try this property in etc/kafka/server.properties: listeners=PLAINTEXT://<your ip here>:<kafka port here>. you will have to restart kafka for this to take affect