I am working on a project where I have to connect local mqtt broker i.e. mosquitto and a cloud based mqtt broker via mqtt bridge. Mosquitto(local broker) is running on raspberry pi4 and I also want to run mosquitto mqtt bridge on same raspberry pi. So the question is, Can I run local mqtt broker i.e. mosquitto and the mqtt bridge both simultaneously in a single system which is raspberry pi4. If yes please tell the process how can i do it.
You only need to run a single MQTT broker (e.g. mosquitto). This will act as the local broker and can also be configured to bridge out to a remote broker.
The bridge can be configured to
mirror messages out to the remote broker
mirror messages in from the remote broker
or both
Depending on what you need. Details of how to configure the bridge can be found in the mosquitto docs here
But if you want to run multiple brokers on the same machine this is also perfectly possible, they will just need to bind to different ports as only one will be able to bind to 1883.
Related
I have to build a modular system where each system could run by itself, or if they are connected to the same network, they could work together. Each system would have a raspberry pi that could potentially run an MQTT broker to handle messaging from other components in the system.
But I am wondering if I connect more than one system to the same network, is there a method to figure out if a broker is already present and disregard spinning up another broker so the 2nd system introduced to the network would use the first systems broker?
I am trying to bridge my mosquitto broker running in raspberry pi to the cloud Mosquitto mqtt broker(test.mosquitto.org:1883).
I am not getting message in my client connected to the local broker, When I publish from a client connected to the cloud broker . But I am getting messages on the client connected to the cloud mosquitto broker from the client connected to the local MQTT Broker. I don't have any firewalls to block messages.
my mosquitto.conf file is like this
connection bridge-01
address test.mosquitto.org:1883
topic # out 0
topic # in 0
And I also have
listener 1883
allow_anonymous true
in my config
How to solve this issue?. Where am I going wrong??
Global wildcard subscriptions (subscribing to #) is disabled on the test.mosquitto.org broker because it generates too much load.
e.g.
$ mosquitto_sub -h test.mosquitto.org -v -t '#'
All subscription requests were denied.
You can change the topic # in 0 line to only pull in only the topics you are actually using.
Please also remember this broker is only really meant for testing, you should not be using it as a free cloud relay for anything long term.
I've started learning some big data tools for a new project, and right now I'm on Kafka and Zookeeper.
I have them both install on my local machine, and I can start them up and start producing and consuming messages just fine. Now, I want to try it having two machines, one with a kafka broker, zookeepr and a producer, and the other with a consumer. Lets call them Machine A and Machine B.
Machine A has runs the Zookeeper server, the broker and a producer. Machine B runs a consumer. From what I think I understand, I should be able to setup the consumer to listen to a topic from the producer on Machine A, using Zookeeper. Since both machines are on the same network (i.e. my local home network), I thought I could change the kafka broker server.properties to use my static ip address for Machine A, and then have the consumer on Machine B connect to it.
My problem, is that zookeeper keeps spinning up on localhost, and connecting to 0.0.0.0/0.0.0.0:2181 so when my broker tries to connect to it using my static ip address (i.e 192.168.x.x), it times out. I have looked all over for a solution, but I cannot find anything that tells me how to configure the Zookeeper sever to start on a different ip address.
Maybe my understanding of these technologies is simply wrong, but I thought this would be a fairly simple thing to do. Does anyone know any way to resolve this? Or else if I'm doing it completely wrong, what is the correct approach
zookeeper keeps spinning up on localhost, and connecting to 0.0.0.0/0.0.0.0:2181
Well, that is the bind address.
You need to also (preferably) have a static IP for Zookeeper, then set zookeeper.connect within the server.properties file of Kafka to reach to that other machine's external address.
From the Zookeeper configuration file, you would make sure you have the myid file and have a line in the property file that looks like this (without the double brackets)
server.{{ myid }}={{ ip_address }}:2888:3888
You wouldn't find this in the Kafka documentation, but it is in the Zookeeper documentation
However, if Kafka and Zookeeper are on the same machine, this isn't necessary.
Your external consumer should be setting bootstrap.servers property and the Kafka IP address(es) w/ port 9092.
Your problem might me related instead to the advertised.listeners setting within Kafka.
For example, start with listeners=PLAINTEXT://:9092
As of Zookeeper 3.3.0 (see Advanced Configuration):
clientPortAddress : New in 3.3.0: the address (ipv4, ipv6 or hostname)
to listen for client connections; that is, the address that clients
attempt to connect to. This is optional, by default we bind in such a
way that any connection to the clientPort for any
address/interface/nic on the server will be accepted
So you could use:
clientPortAddress=127.0.0.1
I have installed node-red on raspberry pi 3 to collect data from sensors and then store them in kafka but now I have some issue with kafka producer node.
I've setup a kafka server on my laptop that correctly works in console: if I send messages on kafka producer console I correctly receive it on consumer console.
Unfortunately when I try to inject a timestamp in kafka producer in node-red on raspberry, server gives no response.
Debug page of node-red says: "BrokerNotAvailableError: Broker not available"
In producer node ZKQuorum field I've typed the ip of the laptop and set port to 9092, as I seen in example on npm site.
I'm sure the topic is correct.
I'm sure zookeeper is running and kafka server also. Indeed if at same time I try to use kafka with laptop console it works great.
I've also tried to reach kafka producer port with telnet: connections are accepted.
I've already posted same question on node-red community, without success for now.
Any hint about this issue?
UPDATE:
Update. I've tried to implement a python function in node-red to send a simple message to the kafka producer and I have obtained a deeper error log in:
/usr/local/lib/python2.7/dist-packages/kafka/client_async.py", line 808
I opened file and at line 808 there is a function with this comment:
It can be helpful?
You have to configure the accepted listeners field on the kafka server properties to the IP address of your laptop. Try to change zookeeper connect to actual ip, not localhost.
Try this property in etc/kafka/server.properties: listeners=PLAINTEXT://<your ip here>:<kafka port here>. you will have to restart kafka for this to take affect
I tried to fetch messages by Java client from host OS . I configured bridge network between host and guest , kafka is running but java client was stuck when try to do ConsumerRecords<String,String> records = consumers.poll(100);
My kafka in guest OS listening on localhost:9092
Could you share your Consumer properties?
There are a couple of properties to validate in you server.properties file: listeners and advertised.listeners. You can check this: https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whycan'tmyconsumers/producersconnecttothebrokers?