I need to bridge two mqtt broker.
The first is mosquitto and it run locally.
The second is another mosquitto instance and run remotely.
I'm using the bridge functionality with persistence enabled to forward the messages from the first broker to the second one.
This is my conf:
# Bridge test
connection br-win-to-deb
bridge_insecure true
address 192.168.138.166:1883
cleansession false
clientid br-win-deb
start_type automatic
notifications false
try_private true
# Forward di tutti i messaggi con topic GATEWAY su topic AZURE
topic # out 2 GATEWAY/ AZURE/
persistence true
persistence_file Messages.db
persistence_location C:/temp/
max_queued_messages 36000000
autosave_interval 10
queue_qos0_messages true
Actually it works but when the file Messages.db reach approx 1Gb and the used ram reach approx 2gb mosquitto crash.
I run mosquitto in windows 7 x64.
The number of messages is something like 10000000 but i need at least 36000000.
Is it a problem of memory? Is there a limit?
Does exists a mqtt broker that support that huge msg quantity?
It's a problem of Windows version.
On Debian the problem is the same.
The mosquitto version I'm using is v.1.4.8 on both the OS.
Related
I am trying to bridge my mosquitto broker running in raspberry pi to the cloud Mosquitto mqtt broker(test.mosquitto.org:1883).
I am not getting message in my client connected to the local broker, When I publish from a client connected to the cloud broker . But I am getting messages on the client connected to the cloud mosquitto broker from the client connected to the local MQTT Broker. I don't have any firewalls to block messages.
my mosquitto.conf file is like this
connection bridge-01
address test.mosquitto.org:1883
topic # out 0
topic # in 0
And I also have
listener 1883
allow_anonymous true
in my config
How to solve this issue?. Where am I going wrong??
Global wildcard subscriptions (subscribing to #) is disabled on the test.mosquitto.org broker because it generates too much load.
e.g.
$ mosquitto_sub -h test.mosquitto.org -v -t '#'
All subscription requests were denied.
You can change the topic # in 0 line to only pull in only the topics you are actually using.
Please also remember this broker is only really meant for testing, you should not be using it as a free cloud relay for anything long term.
I am using 3 server cluster for the Kafka Configuration, with Snowflake connector REST API to push the data to Snowflake database: All are 3 different VMs running on AWS
1.In this, does we require 3 kafka individual server zookeeper-services needs to be up and running in cluster else only 1 is enough, as if it needs to be executed in all the 3 servers zookeeper services, does it require different port configurations like for ex:
1.a:zookeeper.connect=xx.xx.xx.xxx:2181, xx.xx.xx.xxx:2182, xx.xx.xx.xxx:2183 else it should be 2181 in all the servers.properties file
1.b:PLAINTEXT://localhost:9091 in server1, PLAINTEXT://localhost:9092 and PLAINTEXT://localhost:9093 (Even in this it should be localhost else IP Address) that needs to be given?
1.c:server.1=<zookeeper_1_IP>:2888:3888, server.1=<zookeeper_2_IP>:2888:3888, server.1=<zookeeper_3_IP>:2888:3888 (Over here on each server the 2888:3888 needs to be same right?)
1.d:Clientport=2181 needs to be the same across the services in all 3 VMs else it needs to be different?
1.e:Does the listeners = PLAINTEXT://your.host.name:9092 on each server should have separate port like
VM-Server1:9092, VM-Server2:9093, VM-Server3:9094. Else the master server-IP should be given in the worker-nodes that is Server2 and Server3 else the own server IP of that worker-node
What should be the configuration for connector in regards with REST-API for the configuration item "tasks.max":"1". As I am going with 3 server cluster for Kafka and would be starting the 3 distribute-connector on all the 3 machines
I am getting duplicates, if I am starting the services of distributed connector in the 2nd server, how these duplicate records can be avoided. But yes if its only 1 distributed-connector is running the services, then there are no duplicates. Please advice, as the lag gets increased if only 1 distributed-connector services is up and running.
Create /data/zookeeper/myid file and give value 1 for zookeeper1 , 2 for zookeeper2 and 3for zookeeper3. Is this necessary when you are in different VM?
The distributed-connector services once started executing for sometime and then it gets disconnected
Any other parameter for the 3 server cluster architecture and best practices which needs to be followed
Kafka and Zookeeper
You only need one Kafka broker and Zookeeper server, although having more would provide fault tolerance. You don't need to manually create anything in Zookeeper such as myid files.
The ports don't need to be the same, but it is obviously easier to draw a network diagram and automate the configuration if they are.
Regarding Kafka listeners, read this post. For Zookeeper, follow its documentation if you want to create a cluster.
Or use Amazon MSK / Confluent Cloud, etc. instead of EC2, and this is all done for you.
Kafka Connect
tasks.max can be as much as you want, but if you have a source connector, then multiple threads will probably cause duplicates, yes.
I am working on a project where I have to connect local mqtt broker i.e. mosquitto and a cloud based mqtt broker via mqtt bridge. Mosquitto(local broker) is running on raspberry pi4 and I also want to run mosquitto mqtt bridge on same raspberry pi. So the question is, Can I run local mqtt broker i.e. mosquitto and the mqtt bridge both simultaneously in a single system which is raspberry pi4. If yes please tell the process how can i do it.
You only need to run a single MQTT broker (e.g. mosquitto). This will act as the local broker and can also be configured to bridge out to a remote broker.
The bridge can be configured to
mirror messages out to the remote broker
mirror messages in from the remote broker
or both
Depending on what you need. Details of how to configure the bridge can be found in the mosquitto docs here
But if you want to run multiple brokers on the same machine this is also perfectly possible, they will just need to bind to different ports as only one will be able to bind to 1883.
I'm running zookeeper, kafka servers along with a consumer client on a windows machine. When I use a simple producer program in python or c++ in a different machine also running windows, everything works normal. However, when I use the same producer program in a linux machine it does not work, it accuses failed to resolve . All computers are connected on the same network. I have tried to set the listeners and advertised.listeners in the kafka server.properties in several different manners, but with the same result.
Machine running windows with servers and consumer:
IP:192.168.0.149
hostname:something
second Machine running linux or windows (dual boot) with the producer program:
IP:192.168.0.150
hostame:somethingelse
some tried configurations at server.properties:
1
listeners=PLAINTEXT:\0.0.0.0:9092
advertised.listeners=PLAINTEXT:\something:9092
producer at second machine:
producer = KafkaProducer(bootstrap_servers=['192.168.0.149:9092'])
works on windows, does no work on linux
2
listeners=PLAINTEXT:\localhost:9092
advertised.listeners=PLAINTEXT:\something:9092
producer at second machine:
producer = KafkaProducer(bootstrap_servers=['192.168.0.149:9092'])
works on windows, does no work on linux
3
#listeners=
#advertised.listeners=
producer at second machine:
producer = KafkaProducer(bootstrap_servers=['192.168.0.149:9092'])
works on windows, does no work on linux
4
listeners=PLAINTEXT:\192.168.0.149:9092
advertised.listeners=PLAINTEXT:\something:9092
producer at second machine:
producer = KafkaProducer(bootstrap_servers=['192.168.0.149:9092'])
works on windows if consumer passes the ip instead of localhost, does no work on linux
Listeners should always stay as 0.0.0.0. This is the bind address to accept all external connections
If you want to run Kafka on windows, you need to deal with the windows firewall. Try disabling it while testing connections. It would probably be best that you switch the broker to run on Linux.
The advertised listeners should only be the hostname if both systems are able to resolve each other by DNS name. Otherwise, save yourself the extra network hop and use the IP as the advertised listener. If you're getting "failed to resolve", seems like a DNS issue, and ping windows-host shouldn't work either
To debug your connection, I would suggest using kafkacat
kafkacat -b kafka-host:9092 -L
I have a problem with Apache kafka
I have 4 clusters where I want to install kafka instances. On 3 clusters its works, they can product, and consume messages between each other, zookepers work fine. But on 4th cluster I can't run zookeeper connected with others zookeepers. If I set in zoo.cfg only local server (0.0.0.0:2888:3888) zookeeper runs in mode standalone, but if I add others servers I get error
./zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /etc/zookeeper/conf/zoo.cfg
Error contacting service. It is probably not running.
How I can fix this error? I will add, that I can ping servers, so they can see each other.