I have a kafka producer written in Go using the confluent-kafka-go bindings.
I want my kafka producer to establish a connection to a kafka broker that is behind an HTTP proxy that supports HTTP CONNECT tunneling.
Does librdkafka support this? Does confluent-kafka-go support this? If yes, please point me to the relevant documentation or provide a code snippet illustrating how this is done.
If it is not supported, how hard would be it be to add support?
These clients only use Kafka TCP protocol; they don't perform content negotiation or establish a connection with an HTTP CONNECT method.
If you want to use HTTP protocol, you'll need to use an HTTP client
Related
To configure SSL for Kafka connect REST API, i followed the documentation provided in the following link - https://cwiki.apache.org/confluence/display/KAFKA/KIP-208%3A+Add+SSL+support+to+Kafka+Connect+REST+interface
But still i am not clear why listener is used. Is both listener and bootstrap.server are same, because Kafka connect(producer) will be listening to kafka brokers for getting metadata and writing source data.
What should be value for listener and bootstrap.server. Can someone please clarify me the difference between them
Listeners establish how the REST API binds to the host where the Connect server runs. HTTPS://0.0.0.0:8083 would allow all connections on all interfaces to port 8083 over HTTPS, as mentioned in the docs.
List of comma-separated URIs the REST API will listen on. The supported protocols are HTTP and HTTPS. Specify hostname as 0.0.0.0 to bind to all interfaces. Leave hostname empty to bind to default interface. Examples of legal listener lists - HTTP://myhost:8083,HTTPS://myhost:8084
bootstrap.server is the Kafka Connection string. The values for it and the listeners should not be the same, and Kafka Connect is recommended to be ran remotely from the brokers
Note: rest.advertised.listener is also an important setting when setting up a cluster of multiple workers
Sets the advertised listener (HTTP or HTTPS) which will be given to other workers to use
I want to implement a chat application which uses "MQTT protocol" in order to send messages from the device(Android phone). I need a "Kafka broker" which would run on the server and listen to these messages.
For this, I need an MQTT proxy, but even after googling a lot I could not find any open source MQTT proxy. Please suggest if there is an open source MQTT proxy. And if not, then, is it possible to implement one of my own?
There is an MQTT Proxy from Confluent, but it is not open source. You can use it free against a single broker, or 30-day trial for a cluster of more than one broker.
Here is my use case:
I have two endpoints: one with MQ and the second with TCP/IP
I have to replace a legacy server which accepts queries from remote TCP/IP clients. Once the socket is open with the client, data is exchanged in both sides. the server sends asynchronously MQ data through TCP/IP and receive data from clients asynchronously also. Each data message sent has to be acknowledged. The constraint here is that I have to use the same socket.
I created two routes
from("netty4:tcp://ipAddress:port?sync=true").to("wmq:queue:toQueue")
from("wmq:queue:fromQueue").to("netty4:tcp://ipAddress:port?sync=true")
I start the first queue to receive session open request from clients and then I start the second route to start sending data but I cannot use the same channel.
I tried to get the remote port of the first route and used it in the second route but I have a ConnectException because netty4 tries to open a new socket which is already open.
I found that netty4 can be used asynchronously using the AsyncProcessor but I didn't find any example dealing with my use case.
The only idea I found is that I have to create a standalone server which open the sockets with the clients and make it communicate with the two endpoints.
Is there any way to implement this situation using camel only?
any help on this subject is really appreciated.
Your code won't be able to run as it is for your use case. I also suspect you are trying to use Camel as IP server framework and not an integration in this case.
Lets review Apache Camel's concept of producers and consumers. In the integration world we talk about client and servers as consumers and producers. This might seem like a language difference until you realise a consumer(typically a client) can also be a producer(server).
Some helpful definitions:
1. Producer: A producer is an entity capable of creating and sending a message to an endpoint. A typical example would be code like .to("file:data/outbox") as this produces a file.
2. Consumer: A consumer is an entity that receives messages produced by a producer, it wraps these messages in an exchange and sends them to be processed. A typical example would be code like from(jms:topic:xmlOrders)
A rule of thumb is that typically consumers are the source of the messages being routed.
BIG NOTE:
These two definitions are not set in stone a producer can also be an endpoint using the from and a consumer can be an endpoint using the to.
So in your case let's break up the route:
from("netty4:tcp://ipAddress:port?sync=true").to("wmq:queue:toQueue")
In this route you are creating a Netty server that sends a message to a queue. Here your netty endpoint acts as a consumer(yes it is in the from clause) however this creates a Netty4 Server at the IP address and endpoint you specified. This then send a message to another consumer which is the MQ client which act as a consumer again. So two consumers? Where is the producer? The client connecting to the netty server will act as producer.
Let's look at the second piece of the route:
from("wmq:queue:fromQueue").to("netty4:tcp://ipAddress:port?sync=true")
Here you are creating a client/consumer for the MQ services and then creating a client/producer to the netty server. Essentially you are creating a NEW client here that connects to the SERVER you created in the first route.
So in short your route creates a Netty server that send a message to MQ then creates a MQ client that sends a message to a Netty client which connects to the server you have created. It wont work like this.
Go read about message exchange patterns for further reading, but I would suggest that if you are just using Netty and MQ then maybe Camel is a bit overkill as it is a integration platform and not a IP server platform.
If answer is "using xmpp BOSH", then to use BOSH, we need to have in-between CM (Connection Manager) server that would interact with xmpp server. So that xmpp client to CM interaction will then happen over normal http protocol (which will be scalable), but again CM to xmpp server will happen using long-live tcp connection which will cause scalability issue.
How can we make xmpp client-server more scalable??
Bidirectional-streams Over Synchronous HTTP (BOSH) is basically a technology to keep the connection alive to allow the system to send and receive the data to and fro. To be able to use xmpp over the HTTP, you will have to use BOSH.
For you concern about the scalability, you can basically use components which are meant for scalability such as Nginx a component in place of Apache httpd, also there are many connection managers available with Punjab being the most scalable of all. If you are using Openfire as a XMPP server, it has a prebuilt connection manager which has proved scalability.
I understand it's available since MSMQ 3.0, is it available via any of the .NET ESBs?
Is this possible with other MQ transports (ActiveMQ, etc)?
Thanks,
E.
NServiceBus use msmq as its main transport but does not support the http option for Msmq. Can you elaborate on what you're trying to achieve? Perhaps the NServiceBus gateway component is what you need?
http://docs.particular.net/nservicebus/gateway/
Hope this helps!
MassTransit does not support MSMQ over HTTP in any way. RabbitMQ only requires a single port open for communication but it is not the HTTP (80 or 443) port.