I need to create an independent server which continuously polls data from a Kafka Topic. Till now I have created a vertx server and a Kafka Consumer. The issue is when we start the server it drains the queue and polls all the messages currently present but when new messages will arrive later it wont be able to consume them. I need to implement some logic in my server which able it to continuously poll the data and drain the queue.
Related
I'm very much new in Apache Kafka. So, I'm facing some issues.
If one consumer Auto-Commit is enabled and that machine is consumed a data then other new consumers on that group is not receiving those message.
I have 3 consumer, 1 producer. Here, every single consumer is running on different machine. I'm consuming messages from a single machine with auto-commit enabled for testing purpose. And after that when I'm running another two consumer machine then they are not receiving those messages.
As other two consumers did not received those messages so I want to receive those message for this two consumer.
So, How Kafka commit works for a consumer group?
I have to publish messages from a kafka topic to lambda to process them and store in a database using a springboot application, i did some research and found something to consume messages from kafka
public Function<KStream<String, String>, KStream<String, String>> process(){} however, im not sure if this is only used to publish the consumed messages to another kafka topic or can be used as an event source to lambda, I need some guidance on consuming and converting the consumed kafka message to event source.
Brokers do not push. Consumers always poll.
Code shown is for Kafka Streams API, which primarily writes to new Kafka topics. While you could fire HTTP events to start a lambda, that's not recommended.
Alternatively, Kafka is already supported as an event source. You don't need to write any consumer code.
https://aws.amazon.com/about-aws/whats-new/2020/12/aws-lambda-now-supports-self-managed-apache-kafka-as-an-event-source/
This is possible from MSK or a self managed Kafka
process them and store in a database
Your lambda could process the data and send to a new Kafka topic using a producer. You can then use MSK Connect or run your own Kafka Connect cluster elsewhere to dump records into a database. No Spring/Java code would be necessary.
I have noticed that my Kafka Streams application stops working when it has not read new messages from the Kafka topic for a while. It is the third time that I have seen this happen.
No messages have been produced to the topic since 5 days. My Kafka Streams application, which also hosts a spark-java webserver, is still responsive. However, the messages I produce to the Kafka topic are not being read by Kafka Streams anymore. When I restart the application, all messages will be fetched from the broker.
How can I make my Kafka Streams Application more durable to this kind of scenario? It feels that Kafka Streams has an internal "timeout" after which it closes the connection to the Kafka broker when no messages have been received. I could not find such a setting in the documentation.
I use Kafka 1.1.0 and Kafka Streams 1.0.0
Kafka Streams do not have an internal timeout to control when to permanently close a connection to the Kafka broker; Kafka broker, on the other hand, does have some timeout value to close idle connections from clients. But Streams will keep trying to re-connect once it has some processed result data that is ready to be sent to the brokers. So I'd suspect your observed issue came from some other causes.
Could you share your application topology sketch and the config properties you used, for me to better understand your issue?
if I have a kafka producer on a live machine which writes its data to a kafka topic. What happens when there is a network issue or kafka goes down which would simultaneously affect the performance of the live machine by request-timeout set as kafka producer would time out after request timeout millisecs set for each send request requested by the live machine.
Scenario :
Stream create [StreamName] --definition " Kafka -zkconnect=10.10.10.1:2181 --topic=<topic name> | MyCompositeModule " --deploy
We are running this stream in distributed mode and redis is the transport bus.
Per my understanding, kafka source maintains the offsets for messages consumed by MyCompositeModule (which is a sink as its a module created through 'module compose' process) through [streamname]-kafka-offsets topic. Which is unreadable and I would appreciate if there was a way to read the data from this topic.
Also, when I push the messages from kafka source the messages are queued in redis transport, then the module fetches them from this queue.
If kafka consumer module starts consuming 1000 messages from kafka redis queue- and composite module fails after receiving 10 messages or randomly processed 10 messages.So how to identify remaining 990 [ 1000 ( consumed ) - 10 (processed) = 990 ] unprocessed messages .
Even if we check kafka offsets it will show consumed messages count. example: -kafka.offsets - which is unreadable in our process.
So all the unprocessed messages will be in Redis queue as we are using Redis in SpringXD. So can anyone help me out how to identify the unprocessed messages and
re sending it to composite module to process.
Basically, I am looking for the recommendations on an elegant solution for robust delivery, adding failure handling capability in spring xd stream when consuming from from kafka source.
If the messages are effectively consumed from Kafka and moved to the bus, then they will be acknowledged as consumed from the offset manager's perspective.
You can try enabling retry and dead lettering for the Redis Message Bus as described here: http://docs.spring.io/spring-xd/docs/current/reference/html/#error-handling-message-delivery-failures.
Cheers,
Marius