I have a storm setup that picks up messages from a kafka topic and processes and persists them.
I want to understand how storm gurantees message processing in such a scenario
Consider the below scenario:
I have configured multiple supervisors+workers for a storm cluster.
The KafkaSpout is reading message from the topic and then passes on this a bolt. The bolt acks upon completion and the spout moves forward to the next message.
I have 2 supervisors running - each of which are running 3 workers each.
From what I understand - each of the worker on every supervisor is capable to processing a message.
So, at any given time 6 messages are being processed parallely in storm cluster.
what if the second message fails, either due to worker shutdown or due to supervisor shutdown.
the zookeeper is already pointing to the 7 message for the consumer group.
In such a scenario, how will the second message get processed?
I guess there is some miss understanding. The following claims seem to be wrong:
The bolt acks upon completion and the spout moves forward to the next message.
at any given time 6 messages are being processed parallely in storm cluster
=> A spout is not waiting for the acks; it fetches tuples over-and-over again with the maximum speed regardless of the processing speed of the bolts -- as long as new messages are available in Kafka. (Or did you limit the number of tuples in flight via max.spout.pending?). Thus, many messages are processed in parallel (even if only #executors are given to a UDF -- many other messages are buffered in internal Storm queues).
As far as I know (but I am not 100% sure), KafkaSpout "orders" the incoming acks and only move the offset if all consecutive acks are available -- ie, message 7 is not acked to Kafka if the Storm ack of message 6 is not there yet. Thus, KafkaSpout can re-emit message 6 if it fails. Re-call that Storm does not give any ordering guarantees.
Related
Can one have 2 clients reading a single topic such that they never receive the same message? If one client dies, the other keeps reading and gets all the messages.
In a word "redundant clients" - not for performance sake but for client failover.
All I have seen is examples of N partitions and >N clients in a consumer group where N clients get messages and the rest are idle. It's not optimal to have 2 clients on a single partition where one client does nothing until the other client fails.
More than one clients in the same consumer group cannot be assigned the same partition at the same time, therefore will never receive the same messages
The scenario you're asking for is more fault tolerance than load balancing... Assuming one partition, if you run two consumers and one encounters some fatal exception while consuming a message and doesn't commit that offset and the client dies, then the secondary idle consumer will pickup from the last committed offset and try consuming those same messages after the consumer group rebalances
I had a few questions about GoLang Kafka producer using librdkafka -
These are based on the logs I am seeing in the producer log when I set debug: all.
The producer spends some time in building message sets once the batch threshold is recached or linger.ms is passed. However, what happens almost all the time is - messages are moved from partition queue to xmit queue. I was trying to get some documentation on it, but could not find much, so wanted to check if I can get some help on the stack. My questions are following -
a) Does the application produce calls write to a partition specific queue(s)?
b) Are there one xmit queue and one partition queue per partition?
c) What triggers a transfer from partition queue to xmit queue? and why do we need two queues?
d) When the Kafka producer is creating messagesets for a partition - does it block all operations for the partition? (Like moving messages from partition queue to xmit queue)? In short, when message sets are being built for a partition, can new messages sneak in the xmit queue? Is it blocked?
e) How many threads work for creating message sets? Is it one per producer or one per partition?
I am designing an apache storm topology (using streamparse) built with one spout (apache kafka spout) and 1 bolt with a parallelism > 1 that read messages in batch from kafka spout and persist messages in mysql table
The bolt read messages in batches. If the batch complete successfully i manually commit apache kafka offset.
When the bolt insert on mysql fail i don't commit the offset in kafka, but some messages are already in the queue of the messages that the spout has sent to the bolt.
The messages that are already on the queue should be removed because i cannot advance the kafka offset without loosing the previous failed messages.
is there a way in streamparse to clean or fail all the messages that are already in the queue at bolt startup?
I don't know about streamparse, but the impression I get is that you want to bundle up tuples up and write them as a batch. Let's say you've written up to offset 10. Now your bolt receives offset 11-15, and the batch fails to write. Offset 15-20 is queued, and you want to not process them right now, because that would process the batches out of order.
Is this understanding right?
First, I would drop manually committing offsets. You should let the spout handle that. Assuming you are using storm-kafka-client, you can configure it to only commit offsets once the corresponding tuple and all preceding tuples have been acked.
What you should probably do is keep track in the bolt (or even better, in your database) what the highest offset was in the failed batch. Then when your bolt fails to write offset 11-15, you can make the bolt fail every tuple with offset > 15. At some point, you will receive offset 11-15 again, and can retry writing the batch. Since you failed all messages with offset > 15, they will also be retried, and will arrive after the messages in the failed batch.
This solution assumes that you don't do reordering of the message stream between the spout and your writer bolt, so the messages arrive at the bolt in the order they are emitted.
I'm running a Kafka cluster with 4 nodes, 1 producer and 1 consumer. It was working fine until consumer failed. Now after I restart the consumer, it starts consuming new messages but after some minutes it throws this error:
[WARN ]: org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - Auto offset commit failed for group eventGroup: Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. This means that the time between subsequent calls to poll() was longer than the configured session.timeout.ms, which typically implies that the poll loop is spending too much time message processing. You can address this either by increasing the session timeout or by reducing the maximum size of batches returned in poll() with max.poll.records.
And it starts consuming the same messages again and loops forever.
I increased session timeout, tried to change group id and it still does the same thing.
Also is the client version of Kafka consumer a big deal?
I'd suggest you to decouple the consumer and the processing logic, to start with. E.g. let the Kafka consumer only poll messages and maybe after sanitizing the messages (if necessary) delegate the actual processing of each record to a separate thread, then see if the same error is still occurring. The error says, you're spending too much time between the subsequent polls, so this might resolve your issue. Also, please mention the version of Kafka you're using. Kafka had a different heartbeat management policy before version 0.10 which could make this issue easier to reproduce.
I've implemented a heron topology which reads messages from Kafka queue. Hence my topology has a kafka spout and a bolt which counts the number of messages read from the queue.
When I emit say 10000 messages into kafka queue, I can see all the messages being received in the kafka spout in heron topology however there is few messages being lost at the bolt.
Following is the topology settings for heron
Config config = Config.newBuilder()
.setUserConfig("topology.max.spout.pending", 100000)
.setUserConfig("topology.message.timeout.secs", 100000)
.setNumContainers(1)
.setPerContainerCpu(3)
.setPerContainerRamInGigabytes(4)
.setDeliverySemantics("ATLEAST_ONCE")
.build();
Any pointers would be helpful.
EDIT: I'm using streamlet API of heron. I replaced the count bolt with the log bolt but seeing the same issue of message drop in the logs of log bolt
processingGraphBuilder.newSource(kafkaSource)
.log();
EDIT 2: I resolved the issue by completely removing the streamlet API. I reimplemented everything using basic spout and bolt API and had acking at the spout. This fixed the issue. I guess this happened due to no acking happening at spout in streamlet API
Simple answer: shouldn't drop.
A few questions:
- in heronui, what are the all time emit and ack counts of your spout?
- in heronui, what are the all time execute, ack and failure counts of your bolt?
When you say that messages are dropped are you seeing fails registered in the fail count metric or just that your execute count in the bolt does not tally with the emit count of the spout?
In Storm-compatibility mode metrics are calculated based on a sample (default is 5% I think). Therefore counts could be out by that margin. For example, depending on when the stream is sampled, you could send 100 tuples through and the execute count could be 80 or 120.