Kafka returns endOffset 15 for a partition, but the last message that can be consumed from has the offset 13, rather than 14, which I would expect. I wonder why.
The Kafka docs read
In the default read_uncommitted isolation level, the end offset is the high watermark (that is, the offset of the last successfully replicated message plus one). For read_committed consumers, the end offset is the last stable offset (LSO), which is the minimum of the high watermark and the smallest offset of any open transaction.
Here's kafkacat's output. I'm using kafkacat, because it can print the message offsets:
$ kafkacat -Ce -p0 -tTK -f'offset: %o key: %k\n'
offset: 0 key: 0108
offset: 1 key: 0253
offset: 4 key: 0278
offset: 5 key: 0198
offset: 8 key: 0278
offset: 9 key: 0210
offset: 10 key: 0253
offset: 11 key: 1058
offset: 12 key: 0141
offset: 13 key: 1141
% Reached end of topic TK [0] at offset 15: exiting
What's also baffling - and it may very well be related - is that the offsets are not consecutive, although I have not set up compaction etc.
Some more details:
$ kafka-topics.sh --bootstrap-server localhost:9092 --topic TK --describe
Topic: TK PartitionCount: 2 ReplicationFactor: 1 Configs: segment.bytes=1073741824
Topic: TK Partition: 0 Leader: 0 Replicas: 0 Isr: 0
Topic: TK Partition: 1 Leader: 0 Replicas: 0 Isr: 0
Printing the keys via kafka-console-consumer.sh:
$ kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic TK \
--offset earliest --partition 0 --timeout-ms 5000 \
--property print.key=true --property print.value=false
0108
0253
0278
0198
0278
0210
0253
1058
0141
1141
[2021-09-15 10:54:06,556] ERROR Error processing message, terminating consumer process: (kafka.tools.ConsoleConsumer$)
org.apache.kafka.common.errors.TimeoutException
Processed a total of 10 messages
N.B. This topic has been produced to without involvement of transactions, and *) consumption is being done in read_uncommitted mode.
*) Actually, processing.guarantee is set to exactly_once_beta, so that would amount to using transactions.
More info
It turns out I can reliably reproduce this case with my Streams app (1. wipe kafka/zookeeper data, 2. recreate topics, 3. run app), whose output is the topic that shows this problem.
I've meanwhile trimmed down the Streams app to this no-op topology and can still reproduce it:
Topologies:
Sub-topology: 0
Source: KSTREAM-SOURCE-0000000000 (topics: [TK1])
--> KSTREAM-SINK-0000000001
Sink: KSTREAM-SINK-0000000001 (topic: TK)
<-- KSTREAM-SOURCE-0000000000
News
Meanwhile I have replaced the locally running Kafka broker (2.5.0) with one running in a Docker container (wurstmeister/kafka:2.13-2.6.0). The problem persists.
The app is using kafka libraries versioned 6.0.1-ccs, corresponding to 2.6.0.
You should avoid doing calculations on offsets, Kafka ensures any new offset will merely be greater than the last one. You may wish to use keys and track whether or not you have received the proper amount of messages by verifying the proper number of keys have been received.
Kafka has many things to juggle such as Exactly-Once Semantics, re-sending messages, and other internal tasks related to the topic. Those messages will be discard (not share with you). You will only see your messages, and those message offsets will only go up.
These transaction markers are not exposed to applications, but are used by consumers in read_committed mode to filter out messages from aborted transactions and to not return messages which are part of open transactions
When I remove the setting processing.guarantee: exactly_once_beta the problem goes away. In terms of this problem, it doesn't matter whether I use exactly_once_beta or exactly_once.
I still wonder why that happens with exactly_once(_beta) - after all, in my tests there is smooth sailing and no transaction rollbacks etc.
In my latest tests this rule seems to apply to all partitions with at least one item in them:
endOffset == lastMsg.offset + 3
Which is 2 more than expected.
The Kafka docs mentioned in the question say that
For read_committed consumers, the end offset is the last stable offset (LSO), which is the minimum of the high watermark and the smallest offset of any open transaction.
So is Kafka perhaps pre-allocating offsets for 2 (???) transactions per partition?
This seems to be the answer:
https://stackoverflow.com/a/54637004/200445
each commit (or abort) of a transaction writes a commit (or abort) marker into the topic -- those transactional markers also "consume" one offset
Related
I have a Flink streaming application which is consuming data from a Kafka topic which has 3 partitions. Even though, the application is continuously running and working without any obvious errors, I see a lag in the consumer group for the flink app on all 3 partitions.
./kafka-consumer-groups.sh --bootstrap-server $URL --all-groups --describe
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
group-1 topic-test 0 9566 9568 2 - - -
group-1 topic-test 1 9672 9673 1 - - -
group-1 topic-test 2 9508 9509 1 - - -
If I send new records, they get processed but the lag still exists. I tried to view the last few records for partition 0 and this is what I got (ommiting the message part) -
./kafka-console-consumer.sh --topic topic-test --bootstrap-server $URL --property print.offset=true --partition 0 --offset 9560
Offset:9560
Offset:9561
Offset:9562
Offset:9563
Offset:9564
Offset:9565
The log-end-offset value is at 9568 and the current offset is at 9566. Why are these offsets not available in the console consumer and why does this lag exist?
There were a few instances where I noticed missing offsets. Example -
Offset:2344
Offset:2345
Offset:2347
Offset:2348
Why did the offset jump from 2345 to 2347 (skipping 2346)? Does this have something to do with how the producer is writing to the topic?
You can describe your topic for any sort of configuration added while it was created. If the log compaction is enabled through log.cleanup.policy=compact, then the behaviour will be different in the runtime. You can see these lags, due to log compactions lags value setting or missing offsets may be due messages produced with a key but null value.
Configuring The Log Cleaner
The log cleaner is enabled by default. This will start the pool of cleaner threads. To enable log cleaning on a particular topic, add the log-specific property log.cleanup.policy=compact.
The log.cleanup.policy property is a broker configuration setting defined in the broker's server.properties file; it affects all of the topics in the cluster that do not have a configuration override in place. The log cleaner can be configured to retain a minimum amount of the uncompacted "head" of the log. This is enabled by setting the compaction time lag log.cleaner.min.compaction.lag.ms.
This can be used to prevent messages newer than a minimum message age from being subject to compaction. If not set, all log segments are eligible for compaction except for the last segment, i.e. the one currently being written to. The active segment will not be compacted even if all of its messages are older than the minimum compaction time lag.
The log cleaner can be configured to ensure a maximum delay after which the uncompacted "head" of the log becomes eligible for log compaction log.cleaner.max.compaction.lag.ms.
The lag is calculated based on the latest offset committed by the Kafka consumer (lag=latest offset-latest offset committed). In general, Flink commits Kafka offsets when it performs a checkpoint, so there is always some lag if check it using the consumer groups commands.
That doesn't mean that Flink hasn't consumed and processed all the messages in the topic/partition, it just means that it has still not committed them.
I have start working with Kafka few days ago. I am using Kafka on Windows environments, I want to see the data in each partition of a Kafka topic.
I have a topic called ExampleTopic with replication.factor set to 3 and 3 partitions. I am able to see the data into the topic but I want to see which messages are going in which partitions.
Please let me know is it possible if yes then how?
You can use the --partition parameter of the Kafka console consumer to specify which partition to consume from:
bin\windows\kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic ExampleTopic --partition 0
You can also specify an --offset parameter that indicates which offset to start from. If absent, the consumption starts at the end of the partition.
I have got an GUI based tool to view data in each partition of a topic named kafka tool.
http://www.kafkatool.com
It’s a tool to manage our kafka cluster. Also provide many features should try.
Use kafkacat, e.g. :
$ kafkacat -b localhost:9092 -t my_topic -C \
-f '\nKey (%K bytes): %k\t\nValue (%S bytes): %s\n\
Timestamp: %T\tPartition: %p\tOffset: %o\n--\n'
Key (1 bytes): 1
Value (79 bytes): {"uid":1,"name":"Cliff","locale":"en_US","address_city":"St Louis","elite":"P"}
Timestamp: 1520618381093 Partition: 0 Offset: 0
I was just following the quick start guide for Kafka and I decided to test offsets a little bit.
The only modification I did to the default configuration was adding:
log.retention.minutes=5
My test topic was created as basic as possible, as suggested in the quick start guide (1 partition, replication factor 1):
$ bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
I've produced some messages, m1 and m2 (adding date before and after):
$ date
viernes, 21 de julio de 2017, 12:16:06 CEST
$ bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
>m1
>m2
>^C
$ date
viernes, 21 de julio de 2017, 12:16:25 CEST
The thing is I'm able to consume them from the beginning, but I'm not able to consume them given an offset (for instance, offset 0, which I understand points to the first message):
$ date
viernes, 21 de julio de 2017, 12:16:29 CEST
$ bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --offset 0 --partition 0
^CProcessed a total of 0 messages
$ date
viernes, 21 de julio de 2017, 12:17:25 CEST
$ bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning
m1
m2
^CProcessed a total of 2 messages
$ date
viernes, 21 de julio de 2017, 12:17:50 CEST
Most probably I've not understood well this statement from the documentation:
In fact, the only metadata retained on a per-consumer basis is the offset or position of that consumer in the log. This offset is controlled by the consumer: normally a consumer will advance its offset linearly as it reads records, but, in fact, since the position is controlled by the consumer it can consume records in any order it likes. For example a consumer can reset to an older offset to reprocess data from the past or skip ahead to the most recent record and start consuming from "now".
Moreover, I've seen that if a produce a third message (m3) after running the consumer as described above (i.e. pointing to offset 0), this third message is read:
$ bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --offset 0 --partition 0
m3
Could anybody explain this behavior, please? Thanks!
Alright, after a lot of comments and a bit of code searching I think that this is what is happening:
When you configured your retention period with 5 minutes, you caused Kafka to delete a few of your old messages - most notably the one with the offset 0. So at some point in time the smallest offset in partition 0 became lets say 4.
When you start a console consumer with --from-beginning, it internally calls a method that initializes the beginning offset to the smallest offset that can be found in the partition - 4 in this case. With that offset the consumer starts polling and receives that message and all subsequent ones, which is all messages for the partition.
If you start a consumer with --offset 0 that piece of code is bypassed and the consumer polls with an offset of 0 - the broker responds to that with an OFFSET_OUT_OF_RANGE error. The Consumer upon receiving that error resets the offset for the partition in question, and for this it uses the parameter auto.offset.reset which in theory can be earliest or latest.
However, due to the way that the ConsoleConsumer is written, the only way to have this parameter set to earliest is, if you pass the command line parameter --from-beginning - which cannot be combined with --offset - so effectively the only possible value that auto.offset.reset can have here is: latest.
So what happens when you poll with an offset of 0 that does not exist is an unsuccessful poll for data and after that the same behavior as if you hadn't passed any parameter at all.
Hope that helps and makes sense.
Update:
As of Kafka version 1.0 this behavior has been changed by KAFKA-5629 and should now behave a bit more in line with expectations.
You can try: --offset earliest
I'm not sure how to explain the issue I'm facing with Kafka, but I'll try my best. I have a set of 4 consumers in the same consumer group named :
absolutegrounds.helper.processor
consuming from a topic with 5 partitions; therefore each consumer in the group is being assigned to 1 partition and 1 consumer to 2 partitions in order to distribute 5 partitions between 4 consumers fairly.
But for some reason I cannot figure out, the initial assignment turned into only 2 consumers being assigned into all the available partitions, i.e. 1 consumer with 3 partitions and 1 consumer with 2 partitions. However, there are still 4 consumers theoretically in the same consumer group:
[medinen#ocvlp-rks001 kafka_2.11-0.10.0.1]$ ./bin/kafka-run-class.sh kafka.admin.ConsumerGroupCommand --new-consumer --bootstrap-server localhost:9092 --describe --group absolutegrounds.helper.processor
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG OWNER
absolutegrounds.helper.processor AG_TASK_SOURCE 0 27286 31535 4249 consumer-1_/10.132.9.128
absolutegrounds.helper.processor AG_TASK_SOURCE 1 28015 28045 30 consumer-1_/10.132.9.128
absolutegrounds.helper.processor AG_TASK_SOURCE 2 35437 40091 4654 consumer-1_/10.132.9.128
absolutegrounds.helper.processor AG_TASK_SOURCE 3 31765 31874 109 consumer-1_/10.132.8.23
absolutegrounds.helper.processor AG_TASK_SOURCE 4 33279 38003 4724 consumer-1_/10.132.8.23
The most bizarre behaviour is that the other 2 consumers left out of the consumer group (as per the response from Kafka above) seem to be still consuming from the topic as per the logs I see in my application, although I cannot find them anywhere as part of the consumer group. And even more bizarre is the fact that 1 consumer is supposed to be assigned to all the partitions in the topic whereas the other one is assigned only partition 4. See logs from the application (this is a Spring Boot application using Spring Kafka):
First left consumer:
- - 08/05/2017 12:27:29.119 - [-kafka-consumer-1] INFO o.a.k.c.c.i.ConsumerCoordinator - Setting newly assigned partitions [AG_TASK_SOURCE-0, AG_TASK_SOURCE-1, AG_TASK_SOURCE-2, AG_TASK_SOURCE-3, AG_TASK_SOURCE-4] for group absolutegrounds.helper.processor
Second left consumer:
- - 08/05/2017 12:27:19.044 - [-kafka-consumer-1] INFO o.a.k.c.c.i.ConsumerCoordinator - Setting newly assigned partitions [AG_TASK_SOURCE-4] for group absolutegrounds.helper.processor
Trying to understand the reasoning behind this behaviour, I've taken a look into the topic that stores all the offsets for the consumers:
__consumer_offsets
using this command:
kafka/kafka_2.11-0.10.0.1/bin/kafka-console-consumer.sh --consumer.config /tmp/consumer.config --formatter "kafka.coordinator.GroupMetadataManager\$GroupMetadataMessageFormatter" --zookeeper ocvlp-rks003:2181 --topic __consumer_offsets --from-beginning | grep "absolutegrounds.helper.processor"
and this is what I've found:
absolutegrounds.helper.processor::[absolutegrounds.helper.processor,consumer,Stable,Map(consumer-1-170fb8f6-c8d3-4782-8940-350673b859cb -> [consumer-1-170fb8f6-c8d3-4782-8940-350673b859cb,consumer-1,/10.132.8.23,10000], consumer-1-b8d3afc0-159e-4660-bc65-faf68900c332 -> [consumer-1-b8d3afc0-159e-4660-bc65-faf68900c332,consumer-1,/10.132.9.128,10000], consumer-1-dddf10ad-187b-4a29-9996-e05edaad3caf -> [consumer-1-dddf10ad-187b-4a29-9996-e05edaad3caf,consumer-1,/10.132.8.22,10000], consumer-1-2e4069f6-f3a8-4ede-a4f4-aadce6a3adb7 -> [consumer-1-2e4069f6-f3a8-4ede-a4f4-aadce6a3adb7,consumer-1,/10.132.9.129,10000])]
absolutegrounds.helper.processor::[absolutegrounds.helper.processor,consumer,Stable,Map(consumer-1-66de4a46-538c-425f-8e95-5a00ff5eb5fd -> [consumer-1-66de4a46-538c-425f-8e95-5a00ff5eb5fd,consumer-1,/10.132.9.129,10000])]
absolutegrounds.helper.processor::[absolutegrounds.helper.processor,consumer,Stable,Map(consumer-1-5b96166e-e528-48f7-8f6e-18a67328eae6 -> [consumer-1-5b96166e-e528-48f7-8f6e-18a67328eae6,consumer-1,/10.132.9.128,10000], consumer-1-dcfff37a-8ad3-403c-a070-cca82a1f6d21 -> [consumer-1-dcfff37a-8ad3-403c-a070-cca82a1f6d21,consumer-1,/10.132.8.23,10000])]
absolutegrounds.helper.processor::[absolutegrounds.helper.processor,consumer,Stable,Map(consumer-1-5b96166e-e528-48f7-8f6e-18a67328eae6 -> [consumer-1-5b96166e-e528-48f7-8f6e-18a67328eae6,consumer-1,/10.132.9.128,10000], consumer-1-dcfff37a-8ad3-403c-a070-cca82a1f6d21 -> [consumer-1-dcfff37a-8ad3-403c-a070-cca82a1f6d21,consumer-1,/10.132.8.23,10000])]
From the response of Kafka, I can see that at some point in time all 4 consumers where properly distributed between the partitions:
absolutegrounds.helper.processor::[absolutegrounds.helper.processor,consumer,Stable,Map(consumer-1-170fb8f6-c8d3-4782-8940-350673b859cb -> [consumer-1-170fb8f6-c8d3-4782-8940-350673b859cb,consumer-1,/10.132.8.23,10000], consumer-1-b8d3afc0-159e-4660-bc65-faf68900c332 -> [consumer-1-b8d3afc0-159e-4660-bc65-faf68900c332,consumer-1,/10.132.9.128,10000], consumer-1-dddf10ad-187b-4a29-9996-e05edaad3caf -> [consumer-1-dddf10ad-187b-4a29-9996-e05edaad3caf,consumer-1,/10.132.8.22,10000], consumer-1-2e4069f6-f3a8-4ede-a4f4-aadce6a3adb7 -> [consumer-1-2e4069f6-f3a8-4ede-a4f4-aadce6a3adb7,consumer-1,/10.132.9.129,10000])]
However, at some point later, the assignment changed to the current scenario where only 2 of the 4 consumers in the consumer group are assigned partitions.
I've struggled to understand what could have led to this situation, but I cannot find a valid answer to figure it out and fix it.
Anyone can help here? Thanks.
I am going to setup a kafka cluster for our intensive messaging system.
Currently we have setup two kafka clusters, one based in London (LD) as primary and antoher one based in New York (NY) as DR ( backup), and we have made java clients to replicate data from LD to NY.
As Kafka has built-in features such as partitioning, and replication for scalibity, high availability and failover purpose so that we want to create a single bigger cluster comprising of both servers in London and New York
But...
We are having the problem with connectivity between NY and LD servers, the network speed is really bad.
I have performed server tests.
producer config:
- acks=1 ( requires acknowlegement from partition leader only)
- sending Async.
when producers in London sending messages to brokers in LD , the thoughput 100,000 msg /sec, providing message size is : 100bytes => 10MB/sec
when producers in London and sending message to broker in NY, the thoughput 10 msg/sec, providing message size is : 100bytes => 1KB/sec
So...
I am considering any way to make sure the producer/consumer take the advantage of locality that means if they are in the same network will send messages to the neariest broker.
Lets say: consumers in LD will send messages to LD based brokers.
(I understand that the write/read request only happens on partition leader).
Any suggestion would be highly appriciate.
From what I understood your current structure is:
1 Broker located in NY.
1 Broker located in LD.
n number of topics. (I am going to assume the number of topics is 1).
n number of partitions on the topic. (I am going to assume the number of partitions is 2).
Both of the partitions replicated over the brokers.
You want to make broker located in LD leader of all the partitions, so all the producers will interact with this broker and the broker located in NY will be used as replication. If this is the case, then, you can do the following:
Check the configuration of your topic:
./kafka-topics.sh --describe --topic stream-log
Topic:<topic-name> PartitionCount:2 ReplicationFactor:2 Configs:
Topic: stream-log Partition: 0 Leader: 0 Replicas: 0,1 Isr: 0,1
Topic: stream-log Partition: 1 Leader: 1 Replicas: 1,0 Isr: 1,0
And assuming:
LD Broker ID: 0
NY Broker ID: 1
You can observe how the leader of the partition 1 is handled by the broker 1 (NY), we want to modify that, to do so is necessary to reassign the partitions:
./kafka-reassign-partitions.sh --reassignment-json-file manual_assign.json --execute
The contents of the JSON file:
{"partitions": [
{"topic": "<topic-name>", "partition": 0, "replicas": [0,1]},
{"topic": "<topic-name>", "partition": 1, "replicas": [0,1]}
],
"version":1
}
Finally, to force kafka to update the leader, run:
./kafka-preferred-replica-election.sh
The last command will affect all the topics you have created if do not specify a list of topics, that should not be a problem but have it in mind.
Is worth to have a look to this guide, it explains something similar. And if you are curious you can check the official documentation of the tools here.