Partitions define the unit of parallelism in kafka , but increasing partitions may result in decreased producer throughput as due to replication,cluster bandwidth will decrease.
but in experiments it was observed ,
With 3 brokers: When we take 2 partitions on each broker then performance reduces as compared to 1 partition on each broker.
With 9 brokers: When we take 3 partitions on each broker then performance increases as compared to 1 partition on each broker.
Considering the scenerio of 3 brokers the performance should have degraded but it increased.
What can be the reason for such behaviour ??
Experiment details:
kafka-producer-perf-test was used to do benchmarking
Parameters passed to tool: --num-records 12000000 --throughput -1 acks=1 linger.ms=100 buffer.memory=5242880 compression.type=none request.timeout.ms=30000 --record-size 1000
Results of test in attached image
Related
I am testing the scenario as follows.
I am producing the messages to sink which is the Kafka containing the three brokers.
What if brokers are going to down, the producing side have an any issue because of the broker-down?
When I tested it on my local using Flink, I generated the messages and sinked them to Kafka. And I have three kafka brokers. When I deleted the number of brokers to 2, there are no problems. And obviously, when all the brokers are going to down, then the producer-side app gives an exception.
So, according to these fact, I think that the producer-side app can still alive without any errors until one broker remains. Is my assumption correct?
Below is the my producer side configuration.
acks = 1
batch.size = 16384
compression.type = lz4
connections.max.idle.ms = 540000
delivery.timeout.ms = 120000
enable.idempotence = false
key.serializer = class org.apache.kafka.common.serialization.ByteArraySerializer
linger.ms = 0
partitioner.class = class org.apache.kafka.clients.producer.internals.DefaultPartitioner
receive.buffer.bytes = 32768
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
replication is two and I have three partitions for each topic.
Any help will be appreciated.
Thanks.
It all depends on your requirements and your producer configuration. At the moment, yes you can have 2 out of 3 brokers alive and your producer will continue as normal.
This is because you have acks=1 which means only the leader has to acknowledge the message before it is considered successful. The followers don't have to acknowledge the message.
You should also check whether you have changed min.insync.replicas at the broker or topic level configuration. The default is 1, meaning only 1 in-sync replica is needed for a broker to allow acks=all requests.
Side note: you have replication=2, I'd change this so partitions were replicated across all 3 brokers.
I'm not sure if I understood your question, but in Kafka client API there are some retryable Exceptions (like Not Leader, or unreached/unknown host).
So your Producer wil retry until reaching the first limit of these two configs:
retries : https://kafka.apache.org/documentation/#producerconfigs_retries
delivery.timeout.ms : https://kafka.apache.org/documentation/#producerconfigs_delivery.timeout.ms
So using the default values :
retries > 2 billions time &
delivery.timeout.ms = 2 minutes
Your producer will do N retries for only 2 minutes then crashes.
I have a Flink streaming application which is consuming data from a Kafka topic which has 3 partitions. Even though, the application is continuously running and working without any obvious errors, I see a lag in the consumer group for the flink app on all 3 partitions.
./kafka-consumer-groups.sh --bootstrap-server $URL --all-groups --describe
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
group-1 topic-test 0 9566 9568 2 - - -
group-1 topic-test 1 9672 9673 1 - - -
group-1 topic-test 2 9508 9509 1 - - -
If I send new records, they get processed but the lag still exists. I tried to view the last few records for partition 0 and this is what I got (ommiting the message part) -
./kafka-console-consumer.sh --topic topic-test --bootstrap-server $URL --property print.offset=true --partition 0 --offset 9560
Offset:9560
Offset:9561
Offset:9562
Offset:9563
Offset:9564
Offset:9565
The log-end-offset value is at 9568 and the current offset is at 9566. Why are these offsets not available in the console consumer and why does this lag exist?
There were a few instances where I noticed missing offsets. Example -
Offset:2344
Offset:2345
Offset:2347
Offset:2348
Why did the offset jump from 2345 to 2347 (skipping 2346)? Does this have something to do with how the producer is writing to the topic?
You can describe your topic for any sort of configuration added while it was created. If the log compaction is enabled through log.cleanup.policy=compact, then the behaviour will be different in the runtime. You can see these lags, due to log compactions lags value setting or missing offsets may be due messages produced with a key but null value.
Configuring The Log Cleaner
The log cleaner is enabled by default. This will start the pool of cleaner threads. To enable log cleaning on a particular topic, add the log-specific property log.cleanup.policy=compact.
The log.cleanup.policy property is a broker configuration setting defined in the broker's server.properties file; it affects all of the topics in the cluster that do not have a configuration override in place. The log cleaner can be configured to retain a minimum amount of the uncompacted "head" of the log. This is enabled by setting the compaction time lag log.cleaner.min.compaction.lag.ms.
This can be used to prevent messages newer than a minimum message age from being subject to compaction. If not set, all log segments are eligible for compaction except for the last segment, i.e. the one currently being written to. The active segment will not be compacted even if all of its messages are older than the minimum compaction time lag.
The log cleaner can be configured to ensure a maximum delay after which the uncompacted "head" of the log becomes eligible for log compaction log.cleaner.max.compaction.lag.ms.
The lag is calculated based on the latest offset committed by the Kafka consumer (lag=latest offset-latest offset committed). In general, Flink commits Kafka offsets when it performs a checkpoint, so there is always some lag if check it using the consumer groups commands.
That doesn't mean that Flink hasn't consumed and processed all the messages in the topic/partition, it just means that it has still not committed them.
I am trying to come up with a configuration that would enforce producer quota setup based on an average byte rate of producer.
I did a test with a 3 node cluster. The topic however was created with 1 partition and 1 replication factor so that the producer_byte_rate can be measured only for 1 broker (the leader broker).
I set the producer_byte_rate to 20480 on client id test_producer_quota.
I used kafka-producer-perf-test to test out the throughput and throttle.
kafka-producer-perf-test --producer-props bootstrap.servers=SSL://kafka-broker1:6667 \
client.id=test_producer_quota \
--topic quota_test \
--producer.config /myfolder/client.properties \
--record.size 2048 --num-records 4000 --throughput -1
I expected the producer client to learn about the throttle and eventually smooth out the requests sent to the broker. Instead I noticed there is alternate throghput of 98 rec/sec and 21 recs/sec for a period of more than 30 seconds. During this time average latency slowly kept increseing and finally when it hits 120000 ms, I start to see Timeout exception as below
org.apache.kafka.common.errors.TimeoutException : Expiring 7 records for quota_test-0: 120000 ms has passed since batch creation.
What is possibly causing this issue?
The producer is hitting timeout when latency reaches 120 seconds (default value of delivery.timeout.ms )
Why isnt the producer not learning about the throttle and quota and slowing down or backing off
What other producer configuration could help alleviate this timeout issue ?
(2048 * 4000) / 20480 = 400 (sec)
This means that, if your producer is trying to send the 4000 records full speed ( which is the case because you set throughput to -1), then it might batch them and put them in the queue.. in maybe one or two seconds (depending on your CPU).
Then, thanks to your quota settings (20480), you can be sure that the broker won't 'complete' the processing of those 4000 records before at least 399 or 398 seconds.
The broker does not return an error when a client exceeds its quota, but instead attempts to slow the client down. The broker computes the amount of delay needed to bring a client under its quota and delays the response for that amount of time.
Your request.timeout.ms being set to 120 seconds, you then have this timeoutException.
What does running following Kafka tool actually give ?
./bin/kafka-run-class.sh org.apache.kafka.tools.ProducerPerformance --throughput=10000--topic=TOPIC--num-records=50000000 --record-size=200 --producer-props bootstrap.servers=SERVERS buffer.memory=67108864 batch.size=64000
When running with a single producer I get 90MB/s. When I use 3 separate producers on separate nodes I get only around 60 MB/s per producer. ( My Kafka cluster consists of 2 nodes, and topic has 6 partitions )
What does 90 MB/s mean? Is it the maximum rate at which a producer can produce?
Does partition count affect this value?
Why it drops to 60 MB/s when there are 3 producers ( still no network saturation on broker front)?
Thank you
I am going to setup a kafka cluster for our intensive messaging system.
Currently we have setup two kafka clusters, one based in London (LD) as primary and antoher one based in New York (NY) as DR ( backup), and we have made java clients to replicate data from LD to NY.
As Kafka has built-in features such as partitioning, and replication for scalibity, high availability and failover purpose so that we want to create a single bigger cluster comprising of both servers in London and New York
But...
We are having the problem with connectivity between NY and LD servers, the network speed is really bad.
I have performed server tests.
producer config:
- acks=1 ( requires acknowlegement from partition leader only)
- sending Async.
when producers in London sending messages to brokers in LD , the thoughput 100,000 msg /sec, providing message size is : 100bytes => 10MB/sec
when producers in London and sending message to broker in NY, the thoughput 10 msg/sec, providing message size is : 100bytes => 1KB/sec
So...
I am considering any way to make sure the producer/consumer take the advantage of locality that means if they are in the same network will send messages to the neariest broker.
Lets say: consumers in LD will send messages to LD based brokers.
(I understand that the write/read request only happens on partition leader).
Any suggestion would be highly appriciate.
From what I understood your current structure is:
1 Broker located in NY.
1 Broker located in LD.
n number of topics. (I am going to assume the number of topics is 1).
n number of partitions on the topic. (I am going to assume the number of partitions is 2).
Both of the partitions replicated over the brokers.
You want to make broker located in LD leader of all the partitions, so all the producers will interact with this broker and the broker located in NY will be used as replication. If this is the case, then, you can do the following:
Check the configuration of your topic:
./kafka-topics.sh --describe --topic stream-log
Topic:<topic-name> PartitionCount:2 ReplicationFactor:2 Configs:
Topic: stream-log Partition: 0 Leader: 0 Replicas: 0,1 Isr: 0,1
Topic: stream-log Partition: 1 Leader: 1 Replicas: 1,0 Isr: 1,0
And assuming:
LD Broker ID: 0
NY Broker ID: 1
You can observe how the leader of the partition 1 is handled by the broker 1 (NY), we want to modify that, to do so is necessary to reassign the partitions:
./kafka-reassign-partitions.sh --reassignment-json-file manual_assign.json --execute
The contents of the JSON file:
{"partitions": [
{"topic": "<topic-name>", "partition": 0, "replicas": [0,1]},
{"topic": "<topic-name>", "partition": 1, "replicas": [0,1]}
],
"version":1
}
Finally, to force kafka to update the leader, run:
./kafka-preferred-replica-election.sh
The last command will affect all the topics you have created if do not specify a list of topics, that should not be a problem but have it in mind.
Is worth to have a look to this guide, it explains something similar. And if you are curious you can check the official documentation of the tools here.