I know, storm doesn't guarantee total ordering gurantee for kafka topics, but see in many documents, storm guarantees consumption/processing the messages maintaining the order at partition level.
I am looking for a sample storm topology, that consumes/processes the messages of a kafka topic maintaining the order of messages at a kafka partition level.. NOT Total Order!! ONLY partition level ordering guarantee.
please share if you know any sample application. Thanks a lot!!
Have you looked at Apache Storm examples here? https://github.com/apache/storm/tree/master/external/storm-kafka
You may want to consider standard example and scale it based on your needs. Also, while defining Schema for the KafkaSpuout, you may want to output some key as part of the tuple and later use FieldG
rouping.
Related
Is there any API or client library exist which can tell me how much percent topic is filled with data so that I can figure out is there any way to check whether partitions are balanced or not
This is a good strategy to discuss before designing and development on Kafka.
The first point you need to consider how you are defining your key and whats exactly partitioner you are planning to use while producing a message to the topics.
Thumb-rule:
If you not bothering collecting messages in different groups based on key just pass the key as null to redistribute your messages in a round-robin manner.
You can also use a custom partitioner to define partitioning in case you need to do some more refinement.
To check the partition distribution, the best approach is to check the lagging on each partition and rate byte/sec
There many ways to monitor
1.You can use simple API to get various matrices like lagging, rate, etc
You can refer here Kafka Metrices
kafka.server:type=ReplicaFetcherManager,name=MaxLag,clientId=Replica
2.I generally prefer Grafana with JMX exported it will visualize matrices
Grafana
3.We can also use CLI to identify each partition offset and lagging, and really give you the overall figure instantly
bin/kafka-consumer-groups.sh --bootstrap-server broker1:9092 --describe --group consumer-group
TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
test 1 10 30 20 consumer-group
You can also do with programmatically
How to identify partition lagging
Confluent Control-Center is paid one but a very interesting tool to monitor overall Kafka including consumer and its partitions/
Confluent control center
Assume that you created a topic X. Your producers started to push tons of data into your topic. Your topic is growing exponentially. Depending on the configuration log.segment.bytes, Kafka will create a new segment and start writing data into it. Old segment will be kept for log.retention.ms milliseconds. Because of this, 100% of a topic itself is tricky calculate.
However, if you are looking for a tool that can allocate partitions depending on the load on each broker then I would recommend looking into Kafka-kit (https://www.datadoghq.com/blog/engineering/introducing-kafka-kit-tools-for-scaling-kafka/).
I have a single kafka client instance that is consuming from 200 partitions, now I want it to consume only on the first 3 kafka partition for debugging and sampling purpose.
Is there a way I can do that?
Or alternatively I can consume from all partition and drop message from partition that is not from the first 3 partition. Is there a way I can find out which partition is the message from?
You can use KafkaConsumer.assign(java.util.Collection<TopicPartition> partitions) to assign a specific set of partitions. To find out the parttion of the message you can use ConsumerRecord.partition()
if you want consume only partially partition,
implement org.apache.kafka.clients.consumer.internals.PartitionAssignor
already apache kafka's test use MockPartitionAssignor (extends PartitionAssignor)
implement PartitionAssignor and, setup "partition.assignment.strategy"
reference : https://kafka.apache.org/documentation/#newconsumerconfigs
Since you haven't specified which consumer API you use, I am going to give an example based on the Python kafka-python library.
consumer = KafkaConsumer('my-topic',
group_id='my-group',
bootstrap_servers=['host:9092'])
for message in consumer:
if message.partition in [0, 1, 2]:
# do something with message.value
If you really want to read only from a subset of partitions, you can do something like:
consumer.assign([TopicPartition('foobar', 2)])
next_message = next(consumer)
would work for you. But I would suggest the first approach. Irrespective of the language of development you choose, all Kafka consumers must be implementing the above features.
What is maximum limit of topics can a consumer subscribe to in Kafka. Am not able to find this value documented anywhere.
If consumer subscribes 500000 or more topics, will there be downgrade in performance.
500,000 or more topics in a single Kafka cluster would be a bad design from the broker point of view. You typically want to keep the number of topic partitions down to the low tens of thousands.
If you find yourself thinking you need that many topics in Kafka you might instead want to consider creating a smaller number of topics and having 500,000 or more keys instead. The number of keys in Kafka is unlimited.
To be technical the "maximum" number of topics you could be subscribed to would be constrained by the available memory space for your consumer process (if your topics are listed explicitly then a very large portion of the Java String pool will be your topics). This seems the less likely limiting factor (listing that many topics explicitly is prohibitive).
Another consideration is how the Topic assignment data structures are setup at Group Coordinator Brokers. They could run out of space to record the topic assignment depending on how they do it.
Lastly, which is the most plausible, is the available memory on your Apache Zookeeper node. ZK keeps ALL data in memory for fast retrieval. ZK is also not sharded, meaning all data MUST fit onto one node. This means there is a limit to the number of topics you can create, which is constrained by the available memory on a ZK node.
Consumption is initiated by the consumers. The act of subscribing to a topic does not mean the consumer will start receiving messages for that topic. So as long as the consumer can poll and process data for that many topics, Kafka should be fine as well.
Consumer is fairly independent entity than Kafka cluster, unless you are talking about build in command line consumer that is shipped with Kafka
That said logic of subscribing to a kafka topic, how many to subscribe to and how to handle that data is upto the consumer. So scalability issue here lies with consumer logic
Last but not the least, I am not sure it is a good idea to consumer too many topics within a single consumer. The vary purpose of pub sub mechanism that Kafka provides through the segregation of messages into various topics is to facilitate the handling of specific category of messages using separate consumers. So I think if you want to consume many topics like few 1000s of them using a single consumer, why divide the data into separate topics first using Kafka.
We are developing a kafka based streaming system in which the producer would produce to multiple partitions within its topic and a single consumer would consume from the topic. I know that kafka maintains message order within partitions, but can we maintain a global message order between partitions within a topic?
Short answer:
no, Kafka does not provide any ordering guarantees between partitions.
Long answer:
I don't quite understand your problem. If you are saying you have only one consumer consuming your topic, why would you have more than 1 partition in that topic and reinvent the wheel trying to maintain order between partitions? If you want to leave some space for future growth, e.g. adding another consumer to consume a part of partitions, then you'll have to rethink your "global message order" idea.
Do you really need ALL messages to be processed in order? Or maybe you could partition by client/application/whatever and maintain order per partition? In most cases you don't really need that global message order, but just have to partition your data properly.
Maintaining order between multiple consumers is a really tough problem to solve, and even if solved correctly you'll just neglect all Kafka benefits.
You can't benifit from kafka if you want the global ordering in more than one partition. Kafka only supports message ordering in only one partition. In our company, we need only the same catergory messages are sent to the same partition, which can easily partition using partitionId.
The purpose of partitions in Kafka is to create a partial order of messages in a broader topic, where the messages follow a strict total order in any given partition. So the answer is 'no', it would defeat the purpose of partitions if any notion of cross-partition order were to be introduced.
I would suggest instead focusing on how messages (records, in Kafka parlance) are keyed, which effectively determines how they are mapped to a partition. Which partition specifically doesn't matter, as long as the mapping is deterministic and repeatable — all you should care about is that identically keyed records will always appear on the same partition and, hence, will not be assigned to multiple consumers at the same time (within the same consumer group).
If you are publishing updates to persisted entities, the primary key of the entity is typically a good starting point for a Kafka record key. If there needs to be some order of updates across a connected graph of entities, then taking the ID root of the graph and making it the key will likely satisfy your ordering needs.
One of the first things I think about when using a new service (such as a non-RDBMS data store or a message queue) is: "How should I structure my data?".
I've read and watched some introductory materials. In particular, take, for example, Kafka: a Distributed Messaging System for Log Processing, which writes:
"a Topic is the container with which messages are associated"
"the smallest unit of parallelism is the partition of a topic. This implies that all messages that ... belong to a particular partition of a topic will be consumed by a consumer in a consumer group."
Knowing this, what would be a good example that illustrates how to use topics and partitions? When should something be a topic? When should something be a partition?
As an example, let's say my (Clojure) data looks like:
{:user-id 101 :viewed "/page1.html" :at #inst "2013-04-12T23:20:50.22Z"}
{:user-id 102 :viewed "/page2.html" :at #inst "2013-04-12T23:20:55.50Z"}
Should the topic be based on user-id? viewed? at? What about the partition?
How do I decide?
When structuring your data for Kafka it really depends on how it´s meant to be consumed.
In my mind, a topic is a grouping of messages of a similar type that will be consumed by the same type of consumer so in the example above, I would just have a single topic and if you´ll decide to push some other kind of data through Kafka, you can add a new topic for that later.
Topics are registered in ZooKeeper which means that you might run into issues if trying to add too many of them, e.g. the case where you have a million users and have decided to create a topic per user.
Partitions on the other hand is a way to parallelize the consumption of the messages. The total number of partitions in a broker cluster need to be at least the same as the number of consumers in a consumer group to make sense of the partitioning feature. Consumers in a consumer group will split the burden of processing the topic between themselves according to the partitioning so that one consumer will only be concerned with messages in the partition itself is "assigned to".
Partitioning can either be explicitly set using a partition key on the producer side or if not provided, a random partition will be selected for every message.
Once you know how to partition your event stream, the topic name will be easy, so let's answer that question first.
#Ludd is correct - the partition structure you choose will depend largely on how you want to process the event stream. Ideally you want a partition key which means that your event processing is partition-local.
For example:
If you care about users' average time-on-site, then you should partition by :user-id. That way, all the events related to a single user's site activity will be available within the same partition. This means that a stream processing engine such as Apache Samza can calculate average time-on-site for a given user just by looking at the events in a single partition. This avoids having to perform any kind of costly partition-global processing
If you care about the most popular pages on your website, you should partition by the :viewed page. Again, Samza will be able to keep a count of a given page's views just by looking at the events in a single partition
Generally, we are trying to avoid having to rely on global state (such as keeping counts in a remote database like DynamoDB or Cassandra), and instead be able to work using partition-local state. This is because local state is a fundamental primitive in stream processing.
If you need both of the above use-cases, then a common pattern with Kafka is to first partition by say :user-id, and then to re-partition by :viewed ready for the next phase of processing.
On topic names - an obvious one here would be events or user-events. To be more specific you could go with with events-by-user-id and/or events-by-viewed.
This is not exactly related to the question, but in case you already have decided upon the logical segregation of records based on topics, and want to optimize the topic/partition count in Kafka, this blog post might come handy.
Key takeaways in a nutshell:
In general, the more partitions there are in a Kafka cluster, the higher the throughput one can achieve. Let the max throughout achievable on a single partition for production be p and consumption be c. Let’s say your target throughput is t. Then you need to have at least max(t/p, t/c) partitions.
Currently, in Kafka, each broker opens a file handle of both the index and the data file of every log segment. So, the more partitions, the higher that one needs to configure the open file handle limit in the underlying operating system. E.g. in our production system, we once saw an error saying too many files are open, while we had around 3600 topic partitions.
When a broker is shut down uncleanly (e.g., kill -9), the observed unavailability could be proportional to the number of partitions.
The end-to-end latency in Kafka is defined by the time from when a message is published by the producer to when the message is read by the consumer. As a rule of thumb, if you care about latency, it’s probably a good idea to limit the number of partitions per broker to 100 x b x r, where b is the number of brokers in a Kafka cluster and r is the replication factor.
I think topic name is a conclusion of a kind of messages, and producer publish message to the topic and consumer subscribe message through subscribe topic.
A topic could have many partitions. partition is good for parallelism. partition is also the unit of replication,so in Kafka, leader and follower is also said at the level of partition. Actually a partition is an ordered queue which the order is the message arrived order. And the topic is composed by one or more queue in a simple word. This is useful for us to model our structure.
Kafka is developed by LinkedIn for log aggregation and delivery. this scene is very good as a example.
The user's events on your web or app can be logged by your Web sever and then sent to Kafka broker through the producer. In producer, you could specific the partition method, for example : event type (different event is saved in different partition) or event time (partition a day into different period according your app logic) or user type or just no logic and balance all logs into many partitions.
About your case in question, you can create one topic called "page-view-event", and create N partitions through hash keys to distribute the logs into all partitions evenly. Or you could choose a partition logic to make log distributing by your spirit.