I've got sales messages with timestamps and several messages belonging to the same sale share the same ID. But only one contains a field that I want to store in a KTable to enrich follwing messages with the corresponding ID.
I cannot be sure that the message with the necessary field will always be sent first.
Is it possible to do a Join including also the messages prior to populating the KTable (let's say timestamps - 5min)?
(What if your data comes in batches with breaks of x min?)
Thank you!
Not 100% sure if I understand the use case, but it seems you only want to store a message if it contains the corresponding field, but you want to drop the message otherwise. For this case, you could read the data as a KStream and apply a filter before you put the records into a table:
KStream input = builder.stream("table-topic");
KTable table = input.filter(/*contains field or is tombstone*/)
.toTable();
Note, that you might want to ensure that tombstone messages, ie, messages with value==null are not filtered out, to preserve delete semantics (seems to be use-case dependent).
Related
Here is my case:
System produces messages to one topic, and there are two kind of messages:
A. new users messages
produce: every time any user data changed.
markded as: U1, U2, ... Un.
B. user attribute metadata change messages
e.g: user has two attributes name, email, then added a custom attribute profile.
produce: every time user attribute metadata changed.
marked as: M
When we consume this topic, we need to guarantee partial orders:
Same User's data should follow its order.
consumption of metadata change should always: before consuming user data message after this change, after user data message before this change.
Example:
message natural order:
(0:U1)->(1:U2)->(2:U1)->(3:U3)->(4:U1)->(5:M)->(6:U1)->(7:U2)->(8:U2)->(9:M)->(10:U1)
accepted consuming order:
(0:U1)->(2:U1)->(3:U3)->(1:U2)->(4:U1)->(5:M)->(7:U2)->(6:U1)->...
The question
If there is no M in it, I can put different User data into different partitions, to increase throughout, but consider the existence of M's ordering requirement, can I make different partition for this topic?
You can use any kind of user identifier (should be equal for "new users messages" and "user attribute metadata change messages" of the same user and at the same time unique for a particular user) as a key of the Kafka message. That way, the data will get partitioned based on the user identifier and you ensure that the data of one user will go to a single partition while keeping the order. That way you can scale with multiple partitions.
When producing the messages to the topic, make sure to synchronously produce the data, e.g. wait till the first message is received before sending the second.
I have a service calculating reputation scores for accounts. It puts the calculation results in a Kafka topic called "ReputationScores". Each message looks something like this:
{ "account" : 12345, "repScore" : 98765}
I'd like my consumer to be able to consume only those messages for a specific account.
For example, I’d like to have a single instance of a consumer consume only messages with topic “ReputationScore” for account 12345. That instance should probably be the only member of its consumer group.
Can Kafka filter based on message contents? What's the best way to do this?
Thanks for your help.
Can Kafka filter based on message contents?
Since kafka itself doesn't know what's in your data, it cannot index it, therefore it's not readily searchable. You would need to process the full topic and have an explicit check for which deserialized records you want to parse. For example, this is what a stream processing application with a simple filter operation would provide you.
If you want to preserve the ability to do lookups by a particular item, you will either need to make a partitioner that segments all data you're interested in, or create a topic per item (which really only works for certain use cases, not things like individual user accounts).
You could look at inserting all events to an in-memory database, then performing queries against that
Kafka supports the concept of Interceptors, which sit between Kafka and a process that consumes or produces records, such that the records that are read from or written to Kafka can be mutated or custom logging can be performed.
From what I can see, the ConsumerInterceptor allows for records to be filtered, in that it returns a ConsumerRecords object and an implementation could remove (i.e. censor) items from the container class before passing the records to a consumer.
A ProducerInterceptor only takes and returns a ProducerRecord, rather than something like an Optional<ProducerRecord>. What happens if the record that is returned by this method is null? The use case is wanting to prevent the writing of records to Kafka - is this supported by simply dropping the records being written by returning null, or would one have to mutate the input object and zero its fields?
You may get a SerializationException since the interception will happen before serialization given you'd need to do this onSend. If you handle the null case it might work out for you but you'll need to decide if this is the right thing to do. If you want to send essentially a null record to Kafka in the event of filtering occurring then this makes some sense. If you just want to drop the record, better to just put the filtering logic in the producer itself instead of intercepting.
At first glance it seems to me that with a KStream#reduceByKey one can achieve the same functionality as with a KStream to KTable leftJoin. I.e combining records with the same key. What i the difference between the two, also in terms of performance?
Short answer: (What is the difference between the two?)
reduceByKey is applied to a single input stream while leftJoin combines two streams/tables.
Long answer:
If I understand your question correctly, it seems that your incoming KTable changelog stream would be empty, and you want to compute a new join result (ie, update result KTable) for each incoming KStream record? The result KTable of a join is not available as materialized view, but only the changelog topic will be sent downstream. Thus, your input KTable would always be empty and your input KStream record, would always join with "nothing" (because of left join), which would not be really be update the result KTable. You could also do a KStream#map() -- there is no state you can exploit if your input KTable does not provide a state.
In contrast, if you use reduceByKey, the result KTable is available as materialized view, and thus for each KStream input record, the previous result value is available to get updated.
Thus, both operations are fundamentally different. If you have a single input KStream using a join (that required two inputs) would be quite odd, as there is no KTable...
KStream represents a record stream in which each record is self contained. For example, if we are to summarize word occurrences, it would hold the count during a certain frame (e.g. time window or paragraph).
KTable represents a sort of a state and, each record coming in, would normally hold the total occurrences count.
Therefore, the use case to which each method is used is quite different. While KStream#reduceByKey would reduce all records in the same key and summarize the counts for each key, KTable#leftJoin would normally be used in cases when the total count needs to be adjusted according to another information coming in, or combining more data to the record.
The example given in Kafka Stream's documentation is for log compaction. While with KStream, no record could be discarded, in KTable, records that are no longer relevant would be removed.
I'm new to Storm and I'm having problems to figure out how to process records in order.
I have a dataset which contains records with the following fields:
user_id, location_id, time_of_checking
Now, I would like to identify users which have fulfilled the path I specified (for example, users that went from location A to location B to location C).
I'm using Kafka producer and reading this records from a file to simulate live data. Data is sorted by date.
So, to check if my pattern is fulfilled I need to process records in order. The thing is, due to parallelization (bolt replication) I don't get check-ins of user in order. Because of that patterns won't work.
How to overcome this problem? How to process records in order?
There is no general system support for ordered processing in Storm. Either you use a different system that supports ordered steam processing like Apache Flink (Disclaimer, I am a committer at Flink) or you need to take care of it in your bolt code by yourself.
The only support Storm delivers is using Trident. You can put tuples of a certain time period (for example one minute) into a single batch. Thus, you can process all tuples within a minute at once. However, this only works if your use case allows for it because you cannot related tuples from different batches to each other. In your case, this would only be the case, if you know that there are points in time, in which all users have reached their destination (and no other use started a new interaction); ie, you need points in time in which no overlap of any two users occurs. (It seems to me, that your use-case cannot fulfill this requirement).
For non-system, ie, customized user-code based solution, there would be two approaches:
You could for example buffer up tuples and sort on time stamp within a bolt before processing. To make this work properly, you need to inject punctuations/watermarks that ensure that no tuple with larger timestamp than the punctuation comes after a punctuation. If you received a punctuation from each parallel input substream you can safely trigger sorting and processing.
Another way would be to buffer tuples per incoming substream in district buffers (within a substream order is preserved) and merge the tuples from the buffers in order. This has the advantage that sorting is avoided. However, you need to ensure that each operator emits tuples ordered. Furthermore, to avoid blocking (ie, if no input is available for a substream) punctuations might be needed, too. (I implemented this approach. Feel free to use the code or adapt it to your needs: https://github.com/mjsax/aeolus/blob/master/queries/utils/src/main/java/de/hub/cs/dbis/aeolus/utils/TimestampMerger.java)
Storm supports this use case. For this you just have to ensure that order is maintained throughout your flow in all the involved components. So as first step, in Kafka producer, all the messages for a particular user id should go to the same partition in Kafka. For this you can implement a custom Partitioner in your KafkaProducer. Please refer to the link here for implementation details.
Since a partition in Kafka can be read by one and only one kafkaSpout instance in Storm, the messages in that partition come in order in the spout instance. Thereby ensuring that all the messages of the same user id arrive to the same spout.
Now comes the tricky part - to maintain order in bolt, you want to ensure that you use field grouping on bolt based on "user_id" field emitted from the Kafka spout. A provided kafkaSpout does not break the message to emit field, you would have to override the kafkaSpout to read the message and emit a "user_id" field from the spout. One way of doing so is to have an intermediate bolt which reads the message from the Kafkaspout and emits a stream with "user_id" field.
When finally you specify a bolt with field grouping on "user_id", all messages of a particular user_id value would go to the same instance of the bolt, whatever be the degree of parallelism of the bolt.
A sample topology which work for your case could be as follow -
builder.setSpout("KafkaSpout", Kafkaspout);
builder.setBolt("FieldsEmitterBolt", FieldsEmitterBolt).shuffleGrouping("KafkaSpout");
builder.setBolt("CalculatorBolt", CalculatorBolt).fieldsGrouping("FieldsEmitterBolt", new Fields("user_id")); //user_id field emitted by Bolt2
--Beware, there could be case when all the user_id values come to the same CalculatorBolt instance if you have limited number of user_ids. This in turn would decrease the effective 'parallelism'!