I'm trying out some of the operations that I could do on the Observable from Monix. I came across this debounce operator and could not understand its behavior:
Observable.interval(5.seconds).debounce(2.seconds)
This one above just emits a Long every 5 seconds.
Observable.interval(2.seconds).debounce(5.seconds)
This one however does not emit anything at all. So what is the real purpose of the debounce operator and in which cases could I use it?
The term debounce comes from mechanical relays. You can think of it as a frequency filter: o.debounce(5.seconds) filters out any events that are emitted more frequently than once every 5 seconds.
An example of where I've used it is where I expect to get a batch of similar events in rapid succession, and my response to each event is the same. By debouncing I can reduce the amount of work I need to do by making the batch look like just one event.
It isn't useful in situations like your examples where the input frequency is constant, as the only possibilities are that it does nothing or it filters out everything.
Related
Using Fixed Windows in Apache Beam. The watermark is set by the event time.
Some data may arrive out of order and cause the window to close.
How can a trigger be defined in Java to occur say 2 minutes after the last data was seen?
It's not entire clear what behavior you expect. One question is what do you expect to happen if the data arrives within the two minutes? Do you want to restart the two minutes interval, don't restart it, re-emit the data or not?
Looks like the trigger you are trying to describe is something along these lines:
wait until the watermark passed the end of window, in event time;
wait for additional 2 minutes in processing time;
emit the data;
If in step 2 it was event time, i.e. you wanted to re-emit the window if a late element arrives that fits within window + 2min, then you could use withAllowedLateness(). Though it sounds different from what you want, because it can keep re-emitting the window contents every time a matching late element arrives.
With processing-time in step 2 this is not possible in general with basic triggers that are available in Beam. You can probably achieve a behavior you want if you manually manage state and timers in your own ParDo, e.g. you can watch for the incoming elements, keep track on them in the state, and then on timer emit what you want. This can become very complicated and might still be not flexible enough for your specific use case.
One of the major problems is that there is no good way to define processing time triggers in Beam in general. It would be complicated to define a general mechanism of working with timers in this manner. For example, when you want to express "wait for 2 minutes", the framework needs to understand in relation to what these two minutes are, when to start the timer, so you need a mechanism to express that as well. And with composition, continuation and other complications this doesn't seem easy to reason about. So it's not in the framework in this general form.
In order to implement only the "wait for 2 minutes after the last element was seen in the window", the framework has to watch for it and set the timer. Technically it is possible to do something like this but doesn't seem like anyone has done it yet.
There seems to be only one meaningful processing time trigger available in Beam but it's not generic enough and doesn't do what you want. You can look at composite triggers like AfterFirst or AfterAll but they likely won't help you without a better general processing time trigger.
I decided against using Beam and implemented the solution in Kafka Streams.
I basically grouped by, then used fixed windows and the aggregated the result.
The "grace" on the window allows data to arrive late.
KGroupedStream<Long, OxyStreamItem> grouped = input.groupByKey();
TimeWindowedKStream<Long, OxyStreamItem> windowed =
grouped.windowedBy(
TimeWindows.of(WIN_SIZE)
.advanceBy(WIN_SIZE)
.grace(Duration.ofSeconds(5L)));
return windowed
.aggregate(
makeInitializer(),
makeAggregator(),
Materialized
.<Long, Aggregate, WindowStore<Bytes, byte[]>>as("tmp")
.withValueSerde(new AggregateSerde()))
.suppress(
Suppressed.untilWindowCloses(Suppressed.BufferConfig.unbounded()))
.toStream()
.map(calculateAvg());
I have a webflux that gets data from a stream of events, but I want it to end up in a more or less stable state (i.e. the events stop flowing for a few milliseconds) sort of like the debounce operator in RxJS.
I can't seem to find anything like that in the API though.
The webflux equivalent is the bufferTimeout method
.bufferTimeout(20, Duration.ofSeconds(1))
The advantage of it is you get all the events not just the last one which is what debounce does as a list.
This question already has answers here:
How to send final kafka-streams aggregation result of a time windowed KTable?
(3 answers)
Closed 4 years ago.
We're having an issue where upon doing a groupby --> reduce --> toStream, partial reduce values are being sent downstream when a commit happens during the reduce. So if there are 65 keys to be reduced, and say a commit happens half we through, the output will be two messages: one partially reduced, the other with all the values reduced.
So here is our case in more detail:
msg --> leftJoin
leftJoin --> flatMap //break msg into parts so we can join again downstream
flatMap --> leftJoin
leftJoin --> groupByKey
groupByKey --> reduce
reduce --> toStream
toStream --> to
Currently, we've come up with a very ugly fix for this, which has to do with adding an index and out of values to each message created during the flatMap phase...we filter out any message emitted by the reduce where index != out of. My feeling is we're not doing something right here or looking at it the wrong way. Please advise on the correct way of doing this.
Thanks.
So if there are 65 keys to be reduced, and say a commit happens half we through, the output will be two messages: one partially reduced, the other with all the values reduced.
If I understand your description correctly, this is actually intended behavior. For one, it's a tradeoff between processing latency (where you want to see update records as soon as you have a new piece of input data) vs. coalescing multiple update records into fewer or even just a single update record.
The default behavior of Kafka Streams is to favor lower processing latency. That is, it will not wait for "all input data to have arrived" before sending downstream updates. Rather, it will send updates once new data has arrived. Some background information is described at https://www.confluent.io/blog/watermarks-tables-event-time-dataflow-model/.
Today, you have two main knobs to change/tune this default behavior, which is controlled by (1) Kafka Streams record caches (for the DSL) and (2) the configured commit interval (you already mentioned this).
Moving forward, the Kafka community has also been working on a new feature that will allow you to define that you just want a single, final update record to be sent (rather than what you described as "partial" updates). This new feature, in case you are interested, is described in the Kafka Improvement Proposal KIP-328: Ability to suppress updates for KTables. This is actively being worked on, but it will unlikely to be finished in time for the upcoming Kafka v2.1 release in October.
Currently, we've come up with a very ugly fix for this, which has to do with adding an index and out of values to each message created during the flatMap phase...we filter out any message emitted by the reduce where index != out of. My feeling is we're not doing something right here or looking at it the wrong way. Please advise on the correct way of doing this.
In short, in stream processing you should embrace the nature of how streaming works. In general, you will only have partial/incomplete knowledge of the world, so to speak, or rather: you only know what you observed thus far. So, at any given point in time, you must deal with the situation that more, additional data may arrive that you still have to deal with.
A typical situation is having to deal with late-arriving data, where your application logic must decide whether you want to still integrate and process this data (quite likely) or discard (sometimes the way it needs to be).
Going back to your example:
So if there are 65 keys to be reduced [...]
How would one know it's 65, and not 100 or 28, and so on? One can only tell that: "Thus far, at this point in time, I have received 65. So, what do I do? Do I reduce those 65 because I believe that's all the input? Or do I wait some seconds/minutes/hours longer because there might be 35 more to arrive, but this will mean that I will not send an update/answer downstream until this waiting time has elapsed (which results in higher processing latency)?"
In your situation, I would ask: Why do you consider the streaming behavior of how/when updates are being sent a problem? Perhaps it's because you have a downstream system or application that doesn't know how to handle such streaming updates?
Does that make any sense? Again, the above is based on my understanding of what you described as being the issue.
I am working on a project where there is a continuous stream of events. There are many concurrent users who can create/update/delete rules and get alerts when the stream matches those rules.
I used Esper for this where I am generating EPL statements from rules dynamically and registering them. This is very error prone as the rules at times are complex and cannot be dynamically converted to EPL. This is also not scalable.
What would be the correct approach to handle this? Is there a standard way/ library for this kind of use cases.
Thank you
Well complex event processing engines are usually used to address complex requirements. As the name says, we are looking for complex relationships between events and time, so its complex event processing.
Simple event processing would be for simple requirements. Simple requirements can be tested with just two scenarios. For example, assume we want to look for an event with a temperature over 100. The design for this in Esper EPL is “select * from Event(temperature > 100). This can be tested by two or maybe three scenarios: Event={temp=99}, Event={temp=100} and Event={temp=101}, for example.
There are some use cases for medium complexity event processing. Those usually require between 4 and 9 scenarios for testing. For example, assume we want to look for more than 100 events arriving within one sliding minute (sliding, not hopping). The design for this in Esper EPL is “select count() from Event#time(60) having count() > 100”. This now needs a bunch of scenarios for testing this rule, such as no events, 100 events, 101 events, 101 events over 61 seconds etc. etc.. There is time involved here which makes these scenarios more difficult to describe and test.
I would say that complex event processing is for cases of medium to high complexity of event processing. This is the sort of stuff where you need to be clear about the requirements first. You need to nail down your different scenarios of events arriving and time passing before going to design Esper EPL.
Requirements drive the tool. Esper does handle the more complex requirements well but there is a learning curve. In my experience scaling can be done but needs planning. When talking about low latency analysis and scaling in one sentence there is no silver bullet simple approach. There is high availability for Esper but that is not for free but that scales well in a Kafka stack.
I watched the video and I know the general principles - hot happens even when nobody is subscribed, cold happens "on demand".
Also, Publish() converts cold to hot and Defer() converts hot to cold.
But still, I feel I am missing the details. Here are some questions I'd like to have answered:
Can you give a comprehensive definition for these terms?
Does it ever make sense to call Publish on a hot observable or Defer on a cold?
What are the aspects of Hot/Cold conversions - do you lose messages, for example?
Are there differences between hot and cold definitions for IObservable and IEnumerable?
What are the general principles you should take into account when programming for cold or hot?
Any other tips on hot/cold observables?
From: Anton Moiseev's Book “Angular Development with Typescript, Second Edition.” :
Hot and cold observables
There are two types of observables: hot and cold. The main
difference is that a cold observable creates a data
producer for each subscriber, whereas a hot observable
creates a data producer first, and each subscriber gets the
data from one producer, starting from the moment of subscription.
Let’s compare watching a movie on Netflix to going into a
movie theater. Think of yourself as an observer. Anyone who decides to watch Mission: Impossible on Netflix will get the entire
movie, regardless of when they hit the play button. Netflix creates a
new producer to stream a movie just for you. This is a cold
observable.
If you go to a movie theater and the showtime is 4 p.m., the producer
is created at 4 p.m., and the streaming begins. If some people
(subscribers) are late to the show, they miss the beginning of the
movie and can only watch it starting from the moment of arrival. This
is a hot observable.
A cold observable starts producing data when some code invokes a
subscribe() function on it. For example, your app may declare an observable providing a URL on the server to get certain products. The
request will be made only when you subscribe to it. If another script
makes the same request to the server, it’ll get the same set of data.
A hot observable produces data even if no subscribers are
interested in the data. For example, an accelerometer in your
smartphone produces data about the position of your device, even if no
app subscribes to this data. A server can produce the latest stock
prices even if no user is interested in this stock.
Hot observables are ones that are pushing event when you are not subscribed to the observable. Like mouse moves, or Timer ticks or anything like that. Cold observables are ones that start pushing only when you subscribe, and they start over if you subscribe again.
I hope this helps.
Can you give a comprehensive
definition for these terms?
See my blog post at: https://leecampbell.com/2010/08/19/rx-part-7-hot-and-cold-observables
Does it ever make sense to call
Publish on a hot observable or Defer
on a cold?
No, not that I can think of.
What are the aspects of Hot/Cold
conversions - do you lose messages,
for example?
It is possible to "lose" messages when the Observable is Hot, as "events" happen regardless of subscribers.
Are there differences between hot and
cold definitions for IObservable and
IEnumerable?
I dont really understand the question. I hope this analogy helps though. I would compare a Hot Observable to an Eagerly evaluated IEnumerable. ie a List or an Array are both Eagerly evaluated and have been populated even if no-one enuemerates over them. A yield statement that gets values from a file or a database could be lazily evaluated with the Yield keyword. While lazy can be good, it will by default, be reevaluated if a second enumerator runs over it. Comparing these to Observables, a Hot Observable might be an Event (Button click) or a feed of temperatures; these events will happen regardless of a subscription and would also be shared if multiple subscriptions were made to the same observale. Observable.Interval is a good example of a Cold observable. It will only start producing values when a subscription is made. If multiple subscriptions as made then the sequence will be re-evaluated and the "events" will occur at seperate times (depending on the time between subscriptions).
What are the general principles you should take into account when programming for cold or hot?
Refer to the link in point one. I would also recommend you look into Publsh being used in conjunction with RefCount. This allows you to have the ability to have Lazy evaluation semantics of Cold Observables but the sharing of events that Hot Observables get.
Any other tips on hot/cold
observables?
Get your hands dirty and have a play with them. Once you have read about them for more than 30minutes, then time spent coding with them is far more productive to you than reading any more :)
Not pretending to give a comprehensive answer, I'd like to summarize in a simplest form what I have learned since the time of this question.
Hot observable is an exact match for event. In events, values usually are fed into the handler even if no subscribers are listening. All subscribers are receiving the same set of values. Because of following the "event" pattern, hot observables are easier to understand than the cold ones.
Cold observable is also like an an event, but with a twist - Cold observable's event is not a property on a shared instance, it is a property on an object that is produced from a factory each time when somebody subscribes. In addition, subscription starts the production of the values. Because of the above, multiple subscribers are isolated and each receives its own set of values.
The most common mistake RX beginners make is creating a cold observable (well, thinking they are creating a cold observable) using some state variables within a function (f.e. accumulated total) and not wrapping it into a .Defer() statement. As a result, multiple subscribers share these variables and cause side effects between them.