I am using Drools fusion for processing real time events. Each event has a timestamp field.
Issue is events can be out of order sometimes. Can drools fusion handle this situtation and if yes how ?
Thanks
If A is stamped 0:00:00 and B is stamped 0:01:00 and B arrives and you have the rule
rule "A before B"
when
$b: B()
not A( this before $b )
then ... end
it will fire.
You can use fact insertion time as a timestamp.
Keep the original timestamp as a property. Maybe you'll want to look at it if a "situation" occurs. It depends.
Addition If you have a maximum delay dt you can put all arriving events into "quarantine" for this delay. Before you insert A, check all other streams (sources) for an event B that precedes A and react accordingly. Everything will react at least delayed by dt.
Related
In accordance, I have a question in the modeling process. I want my agent to have an event that is triggered by time and condition, example: goToSchool if it is more than 6 am and there is a school bus. I am confused about whether to use the timeout trigger (but cannot use the condition) or the condition (but cannot use the timeout) or is there any possible alternative?
In your example, "if it is more than 6 am" is a condition and not a timeout. A timeout trigger is used when you want an event to happen at an exact time. In your case, while "more than 6 am" is time related, it is still a condition. So I would use a condition triggered event with two conditions:
getHourOfDay() > 6 && <bus condition>
getHourOfDay() function returns the hour of the day in a 24-hr format.
You need to keep in mind something important related to condition triggered events, they are only evaluated "on change". I recommend you read this carefully:
https://help.anylogic.com/index.jsp?topic=%2Fcom.anylogic.help%2Fhtml%2Fstatecharts%2Fcondition-event.html
My recommendation would be to use the onChange() function in the block controlling your bus arrival so that the condition is evaluated each time a bus arrives.
I want to create an EventTime based session-window in Flink, such that it triggers when the event time of a new message is more than 180 seconds greater than the event time of the message, that created the window.
For example:
t1(0 seconds) : msg1 <-- This is the first message which causes the session-windows to be created
t2(13 seconds) : msg2
t3(39 seconds) : msg3
.
.
.
.
t7(190 seconds) : msg7 <-- The event time (t7) is more than 180 seconds than t1 (t7 - t1 = 190), so the window should be triggered and processed now.
t8(193 seconds) : msg8 <-- This message, and all subsequent messages have to be ignored as this window was processed at t7
I want to create a trigger such that the above behavior is achieved through appropriate watermark or onEventTime trigger. Can anyone please provide some examples to achieve this?
The best way to approach this might be with a ProcessFunction, rather than with custom windowing. If, as shown in your example, the events will be processed in timestamp order, then this will be pretty straightforward. If, on the other hand, you have to handle out-of-order events (which is common when working with event time data), it will be somewhat more complex. (Imagine that msg6 with for time 187 arrives after t8. If that's possible, and if that will affect the results you want to produce, then this has to be handled.)
If the events are in order, then the logic would look roughly like this:
Use an AscendingTimestampExtractor as the basis for watermarking.
Use Flink state (perhaps ListState) to store the window contents. When an event arrives, add it to the window and check to see if it has been more than 180 seconds since the first event. If so, process the window contents and clear the list.
If your events can be out-of-order, then use a BoundedOutOfOrdernessTimestampExtractor, and don't process the window's contents until currentWatermark indicates that event time has passed 180 seconds past the window's start time (you can use an event time timer for this). Don't completely clear the list when triggering a window, but just remove the elements that belong to the window that is closing.
How would a q implementation of Windows' waitfor look like?
I have a q process that feeds rows of a table, one at a time, to another process (say tickerplant).
feed:{h`.u.upd,x;};
feed each tbl;
I'd like to feed the next row if any one of the conditions is met:
the process receives the signal (next or ready) from the other process to do so;
OR
time waiting for signal runs out (timeout is specific to each row of tbl and will be provided as a column of tbl)
Think of tbl as containing the list of events that are to be published sequentially either when the rest of CEP/tickerplant is ready OR when the next event is due (timeout measured as deltas timespan between the last published event and the next one, i.e. update timeout:((1 _deltas time),0Wp) from tbl), whichever happens sooner.
I've tried while[.z.N<prevtstamp+timeout;wait()];feed x but while blocks the process from listening to async messages from the other process AFAIK.
Other solutions considered:
Checking for signals with .z.ts is too slow as \t can't go below 1ms precision.
Polling the tickerplant for next(ready) signal continuously from within the while loop would slow down the tickerplant.
One solution is to maintain i index of the current row of tbl, separate the feeder out into two processes each handling one condition separately and polling for the current i index. This sounds slow, compared to eaching rows.
KDB is a single thread application so any waitFor alike things will do nothing more than while loop.
1ms precision is pretty high already...And KDB is NOT design for a real time stuff. So even 1ms can be easily missed if the process is busying on something else.
If u need higher precision than that.. U might need C or even Driver level help to improve precision.
Maybe U can consider a better design to avoid the real time requirement :)
I want to test drools 6.3 with a scenario, But I have a problem in a special situation.
Here is my scenario in a simple form:
I have two systems, A and B, in a simulated Network that generate events. I want to write two rules to find out patterns in these events. Two rules for testing this scenario is:
declare A
#timestamp(timestampA)
end
declare B
#timestamp(timestampB)
end
Rule “1”
When
accumulate( A() over window:time( 10s ) ; s:count(1) ; s>1)
Then
System.out.println( " Rule 1 matched " );
Rule “2”
When
B()
Then
System.out.println( " Rule 2 matched " );
Timestamp of each event is the timestamp from log generated in each system on when received by drools and inserted in working memory.
I’m using STREAM mode with pseudo clock, because events from System B receives with 25min delay due to network congestion and I should adjust session clock manually. Session clock set with the timestamp of every event inserted into the session. And All rules fire when every event inserted.
When order of receiving and inserting events are like below matched correctly.
Event A received at 10:31:21 – Session clock : 10:31:21 – insert A and fire
Event A received at 10:31:23 - Session clock : 10:31:23 – insert A and fire
Rule 1 matched
Event B received at 10:06:41 - Session clock : 10:06:41 – insert B and fire
Rule 2 matched
But when order of receiving and inserting events are like below matched incorrectly:
Event A received at 10:31:21 – Session clock : 10:31:21 – insert A and fire
Event B received at 10:06:41 - Session clock : 10:06:41 – insert B and fire
Rule 2 matched
Event A received at 10:31:23 - Session clock : 10:31:23 – insert A and fire
When second A event inserted tow A events in last 10s are in working memory but rule 1 does not match. Why?
What you are doing is somewhat in conflict with the assumptions underlying the CEP (Continuous Event Processing) of Drools. STREAM mode implies that events should be inserted in the order of their timestamps, irrespective of their origin. Setting the pseudo clock back and forth in big jumps is another good way to confuse the Engine.
Don't use STREAM mode, window:time and forget about session clocks.
You have facts containing time stamps, and you can easily write your rules by referring to these time stamps, either using plain old arithmetic or by applying the temporal operators (which are nothing but syntactic sugar for testing the relation of long (as in java.lang) values.
My use-case is to identify entities from which expected events have not been received after X amount of time in real-time rather than using batch jobs. For Example:
If we have received PaymentInitiated event at time T but didn't receive either of PaymentFailed / PaymentAborted / PaymentSucedded by T+X, then raise a trigger saying PaymentStuck along with details of PaymentIntitiated event.
How can I model such use-cases in Apache Storm as it is rolling time period X on each event, rather than fixed time interval.
Thanks,
Harish
For Storm, would need to put all your logic into your UDF code using low level Java API (I doubt that Trindent is helpful). I never worked with Samza and cannot provide any help for it (or judge which system would be the better fit for your problem).
In Storm for example, you could assign a timestamp to each tuple in Spout.nextTuple(), and buffer all tuples of an incomplete payment within a Bolt in descending order of the timestamp. Each time Bolt.execute() is called, you can compare the timestamp of the new tuple with the head (ie, oldest tuple) of your queue. If the input tuple has a larger timestamep than head-T plus X, you know that your head tuple times out and you can raise your trigger for it.
Of course, you need to do fieldsGrouping() to ensure that all tuples belonging to the same payment are processed by the same Bolt instance. You might also need to somewhat order the incoming bolt tuples by timestamp or use more advance time-out logic to deal with out-of-order tuples (with regard to increasing timestamps).
Depending on you latency requirement and input stream rate you might also use "tick tuples" to trigger the comparison of the head tuple with this dummy tick tuple. Or as an ever stricter implementation, do all this logic directly in Spout.next() (if you know that all tuples of a payment go through the same Spout instance).