Use External Window Time Stamp to Debug Siddhi Stream Query - complex-event-processing

I am planning to use the historical event traces (stored in JSON with my own event time stamp recorded for each event) to debug the Siddhi stream queries that I have just created. My stream starts with:
from MyInputEventStream#window.externalTime(my_own_timestamp, 10 min)
select some_fields
insert into MyOutpuStream;
and I will input my events from traces, one by one.
Supposed event 1 arrives at the specified my_own_timestamp = 1528905600000, which is 9 am PST time, June 13. and event 2 arrives at 11 minutes later, my_own_timestamp = 1528906260000. I believe that I will get the output at MyOutpuStream at 9:10 am, as time_stamp(e2) - time_stamp(e1) > 10 min, and e2 will trigger the system after the windows passes.
Now supposed event 1 arrives at my_own_timestamp = 1528905600000, that is, 9:00 am. But no events will arrive in the next 2 hours. Do I still get the output at 9:10 am, as in reality, the window time should expire at 9:10 am, independent of when the next event should arrive? But it seems that in this case, the internal timing system of Siddhi will have to incorporate my event input's time stamp, and then set the expiration time of the events based on the clock system of the process on which the Siddhi is running. Is this correct? could you help clarify it.

You won't get an output at 9:10 am. Because if you use externalTime, the event expiration logic will entirely base on the timestamp that you defined. And it will wait for a timestamp that satisfies the time difference which is greater than or equal to expire the previous event.
What internally happens is;
def array previousEvents;
foreach currentEvent in currentEvents (events that are coming in):
def currentTime = currentEvent.timestamp;
foreach previousEvent in previousEvents:
def previousTime = previousEvent.timestamp;
def timeDiff = previousTime - currentTime + windowLength;
if (timeDiff <= 0) {
remove previousEvent from previousEvents;
set expired timestamp of previousEvent to currentTime;
expire previousEvent;
}
previousEvents.add(currentEvent);

Related

Add milliseconds to current time

So I currently have a now playing swift app, that requests to the server every 30 seconds, and seems to be overloading the server.
Since we request data from iTunes API about the current song that is playing, I was thinking that I could possibly limit the request if a result was found by iTunes API.
Using trackTimeMillis I could add that trackTimeMillis to the current time and if not greater than the time don't ping the server. If greater than ping server for latest track.
I am thinking I will need to wrap the function that pings the server in a
Calendar.current.dateComponents([.hour], from: Date(), to: Date()), diff > Date())+ trackTimeMillis {
But that does not work
basically I need to add trackTimeMillis to the current Date() and if that users current timestamp (date) is larger than the Date() + trackTimeMillis then allow request.
I don't really understand what you are trying to do with the code snippet you put in your question.
Focusing on how you add some number of milliseconds to a Date():
Date has the function addingTimeInterval(_:)
If you need to add a certain number of milliseconds do a date, use this:
newDate = Date().addingTimeInterval( Double(millis) / 1000)
Edit:
As Matt pointed out, you can do that even more simply by writing
newDate = Date() + Double(millis) / 1000
Since there is an override of the + operator that takes a Date and a Double and returns a Date.
Both versions of this add a Double containing a specified number of seconds to the Date. Since Doubles can represent very small and very large numbers you can add or subtract values ranging from tiny fractions of a second to thousands of years to any Date.
A simpler alternative. When I am doing a task like this, I usually store the Date of the previous request, e.g.:
let lastRequestTime = Date()
and then simply check the elapsed time:
if Date().timeIntervalSince(lastRequestTime) > 30 {
...
}
Another option is to use Timer. Basically, you can schedule a Timer that will send the request after X seconds. If you receive a response from a different source first, you simply cancel the Timer and possibly the request if it is already running.

Esper - handle out of order events

I want to monitor if an event does NOT arrive within 10 minutes of arrival of event with the same id.
This is the EPL I am currently using:
SELECT * FROM pattern[ every s=Order_Status(status="placed") -> (timer:interval(600 sec) and not e=Order_Status(status="delivered", id=s.id))]
Usually placed event arrives before delivered, but sometime because of some lag in our systems, delivered event happens to come before placed for some id.
Cases
time: 8:00 event: Order_Status{id=167, status="placed"}
time: 8:07 event: Order_Status{id=167, status="delivered"}
< No alert > (delivered within 10 minutes)
time: 8:00 event: Order_Status{id=189, status="placed"}
time: 8:17 event: Order_Status{id=189, status="delivered"}
< Alert> (delivered after 10 minutes)
time: 8:00 event: Order_Status{id=2637, status="delivered"}
time: 8:08 event: Order_Status{id=2637, status="placed"}
< Alert > (but shouldn't alert, the problem is delivered event for this id has arrived before placed)
As stated, I would get a false alert as the EPL pattern starts the window after the placed event and waits for delivered event which has already arrived.
How do I handle this scenario of out of order events ?
Note:
(Basically I want to check for every id if the time difference between placed and delivered is above a certain threshold.
I also have the timestamp fields inside each event)
You have a "not" in your pattern that is used to detect the absence of events. Your requirement doesn't search for absence so the "not" isn't right.
There is a requirement questions as well. You don't state what happens when there are many A events and just one B event. Are there many matches or just the match for the last A event or the first A event or something else?
Sample pattern:
pattern [A -> (timer:interval(10 minutes) and B)]
Or this is a join that would seem to match what you want:
select * from B unidirectional, A#window(10 minutes)#lastevent
The "A#window(10 minutes)#lastevent" keeps the last event for up to 10 minutes.

Count and Time window in Esper EPL

I have the following use case, which I'm trying to write in EPL, without success. I'm generating analytics events of different types, generated in different intervals (1min, 5min, 10min, ...). In special kind of analytics, I need to collect 4 specific
Analytics events (from which I will count another analytic event) of different types, returned every interval (1min, 5min, 10min, ...). The condition there is, that on every whole interval, e.g., every whole minute 00:01:00, 00:02:00 I want to have returned either 4 events or nothing if the events don't arrive in some slack period after (e.g., 2s).
case 1: events A,B,C,D arrive at times 00:01:00.500, 00:01:00.600, 00:01:00.700, 00:01:00.800 - right after fourth event arrives to esper, the aggregated event with all 4 events is returned
case 2: slack period is 2seconds, events A,B,C,D arrives at 00:01:00.500, 00:01:00.600, 00:01:00.700, 00:01:02.200 - nothing is arrived, as the last event is out of the slack period
You could create a trigger event every minute like this:
insert into TriggerEvent select * from pattern[timer:schedule(date:'1970-01-01T00:00:00.0Z', period: 1 minute, repetitions: -1)]
The trigger that arrives every minute can kick off a pattern or context. A pattern would seem to be good enough. Here is something like that:
select * from pattern [every TriggerEvent -> (a=A -> b=B -> c=C -> d=D) where timer:within(2 seconds)]

DHTMLX Scheduler Start_hour

I have two questions related to DHTMLX
QUESTION 1: how to start_hour with minutes.
Start_hour_image
QUESTION 2: how to start_hour with PM (Post Meridiem)
Start_hour_with_pm
Minutes are not supported in first_hour config, in Day/Week/Unit views you can set only hour as a minimum value for the scale.
You can block time from 05:00 to 05:30 to prevent events creation.
Start minutes on the scale can be configured only in Timeline view.
To start from 5pm, set
scheduler.config.first_hour = 17;
It doesn't mean that current date 17:00-24:00 and the next day till 10:00 (if you set, for example, "last_hour = 10") will be rendered on the scale. You need to set values for these configs within one day. I.e. 0:00 <= first_hour < last_hour <= 24:00.

Is it possible to join 2 Kafka KStreams where the JoinWindows duration is stored in the object of 1 of the streams?

Let say I have 2 streams:
TimeWindow (with begin time, end time)
Numbers (with time stamp)
Is it possible to user either DSL API or Process API to join the streams such that the output will contain TimeWindow object that contains the sum of the numbers that is within the time range specified in TimeWindow?
To be specific, how do you set XXX where it is the duration store in win.getDuration() where win is the one referenced in ValueJoiner.
timeWindow.join(
numbers,
(ValueJoiner<TimeWindow, Number, TimeWindow>) (win, num) -> win.addToTotal(num),
new JoinWindows(XXX, 0)
).to("output_Topic");
The JoinWindows after is 0 because TimeWindow's timestamp is endtime. XXX duration should be calculate as TimeWindows end time - begin time in milli seconds.
Many thanks for any help!
Thanks to Matthias' incite, I end up roll back to use Processor API with the implementation of TimestampExtractors and usage of in memory state store (default to use RockDB) to implemented this function.