Apache Samza aggregation rules for missing expected events in rolling time-period - real-time

Question

My use-case is to identify entities from which expected events have not been received after X amount of time in real-time.
For example:
If we have received PaymentInitiated event at time T but didn't receive either of PaymentFailed / PaymentAborted / PaymentSucedded by T+X, then raise a trigger saying PaymentStuck along with details of PaymentIntitiated event.
How can I model such use-cases in Apache Samza as it is rolling time period X on each event, rather than fixed time interval.
Thanks, Harish

Answer 1

I'm not aware of any native support for this in Samza, but I can imagine a work-around that uses WindowableTask.
public class PaymentEvent implements Comparable<PaymentEvent> {
// if current time > timestamp, payment is stuck
public long timestamp;
// we want a corresponding PaymentFailed... event with the same id
public long interactionId;
// PaymentRequest, PaymentAborted, PaymentSucceeded...
public enum type;
...
#Override
public int compareTo(PaymentEvent o){
return timestamp - o.timestamp;
}
}
Now in your process method you would have something like:
PriorityQueue<PaymentEvent> pqueue;
Map<Long, PaymentEvent> responses;
public void process(...) {
PaymentEvent e = new PaymentEvent(envelope.getMessage());
if (e.enum == PAYMENT_REQUEST) {
pqueue.add(e);
} else {
responses.put(e.interactionId, e);
}
}
And finally during your window you would pop off from the priority queue everything with timestamp > current time and check if there is a corresponding event in the Map.
public void window(...) {
while(pqueue.peek().timestamp <= currentTime) {
if (!map.containsKey(pqueue.poll().interactionId) {
// send the trigger via the collector
}
}
}
Then lastly you would set the window time in your configuration to however long you wanted to poll. The config is task.window.ms.