Test Event expiration in Drools Fusion CEP - drools

Ciao, I have tested in several ways, but I'm still unable to test and verify the Event expiration mechanism in Drools Fusion, so I'm looking for some little guidance, please?
I've read the manual and I'm interested in this feature:
In other words, one an event is inserted into the working memory, it is possible for the engine to find out when an event can no longer match other facts and automatically retract it, releasing its associated resources.
I'm using the Drools IDE in Eclipse, 5.4.0.Final and I modified the template code created by the "New Drools Project" wizard to test and verify for Event expiration.
The code below. The way I understood to make the "lifecycle" to work correctly is that:
You must setup the KBase in STREAM mode - check
You must Insert the Events in temporal order - check
You must define temporal constraints between Events - check in my case is last Message()
However, when I inspect the EventFactHandle at the end, none of the Event() has expired.
Thanks for your help.
Java:
public class DroolsTest {
public static final void main(String[] args) {
try {
KnowledgeBase kbase = readKnowledgeBase();
// I do want the pseudo clock
KnowledgeSessionConfiguration conf = KnowledgeBaseFactory.newKnowledgeSessionConfiguration();
conf.setOption(ClockTypeOption.get("pseudo"));
StatefulKnowledgeSession ksession = kbase.newStatefulKnowledgeSession(conf, null);
SessionPseudoClock clock = ksession.getSessionClock();
KnowledgeRuntimeLogger logger = KnowledgeRuntimeLoggerFactory.newFileLogger(ksession, "test");
// Insert of 2 Event:
Message message = new Message();
message.setMessage("Message 1");
message.setStatus(Message.HELLO);
ksession.insert(message);
ksession.fireAllRules();
clock.advanceTime(1, TimeUnit.DAYS);
Message message2 = new Message();
message2.setMessage("Message 2");
message2.setStatus(Message.HELLO);
ksession.insert(message2);
ksession.fireAllRules();
clock.advanceTime(1, TimeUnit.DAYS);
ksession.fireAllRules();
// Now I do check what I have in the working memory and if EventFactHandle if it's expired or not:
for (FactHandle f : ksession.getFactHandles()) {
if (f instanceof EventFactHandle) {
System.out.println(((EventFactHandle)f)+" "+((EventFactHandle)f).isExpired());
} else {
System.out.println("not an Event: "+f);
}
}
logger.close();
} catch (Throwable t) {
t.printStackTrace();
}
}
private static KnowledgeBase readKnowledgeBase() throws Exception {
KnowledgeBuilder kbuilder = KnowledgeBuilderFactory.newKnowledgeBuilder();
kbuilder.add(ResourceFactory.newClassPathResource("Sample.drl"), ResourceType.DRL);
KnowledgeBuilderErrors errors = kbuilder.getErrors();
if (errors.size() > 0) {
for (KnowledgeBuilderError error: errors) {
System.err.println(error);
}
throw new IllegalArgumentException("Could not parse knowledge.");
}
KnowledgeBase kbase = KnowledgeBaseFactory.newKnowledgeBase();
kbase.addKnowledgePackages(kbuilder.getKnowledgePackages());
// following 2 lines is the template code modified for STREAM configuration
KnowledgeBaseConfiguration config = KnowledgeBaseFactory.newKnowledgeBaseConfiguration();
config.setOption( EventProcessingOption.STREAM );
return kbase;
}
/*
* This is OK from template, as from the doc:
* By default, the timestamp for a given event is read from the Session Clock and assigned to the event at the time the event is inserted into the working memory.
*/
public static class Message {
public static final int HELLO = 0;
public static final int GOODBYE = 1;
private String message;
private int status;
public String getMessage() {
return this.message;
}
public void setMessage(String message) {
this.message = message;
}
public int getStatus() {
return this.status;
}
public void setStatus(int status) {
this.status = status;
}
}
}
Drools:
package com.sample
import com.sample.DroolsTest.Message;
declare Message
#role(event)
end
declare window LastMessageWindow
Message() over window:length(1)
end
rule "Hello World"
when
accumulate( $m : Message(status==Message.HELLO) from window LastMessageWindow,
$messages : collectList( $m ) )
then
System.out.println( ((Message)$messages.get(0)).getMessage() );
end
Please note: even if I add expiration of 1second to the Message event, by
#expires(1s)
I still don't get the expected result that the very first Message event inserted, I would have expected is now expired? Thanks for your help.

Found solution! Obviously it was me being stupid and not realizing I was using Drools 5.4.0.Final while still referring to old documentation of 5.2.0.Final. In the updated documentation for Drools Fusion 5.4.0.Final, this box is added for 2.6.2. Sliding Length Windows:
Please note that length based windows do not define temporal constraints for event expiration from the session, and the engine will not consider them. If events have no other rules defining temporal constraints and no explicit expiration policy, the engine will keep them in the session indefinitely.
Therefore the 3rd requirement I originally enlisted of "You must define temporal constraints between Events" is obviously NOT met because I now understand Sliding Length Window in Drools 5.4.0.Final:
Message() over window:length(1)
are indeed NOT a definition of a temporal constraints for event expiration from the session.
Updating this answer hopefully somebody will find it helpful. Also, just so for your know, me being stupid actually for relying on googling in order to reach the doc, and sometimes you don't get redirected to the current release documentation, so it seems...

Related

Esper EPL window select not working for a basic example

Everything I read says this should work: I need my listener to trigger every 10 seconds with events. What I am getting now is every event in, it a listener trigger. What am I missing? The basic requirements are to create summarized statistics every 10s. Ideally I just want to pump data into the runtime. So, in this example, I would expect a dump of 10 records, once every 10 seconds
class StreamTest {
private final Configuration configuration = new Configuration();
private final EPRuntime runtime;
private final CompilerArguments args = new CompilerArguments();
private final EPCompiler compiler;
public DatadogApplicationTests() {
configuration.getCommon().addEventType(CommonLogEntry.class);
runtime = EPRuntimeProvider.getRuntime(this.getClass().getSimpleName(), configuration);
args.getPath().add(runtime.getRuntimePath());
compiler = EPCompilerProvider.getCompiler();
}
#Test
void testDisplayStatsEvery10S() throws Exception{
// Display stats every 10s about the traffic during those 10s:
EPCompiled compiled = compiler.compile("select * from CommonLogEntry.win:time(10)", args);
runtime.getDeploymentService().deploy(compiled).getStatements()[0].addListener(
(old, newEvents, epStatement, epRuntime) ->
Arrays.stream(old).forEach(e -> System.out.format("%s: received %n", LocalTime.now()))
);
new BufferedReader(new InputStreamReader(this.getClass().getResourceAsStream("/access.log"))).lines().map(CommonLogEntry::new).forEachOrdered(e -> {
runtime.getEventService().sendEventBean(e, e.getClass().getSimpleName());
try {
Thread.sleep(TimeUnit.SECONDS.toMillis(1));
} catch (InterruptedException ex) {
System.err.println(ex);
}
});
}
}
Which currently outputs every second, corresponding to the sleep in my stream:
11:00:54.676: received
11:00:55.684: received
11:00:56.689: received
11:00:57.694: received
11:00:58.698: received
11:00:59.700: received
A time window is a sliding window. There is a chapter on basic concepts that explains how they work. Here is the link to the basic concepts chapter.
It is not clear what the requirements are but I think what you want to achieve is collecting events for a while and then releasing them. You can draw inspiration from the solution patterns.
This will collect events for 10 seconds.
create schema StockTick(symbol string, price double);
create context CtxBatch start #now end after 10 seconds;
context CtxBatch select * from StockTick#keepall output snapshot when terminated;

pgjdbc-ng Ill-formed region:

Currently i'm trying to make a module which will listen to any changes via trigger on Postgres. I'm using pgjdbc-ng ver 0.8.2 ,download the JAR from maven repo central and add it as project reference.
Following is the code that i used :
public class ListenNotify
{
// Create the queue that will be shared by the producer and consumer
private BlockingQueue queue = new ArrayBlockingQueue(10);
// Database connection
PGConnection connection;
public ListenNotify()
{
// Get database info from environment variables
/*
String DBHost = System.getenv("DBHost");
String DBName = System.getenv("DBName");
String DBUserName = System.getenv("DBUserName");
String DBPassword = System.getenv("DBPassword");
*/
String DBHost = "127.0.0.1";
String DBName = "dbname";
String DBUserName = "postgres";
String DBPassword = "postgres";
// Create the listener callback
PGNotificationListener listener = new PGNotificationListener()
{
#Override
public void notification(int processId, String channelName, String payload)
{
// Add event and payload to the queue
queue.add("/channels/" + channelName + " " + payload);
}
};
try
{
// Create a data source for logging into the db
PGDataSource dataSource = new PGDataSource();
dataSource.setHost(DBHost);
dataSource.setPort(5432);
dataSource.setDatabaseName(DBName);
dataSource.setUser(DBUserName);
dataSource.setPassword(DBPassword);
// Log into the db
connection = (PGConnection) dataSource.getConnection();
// add the callback listener created earlier to the connection
connection.addNotificationListener(listener);
// Tell Postgres to send NOTIFY q_event to our connection and listener
Statement statement = connection.createStatement();
statement.execute("LISTEN q_event");
statement.close();
}
catch (Exception e)
{
e.printStackTrace();
}
}
/**
* #return shared queue
*/
public BlockingQueue getQueue()
{
return queue;
}
/**
*
* main entry point
*
* #param args
*/
public static void main(String[] args)
{
// Create a new listener
ListenNotify ln = new ListenNotify();
// Get the shared queue
BlockingQueue queue = ln.getQueue();
// Loop forever pulling messages off the queue
while (true)
{
try
{
// queue blocks until something is placed on it
String msg = queue.take().toString();
// Do something with the event
System.out.println(msg);
}
catch (InterruptedException e)
{
e.printStackTrace();
}
}
}
}
Upon running, i got exception :
Ill-formed region: Indonesia [at index 0]
I have read the official git, saying that it should be fixed within some release number.
How do i apply those fix ?
Thank you
I know its a little bit late ;)
I had the same problem and also read that the problem was solved. But it does not seem that way.
Anyway the problem is this when creating the postgres database, LC_COLLATE is probably set to Indonesian_Indonesia.1252. When trying to establish a connection this value is compared with the java locales. In the Java Locales class, the value is probably in your language so the entry can not be found. However, to solve the problem you can set the default value of the Java locales to English. This is certainly not the best way to solve the problem, but it works. For security, I would put back after the connection is established
you can set the default value as follows:
Locale.setDefault(Locale.ENGLISH)

Vert.x: How to wait for a future to complete

Is there a way to wait for a future to complete without blocking the event loop?
An example of a use case with querying Mongo:
Future<Result> dbFut = Future.future();
mongo.findOne("myusers", myQuery, new JsonObject(), res -> {
if(res.succeeded()) {
...
dbFut.complete(res.result());
}
else {
...
dbFut.fail(res.cause());
}
}
});
// Here I need the result of the DB query
if(dbFut.succeeded()) {
doSomethingWith(dbFut.result());
}
else {
error();
}
I know the doSomethingWith(dbFut.result()); can be moved to the handler, yet if it's long, the code will get unreadable (Callback hell ?) It that the right solution ? Is that the omny solution without additional libraries ?
I'm aware that rxJava simplifies the code, but as I don't know it, learning Vert.x and rxJava is just too much.
I also wanted to give a try to vertx-sync. I put the dependency in the pom.xml; everything got downloaded fine but when I started my app, I got the following error
maurice#mickey> java \
-javaagent:~/.m2/repository/co/paralleluniverse/quasar-core/0.7.5/quasar-core-0.7.5-jdk8.jar \
-jar target/app-dev-0.1-fat.jar \
-conf conf/config.json
Error opening zip file or JAR manifest missing : ~/.m2/repository/co/paralleluniverse/quasar-core/0.7.5/quasar-core-0.7.5-jdk8.jar
Error occurred during initialization of VM
agent library failed to init: instrument
I know what the error means in general, but I don't know in that context... I tried to google for it but didn't find any clear explanation about which manifest to put where. And as previously, unless mandatory, I prefer to learn one thing at a time.
So, back to the question : is there a way with "basic" Vert.x to wait for a future without perturbation on the event loop ?
You can set a handler for the future to be executed upon completion or failure:
Future<Result> dbFut = Future.future();
mongo.findOne("myusers", myQuery, new JsonObject(), res -> {
if(res.succeeded()) {
...
dbFut.complete(res.result());
}
else {
...
dbFut.fail(res.cause());
}
}
});
dbFut.setHandler(asyncResult -> {
if(asyncResult.succeeded()) {
// your logic here
}
});
This is a pure Vert.x way that doesn't block the event loop
I agree that you should not block in the Vertx processing pipeline, but I make one exception to that rule: Start-up. By design, I want to block while my HTTP server is initialising.
This code might help you:
/**
* #return null when waiting on {#code Future<Void>}
*/
#Nullable
public static <T>
T awaitComplete(Future<T> f)
throws Throwable
{
final Object lock = new Object();
final AtomicReference<AsyncResult<T>> resultRef = new AtomicReference<>(null);
synchronized (lock)
{
// We *must* be locked before registering a callback.
// If result is ready, the callback is called immediately!
f.onComplete(
(AsyncResult<T> result) ->
{
resultRef.set(result);
synchronized (lock) {
lock.notify();
}
});
do {
// Nested sync on lock is fine. If we get a spurious wake-up before resultRef is set, we need to
// reacquire the lock, then wait again.
// Ref: https://stackoverflow.com/a/249907/257299
synchronized (lock)
{
// #Blocking
lock.wait();
}
}
while (null == resultRef.get());
}
final AsyncResult<T> result = resultRef.get();
#Nullable
final Throwable t = result.cause();
if (null != t) {
throw t;
}
#Nullable
final T x = result.result();
return x;
}

Kafka listener, get all messages

Good day collegues.
I have Kafka project using Spring Kafka what listen a definite topic.
I need one time in a day listen all messages, put them into a collection and find specific message there.
I couldn't understand how to read all messages in one #KafkaListener method.
My class is:
#Component
public class KafkaIntervalListener {
public CountDownLatch intervalLatch = new CountDownLatch(1);
private final SCDFRunnerService scdfRunnerService;
public KafkaIntervalListener(SCDFRunnerService scdfRunnerService) {
this.scdfRunnerService = scdfRunnerService;
}
#KafkaListener(topics = "${kafka.interval-topic}", containerFactory = "intervalEventKafkaListenerContainerFactory")
public void intervalListener(IntervalEvent event) throws UnsupportedEncodingException, JSONException {
System.out.println("Recieved interval message: " + event);
IntervalType type = event.getType();
Instant instant = event.getInterval();
List<IntervalEvent> events = new ArrayList<>();
events.add(event);
events.size();
this.intervalLatch.countDown();
}
}
My events collection always has size = 1;
I tried to use different loops, but then, my collection become filed 530 000 000 times the same message.
UPDATE:
I have found a way to do it with factory.setBatchListener(true); But i need to find launch it with #Scheduled(cron = "${kafka.cron}", zone = "Europe/Moscow"). Right now this method is always is listening. Now iam trying something like this:
#Scheduled(cron = "${kafka.cron}", zone = "Europe/Moscow")
public void run() throws Exception {
kafkaIntervalListener.intervalLatch.await();
}
It doesn't work, in debug mode my breakpoint never works on this site.
The listener container is, by design, message-driven.
For fetching messages on-demand, it's better to use the Kafka Consumer API directly and fetch messages using the poll() method.

Apache Samza aggregation rules for missing expected events in rolling time-period

My use-case is to identify entities from which expected events have not been received after X amount of time in real-time.
For example:
If we have received PaymentInitiated event at time T but didn't receive either of PaymentFailed / PaymentAborted / PaymentSucedded by T+X, then raise a trigger saying PaymentStuck along with details of PaymentIntitiated event.
How can I model such use-cases in Apache Samza as it is rolling time period X on each event, rather than fixed time interval.
Thanks, Harish
I'm not aware of any native support for this in Samza, but I can imagine a work-around that uses WindowableTask.
public class PaymentEvent implements Comparable<PaymentEvent> {
// if current time > timestamp, payment is stuck
public long timestamp;
// we want a corresponding PaymentFailed... event with the same id
public long interactionId;
// PaymentRequest, PaymentAborted, PaymentSucceeded...
public enum type;
...
#Override
public int compareTo(PaymentEvent o){
return timestamp - o.timestamp;
}
}
Now in your process method you would have something like:
PriorityQueue<PaymentEvent> pqueue;
Map<Long, PaymentEvent> responses;
public void process(...) {
PaymentEvent e = new PaymentEvent(envelope.getMessage());
if (e.enum == PAYMENT_REQUEST) {
pqueue.add(e);
} else {
responses.put(e.interactionId, e);
}
}
And finally during your window you would pop off from the priority queue everything with timestamp > current time and check if there is a corresponding event in the Map.
public void window(...) {
while(pqueue.peek().timestamp <= currentTime) {
if (!map.containsKey(pqueue.poll().interactionId) {
// send the trigger via the collector
}
}
}
Then lastly you would set the window time in your configuration to however long you wanted to poll. The config is task.window.ms.