Calculating Time between two events using Siddhi - complex-event-processing

I am new to CEP and Siddhi. I need to calculate time on task using 2 events.
First event will have {taskId:"100", startTime="2014-11-03T18:35:00.000Z"}
Second event will have {taskId:"100", endTime="2014-11-03T18:35:57.000Z"}
Logic should be for the same taskId do endTime - startTime, will give time on task.
I am struggling to form query that can do this.
Please advice

You can do something like following siddhi query
define stream sensorStream (taskId string, timestamp string);
from every e1=sensorStream -> e2=sensorStream[taskId == e1.taskId]
select e1.taskId,
time:timestampInMilliseconds(e2.timestamp, "yyyy-MM-dd HH:mm:ss")
- time:timestampInMilliseconds(e1.timestamp, "yyyy-MM-dd HH:mm:ss")
as duration
insert into outputStream;
Here i have done a pattern matching [1] to catch sequence of occurring of 2 events of same task ID, then those 2 events will be assigned to e1 and e2 respectively. Finally inside select, e2.timestamp and e1.timestamp and string values will be converted to milliseconds using time extensions timestampInMilliseconds function [2] and subtracted e1 timestamp value from e2 timestamp to get the duration
[1] https://docs.wso2.com/display/CEP420/SiddhiQL+Guide+3.1#SiddhiQLGuide3.1-Pattern
[2] https://docs.wso2.com/display/CEP420/Time+Extension#TimeExtension-TimestampInMillisecondsfunction

Related

Kafka source connector is not pulling the record as expected when records are inserted in source topic from multiple sources

In one of my use case i am trying to create a pipeline
whenever i sent the message from custom partition, i sent the timestamp in milliseconds with LONG data type because in the schema, the timestamp column has been defined as long.
Code that i had earlier in custom partition:
Date date = new Date();
long timeMilli = date.getTime();
System.out.println("date = " + date.toString() + " , time in millis = " + timeMilli);
Display result before i sent the record:
date = Tue Mar 26 22:02:04 EDT 2019 , time in millis = 1553652124063
value inserted in timestamp column in table2:
3/27/2019 2:02:04.063000 AM
Since its taking UK timezone (i believe), i put temporary fix for time being to subtract 4 hours from the current timestamp so that i can match with USA EST timestamp.
Date date = new Date();
Date adj_date = DateUtils.addHours(date,-4);
long timeMilli = adj_date.getTime();
System.out.println("date = " + date.toString() + " , time in millis = " + timeMilli);
Display result:
date = Tue Mar 26 22:04:43 EDT 2019 , time in millis = 1553637883826
value inserted in timestamp column in table2:
3/26/2019 10:04:43.826000 PM
Please let me know if i am missing anything as i am not sure why this is happening when i sent message from custom partition.
Under the hood Jdbc Source Connector use following query:
SELECT * FROM someTable
WHERE
someTimestampColumn < $endTimetampValue
AND (
(someTimestampColumn = $beginTimetampValue AND someIncrementalColumn > $lastIncrementedValue)
OR someTimestampColumn > $beginTimetampValue)
ORDER BY someTimestampColumn, someIncrementalColumn ASC
Summarizing: The query retrieve rows if their timestamp column's value is earlier the current timestamp and is later than last checked.
Above parameters are:
beginTimetampValue - value of timestamp column of last imported record
endTimetampValue - current timestamp according to the Database
lastIncrementedValue - value of incremental column of last imported record
I think in your case Producer put to the Tables records with higher timestamp, than you later insert manually (using the query).
When Jdbc Connector checks for new records to import to Kafka it skips them (because they don't fullfil someTimestampColumn < $endTimetampValue timestamp condition)
You can also change log level to DEBUG and see what is going on in logs

Scala/Java joda.time not converting date in 24 hours format

I am trying to convert a long utc value into "yyyy-MM-dd HH:mm:ss" formatted pattern. I am expecting my data to be converted on 24 hours range scale and in GMT. My code passes all the test cases, I push the data into database using the jar that is newly built with this code -
dbRecord("order_dt_utc") = if (orderTs.isDefined) Some(new DateTime(orderTs.get, DateTimeZone.UTC).toString("yyyy-MM-dd HH:mm:ss")) else None
and now, when I query my database, I find that the data is still converting on 12 hours range. The query -
SELECT order_id, order_dt, order_dt_utc, order_ts_utc, from_unixtime(order_ts_utc/1000) FROM order_items where order_dt >= '2018-08-01' AND order_dt <= '2018-08-02' ORDER BY order_dt_utc LIMIT 1000;
And you can see the the values are not matching in the columns from_unixtime(order_ts_utc/1000) and order_dt_utc -
I am not able to figure the reason for this behaviour.
To convert Time Zone use the function first:
CONVERT_TZ (dateobj, oldtz, newtz)
After that use the date_format function:
date_format(from_unixtime(order_ts_utc), '%Y-%m-%d %H:%i:%s');
to format your time to 00-23 format.

CrudRepository,Can't get the correct result query with And Between

I am having trouble using spring-boot CrudRepository,What I want to do is to look up the data for a period of time
I make a query by this and result is ok
List<T> findByTimestampAfter(#Param("timestamp") Date start)
This is also ok
List<T> findByTimestampBefore(#Param("timestamp") Date end)
But I tried the following three methods do not work:
List<T> findByTimestampBetween(#Param("timestamp") Date start,#Param("timestamp") Date end)
List<T> findByTimestampAfterAndTimestampBefore(#Param("timestamp") Date start,#Param("timestamp") Date end)
List<T> findByTimestampGreaterThanEqualAndTimestampLessThanEqual(#Param("timestamp") Date start,#Param("timestamp") Date end)
What I want to get is more than 100 data in the time frame, but there is only one which timestamp euqal end
And i try execute this sql statementselect:
select * from exchange where timestamp > "2018-01-02 17:30:00" AND timestamp < "2018-01-02 17:48:00";
i can see the more than 100 data.
I think I can not solve this problem
The keyword between is not supported inside method name. You should define your custom query, for example:
#Query("select * from exchange where timestamp > ?1 AND timestamp < ?2")
Object[] findByTimeCustomQuery(Date dFrom, Date dTo);
See the query executing from the jpa automatically.If you just use proper debugging logger .Also tell the proper result achieved from above three methods.

Is it possible to find data from MySQL by month using JPA and java.time.LocalDate date format?

I creating an application, for that I need to find data by month using JPA and java.time.LocalDate. So, is it possible to retrieve data by month from mysql?
Thanks in advance for help.
First find start and end date of month and use between method of JPA to find data of current month.
LocalDate start = LocalDate.ofEpochDay(System.currentTimeMillis() / (24 * 60 * 60 * 1000) ).withDayOfMonth(1);
LocalDate end = LocalDate.ofEpochDay(System.currentTimeMillis() / (24 * 60 * 60 * 1000) ).plusMonths(1).withDayOfMonth(1).minusDays(1);
In Repository
List<Object> findByCreatedateGreaterThanAndCreatedateLessThan(LocalDate start,LocalDate end);
Its better to use the between keyword, it makes things allot shorter.
List<Object> findByCreatedateBetween(LocalDate start,LocalDate end);
Also if you want to use the LocalDate or LocalDateTime objects with Spring Data you should use the converter class Jsr310JpaConverters or else the documents will be stored as Blobs instead of Dates (which is bad for portability of the database). Please see this tutorial on how to implement the Converter.
https://www.mkyong.com/spring-boot/spring-boot-spring-data-jpa-java-8-date-and-time-jsr310/
tl;dr
YearMonth.now( ZoneId.of( "Pacific/Auckland" ) ) // Get current month for particular time zone.
.atDayOfMonth( 1 ) // Get the first date of that month.
.plusMonths( 1 ) // Get first of next month for Half-Open query.
Details
Assuming your column in MySQL is of DATE type…
LocalDate
The LocalDate class represents a date-only value without time-of-day and without time zone.
Time zone
A time zone is crucial in determining a date. For any given moment, the date varies around the globe by zone. For example, a few minutes after midnight in Paris France is a new day while still “yesterday” in Montréal Québec.
Specify a proper time zone name in the format of continent/region, such as America/Montreal, Africa/Casablanca, or Pacific/Auckland. Never use the 3-4 letter abbreviation such as EST or IST as they are not true time zones, not standardized, and not even unique(!).
ZoneId z = ZoneId.of( "America/Montreal" );
LocalDate today = LocalDate.now( z );
YearMonth
The YearMonth class represents an entire month. Getting the current month requires a time zone as discussed above. Around the beginning/ending of the month, the current moment could be “next” month in Auckland New Zealand while still “previous” month in Kolkata India.
YearMonth currentMonth = YearMonth.now( z ) ;
Get the first date of the month.
LocalDate start = currentMonth.atDayOfMonth( 1 ) ;
Half-Open
Generally best to use the Half-Open [) approach to defining a span of time, where the beginning is inclusive while the ending is exclusive. So defining a month means starting with the first date of the month and running up to, but not including, the first date of the following month.
LocalDate stop = start.plusMonths( 1 ) ;
Query
Do not use the BETWEEN command in SQL as it is fully closed [], both beginning and ending being inclusive. Half-Open uses >= & < logic.
SELECT when FROM tbl
WHERE when >= start
AND when < stop
;
it's also useful
#Query("from PogWorkTime p where p.codePto = :codePto and month(p.dateApply) = :month and year(p.dateApply) = :year")
Iterable<PtoExceptWorkTime> findByCodePtoAndDateApply_MonthAndDateApply_Year(#Param("codePto") String codePto,#Param("month") int month, #Param("year") int year);

Aggregating date data with entity framework grouped by day, month, qtr, year, etc

I have a table that records activities. All activities have an activitydate. I want to count how many activities for a given period of time (day, month, qtr, etc.). I want to include all dates even those that have zero activities. I could do this in the Data Tier with a DateDimension table where the date table has a single column called day containing one row for each calendar day and a outer join, group by query:
DateDimension Table
| Day |
|1/1/2013 00:00:00 |
|1/1/2013 00:00:00 |
|1/1/2013 00:00:00 |
Query
SELECT CAST(Day AS DATE), COUNT() AS CountOfActivities
FROM DateDimension dd LEFT OUTER JOIN Activities a
ON CAST(dd.Day AS DATE) = CAST(a.ActivityDate AS DATE)
WHERE Day BETWEEN MyStartDate AND MyEndDate
GROUP BY CAST(Day AS DATE)
ORDER BY CAST(Day AS DATE)
I'm using EntityFramework so I'd like to execute this query using Linq. The DateDimension table has no business value residing in the database. It exists only to support these aggregate queries by providing a list of dates so I can ensure a row is returned if no activities exist for a given day.
I have the idea that I could manufacture a list of days in memory and weave them in to the results of a much simpler database query at runtime. By perhaps Concatenating the results from 2 IEnumerables - 1 from the in memory enemurable of dates and the other from the database results. How could I do that? Should I do that?
How about something like this:
Example date range:
var from = DateTime.Today.AddDays(-30);
var to = DateTime.Today;
Dictionary to hold your tally of activities per day:
var activityCounts = new Dictionary<DateTime, int>();
Seed with a zero count for each day in the range (this is equivalent to setting up your date dimensions table):
Enumerable.Range(0, (to - from).Days + 1)
.ToList()
.ForEach(x => activityCounts[from.AddDays(x)] = 0);
Add in the real activity counts for each day in the range:
context.Activities.Where(a => a.DateTime >= from && a.DateTime <= to)
.GroupBy(a => a.DateTime)
.ToList()
.ForEach(x => activityCounts[x.Key] = x.Count());
In this way, you only hit the database for the aggregation of activities for dates with activities. The padding out of the resultset with contiguous dates within the date range is then performed on the app server.
Just need to be careful how your dates are stored in the database. This code example is expecting to be able to match keys in the activity dictionary based on the the format of the calls to DateTime.Today. You will need to shape your dates in your database query accordingly.