ESPER: 'Partition by' CLAUSE ERROR - complex-event-processing

The issue that I have is using the clause 'partition by' in 'Match Recognize', the 'partition by' clause seems to support just 99 different events because when I have 100 or more different events it does not group correctly. to test this I have the following EPL query:
select * from TemperatureSensorEvent
match_recognize (
partition by id
measures A.id as a_id, A.temperature as a_temperature
pattern (A)
define
A as prev(A.id) is null
)
I am using this query basically to get the first event (first temperature) of each device, however testing with 10, 20, 50, ... 99 different devices it works fine but when I have more than 99, it seems that ESPER resets all the events send before the device with id=100, and if I send a event that is of the device with id=001, ESPER takes it as if it was the first event.
it seems that 'partition by' just supports 99 different events and if you add one more the EPL is reset or something like that. Is it a restriction that 'partition by' clause has?, how I can increase this threshold because I have more than 100 devices?.
ESPER version: 5.1.0
Thanks in advance
Demo Class:
public class EsperDemo
{
public static void main(String[] args)
{
Configuration config = new Configuration();
config.addEventType("TemperatureSensorEvent", TemperatureSensorEvent.class.getName());
EPServiceProvider esperProvider = EPServiceProviderManager.getProvider("EsperDemoEngine", config);
EPAdministrator administrator = esperProvider.getEPAdministrator();
EPRuntime esperRuntime = esperProvider.getEPRuntime();
// query to get the first event of each temperature sensor
String query = "select * from TemperatureSensorEvent "
+ "match_recognize ( "
+ " partition by id "
+ " measures A.id as a_id, A.temperature as a_temperature "
+ " after match skip to next row "
+ " pattern (A) "
+ " define "
+ " A as prev(A.id) is null "
+ ")";
TemperatureSubscriber temperatureSubscriber = new TemperatureSubscriber();
EPStatement cepStatement = administrator.createEPL(query);
cepStatement.setSubscriber(temperatureSubscriber);
TemperatureSensorEvent temperature;
Random random = new Random();
int sensorsQuantity = 100; // it works fine until 99 sensors
for (int i = 1; i <= sensorsQuantity; i++) {
temperature = new TemperatureSensorEvent(i, random.nextInt(20));
System.out.println("Sending temperature: " + temperature.toString());
esperRuntime.sendEvent(temperature);
}
temperature = new TemperatureSensorEvent(1, 64);
System.out.println("Sending temperature: sensor with id=1 again: " + temperature.toString());
esperRuntime.sendEvent(temperature);
}
}

Related

Pagination in jpa query giving wrong number of elements in a page

I am getting the wrong number of items (different from the page size I am passing). The code is as follows,
Query query = entityManager.createQuery("SELECT new com.one97.one97pay.web.dto.CustomerReportResponseDTO(cw.walletId, cw.balance, cw.isEnabled,ca.isEnabled,ca.accountLockType,ck.isEnabled AS kycStatus,cp.mobile,cp.residentId,cp.ridExpiryDate, ck.kycStatus,ck.kycType,ca.createdOn "
+ ",TRIM(UPPER(CONCAT (cp.firstName,CASE WHEN cp.middelName is NULL THEN '' ELSE CONCAT(' ', cp.middelName) END,CASE WHEN cp.lastName is NULL THEN '' ELSE CONCAT(' ', cp.lastName) END ))), ck.creationDateTime)"
+ " FROM CustomerProfile AS cp JOIN CustomerAuthModel AS ca ON cp.customerId=ca.customerId "
+ " JOIN CustomerWalletModel AS cw ON cp.customerId = cw.customerId "
+ " JOIN CustomerKycModel AS ck ON ck.customerId = cw.customerId"
+ " ORDER BY ck.creationDateTime DESC");
int pageSize = requestDto.getPageSize();
int pageNumber = requestDto.getPageNum();
query.setFirstResult((pageNumber-1) * pageSize);
query.setMaxResults(pageSize);
List <CustomerReportResponseDTO> customerReportLst = query.getResultList();
I have checked through another api without pagination, the number of results are 100+, but when I pass page number as 1 and page size 10, I get 8 elements in return. What am I doing wrong in this or is there any other way of doing this?
By looking at your code i think * pageSize will not come at query.setFirstResult because i think you are saving the page no in setFirstResult so as per that logic write this query.setFirstResult(pageNumber-1);

EF Core to make SQL Database MERGE call and get output of new inserted record along with identify column

We are using EF Core to make database call and we have a MERGE call to conditionally insert a record and what we want is once the record is inserted we want to get that new RECORD in the OUTPUT (along with the identify column). I am doing this avoid additional Select query post the insert. Following is our code, any guidance on how we can get the OUTPUT of the new record along with the identify column (auto generated value):
string query = $"MERGE INTO [ReceiptMaster] AS old " +
$"USING (VALUES ({receipt.ProfileID},'{receipt.TranscribeID}','{receipt.ReceiptStatusID}'," +
$"'{receipt.ReceiptTypeID}','{receipt.TransactionDate}','{receipt.ProcessTypeID}'," +
$"'{receipt.TripTranscriptionVendorID}','{receipt.AcquireTypeID}',{receipt.CategoryTypeID}," +
$"'{receipt.IsResubmitted}','{receipt.IsResubmittedByPanel}', {receipt.ScrapeJobID}, " +
$"'{receipt.TripTranscriptionPayloadID}')) " +
$"AS new (ProfileID, TranscribeID, ReceiptStatusID, ReceiptTypeID, TransactionDate, ProcessTypeID, " +
$"TripTranscriptionVendorID, AcquireTypeID, CategoryTypeID, IsResubmitted, IsResubmittedByPanel, " +
$"ScrapeJobID, TripTranscriptionPayloadID) " +
$"ON new.TranscribeID = old.TranscribeID " +
$"WHEN NOT MATCHED BY TARGET THEN INSERT (ProfileID, TranscribeID, ReceiptStatusID, ReceiptTypeID, " +
$"TransactionDate, ProcessTypeID, TripTranscriptionVendorID, AcquireTypeID, CategoryTypeID, IsResubmitted, " +
$"IsResubmittedByPanel, ScrapeJobID, TripTranscriptionPayloadID) VALUES (ProfileID, TranscribeID, ReceiptStatusID, " +
$"ReceiptTypeID, TransactionDate, ProcessTypeID, TripTranscriptionVendorID, AcquireTypeID, CategoryTypeID, " +
$"IsResubmitted, IsResubmittedByPanel, ScrapeJobID, TripTranscriptionPayloadID);";
var context = _dbContext.Database.ExecuteSqlRaw(query);
if (context == 0) return false;
else return true;
}

Syntax error when querying results directly to a DTO

My native query -
interface PodcastRepository: JpaRepository<Podcast, Long> {
#Query(value = "SELECT new com.krtkush.sample.modules.podcast.models.PodcastDTO" +
"(p.id, p.author, p.title, p.description c.name, c2.name) " +
"AS sub_category_name FROM podcasts p " +
"LEFT JOIN categories c ON p.podcast_category_id = c.category_id " +
"LEFT JOIN categories c2 ON p.podcast_subcategory_id = c2.category_id " +
"WHERE p.podcast_owner = :ownerId", nativeQuery = true)
fun getPodcastsByOwner(#Param("ownerId")owner: Long): List<PodcastDTO>
}
However, when I execute the function I get the following error -
org.postgresql.util.PSQLException: ERROR: syntax error at or near "." Position: 15
position 15 is . after SELECT new com
I'm following this tutorial - https://smarterco.de/spring-data-jpa-query-result-to-dto/
The difference is that I'm using SQL rather than JPQL.

UPDATE and JOIN with JPQL

Tutorials and samples about JPQL always deal with SELECT statement and sometimes, simple UPDATE statements. I need to update a table with a join.
I have simplified my env :
KEY
= id
- counter
APPLET
= id
! key_id (1-1)
DEVICE
= id
! applet_id (1-1)
! user_id (1-n)
USER
= id
- login
A device has a unique applet, which has a unique keyset. But an user can own several devices.
I need to reset the counter of every KEY attached to the USER login "x".
I tried some syntax with UPDATE and JOIN, without success. Any clue ?
Thank you.
What did you try and what error did you get? What is your object model?
Perhaps something like,
Update Key k set k.counter = 0 where exists (Select u from User u join u.devices d where u.login = "x" and d.applet.key = k)
See,
http://en.wikibooks.org/wiki/Java_Persistence/JPQL_BNF#Update
You could also select the objects and reset the counter in memory and commit the changes.
JPQL does not support join operations in bulk update operations.
When you edit query, nativeQuery = true, you can join.
Query should be written according to the fields in the database.
#Transactional
#Modifying
#Query(nativeQuery = true,
value = "UPDATE Team t" +
" SET current = :current " +
" FROM " +
" members," +
" account" +
" WHERE " +
" members.members_id = t.members_id " +
" AND members.account_id = :account " +
" AND t.current = :current_true ")
int updateTeam(
#Param("current") String current,
#Param("account") Long account,
#Param("current_true") Integer current_true);

Uploading data using C# console application

I have a C# console application that uploads data into SQL Server database after doing a bit of calculation which is done using various C# functions. Now the problem is it is taking almost 1 sec to calculate and upload one line of data and I have to upload 50,000 lines of data in the same way.
Please suggest me a way to solve this problem.
P.S. : I am using stringbuilder to compose separate insert statements and upload in bulk. This process is taking only 1 min.
Inserting or updating to database is hardly taking any time as I have mentioned in my question. Calculation is taking most of the time. I am attaching the code sample of a function below:
public void EsNoMinLim()
{
ds = new DataSet();
ds = getDataSet("select aa.Country, aa.Serial_No from UEM_Data aa inner join (select distinct " +
"IId, Country from UEM_Data where Active_Status is null) bb on aa.iid = bb.iid where aa.Serial_No <> '0'").Copy();
execDML("Delete from ProMonSys_Grading");
StringBuilder strCmd = new StringBuilder();
foreach (DataRow dRow in ds.Tables[0].Rows)
{
SiteCode = dRow["Country"].ToString();
Serial_No = dRow["Serial_No"].ToString();
ds_sub = new DataSet();
ds_sub = getDataSet("select EsNo_Abs_Limit from EsNo_Absolute_Limit where Fec_Coding_Rate in "+
"(select MODCOD from FEC_Master where NMS_Value in (select Top 1 FEC_Rate from "+
"DNCC_Billing_Day where Serial_No = '" + Serial_No + "' and [Date] = (select max([Date]) "+
"from DNCC_Billing_Day where Serial_No = '" + Serial_No + "')))").Copy();
if (ds_sub.Tables[0].Rows.Count > 0 && Convert.ToString(ds_sub.Tables[0].Rows[0][0]) != "")
{
Min_EsNo = Convert.ToString(ds_sub.Tables[0].Rows[0][0]);
}
else
{
Min_EsNo = "a";
}
if (Min_EsNo != "a")
{
ds_sub = new DataSet();
ds_sub = getDataSet("select Top 1 modal_Avg_EsNo from DNCC_Billing_Day where " +
"Serial_No = '" + Serial_No + "' and [Date] = (select max([Date]) from DNCC_Billing_Day " +
"where Serial_No = '" + Serial_No + "')").Copy();
if (ds_sub.Tables[0].Rows.Count > 0 && Convert.ToString(ds_sub.Tables[0].Rows[0][0]) != "")
{
Avg_EsNo = Convert.ToString(ds_sub.Tables[0].Rows[0][0]);
}
else
{
Avg_EsNo = "-1";
}
ds_sub = new DataSet();
ds_sub = getDataSet("select Top 1 Transmit_Power from ProMonSys_Threshold where Serial_No = '" + Serial_No + "'").Copy();
if (ds_sub.Tables[0].Rows.Count > 0 && Convert.ToString(ds_sub.Tables[0].Rows[0][0]) != "")
{
Threshold_EsNo = Convert.ToString(ds_sub.Tables[0].Rows[0][0]);
}
else
{
Threshold_EsNo = "-1";
}
getGrade = EsNoSQFGrading(Min_EsNo, Avg_EsNo, Threshold_EsNo);
strCmd.Append("insert into ProMonSys_Grading(SiteCode, Serial_No, EsNo_Grade) " +
"values('" + SiteCode + "','" + Serial_No + "','" + getGrade + "')");
}
}
execDML_StringBuilder(strCmd);
}
Find out, what part of the process is the expensive one. Use StopWatch to check how long loading, calculating and saving takes separately. Then you which part to improve (and can tell us).