I have around 20k records to be deleted from Spring data JPA query , query is related to deleting all records before some particular date.
I am using below query
dao.deleteByCreationDateBefore(new Date());
I think this query hits database for each row deletion.
Please let me know is there any way I can use batch deletion here?
Regards
spting data doen't support batch operation.
try to do it with simple delete if it's possible , like : it be very fast operation (faster than butch delete)
delete from SOME_ENTITY/TABLE where CreationDate < new Date()/curentDate
if your dao method delete record by record :
But you can do it with hibernate/jpa in dao level , like (example from site with persist) from Hibernate/JPA Batch Insert and Batch Update Example:
em.getTransaction().begin();
for (int i = 0; i < 100; i++){
Book book = new Book(i, "Hibernate/JPA Batch Insert Example: " + i);
em.persist(book);
if (i % batchSize == 0 && i > 0) {
em.flush();
em.clear();
}
}
em.getTransaction().commit();
for hibernate :
and here is article How to batch INSERT and UPDATE statements with Hibernate , read about 'Configuring hibernate.jdbc.batch_size'. And Hibernate JDBC and Connection Properties options for hibernate for hibernate.jdbc.fetch_size and hibernate.jdbc.batch_size
Related
I am trying to call a stored procedure, which is built in mysql, in my Spring boot app using JPA. My stored procedure returns the result which cant be contain in single model as it fetches data from combination of tables.
I can do this with "call " but i guess that is not JPA's way. COuld you please let me know what is the best way to do it?
In case you're using plain JPA you need to do a native query call. Something like below.
Query q = em.createNativeQuery("select my_store_pro(?, ?)");
List<Object[]> results = q.getResultList();
for (Object[] a : results) {
System.out.println("result " + a[0] + " " + a[1]);
}
If you're using Spring Data repositories then you want something like below.
#Query(nativeQuery = true, value = "select my_store_pro(?, ?)")
Date callMyStoreProc(int val1, int val2);
I have a method on my Spring Data repository interface defined like this:
#Query("SELECT MAX(e.index) FROM entity e WHERE e.companyId = :companyId AND e.userId = :userId")
public Integer getMaxIndex(#Param("companyId") Long companyId, #Param("userId") Long userId);
The calling code looks like this:
int currIndex = 0;
Integer maxIndex = userActivityQueueRepository.getMaxIndex(companyId, user.getId());
if (null != maxIndex)
{
currIndex = maxIndex.intValue() + 1;
}
//else no records exist yet, so currIndex is 0
//create new records starting at currIndex and incrementing it along the way
The problem is that when no records exist yet, it is returning 0 and not null. So, my currIndex gets set to 1 instead of 0. Is there something I'm doing wrong in the JPQL? Is there something I can add to the JPQL so it behaves as I was expecting it would?
I'm using version 1.7.2 of Spring Data JPA with PostreSQL and EclipseLink. I turned on the logging of SQL and ran the query manually in the database, and it gives the results I expect. I just don't understand why it isn't returning null when there are no records.
For MAX function (MIN too) the result type is the type of the field, so to return null instead of 0, entity.index should be like Integer, Long but no int, long
See this table of the official documentation.
Following is the code which is blowing up if the list which is being passed in to "IN" clause has several values. In my case the count is 1400 values. Also the customer table has several thousands (arround 100,000) of records in it. The query is executing against DERBY database.
public List<Customer> getCustomersNotIn(String custType, List<Int> customersIDs) {
TypedQuery<Customer> query = em.createQuery("from Customer where type=:custType and customerId not in (:customersIDs)", Customer.class);
query.setParameter("custType", custType);
query.setParameter("customersIDs", customersIDs);
List<Customer> customerList = query.getResultList();
return customerList;
}
The above mentioned method perfectly executes if the list has less values ( probably less than 1000 ), if the list customersIDs has more values since the in clause executes based on it, it throws an error saying "Statement too complex"
Since i am new to JPA can any one please tell me how to write the above mention function in the way described below.. * PLEASE READ COMMENTS IN CODE *
public List<Customer> getCustomersNotIn(String custType, List<Int> customersIDs) {
// CREATE A IN-MEMORY TEMP TABLE HERE...
// INSERT ALL VALUES FROM customerIDs collection into temp table
// Change following query to get all customers EXCEPT THOSE IN TEMP TABLE
TypedQuery<Customer> query = em.createQuery("from Customer where type=:custType and customerId not in (:customersIDs)", Customer.class);
query.setParameter("custType", custType);
query.setParameter("customersIDs", customersIDs);
List<Customer> customerList = query.getResultList();
// REMOVE THE TEMP TABLE FROM MEMORY
return customerList;
}
The Derby IN clause support does have a limit on the number of values that can be supplied in the IN clause.
The limit is related to an underlying limitation in the size of a single function in the Java bytecode format; Derby currently implements IN clause execution by generating Java bytecode to evaluate the IN clause, and if the generated bytecode would exceed the JVM's basic limitations, Derby throws the "statement too complex" error.
There have been discussions about ways to fix this, for example see:
DERBY-6784
DERBY-6301, or
DERBY-216
But for now, your best approach is probably to find a way to express your query without generating such a large and complex IN clause.
Ok here is my solution that worked for me. I could not change the part generating the customerList since it is not possible for me, so the solution has to be from within this method. Bryan your explination was the best one, i am still confuse how "in" clause worked perfectly with table. Please see below solution.
public List<Customer> getCustomersNotIn(String custType, List<Int> customersIDs) {
// INSERT customerIds INTO TEMP TABLE
storeCustomerIdsIntoTempTable(customersIDs)
// I AM NOT SURE HOW BUT, "not in" CLAUSE WORKED INCASE OF TABLE BUT DID'T WORK WHILE PASSING LIST VALUES.
TypedQuery<Customer> query = em.createQuery("select c from Customer c where c.customerType=:custType and c.customerId not in (select customerId from TempCustomer)");
query.setParameter("custType", custType);
List<Customer> customerList = query.getResultList();
// REMOVE THE DATA FROM TEMP TABLE
deleteCustomerIdsFromTempTable()
return customerList;
}
private void storeCustomerIdsIntoTempTable(List<Int> customersIDs){
// I ENDED UP CREATING TEMP PHYSICAL TABLE, INSTEAD OF JUST IN MEMORY TABLE
TempCustomer tempCustomer = null;
try{
tempCustomerDao.deleteAll();
for (Int customerId : customersIDs) {
tempCustomer = new TempCustomer();
tempCustomer.customerId=customerId;
tempCustomerDao.save(tempCustomer);
}
}catch(Exception e){
// Do logging here
}
}
private void deleteCustomerIdsFromTempTable(){
try{
// Delete all data from TempCustomer table to start fresh
int deletedCount= tempCustomerDao.deleteAll();
LOGGER.debug("{} customers deleted from temp table", deletedCount);
}catch(Exception e){
// Do logging here
}
}
JPA and the underlying Hibernate simply translate it into a normal JDBC-understood query. You wouldn't write a query with 1400 elements in the IN clause manually, would you? There is no magic. Whatever doesn't work in normal SQL, wouldn't in JPQL.
I am not sure how you get that list (most likely from another query) before you call that method. Your best option would be joining those tables on the criteria used to get those IDs. Generally you want to execute correlated filters like that in one query/transaction which means one method instead of passing long lists around.
I also noticed your customerId is double - a poor choice for a PK. Typically people use long (autoincremented/sequenced, etc.) And I don't get the "temp table" logic.
I'm using Azure SQL database and Entity Framework for performing database operations on it.
How can I convert this SQL query to Entity Framework ?
begin tran set transaction isolation level serializable
go
select top 1 * from Employee with (UPDLOCK) where EmpID = #id;
......
commit
I want to lock a row from other threads when some thread is reading it and perform some operation on it.
I cannot use stored procedures as I'm using Azure SQL database.
I don't know why you could not use a stored procedure for this. As marc_s mentioned Azure SQL DB supports stored procedures. That said you can always execute the query from EF if you want. EF does not support specifying query hints for LINQ queries, so the easiest would be to use raw SQL execution APIs. It would look like this using EF6:
using (var context = new MyContext())
{
using (var transaction = context.Database.BeginTransaction())
{
try
{
var employee = context.Employees.SqlQuery(
"select top 1 * from Employee with (UPDLOCK) where EmpID = #id", id);
// ...
transaction.Commit();
}
catch (Exception)
{
transaction.Rollback();
}
}
}
I'm trying to write a trigger where inserting, updating, or deleting a record(line item) will update an amount field. Now all this records will have same ParentID (expense) and Name(of the line items). Basically duplicate records except the Contact name will be different. So, when I add a new line item with new amount with same Parents and name, the trigger should go off and query all the line items with same Parent and should re-calculate the amount.
So, if I enter first line item and say Total amount should be 100. Then I enter 2nd line Item, trigger should fire and update Amount on both record saying '50.00'. For some reason, my trigger is not updating even though it's calculating it correctly. Where is the bug? Please help!!!
trigger Test on Expense_Line_Item__c (after insert, after update, after delete) {
set<id>testlist = new set<id>();
//List<Expense_Line_Item__c> listItem= new List<Expense_Line_Item__c>();
for (Expense_Line_Item__c a : trigger.new) {
testlist.add(a.expense__c);
}
list<Expense_Line_Item__c> mapParent =
new list<Expense_Line_Item__c>([SELECT name,
id,
Amount__c
FROM Expense_Line_Item__c
WHERE expense__c IN:testlist]);
Decimal Total = 0.0;
Integer Count = 0;
for (Expense_Line_Item__c exp : mapParent) {
Total = Total + exp.Amount__c;
Count++;
System.debug('Total during iterator::::::::::::::::::::::' + Total);
System.debug('Counter:::::::::::::::::::::::::::::::::::::' + Count);
}
if (Count > = 1)
Total = Total / Count;
System.debug('Total count after division::::::::::::::::::::::' + Total);
List <Expense_Line_Item__c> insertLineItem = new List <Expense_Line_Item__c>();
for (Expense_Line_Item__c lineItem : MapParent) {
lineItem.Amount__c = Total;
//insertLineItem.add(lineItem);
//System.debug('LineItem Amount getting inserted::::::::::::::::::::::'+lineItem.Amount__c);
}
// upsert insertLineItem;
}
It looks like your code is an After trigger. As you're in after trigger context you have to do DML to update the records.
It also seems to me you're trying to do too much in a single trigger, its always better to take each method individually so that you can fully control what you're doing.
I see you do have an Upsert DML statement commented out but you'd be better off using Update for this particular instance as you're not using an external id or making a decision to create or insert which is the purpose of upsert