"Column count does not match" for h2 database Select query using projection in Spring-Boot #DataJpaTest - spring-data-jpa

I have a Spring-Boot app that has some native queries that use content projection. It runs Postgres in production and works fine. I'm trying to set up integration tests for the repositories using #DataJpaTest and a h2 in-memory database, but my queries that are using content projection are failing with a JdbcSQLException out of the driver:
org.h2.jdbc.JdbcSQLException: Column count does not match
I successfully save to the TestEntityManager, so there are records in the database, but I am unable to invoke the SELECT via the repository method. It works properly in production on Postgres -- is this a limitation to h2 and is there a workaround I could apply so I can properly test this?
The repository method looks like this (one inner join, two params in the where clause, table names and columns changed to protect the guilty):
public interface OrderRepository extends PagingAndSortingRepository<Order, Long> {
#Query(nativeQuery = true,
value = "SELECT order.id, order.total, pizza.name " +
"FROM example.order " +
"INNER JOIN example.pizza USING (pizza_id) " +
"WHERE order.customer_id = :custId " +
"AND order.order_date = :orderDate ",
countQuery = "SELECT count(order.id) " +
"FROM example.order " +
"INNER JOIN example.pizza USING (pizza_id) " +
"WHERE order.customer_id = :custId " +
"AND order.order_date = :orderDate")
<T> Page<T> findAllByCustIdAndOrderDate(String custId, OffsetDateTime orderDate, Pageable paging, Class<T> type);
}
And the projection looks like this:
public interface PizzaOrderProjection {
Long getId();
Double getTotal();
String getName();
}
The exception triggers when I call findAllByCustIdAndOrderDate, and the SQL statement that it prints is causing it is the SELECT. The SELECT it prints looks perfectly normal:
Hibernate:
/* dynamic native SQL query */ SELECT
order.id,
order.total,
pizza.name
FROM
example.order
INNER JOIN
example.pizza USING (pizza_id)
WHERE
order.customer_id = ?
AND order.order_date = ? limit ?
2019-04-09 12:42:18.704 WARN 17568 --- [ main] o.h.engine.jdbc.spi.SqlExceptionHelper : SQL Error: 21002, SQLState: 21S02
2019-04-09 12:42:18.708 ERROR 17568 --- [ main] o.h.engine.jdbc.spi.SqlExceptionHelper : Column count does not match; SQL statement:

It turns out that the error message actually has nothing to do with the underlying issue.
The H2 database does not support the using keyword on the inner join clause, only the on keyword.
The issue was resolved by changing the inner join to use on instead, like this:
public interface OrderRepository extends PagingAndSortingRepository<Order, Long> {
#Query(nativeQuery = true,
value = "SELECT order.id, order.total, pizza.name " +
"FROM example.order " +
"INNER JOIN example.pizza ON order.pizza_id = pizza.pizza_id " +
"WHERE order.customer_id = :custId " +
"AND order.order_date = :orderDate ",
countQuery = "SELECT count(order.id) " +
"FROM example.order " +
"INNER JOIN example.pizza ON order.pizza_id = pizza.pizza_id " +
"WHERE order.customer_id = :custId " +
"AND order.order_date = :orderDate")
<T> Page<T> findAllByCustIdAndOrderDate(String custId, OffsetDateTime orderDate, Pageable paging, Class<T> type);
}
This change makes the queries valid in both postgres and h2.

Related

How to parameterise Postgresql Interval for TimescaleDB `time_bucket` function with JPQL, Spring Data Repositories and Hibernate

I am using Spring Data JPA (with Hibernate underneath, JPA 2.1) with TimescaleDB extension on PostgreSQL 13, and wish to use the time_bucket function. This takes the bucket_width which is an INTERVAL and time which is the TIMESTAMP column of the data.
I want to put this in a Spring Data Repository and want to use a JPQL #Query to extract the data into a projection that represents the aggregate counts, averages etc. for the returned time buckets. I don't want to use a native query, because I want to join with some other tables, and populate their entities automatically.
I registered the time_bucket function to the PostgisPG95Dialect I am extending, like this:
public class CustomPostgresqlDialect extends PostgisPG95Dialect {
public CustomPostgresqlDialect() {
super();
this.registerFunction("time_bucket", new StandardSQLFunction("time_bucket", new OffsetDateTimeType()));
}
}
If the bucket_width is hardcoded, all this works fine. But I want the bucket_width to be a parameter of the query method.
The following works fine:
#Query("select sys as system, "
+ "function('time_bucket', '10 mins', vt.ts) as startTime, "
+ "count(vt) as total, avg(vt.speed) as avgSpeed "
+ "from Data vt "
+ "JOIN vt.system sys "
+ "where sys.sysId = :sysId and "
+ "function('time_bucket', '10 mins', vt.ts) between :from and :to "
+ "group by system, startTime "
+ "order by startTime")
List<SummaryAggregate> getSummaryData(
#Param("sysId") String sysId,
#Param("from") OffsetDateTime from,
#Param("to") OffsetDateTime to);
But when I try to parameterise the interval I can't get it to work. I tried passing the interval as a string, since that is how it is being written in the hardcoded version:
#Query("select sys as system, "
+ "function('time_bucket', :grouping, vt.ts) as startTime, "
+ "count(vt) as total, avg(vt.speed) as avgSpeed "
+ "from Data vt "
+ "JOIN vt.system sys "
+ "where sys.sysId = :sysId and "
+ "function('time_bucket', :grouping, vt.ts) between :from and :to "
+ "group by system, startTime "
+ "order by startTime")
List<SummaryAggregate> getSummaryData(
#Param("sysId") String sysId,
#Param("from") OffsetDateTime from,
#Param("to") OffsetDateTime to,
#Param("grouping") String grouping);
where grouping is passed a value like 10 mins.
But for this I get this error:
SQL Error: 0, SQLState: 42883
ERROR: function time_bucket(character varying, timestamp with time zone) does not exist
Hint: No function matches the given name and argument types. You might need to add explicit type casts.
Position: 61
I then tried to change it to a Duration, since Hibernate translates Duration to PostgreSQL Interval types
#Query("select sys as system, "
+ "function('time_bucket', :grouping, vt.ts) as startTime, "
+ "count(vt) as total, avg(vt.speed) as avgSpeed "
+ "from Data vt "
+ "JOIN vt.system sys "
+ "where sys.sysId = :sysId and "
+ "function('time_bucket', :grouping, vt.ts) between :from and :to "
+ "group by system, startTime "
+ "order by startTime")
List<SummaryAggregate> getSummaryData(
#Param("sysId") String sysId,
#Param("from") OffsetDateTime from,
#Param("to") OffsetDateTime to,
#Param("grouping") Duration grouping);
But I still got the same error, this time it is thinking that the Duration is a bigint not an Interval.
SQL Error: 0, SQLState: 42883
ERROR: function time_bucket(bigint, timestamp with time zone) does not exist
Hint: No function matches the given name and argument types. You might need to add explicit type casts.
Position: 61
Is there a way to parameterise an Interval using JPQL?
There is a way, but you will have to register a custom function for this purpose because you can't cast to an arbitrary SQL type.
public class CastInterval implements SQLFunction {
#Override
public boolean hasArguments() {
return true;
}
#Override
public boolean hasParenthesesIfNoArguments() {
return true;
}
#Override
public Type getReturnType(Type firstArgumentType, Mapping mapping) throws QueryException {
return firstArgumentType;
}
#Override
public String render(Type firstArgumentType, List args, SessionFactoryImplementor factory) throws QueryException {
return "cast(" + args.get(0) + " as interval)";
}
}
You will have to register the function within the Dialect.
So if the Dialect is being extended as indicated, this would be done with something like:
this.registerFunction("castInterval", new CastInterval());
Then you can use it like this: function('time_bucket', castInterval(:grouping), vt.ts)

JPQL Create new object using a query with dynamic order by and limit

I am able to create my custom DTO by using this JPQL query:
#Query("SELECT new com.mycompany.dto.UserDetailsDTO(vu.id, ru.active, ru.firstname, ru.lastname, ru.username, vu.logins, ru.email, COUNT(li.creator)) "
+ "FROM User vu inner join RemoteUser ru on vu.remoteUser = ru.username "
+ "inner join Item li on li.creator = vu.id "
+ "group by li.creator, ru.active, ru.firstname, ru.lastname, ru.username, vu.logins, vu.id, ru.email")
List<UserDetailsDTO> getAllUsers();
Now I want to add order by, ASC/DESC, limit, offset to the above query to get the result based on dynamic params something like this:
#Query("SELECT new com.mycompany.dto.UserDetailsDTO(vu.id, ru.active, ru.firstname, ru.lastname, ru.username, vu.logins, ru.email, COUNT(li.creator)) "
+ "FROM User vu inner join RemoteUser ru on vu.remoteUser = ru.username "
+ "inner join Item li on li.creator=vu.id "
+ "group by li.creator, ru.active, ru.firstname, ru.lastname, ru.username, vu.logins, vu.id, ru.email "
+ "order by = :orderBy :orderDir and offset = :pageNo and limit = :pageSize")
List<UserDetailsDTO> getAllUsers(#Param("pageNo") int pageNo,
#Param("pageSize") int pageSize,
#Param("orderDir") String orderDir,
#Param("orderBy") String orderBy);
But it is not working and the error is:
Caused by: org.hibernate.hql.internal.ast.QuerySyntaxException: unexpected token: = near line 1, column 427
Already tried with passing pageable as a param:
Page<UserDetailsDTO> getAllUsers(Pageable pageable)
and preparing page request like:
PageRequest pageRequest = PageRequest.of(pageNo, pageSize, Sort.by(Sort.Direction.valueOf(orderDir), orderFieldUid));
Also, I tried with
+ "order by ?4 ?3 and offset ?1 and limit ?2")
Is there any way to add dynamic params as order by and offset to it?
I want similar to something like this in JPQL.
Finally, I fixed it by using Page<UserDetailsDTO> getAllUsers(Pageable pageable) No need to pass params in the query. Hibernate will take the params internally from pageable

JPA - Find items from a list that don't exist in a table

Given a list of emails, I need to find which ones don't exist in a table. Using SQL, I can do the following:
SELECT e.email
FROM
(
VALUES('email1'),('email2'),('email3'),('email4')
) AS e(email)
EXCEPT
SELECT username FROM dbo.UsersTbl;
How can I write equivalent JPQL? In the application, values email1, email2... need be dynamically built (not hardcoded) based on passed in list. Using a Spring Data JPA native query, I can do the following:
#Query( value =
"SELECT e.email " +
" FROM " +
"( " +
" VALUES('email1'),('email2'),('email3'),('email4') " +
" ) AS e(email) " +
" EXCEPT " +
" SELECT username FROM dbo.UsersTbl ",
nativeQuery=true)
List<String> findMissingEmails(List<String> emails);
But how can I pass in the list of emails to the query?
For fixed number of email arguments, this could work:
#Query( value =
"SELECT e.email " +
" FROM " +
"( " +
" VALUES(:email1),(:email2),(:email3),(:email4) " +
" ) AS e(email) " +
" EXCEPT " +
" SELECT username FROM dbo.UsersTbl ",
nativeQuery=true)
List<String> findMissingEmails(String email1, String email2, String email3, String email4);
For high and/or dynamic number of emails, a better approach could be to use NativeQuery constructed at runtime.
Old answer - more or less exactly the opposite of what was asked for, but I'll keep it here as reference.
Using of named parameter:
#Query("SELECT u.email FROM User AS u WHERE u.email NOT IN (:emails)")
List<String> findMissingEmails(#Param("emails")Collection<String> emails);
Alternatively, you could use a JPA query method:
#Repository
public interface UserRepository extends JpaRepository<User, Long> {
List<User> findAllByEmailNotIn(Collection<String> emails);
}
Unfortunately that method would fetch and return a list of Users instead of list of their emails.
To fetch just emails you could use a JPA projection.
Assuming that User entity has a field of type String named email, the following projection could be used:
public interface UserEmail {
String getEmail();
}
And this is the repository method:
#Repository
public interface UserRepository extends JpaRepository<User, Long> {
List<UserEmail> findAllByEmailNotIn(Collection<String> emails);
}

ERROR: operator does not exist: timestamp without time zone >= boolean Hint: No operator matches the given name and argument type(s)

Am trying to execute a Named Query using JPA. Query has to search and pull the records based on criteria of the query. Criteria is query should pull the records between specific given times(Records between From and To Dates provided) and name of the Application.
Query works fine, when executed in Postgresql. But, through JPA it gives error while executing the query
Here is my Query in PostgreSQL :
SELECT auditLog.busn_sys_id
, sourceSystem.busn_sys_full_nm
, auditLog.purge_ts
, auditLog.rec_purge_cnt
FROM rec_ret.rec_rtn_purge_adt auditLog
LEFT JOIN gbl_dm.gbl_busn_sys_dm sourceSystem on (auditLog.busn_sys_id = sourceSystem.busn_sys_id)
WHERE (auditLog.purge_ts BETWEEN '2019-08-19' AND '2019-08-25')
and sourceSystem.busn_sys_full_nm like 'PROFILE'
order by auditLog.busn_sys_id ;
Here is my JPA implementation to get Results List
try{
Query query = entityManager.createNativeQuery("SELECT auditLog.busn_sys_id, sourceSystem.busn_sys_full_nm, auditLog.purge_ts, auditLog.rec_purge_cnt " +
"FROM rec_ret.rec_rtn_purge_adt auditLog " +
"LEFT JOIN gbl_dm.gbl_busn_sys_dm sourceSystem on (auditLog.busn_sys_id = sourceSystem.busn_sys_id) " +
"WHERE (auditLog.purge_ts BETWEEN auditLog.purge_ts = :requestedFrom AND auditLog.purge_ts = :requestedTo) " +
//"WHERE (auditLog.purge_ts BETWEEN requestedFrom = (?) AND requestedTo = (?)) " +
"and sourceSystem.busn_sys_full_nm = :sourceSystem " +
"order by auditLog.busn_sys_id ");
query.setParameter("requestedFrom",auditLogCriteria.getFromDate());
query.setParameter("requestedTo",auditLogCriteria.getToDate());
query.setParameter("sourceSystem",auditLogCriteria.getSourceSystem());;
query.executeUpdate();
return query.getResultList();
}catch (Exception e){
logger.error("Fetching of logs failed with message : " + e.getMessage());
throw e;
}
Here is my implementation of Entity/Model class code
#NotNull (message = "Name of the Source System should be entered")
private String sourceSystem;
#NotNull (message = "Specify FromDate to filter records")
#JsonFormat(shape = JsonFormat.Shape.STRING, pattern="yyyy-MM-dd HH:mm:ss[.SSS]")
#Convert(converter = LocalDateTimeConverter.class)
private LocalDateTime fromDate;
#NotNull (message = "Specify ToDate to filter records")
#JsonFormat(shape = JsonFormat.Shape.STRING, pattern="yyyy-MM-dd HH:mm:ss[.SSS]")
#Convert(converter = LocalDateTimeConverter.class)
private LocalDateTime toDate;
\\ Getters and Setters
Here is the Error :
Hibernate: SELECT auditLog.busn_sys_id, sourceSystem.busn_sys_full_nm, auditLog.purge_ts, auditLog.rec_purge_cnt FROM rec_ret.rec_rtn_purge_adt auditLog LEFT JOIN gbl_dm.gbl_busn_sys_dm sourceSystem on (auditLog.busn_sys_id = sourceSystem.busn_sys_id) WHERE (auditLog.purge_ts BETWEEN auditLog.purge_ts = ? AND auditLog.purge_ts = ?) and sourceSystem.busn_sys_full_nm = ? order by auditLog.busn_sys_id
[ERROR] 2019-08-29 10:02:00.558 [http-nio-8080-exec-2] SqlExceptionHelper - ERROR: operator does not exist: timestamp without time zone >= boolean
Hint: No operator matches the given name and argument type(s). You might need to add explicit type casts.
Position: 267
javax.persistence.PersistenceException: org.hibernate.exception.SQLGrammarException: could not execute statement
at org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:154)
at org.hibernate.internal.ExceptionConverterImpl.convert(ExceptionConverterImpl.java:181)
at org.hibernate.query.internal.AbstractProducedQuery.executeUpdate(AbstractProducedQuery.java:1593)
So, result should be all the records matching with the criteria has to be returned as a response to JSON request
This part of your JPA query looks wrong:
BETWEEN auditLog.purge_ts = :requestedFrom AND auditLog.purge_ts = :requestedTo
I think it should be:
BETWEEN :requestedFrom AND :requestedTo

JPQL JOIN FETCH multiple tables also with empty collections

I am simply trying to get a whole entity with all attributes including empty collections, if they are empty.
public class Users extends BaseEntity {
...
#Column(name = "Prename")
private String prename;
#Column(name = "session")
private String session;
#JsonView({ View.Users.class })
#ManyToMany(cascade = { CascadeType.PERSIST, CascadeType.MERGE }, fetch = FetchType.LAZY)
#JoinTable(
name="USERS_PROJECTS",
joinColumns={#JoinColumn(name="USERS_ID", referencedColumnName="id")},
inverseJoinColumns={#JoinColumn(name="PROJECT_ID", referencedColumnName="id")})
private Set<Project> projects = new HashSet<>();
#JsonView({ View.Users.class })
#ManyToMany(mappedBy = "projectManager",
cascade = { CascadeType.PERSIST, CascadeType.MERGE },
fetch = FetchType.LAZY)
private Set<Project> managingProjects = new HashSet<>();
#JsonView({ View.Users.class })
#ManyToMany(mappedBy = "projectManager",
cascade = { CascadeType.PERSIST, CascadeType.MERGE },
fetch = FetchType.LAZY)
private Set<Project> watchingProjects = new HashSet<>();
...
}
The corresponding named query looks like this:
#NamedQueries({
#NamedQuery(
name = Users.QUERY_GET_ALL_USERS_ASC,
query = "SELECT u " +
"FROM Users u " +
"LEFT JOIN FETCH u.projects proj " +
"LEFT JOIN FETCH u.managingProjects manProj " +
"LEFT JOIN FETCH u.watchingProjects watProj " +
"ORDER BY u.email ASC")
})
This query returns about 350 objects whereas it should return only 17, probably because it is not "GROUPED".
#NamedQueries({
#NamedQuery(
name = Users.QUERY_GET_ALL_USERS_ASC,
query = "SELECT u " +
"FROM Users u " +
"JOIN FETCH u.projects proj " +
"JOIN FETCH u.managingProjects manProj " +
"JOIN FETCH u.watchingProjects watProj " +
"ORDER BY u.email ASC")
})
This query returns 0 objects, because not every object relates to a not empty watchingProjects or managingProjects collection.
#NamedQuery(
name = Users.QUERY_GET_ALL_USERS_ASC,
query = "SELECT u " +
"FROM Users u " +
"LEFT JOIN FETCH u.projects proj " +
"LEFT JOIN FETCH u.managingProjects manProj " +
"LEFT JOIN FETCH u.watchingProjects watProj " +
"GROUP BY u " // + ", manProj, watProj " + // does not change anything
"ORDER BY u.email ASC")
This query evokes the following exception:
Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.6.1.v20150605-31e8258): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: org.postgresql.util.PSQLException: ERROR: column "t1.id" must appear in the GROUP BY clause or be used in an aggregate function
Position: 99
Error Code: 0
Call: SELECT t0.ID, t0.CreateDate, t0.EMail, t0.Lastname, t0.Password, t0.Prename, t0.Role, t0.session, t1.ID, t1.Name, t1.Scheme, t2.ID, t2.Name, t2.Scheme, t3.ID, t3.Name, t3.Scheme FROM Users t0 LEFT OUTER JOIN (USERS_PROJECTS t6 JOIN PROJECT t1 ON (t1.ID = t6.PROJECT_ID)) ON (t6.USERS_ID = t0.ID) LEFT OUTER JOIN (PROJECTS_MANAGER t7 JOIN PROJECT t2 ON (t2.ID = t7.PROJECT_ID)) ON (t7.MANAGER_ID = t0.ID) LEFT OUTER JOIN (PROJECTS_MANAGER t8 JOIN PROJECT t3 ON (t3.ID = t8.PROJECT_ID)) ON (t8.MANAGER_ID = t0.ID) LEFT OUTER JOIN (PROJECTS_MANAGER t9 JOIN PROJECT t4 ON (t4.ID = t9.PROJECT_ID)) ON (t9.MANAGER_ID = t0.ID) LEFT OUTER JOIN (PROJECTS_MANAGER t10 JOIN PROJECT t5 ON (t5.ID = t10.PROJECT_ID)) ON (t10.MANAGER_ID = t0.ID) GROUP BY t0.ID, t0.CreateDate, t0.EMail, t0.Lastname, t0.Password, t0.Prename, t0.Role, t0.session, t4.ID, t4.Name, t4.Scheme, t5.ID, t5.Name, t5.Scheme ORDER BY t0.EMail ASC
Setting the FetchType to LAZY is not an option, because there are other queries which should not fetch these attributes. So the question is how is it possible to return all the entities with a JPQL query with possible empty collections?
I am using EclipseLink 2.6.2 and JPA 2.1.
How many objects are returned if you use SELECT u FROM Users u? I think you need to use 'DISTINCT'. i.e.
SELECT distinct u FROM Users u LEFT JOIN FETCH u.projects proj ...