I have a query which is executed from java application like this:
Query query = getEntityManager().createQuery(hql);
The query looks like this:
String hql = "select * from table a where round(column1, 3) = round(parameter, 3)";
Here column1 is of type Double. The value it holds is like 143.02856666. I need to retain the value as it is, but for some business logic just need to round and compare.
The initial database configured was H2 and this worked fine. Now the database has been changed to Postgres and this query now errors out.
ERROR: function round(double precision, integer) does not exist Hint: No function matches the given name and argument types. You might
need to add explicit type casts.
The round() function in Postgres takes a numeric datatype and needs a cast.
The below query works fine if executed directly in Postgres console.
select * from table a where round(cast(column1 as numeric), 3) = round(cast(parameter as numeric), 3);
The same from java application errors out.
java.lang.IllegalArgumentException: org.hibernate.QueryException: Could not resolve requested type for CAST : numeric
Also tried Query query = getEntityManager().createNativeQuery(hql);
This results in a new error.
org.hibernate.engine.jdbc.spi.SqlExceptionHelper - ERROR: syntax error at or near "where"
If I debug, this errors out when the below line is executed.
List resultList = query.getResultList();
How do I rewrite the query so that it works against Postgres ?
What you are doing with Query query = getEntityManager().createQuery(hql); is calling a jpql-query, which does not support all db-functions like round(v numeric, s integer).
Two Suggestions:
Use BETWEEN and maintain jpql-mapping
Write a NativeQuery -> Query query = em.createNativeQuery(queryString);
Your queryString just has to be altered by your parameters.
Related
For context, this issue occurred in a Go program I am writing using the default postgres database driver.
I have been building a service to talk to a postgres database which has a table similar to the one listed below:
CREATE TABLE object (
id SERIAL PRIMARY KEY NOT NULL,
name VARCHAR(255) UNIQUE,
some_other_id BIGINT UNIQUE
...
);
I have created some endpoints for this item including an "Install" endpoint which effectively acts as an upsert function like so:
INSERT INTO object (name, some_other_id)
VALUES ($1, $2)
ON CONFLICT name DO UPDATE SET
some_other_id = COALESCE(NULLIF($2, 0), object.some_other_id)
I also have an "Update" endpoint with an underlying query like so:
UPDATE object
SET some_other_id = COALESCE(NULLIF($2, 0), object.some_other_id)
WHERE name = $1
The problem:
Whenever I run the update query I always run into the error, referencing the field "some_other_id":
pq: value "1010101010144" is out of range for type integer
However this error never occurs on the "upsert" version of the query, even when the row already exists in the database (when it has to evaluate the COALESCE statement). I have been able to prevent this error by updating COALESCE statement to be as follows:
COALESCE(NULLIF($2, CAST(0 AS BIGINT)), object.some_other_id)
But as it never occurrs with the first query I wondered if this inconsitency had come from me doing something wrong or something that I don't understand? And also what the best practice is with this, should I be casting all values?
I am definitely passing in a 64 bit integer to the query for "some_other_id", and the first query works with the Go implementation even without the explicit type cast.
If any more information (or Go implementation) is required then please let me know, many thanks in advance! (:
Edit:
To eliminate confusion, the queries are being executed directly in Go code like so:
res, err := s.db.ExecContext(ctx, `UPDATE object SET some_other_id = COALESCE(NULLIF($2, 0), object.some_other_id) WHERE name = $1`,
"a name",
1010101010144,
)
Both queries are executed in exactly the same way.
Edit: Also corrected parameter (from $51 to $2) in my current workaround.
I would also like to take this opportunity to note that the query does work with my proposed fix, which suggests that the issue is in me confusing postgres with types in the NULLIF statement? There is no stored procedure asking for an INTEGER arg inbetween my code and the database, at least that I have written.
This has to do with how the postgres parser resolves types for the parameters. I don't know how exactly it's implemented, but given the observed behaviour, I would assume that the INSERT query doesn't fail because it is clear from (name,some_other_id) VALUES ($1,$2) that the $2 parameter should have the same type as the target some_other_id column, which is of type int8. This type information is then also used in the NULLIF expression of the DO UPDATE SET part of the query.
You can also test this assumption by using (name) VALUES ($1) in the INSERT and you'll see that the NULLIF expression in DO UPDATE SET will then fail the same way as it does in the UPDATE query.
So the UPDATE query fails because there is not enough context for the parser to infer the accurate type of the $2 parameter. The "closest" thing that the parser can use to infer the type of $2 is the NULLIF call expression, specifically it uses the type of the second argument of the call expression, i.e. 0, which is of type int4, and it then uses that type information for the first argument, i.e. $2.
To avoid this issue, you should use an explicit type cast with any parameter where the type cannot be inferred accurately. i.e. use NULLIF($2::int8, 0).
COALESCE(NULLIF($51, CAST(0 AS BIGINT)), object.some_other_id)
Fifty-one? Realy?
pq: value "1010101010144" is out of range for type integer
Pay attention, the data type in the error message is an integer, not bigint.
I think the reason for the error is out of showed code. So I take out a magic crystal ball and make a pass with my hands.
an "Install" endpoint which effectively acts as an upsert function like so
I also have an "Update" endpoint
Do you call endpoint a PostgreSQL function (stored procedure)? I think yes.
Also $1, $2 looks like PostgreSQL function arguments.
The magic crystal ball says: you have two PostgreSQL function with different data types of arguments:
"Install" endpoint has $2 function argument as a bigint data type. It looks like CREATE FUNCTION Install(VARCHAR(255), bigint)
"Update" endpoint has $2 function argument as an integer data type, not bigint. It looks like CREATE FUNCTION Update(VARCHAR(255), integer).
At last, I would rewrite your condition more understandable:
UPDATE object
SET some_other_id =
CASE
WHEN $2 = 0 THEN object.some_other_id
ELSE $2
END
WHERE name = $1
I am working with a Java application which uses JPA and a Postgres database, and I am trying to create a flexible prepared statement which can handle a variable number of input parameters. An example query would best explain this:
SELECT *
FROM my_table
WHERE
(string_col = :param1 OR :param1 IS NULL) AND
(double_col = :param2 OR :param2 IS NULL);
The idea behind this "trick" is that if a user specifies only one parameter, say :param1, we can just bind null to :param2, and the WHERE clause would then behave as if only the first parameter were even being checked. This approach lets us handle, in theory, any number of input parameters using a single prepared statement, instead of needing to maintain many different statements.
I have gotten a simple POC working locally using pure JDBC prepared statements. However, doing so required casting the parameter before comparing it to NULL, e.g.
WHERE (double_col = ? OR ?::numeric IS NULL)
^^ does not work without cast
However, my actual application is using JPA, and I keep getting the following persistent error:
Caused by: org.postgresql.util.PSQLException: ERROR: operator does not exist: double precision = bytea
Hint: No operator matches the given name and argument type(s). You might need to add explicit type casts.
The problem does not occur with string/text columns, but only with columns which are double precision in my Postgres table. I have tried all combinations of casting, and nothing works:
(double_col = :param2 OR CAST(:param2 AS double precision) IS NULL);
(CAST(double_col AS double precision) = :param2 OR :param2 IS NULL);
(CAST(double_col AS double precision) = :param2 OR CAST(:param2 AS double precision) IS NULL);
The error seems to be saying that JDBC is sending Postgres a bytea type for the double columns, and then Postgres is rolling over because it can't find a way to cast byte to double precision.
The Java code looks something like:
Query query = entityManager.createNativeQuery(sqlString, MyEntity.class);
query.setParameter("param1", "some value");
// bind other parameters here
List<MyEntity> = query.getResultList();
For reference, here are the versions of everything I am using:
Hibernate version | 4.3.7.Final
Spring data JPA vesion | 1.7.1.RELEASE
Postgres driver version | 42.2.2
Postgres database version | 9.6.10
Java version | 1.8.0_171
Not having received any feedback in the form of answers or even a comment, I was getting ready to give up, when I stumbled onto this excellent blog post:
How to bind custom Hibernate parameter types to JPA queries
The post gives two options for controlling the types which JPA passes through the driver to Postgres (or whatever the underlying database actually is). I went with the approach using TypedParameterValue. Here is what my code looks like continuing with the example given above:
Query query = entityManager.createNativeQuery(sqlString, MyEntity.class);
query.setParameter("param1", new TypedParameterValue(StringType.INSTANCE, null));
query.setParameter("param2", new TypedParameterValue(DoubleType.INSTANCE, null));
List<MyEntity> = query.getResultList();
Of course, it is trivial to be passing null for every parameter in the query, but I am doing this mainly to show the syntax for the text and double columns. In practice, we would expect at least a few of the parameters to be non null, but the above syntax handles all values, null or otherwise.
If you want to keep using plain queries with automatic parameter binding, you could try the following.
WHERE (? IS NULL OR (CAST(CAST(? AS TEXT) AS DOUBLE PRECISION) = double_col
This seems to satisfy the PostgreSQL driver's type checks as well as yielding the correct results. I haven't done much testing, but the performance hit seems minimal because the CASTs happen on a constant value rather than rows from the database.
I tried to use postgreSQL stored function in ssrs, firstly tried using like this :
select *
from reports.get_daily_cash_flow_test('01.01.2017'::timestamp with time zone,
'01.01.2019'::timestamp with time zone, True, Offset)
WHERE Offset = #Offset
It causes syntax error, near #Offset. #Offset is integer type variable, added in Parameters, and also added in dataset parameters (also tried using query like :
select *
from reports.get_daily_cash_flow_test('01.01.2017'::timestamp with time zone,
'01.01.2019'::timestamp with time zone, True, #Offset)
Just because #Offset is already added in dataset parameters but still syntax error.
After that I tried using function as stored procedure. Still the problem is that my function returns Table, and in dataset parameters it requires me to add output parameters (returned table columns) values.
After that tried using simple query without stored function which looks like that:
SELECT COUNT(DISTINCT (l.player_id))
FROM public.logins l
WHERE (l.login_date between #DateFrom AND #DateTo)
Error message is
An error occurred while executing the query.
ERROR [42703] ERROR: column "datefrom" does not exist;
Error while executing the query
And I found the solution, once executing query you should use it like :
SELECT * FROM reports.get_devices(?,?,?)
question marks instead of parameters, that does the trick.
I am using
length(ze.string)>2 in openJpa query. but i am getting
SQLCODE=-440, SQLSTATE=42884, SQLERRMC=CHAR_LENGTH;FUNCTION, DRIVER=3.53.95 {prepstmnt 1776269692 SELECT t0.f1, t0.f2, t0.f3, t0.f4, t0.f5, t0.f6, t0.f7, t0.f8, t0.f9, t0.f10, t0.f11, t0.f12, t0.f13, t0.f14, t0.f15, t0.f16, t0.f17 FROM table t0 WHERE (t0.f1 = ? AND CHAR_LENGTH(?) > ? AND .....
In plain query when i do length operation i am getting record but using jpa its not working. I looked Here used size it doesn't work. and the field is varchar and db2. trying from past 1 hour.
DB2 requires use of the SQL function LENGTH, yet OpenJPA seems to be incorrectly converting your JPQL to use SQL function CHAR_LENGTH (hence the error message - not that DB2 gives out clear messages saying what is wrong, who knows what SQLCODE=-440 is without having to search!!).
Raise a bug on your JPA provider.
See https://www.ibm.com/support/knowledgecenter/SSEPGG_9.7.0/com.ibm.db2.luw.sql.ref.doc/doc/r0000818.html
You would need to give more details about your entity, persistence.xml, and query to get to the bottom or this. However, I do not see how OpenJPA would use CHAR_LENGTH instead of LENGTH for DB2. Let me explain. If you look at DBDictionary here:
https://svn.apache.org/viewvc/openjpa/branches/2.2.x/openjpa-jdbc/src/main/java/org/apache/openjpa/jdbc/sql/DBDictionary.java?view=markup
You can see it defines something called "stringLengthFunction" as follows:
public String stringLengthFunction = "CHAR_LENGTH({0})";
This is the string length function which should be used for each individual dictionary (i.e. Database config). However, for DB2, the AbstractDB2Dictionary, see here:
https://svn.apache.org/viewvc/openjpa/branches/2.2.x/openjpa-jdbc/src/main/java/org/apache/openjpa/jdbc/sql/AbstractDB2Dictionary.java?view=markup
overrides this as follows:
stringLengthFunction = "LENGTH({0})";
Given this, for DB2, LENGTH should be used. I took the following simple query:
"select me.id from MyEntity me where length(me.name)>2"
And executed it on OpenJPA using DB2, and I got this:
SELECT t0.ID FROM MYENTITY t0 WHERE (CAST(LENGTH(t0.ID) AS BIGINT) > CAST(? AS BIGINT)) [params=(long) 2]
Thanks,
Heath Thomann
I have a string that I want to pass to SQL. To prevent SQL injection and other quoting and escaping problems, the best practice is to use a PreparedStatement with ?. For example:
val ps = conn.prepareStatement("select * from foo where name = ?")
ps.setString(1, name)
But for some SQL code, this won't work. For example, here is PostgreSQL, trying to create a view.
val ps = conn.prepareStatement("create temp view v1 as select * from foo where name = ?")
ps.setString(1, name)
val rs = ps.execute()
This throws an error:
org.postgresql.util.PSQLException: ERROR: there is no parameter $1
It apparently doesn't allow parameters to create view. How do you get around this and safely escape the string?
Prepared statements are used to plan a complex statement once and then execute it multiple (= very many) times with different parameter values. Simple statements have no noticeable benefit from using a prepared statement because planning them is trivial. DDL statements are not supported at all, so that is most likely the cause of the error, although the error message is confusing.
From the documentation:
PREPARE name [ ( data_type [, ...] ) ] AS statement
statement
Any SELECT, INSERT, UPDATE, DELETE, or VALUES statement.
The PreparedStatement class does document that you can use DDL in the executeUpdate() method, but from a logical standpoint that is just nonsense, at least in PostgreSQL.
Instead, you should use a Statement and then call execute() or executeUpdate() (a bit odd that the latter method would support DDL statements because there is no update being performed).
Preventing SQL-injection
In order to prevent SQL-injection you can use a few PostgreSQL functions:
quote_literal() - As can be expected, this will quote a literal parameter value to be safe in the query. Not only does this prevent you from Bobby Tables, but also from the likes of Mr. O'Brien.
quote_nullable() - For literals like above, but will generate proper code when the parameter IS NULL.
quote_identifier() - Will double quote any table or column name that might cause problems for the planner, like table name from with columns int, type and from: SELECT int, type, from FROM from WHERE int = type becomes SELECT "int", "type", "from" FROM "from" WHERE "int" = "type".
You can use these functions directly in your SQL statements and then let PostgreSQL deal with nasty input.