I am using Postgresql together with HikariCP and my query is something like
SELECT * FROM my_table WHERE int_val = ? ...
Now, I would like to set NULL value to my variables - I have tried
ps.setNull(1, Types.INTEGER); // ps is instance of PreparedStatement
try (ResultSet rs = ps.executeQuery()) {
... // get result from resultset
}
Although I have rows matching the conditions ( NULL in column 'int_val'), I have not received any records..
The problem is (I think) in query produced by the Statement, looks like:
System.out.println(ps.toString());
// --> SELECT * FROM my_table WHERE int_val = NULL ...
But the query should look like:
"SELECT * FROM my_table WHERE int_val IS NULL ..." - this query works
I need to use dynamically create PreparedStatements which will contain NULL values, so I cannot somehow easily bypass this.
I have tried creating connection without the HikariCP with the same result, so I thing the problem is in the postgresql driver? Or am I doing something wrong?
UPDATE:
Based on answer from #Vao Tsun I have set transform_null_equals = on in postgresql.conf , which started changing val = null --> val is null in 'simple' Statements, but NOT in PreparedStatements..
To summarize:
try (ResultSet rs = st.executeQuery(SELECT * FROM my_table WHERE int_val = NULL)){
// query is replaced to '.. int_val IS NULL ..' and gets correct result
}
ps.setNull(1, Types.INTEGER);
try (ResultSet rs = ps.executeQuery()) {
// Does not get replaced and does not get any result
}
I am using JVM version 1.8.0_121, the latest postgres driver (42.1.4), but I have also tried older driver (9.4.1212). Database version -- PostgreSQL 9.6.2, compiled by Visual C++ build 1800, 64-bit.
It is meant behaviour that comparison x = null is equal to null (no matter what x is equal to). Basically for SQL NULL is unknown, not the actual value... To bypass it you can set transform_null_equals to on or true. Please checkout docs:
https://www.postgresql.org/docs/current/static/functions-comparison.html
Some applications might expect that expression = NULL returns true if
expression evaluates to the null value. It is highly recommended that
these applications be modified to comply with the SQL standard.
However, if that cannot be done the transform_null_equals
configuration variable is available. If it is enabled, PostgreSQL will
convert x = NULL clauses to x IS NULL.
I have just found a solution, which works the same for "values" and "NULLs" by using IS NOT DISTINCT FROM instead of =.
More on postgresql wiki
It is important to recognize that null is not a value with SQL. It is encoding the logical notion of "unknown". This is why null = var results in false always, even for cases where var has a value of null. So even if if you are replacing the value of your variable (aka ? in your case) with a value of null, the result be definition must not be what you do expect as long as SQL standard is complied with.
Now there are some databases around that try to outsmart SQL standard by assuming a column value of null should be taken as a programming language null (nil, undef or whatever is used for that purpose).
This creates some convenience for the unwary programmer, but in the long run causes grieve as soon as you need a true distinction between a SQL null and a programming language null.
Nevertheless, for ease of porting from such databases to PostgresQL (or simple for ease of lazy programming) you may resort to setting transform_null_equals.
BUT, you are using prepared statements. As such, prepared statements are converted to query plan once and such query plan needs to be valid for all potential values of the variables used in the prepared statement query. Now, a VAR is null is fundamentally different from a VAR = ?. So there is no chance for the query parser, query optimizer or even query execution engine to dynamically rewrite the (already prepared) query based on the actual parameter values passed in.
From this, you should take the recommendation serious that is given with the documentation of transform_null_equals and change your code to use VAR is null when a null value is to be searched for and a VAR = ? for other cases.
Related
For context, this issue occurred in a Go program I am writing using the default postgres database driver.
I have been building a service to talk to a postgres database which has a table similar to the one listed below:
CREATE TABLE object (
id SERIAL PRIMARY KEY NOT NULL,
name VARCHAR(255) UNIQUE,
some_other_id BIGINT UNIQUE
...
);
I have created some endpoints for this item including an "Install" endpoint which effectively acts as an upsert function like so:
INSERT INTO object (name, some_other_id)
VALUES ($1, $2)
ON CONFLICT name DO UPDATE SET
some_other_id = COALESCE(NULLIF($2, 0), object.some_other_id)
I also have an "Update" endpoint with an underlying query like so:
UPDATE object
SET some_other_id = COALESCE(NULLIF($2, 0), object.some_other_id)
WHERE name = $1
The problem:
Whenever I run the update query I always run into the error, referencing the field "some_other_id":
pq: value "1010101010144" is out of range for type integer
However this error never occurs on the "upsert" version of the query, even when the row already exists in the database (when it has to evaluate the COALESCE statement). I have been able to prevent this error by updating COALESCE statement to be as follows:
COALESCE(NULLIF($2, CAST(0 AS BIGINT)), object.some_other_id)
But as it never occurrs with the first query I wondered if this inconsitency had come from me doing something wrong or something that I don't understand? And also what the best practice is with this, should I be casting all values?
I am definitely passing in a 64 bit integer to the query for "some_other_id", and the first query works with the Go implementation even without the explicit type cast.
If any more information (or Go implementation) is required then please let me know, many thanks in advance! (:
Edit:
To eliminate confusion, the queries are being executed directly in Go code like so:
res, err := s.db.ExecContext(ctx, `UPDATE object SET some_other_id = COALESCE(NULLIF($2, 0), object.some_other_id) WHERE name = $1`,
"a name",
1010101010144,
)
Both queries are executed in exactly the same way.
Edit: Also corrected parameter (from $51 to $2) in my current workaround.
I would also like to take this opportunity to note that the query does work with my proposed fix, which suggests that the issue is in me confusing postgres with types in the NULLIF statement? There is no stored procedure asking for an INTEGER arg inbetween my code and the database, at least that I have written.
This has to do with how the postgres parser resolves types for the parameters. I don't know how exactly it's implemented, but given the observed behaviour, I would assume that the INSERT query doesn't fail because it is clear from (name,some_other_id) VALUES ($1,$2) that the $2 parameter should have the same type as the target some_other_id column, which is of type int8. This type information is then also used in the NULLIF expression of the DO UPDATE SET part of the query.
You can also test this assumption by using (name) VALUES ($1) in the INSERT and you'll see that the NULLIF expression in DO UPDATE SET will then fail the same way as it does in the UPDATE query.
So the UPDATE query fails because there is not enough context for the parser to infer the accurate type of the $2 parameter. The "closest" thing that the parser can use to infer the type of $2 is the NULLIF call expression, specifically it uses the type of the second argument of the call expression, i.e. 0, which is of type int4, and it then uses that type information for the first argument, i.e. $2.
To avoid this issue, you should use an explicit type cast with any parameter where the type cannot be inferred accurately. i.e. use NULLIF($2::int8, 0).
COALESCE(NULLIF($51, CAST(0 AS BIGINT)), object.some_other_id)
Fifty-one? Realy?
pq: value "1010101010144" is out of range for type integer
Pay attention, the data type in the error message is an integer, not bigint.
I think the reason for the error is out of showed code. So I take out a magic crystal ball and make a pass with my hands.
an "Install" endpoint which effectively acts as an upsert function like so
I also have an "Update" endpoint
Do you call endpoint a PostgreSQL function (stored procedure)? I think yes.
Also $1, $2 looks like PostgreSQL function arguments.
The magic crystal ball says: you have two PostgreSQL function with different data types of arguments:
"Install" endpoint has $2 function argument as a bigint data type. It looks like CREATE FUNCTION Install(VARCHAR(255), bigint)
"Update" endpoint has $2 function argument as an integer data type, not bigint. It looks like CREATE FUNCTION Update(VARCHAR(255), integer).
At last, I would rewrite your condition more understandable:
UPDATE object
SET some_other_id =
CASE
WHEN $2 = 0 THEN object.some_other_id
ELSE $2
END
WHERE name = $1
I am working with a Java application which uses JPA and a Postgres database, and I am trying to create a flexible prepared statement which can handle a variable number of input parameters. An example query would best explain this:
SELECT *
FROM my_table
WHERE
(string_col = :param1 OR :param1 IS NULL) AND
(double_col = :param2 OR :param2 IS NULL);
The idea behind this "trick" is that if a user specifies only one parameter, say :param1, we can just bind null to :param2, and the WHERE clause would then behave as if only the first parameter were even being checked. This approach lets us handle, in theory, any number of input parameters using a single prepared statement, instead of needing to maintain many different statements.
I have gotten a simple POC working locally using pure JDBC prepared statements. However, doing so required casting the parameter before comparing it to NULL, e.g.
WHERE (double_col = ? OR ?::numeric IS NULL)
^^ does not work without cast
However, my actual application is using JPA, and I keep getting the following persistent error:
Caused by: org.postgresql.util.PSQLException: ERROR: operator does not exist: double precision = bytea
Hint: No operator matches the given name and argument type(s). You might need to add explicit type casts.
The problem does not occur with string/text columns, but only with columns which are double precision in my Postgres table. I have tried all combinations of casting, and nothing works:
(double_col = :param2 OR CAST(:param2 AS double precision) IS NULL);
(CAST(double_col AS double precision) = :param2 OR :param2 IS NULL);
(CAST(double_col AS double precision) = :param2 OR CAST(:param2 AS double precision) IS NULL);
The error seems to be saying that JDBC is sending Postgres a bytea type for the double columns, and then Postgres is rolling over because it can't find a way to cast byte to double precision.
The Java code looks something like:
Query query = entityManager.createNativeQuery(sqlString, MyEntity.class);
query.setParameter("param1", "some value");
// bind other parameters here
List<MyEntity> = query.getResultList();
For reference, here are the versions of everything I am using:
Hibernate version | 4.3.7.Final
Spring data JPA vesion | 1.7.1.RELEASE
Postgres driver version | 42.2.2
Postgres database version | 9.6.10
Java version | 1.8.0_171
Not having received any feedback in the form of answers or even a comment, I was getting ready to give up, when I stumbled onto this excellent blog post:
How to bind custom Hibernate parameter types to JPA queries
The post gives two options for controlling the types which JPA passes through the driver to Postgres (or whatever the underlying database actually is). I went with the approach using TypedParameterValue. Here is what my code looks like continuing with the example given above:
Query query = entityManager.createNativeQuery(sqlString, MyEntity.class);
query.setParameter("param1", new TypedParameterValue(StringType.INSTANCE, null));
query.setParameter("param2", new TypedParameterValue(DoubleType.INSTANCE, null));
List<MyEntity> = query.getResultList();
Of course, it is trivial to be passing null for every parameter in the query, but I am doing this mainly to show the syntax for the text and double columns. In practice, we would expect at least a few of the parameters to be non null, but the above syntax handles all values, null or otherwise.
If you want to keep using plain queries with automatic parameter binding, you could try the following.
WHERE (? IS NULL OR (CAST(CAST(? AS TEXT) AS DOUBLE PRECISION) = double_col
This seems to satisfy the PostgreSQL driver's type checks as well as yielding the correct results. I haven't done much testing, but the performance hit seems minimal because the CASTs happen on a constant value rather than rows from the database.
I'm using a PostgreSQL ODBC data source in an SSRS report.
From my previous knowledge with MS SQL Server data sources, to pass an optional parameter to a SSRS dataset query, you would do this:
SELECT *
FROM MyTable
WHERE SomeField = #Param1 OR #Param1 IS NULL
The key was using OR #Param1 IS NULL which basically bypasses the SomeField = #Param1 part and effectively acts as an "optional" filter. However, this doesn't seem to work when using a PostgreSQL ODBC data source (not sure if it's due to Postgres or ODBC or both). I get an error when clicking on Refresh Fields, or when executing/previewing the report.
This might be partly due to not being able to pass named parameters into the SQL, but instead having to use "?" like so:
SELECT *
FROM MyTable
WHERE SomeField = ?
...which makes it impossible to do OR ? IS NULL basically.
So, anybody know how I might be able to pass optional parameters to a PostgreSQL data set in SSRS?
Not a specific ODBC solution but a workaround. This will work if SomeField can not be null:
select *
from mytable
where somefield = coalesce(?, somefield)
If SomeField can be null and you want to return nulls:
where somefield is not distinct from coalesce(?, somefield)
Check is distinct from
I'm not using a full on DB Abstraction library, and am using raw sql templates in psycopg2 that look like this :
SELECT id FROM table WHERE message = %(message)s ;
The ideal query to retrieve my intended results looks something like this :
SELECT id FROM table WHERE message = 'a3cbb207' ;
SELECT id FROM table WHERE message IS NULL ;
Unfortunately... the obvious problem is that my NULL comparisons come out looking like this:
SELECT id FROM table WHERE message = NULL ;
... which is not the correct comparison - and doesn't give me the intended result set.
My actual queries are much more complex than the illustration above - so I can't change them easily. ( which would be the correct solution , i agree. i'm looking for an emergency fix right now )
Does anyone know of a workaround , so I can keep the same singular templates going until a proper fix is in place ? I was trying to get coalesce and/or cast to work , but I struck out with my attempts.
What you want is IS NOT DISTINCT FROM.
SELECT id FROM table WHERE message IS NOT DISTINCT FROM 'the text';
SELECT id FROM table WHERE message IS NOT DISTINCT FROM NULL;
NULL IS NOT DISTINCT FROM NULL is true, not NULL, so it's like = but with different NULL comparison semantics. Great in trigger functions.
AFAIK can't use IS DISTINCT FROM for index lookups though, so be careful there. It can be better to use separate tests for null and value.
You can try writing your query clause as follows:
WHERE message = %(message)s OR ((%message)s IS NULL AND message IS NULL))
It's a bit rough, but it means "select the message that match my parameter, or all the messages that are null if my parameter is null". It should do the trick.
Unfortunately, NULL does not actually equal anything (not even another NULL) as the value of NULL is intended to represent an unknown. Your best bet is to change your templates to handle this correctly.
If it's possible that you can pass in separate values for the left and right operand in your template, one way to still use an equal sign would be:
SELECT id FROM table WHERE true = (message is null);
I'm trying to construct a T-SQL statement with a WHERE clause determined by an input parameter. Something like:
SELECT * FROM table
WHERE id IN
CASE WHEN #param THEN
(1,2,4,5,8)
ELSE
(9,7,3)
END
I've tried all combination of moving the IN, CASE etc around that I can think of. Is this (or something like it) possible?
try this:
SELECT * FROM table
WHERE (#param='??' AND id IN (1,2,4,5,8))
OR (#param!='??' AND id in (9,7,3))
this will have a problem using an index.
The key with a dynamic search conditions is to make sure an index is used, instead of how can I easily reuse code, eliminate duplications in a query, or try to do everything with the same query. Here is a very comprehensive article on how to handle this topic:
Dynamic Search Conditions in T-SQL by Erland Sommarskog
It covers all the issues and methods of trying to write queries with multiple optional search conditions. This main thing you need to be concerned with is not the duplication of code, but the use of an index. If your query fails to use an index, it will preform poorly. There are several techniques that can be used, which may or may not allow an index to be used.
here is the table of contents:
Introduction
The Case Study: Searching Orders
The Northgale Database
Dynamic SQL
Introduction
Using sp_executesql
Using the CLR
Using EXEC()
When Caching Is Not Really What You Want
Static SQL
Introduction
x = #x OR #x IS NULL
Using IF statements
Umachandar's Bag of Tricks
Using Temp Tables
x = #x AND #x IS NOT NULL
Handling Complex Conditions
Hybrid Solutions – Using both Static and Dynamic SQL
Using Views
Using Inline Table Functions
Conclusion
Feedback and Acknowledgements
Revision History
if you are on the proper version of SQL Server 2008, there is an additional technique that can be used, see: Dynamic Search Conditions in T-SQL Version for SQL 2008 (SP1 CU5 and later)
If you are on that proper release of SQL Server 2008, you can just add OPTION (RECOMPILE) to the query and the local variable's value at run time is used for the optimizations.
Consider this, OPTION (RECOMPILE) will take this code (where no index can be used with this mess of ORs):
WHERE
(#search1 IS NULL or Column1=#Search1)
AND (#search2 IS NULL or Column2=#Search2)
AND (#search3 IS NULL or Column3=#Search3)
and optimize it at run time to be (provided that only #Search2 was passed in with a value):
WHERE
Column2=#Search2
and an index can be used (if you have one defined on Column2)
if #param = 'whatever'
select * from tbl where id in (1,2,4,5,8)
else
select * from tbl where id in (9,7,3)