Postgres: getting "... is out of range for type integer" when using NULLIF - postgresql

For context, this issue occurred in a Go program I am writing using the default postgres database driver.
I have been building a service to talk to a postgres database which has a table similar to the one listed below:
CREATE TABLE object (
id SERIAL PRIMARY KEY NOT NULL,
name VARCHAR(255) UNIQUE,
some_other_id BIGINT UNIQUE
...
);
I have created some endpoints for this item including an "Install" endpoint which effectively acts as an upsert function like so:
INSERT INTO object (name, some_other_id)
VALUES ($1, $2)
ON CONFLICT name DO UPDATE SET
some_other_id = COALESCE(NULLIF($2, 0), object.some_other_id)
I also have an "Update" endpoint with an underlying query like so:
UPDATE object
SET some_other_id = COALESCE(NULLIF($2, 0), object.some_other_id)
WHERE name = $1
The problem:
Whenever I run the update query I always run into the error, referencing the field "some_other_id":
pq: value "1010101010144" is out of range for type integer
However this error never occurs on the "upsert" version of the query, even when the row already exists in the database (when it has to evaluate the COALESCE statement). I have been able to prevent this error by updating COALESCE statement to be as follows:
COALESCE(NULLIF($2, CAST(0 AS BIGINT)), object.some_other_id)
But as it never occurrs with the first query I wondered if this inconsitency had come from me doing something wrong or something that I don't understand? And also what the best practice is with this, should I be casting all values?
I am definitely passing in a 64 bit integer to the query for "some_other_id", and the first query works with the Go implementation even without the explicit type cast.
If any more information (or Go implementation) is required then please let me know, many thanks in advance! (:
Edit:
To eliminate confusion, the queries are being executed directly in Go code like so:
res, err := s.db.ExecContext(ctx, `UPDATE object SET some_other_id = COALESCE(NULLIF($2, 0), object.some_other_id) WHERE name = $1`,
"a name",
1010101010144,
)
Both queries are executed in exactly the same way.
Edit: Also corrected parameter (from $51 to $2) in my current workaround.
I would also like to take this opportunity to note that the query does work with my proposed fix, which suggests that the issue is in me confusing postgres with types in the NULLIF statement? There is no stored procedure asking for an INTEGER arg inbetween my code and the database, at least that I have written.

This has to do with how the postgres parser resolves types for the parameters. I don't know how exactly it's implemented, but given the observed behaviour, I would assume that the INSERT query doesn't fail because it is clear from (name,some_other_id) VALUES ($1,$2) that the $2 parameter should have the same type as the target some_other_id column, which is of type int8. This type information is then also used in the NULLIF expression of the DO UPDATE SET part of the query.
You can also test this assumption by using (name) VALUES ($1) in the INSERT and you'll see that the NULLIF expression in DO UPDATE SET will then fail the same way as it does in the UPDATE query.
So the UPDATE query fails because there is not enough context for the parser to infer the accurate type of the $2 parameter. The "closest" thing that the parser can use to infer the type of $2 is the NULLIF call expression, specifically it uses the type of the second argument of the call expression, i.e. 0, which is of type int4, and it then uses that type information for the first argument, i.e. $2.
To avoid this issue, you should use an explicit type cast with any parameter where the type cannot be inferred accurately. i.e. use NULLIF($2::int8, 0).

COALESCE(NULLIF($51, CAST(0 AS BIGINT)), object.some_other_id)
Fifty-one? Realy?
pq: value "1010101010144" is out of range for type integer
Pay attention, the data type in the error message is an integer, not bigint.
I think the reason for the error is out of showed code. So I take out a magic crystal ball and make a pass with my hands.
an "Install" endpoint which effectively acts as an upsert function like so
I also have an "Update" endpoint
Do you call endpoint a PostgreSQL function (stored procedure)? I think yes.
Also $1, $2 looks like PostgreSQL function arguments.
The magic crystal ball says: you have two PostgreSQL function with different data types of arguments:
"Install" endpoint has $2 function argument as a bigint data type. It looks like CREATE FUNCTION Install(VARCHAR(255), bigint)
"Update" endpoint has $2 function argument as an integer data type, not bigint. It looks like CREATE FUNCTION Update(VARCHAR(255), integer).
At last, I would rewrite your condition more understandable:
UPDATE object
SET some_other_id =
CASE
WHEN $2 = 0 THEN object.some_other_id
ELSE $2
END
WHERE name = $1

Related

How to use sub-queries correctly inside a Postgresql query

I'm having troubles resetting the sequences as automatically as possible.
I'm trying to use the next query from phpPgAdmin:
SELECT SETVAL('course_subjects_seq', (SELECT MAX(subject_id) FROM course_subjects));
Somehow this query returns:
> HINT: No function matches the given name and argument types. You might need to add explicit type casts.
pointing to the first SELECT SETVAL
The next query will give the same error:
SELECT setval("course_subjects_seq", COALESCE((SELECT MAX(subject_id) FROM course_subjects), 1))
Can anyone point me to what am I doing wrong?
Fixed by doing so:
setval function requires regclass, bigint and boolean as arguments, therefore I added the type casts:
SELECT setval('course_subjects_seq'::regclass, COALESCE((SELECT MAX(subject_id) FROM course_subjects)::bigint, 1));
::regclass
and ::bigint
You don't need a subquery at all here. Can be a single SELECT:
SELECT setval(pg_get_serial_sequence('course_subjects', 'subject_id')
, COALESCE(max(subject_id) + 1, 1)
, false) -- not called yet
FROM course_subjects;
Assuming subject_id is a serial column, pg_get_serial_sequence() is useful so you don't have to know the sequence name (which is an implementation detail, really).
SELECT with an aggregate function like max() always returns a single row, even if the underlying table has no rows. The value is NULL in this case, that's why you have COALESCE in there.
But if you call setval() with 1, the next sequence returned number will be 2, not 1, since that is considered to be called already. There is an overloaded variant of setval() with a 3rd, boolean parameter: is_called, which makes it possible to actually start from 1 in this case like demonstrated.
Related:
How to reset postgres' primary key sequence when it falls out of sync?

DB2: invalid use of one of the following: an untyped parameter marker, the DEFAULT keyword, or a null value

A user can only modify the ST_ASSMT_NM and , CAN_DT columns in the ST_ASSMT_REF record. In our system, we keep history in the same table and we never really update a record, we just insert a new row to represent the updated record. As a result, the "active" record is the record with the greatest LAST_TS timestamp value for a VENDR_ID. To prevent the possibility of an update to columns that cannot be changed, I wrote the logical UPDATE so that it retrieves the non-changable values from the original record and copies them to the new one being created. For the fields that can be modified, I pass them as params,
INSERT INTO GSAS.ST_ASSMT_REF
(
VENDR_ID
,ST_ASSMT_NM
,ST_CD
,EFF_DT
,CAN_DT
,LAST_TS
,LAST_OPER_ID
)
SELECT
ORIG_ST_ASSMT_REF.VENDR_ID
,#ST_ASSMT_NM
,ORIG_ST_ASSMT_REF.ST_CD
,ORIG_ST_ASSMT_REF.EFF_DT
,#CAN_DT
,CURRENT TIMESTAMP
,#LAST_OPER_ID
FROM
(
SELECT
ST_ASSMT_REF_ACTIVE_V.VENDR_ID
,ST_ASSMT_REF_ACTIVE_V.ST_ASSMT_NM
,ST_ASSMT_REF_ACTIVE_V.ST_CD
,ST_ASSMT_REF_ACTIVE_V.EFF_DT
,ST_ASSMT_REF_ACTIVE_V.CAN_DT
,CURRENT TIMESTAMP
,ST_ASSMT_REF_ACTIVE_V.LAST_OPER_ID
FROM
G2YF.ST_ASSMT_REF_ACTIVE_V ST_ASSMT_REF_ACTIVE_V --The view of only the most recent, active records
WHERE
ST_ASSMT_REF_ACTIVE_V.VENDR_ID = #VENDR_ID
) ORIG_ST_ASSMT_REF;
However, I am getting this error:
DB2 SP
:
ERROR [42610] [IBM][DB2] SQL0418N The statement was not processed because the statement contains an invalid use of one of the following: an untyped parameter marker, the DEFAULT keyword, or a null value.
It appears as though DB2 will not allow me to use a variable in a SELECT statement. For example, when I do this in TOAD for DB2:
select 1, #vendorId from SYSIBM.SYSDUMMY1
I get a popup dialog box. When I provide any string value, I get the same error.
I usually use SQL Server and I'm pretty sure I wouldn't have an issue doing this but I am not sure how to handle it get.
Suggestions? I know that I could do this in two seperate commands, 1 query SELECT to retreive the original VALUES and then supply the returned values and the modified ones to the INSERT command, but I should be able to do thios in one. Why can't I?
As you mentioned in your comment, DB2 is really picky about data types, and it wants you to cast your variables into the right data types. Even if you are passing in NULLs, sometimes DB2 wants you to cast the NULL to the data type of the target column.
Here is another answer I have on the topic.

PostgreSQL function call

I have PostgreSQL function named test(integer) taking an integer parameter and an overloaded function of the same name test(character varying).
When calling this function with a null value, Postgres always executes the function taking an integer parameter. Why does this happen? Why doesn't Postgres chose the function with a varchar parameter?
Function call example:
select test(null);
That's decided by the rules of Function Type Resolution. Detailed explanation in the manual. Related:
Is there a way to disable function overloading in Postgres
NULL without explicit type cast starts out as type "unknown":
SELECT pg_typeof(NULL)
pg_typeof
-----------
unknown
Actually, I got suspicious and ran a quick test, just to find different results in Postgres 9.3 and 9.4. varchar is picked over integer (which oddly contradicts your findings):
SQL Fiddle.
I would think the according rule is point 4e in the list (none of the earlier points decide the match):
At each position, select the string category if any candidate accepts
that category. (This bias towards string is appropriate since an
unknown-type literal looks like a string.)
If you added another function with input type text to the overloaded mix, text would be picked over varchar.
Personally I almost always use text instead of varchar. While being binary compatible (so almost but not quite the same), text is closer to the heart of Postgres in every respect.
I added that to the fiddle, as well as another example where Postgres cannot decide and throws a tantrum.
If you want to pick a particular function, add an explicit type cast (that's the way to go here!):
select test(null::int) AS func_int
, test(null::varchar) AS func_vc;

Pentaho Data Integration Input / Output Bit Type Error

I am using Pentaho Data Integration for numerous projects at work. We predominantly use Postgres for our database's. One of our older tables has two columns that are set to type bit(1) to store 0 for false and 1 for true.
My task is to synchronize a production table with a copy in our development environment. I am reading the data in using Table Input and immediately trying to do an Insert/Update. However, it fails because of the conversion to Boolean by PDI. I updated the query to cast the values to integers to retain the 0 and 1 but when I run it again, my transformation fails because an integer cannot be a bit value.
I have looked for several days trying different things like using the javascript step to convert to a bit but I have not been able to successfully read in a bit type and use the Insert/Update step to store the data. I also do not believe the Insert/Update step has the capabilities of updating the SQL that is being used to define the data type for the column.
The database connection is set up using:
Connection Type: PostgreSQL
Access: Native (JDBC)
Supports the boolean data type: true
Quote all in database: true
Note: Altering the table to change the datatype is not optional at this point in time. Too many applications currently depend on this table so altering it in this way could cause undesirable affects
Any help would be appreciated. Thank you.
You can create cast object (for example from character varying to bit) in your destination database with "as assignment" option. AS ASSIGNMENT allows to apply this cast automatically during inserts.
http://www.postgresql.org/docs/9.3/static/sql-createcast.html
Here is some proof-of-concept for you:
CREATE FUNCTION cast_char_to_bit (arg CHARACTER VARYING)
RETURNS BIT(1) AS
$$
SELECT
CASE WHEN arg = '1' THEN B'1'
WHEN arg = '0' THEN B'0'
ELSE NULL
END
$$
LANGUAGE SQL;
CREATE CAST (CHARACTER VARYING AS BIT(1))
WITH FUNCTION cast_char_to_bit(CHARACTER VARYING)
AS ASSIGNMENT;
Now you should be able to insert/update single-character strings into bit(1) column. However, you will need to cast your input column to character varying/text, so that it would be converted to String after in the table input step and to CHARACTER VARYING in the insert/update step.
Probably, you could create cast object using existing cast functions, which are defined in postgres already (see pg_cast, pg_type and pg_proc tables, join by oid), but I haven't managed to do this, unfortunately.
Edit 1:
Sorry for the previous solution. Adding a cast from boolean to bit looks much more reasonable: you will not even need to cast data in your table input step.
CREATE FUNCTION cast_bool_to_bit (arg boolean)
RETURNS BIT(1) AS
$$
SELECT
CASE WHEN arg THEN B'1'
WHEN NOT arg THEN B'0'
ELSE NULL
END
$$
LANGUAGE SQL;
CREATE CAST (BOOLEAN AS BIT(1))
WITH FUNCTION cast_bool_to_bit(boolean)
AS ASSIGNMENT;
I solved this by writing out the Postgres insert SQL (with B'1' and B'0' for the bit values) in a previous step and using "Execute row SQL Script" at the end to run each insert as individual SQL statements.

Rails 4, migration to change datatype of column from daterange to tsrange causing PG::DatatypeMismatch: ERROR:

I'm trying to change a column of type daterange to tsrange (I realized I need time as well as date) using a vanilla Rails migration
def self.up
change_column :events, :when, :tsrange
end
After running rake db:migrate the error is
PG::DatatypeMismatch: ERROR: column "when" cannot be cast automatically to type tsrange
HINT: Specify a USING expression to perform the conversion.
: ALTER TABLE "events" ALTER COLUMN "when" TYPE tsrange
I tried following the hint and used the following
def self.up
change_column :events, :when, :tsrange, 'tsrange USING CAST(when AS tsrange)'
end
but then got
no implicit conversion of Symbol into Integer
From what I can tell, USING CAST is mainly meant for use with ints. Assuming I don't want to drop and then recreate the column, what do you have to specify to alter the type from daterange to tsrange?
I'm using
Rails 4.0.1
ruby-2.0.0-p247
psql (9.2.4)
Some background, daterange and tsrange were introduced to Rails 4 in the following PR: https://github.com/rails/rails/pull/7345. Thanks.
The USING clause is used to specify how to convert the old values to the new ones:
The optional USING clause specifies how to compute the new column value from the old; if omitted, the default conversion is the same as an assignment cast from old data type to new. A USING clause must be provided if there is no implicit or assignment cast from old to new type.
So USING shows up any time there is no default cast from the old type to the new type. Also note that USING is specified as USING expression so any expression (whose value is of the correct type) can be used with a USING, the most common is USING CAST(...) but the expression can be pretty much anything.
Hopefully that should clear up some confusion about USING.
So what's up with the ActiveRecord error? Well, change_column is expecting to see an options Hash in the fourth argument but you're sending in a string. If you look at the change_column source, you'll see things like options[:limit] but String#[] expects integer arguments so your string argument is triggering odd looking complains about Symbols.
AFAIK there is no way to get AR to add a USING clause to the ALTER TABLE ... ALTER COLUMN that change_column generates. This leaves connection.execute(some_sql) if you need a USING clause. Of course this is further complicated by the (apparent) lack of a built-in cast from daterange to tsrange but building the necessary expression isn't terribly difficult if you pull the daterange apart with the upper and lower functions:
connection.execute(%q{
alter table events
alter column "when"
type tsrange using tsrange(lower("when"), upper("when"))
})
You can see the table change in action over here: http://sqlfiddle.com/#!15/fb047/2
That assumes that you're using the default half-open intervals ([...)) for your ranges; if you have ranges that aren't closed on the left and open on the right then you'll have to build a more complicated USING expression using the other range functions to see if the left and right ends of the ranges are open or closed.
BTW, when is a PostgreSQL keyword so it isn't the best choice for an identifier, you'll have to say "when" every time you refer to that column in SQL snippets and that might get tiring. I'd recommend using a different name for that column so that you don't have to worry about quoting.