SELECT from result of UPDATE ... RETURNING in jOOQ - postgresql

I'm transforming some old PostgreSQL code to jOOQ, and I'm currently struggling with SQL that has multiple WITH clauses, where each one depends on previous. It would be best to keep the SQL logic the way it was written and not to change it (e.g. multiple queries to DB).
As it seems, there is no way to do SELECT on something that is UPDATE ... RETURNING, for example
dsl.select(DSL.asterisk())
.from(dsl.update(...)
.returning(DSL.asterisk())
)
I've created some test tables, trying to create some sort of MVCE:
create table dashboard.test (id int primary key not null, data text); --test table
with updated_test AS (
UPDATE dashboard.test SET data = 'new data'
WHERE id = 1
returning data
),
test_user AS (
select du.* from dashboard.dashboard_user du, updated_test -- from previous WITH
where du.is_active AND du.data = updated_test.data
)
SELECT jsonb_build_object('test_user', to_jsonb(tu.*), 'updated_test', to_jsonb(ut.*))
FROM test_user tu, updated_test ut; -- from both WITH clauses
So far this is my jOOQ code (written in Kotlin):
dsl.with("updated_test").`as`(
dsl.update(Tables.TEST)
.set(Tables.TEST.DATA, DSL.value("new data"))
.returning(Tables.TEST.DATA) //ERROR is here: Required Select<*>, found UpdateResultStep<TestRecord>
).with("test_user").`as`(
dsl
.select(DSL.asterisk())
.from(
Tables.DASHBOARD_USER,
DSL.table(DSL.name("updated_test")) //or what to use here?
)
.where(Tables.DASHBOARD_USER.IS_ACTIVE.isTrue
.and(Tables.DASHBOARD_USER.DATA.eq(DSL.field(DSL.name("updated_test.data"), String::class.java)))
)
)
.select() //here goes my own logic for jsonBBuildObject (which is tested and works for other queries)
.from(
DSL.table(DSL.name("updated_test")), //what to use here
DSL.table(DSL.name("test_user")) //or here
)
Are there any workarounds for this? I'd like to avoid changing SQL if possible.
Also, in this project this trick is used very often to get JSON(B) from UPDATE clause (table has JSON(B) columns too):
with _updated AS (update dashboard.test SET data = 'something' WHERE id = 1 returning *)
select to_jsonb(_updated.*) from _updated;
and it will be a real step back for us if there is no workaround for this.
I'm using JOOQ version 3.13.3, and Postgres 12.0.

This is currently not supported in jOOQ, see:
https://github.com/jOOQ/jOOQ/issues/3185
https://github.com/jOOQ/jOOQ/issues/4474
The workaround is, as always, when some vendor specific syntax is unsupported, to resort to plain SQL templating
E.g.
// If you don't need to map data types
dsl.fetch("with t as ({0}) {1}", update, select);
// If you need to map data types
dsl.resultQuery("with t as ({0}) {1}", update, select).coerce(select.getSelect()).fetch();

Related

With PostgREST, convert a column to and from an external encoding in the API

We are using PostgREST to automatically generate a REST API for a Postgres database. Our primary keys have an external representation that's different from how we store them internally. For simplicity's sake lets pretend the ids are stored as integers but we represent them as hexadecimal strings outwardly.
It's simple enough to get PostgREST to convert to the external representation for read operations:
CREATE DOMAIN hexid AS bigint;
CREATE TABLE fruits (
fruit_id hexid PRIMARY KEY,
name text
);
CREATE OR REPLACE VIEW api_fruits AS
SELECT to_hex(fruit_id) as fruit_id, name FROM fruits;
INSERT INTO fruits(fruit_id, name) VALUES('51955', 'avocado');
PostgREST generates the expected representation when we GET api_fruits:
[
{
"fruit_id": "caf3",
"name": "avocado"
}
]
But that's about as far as we get with this solution. It's a one way transformation so we won't be able to POST/PATCH records this way. The way PostgREST works is to transform such requests into equivalent INSERT and UPDATE statements. But this view with its custom formatting is not updatable. This is what would happen if we tried:
ERROR: cannot insert into column "fruit_id" of view "api_fruits"
DETAIL: View columns that are not columns of their base relation are not updatable.
STATEMENT: WITH pgrst_source AS (WITH pgrst_payload AS (SELECT $1::json AS json_data), pgrst_body AS ( SELECT CASE WHEN json_typeof(json_data) = 'array' THEN json_data ELSE json_build_array(json_data) END AS val FROM pgrst_payload) INSERT INTO "api_x"."api_fruits"("fruit_id", "name") SELECT "fruit_id", "name" FROM json_populate_recordset (null::"api_x"."api_fruits", (SELECT val FROM pgrst_body)) _ RETURNING "api_x"."api_fruits".*) SELECT '' AS total_result_set, pg_catalog.count(_postgrest_t) AS page_total, CASE WHEN pg_catalog.count(_postgrest_t) = 1 THEN coalesce((
WITH data AS (SELECT row_to_json(_) AS row FROM pgrst_source AS _ LIMIT 1)
SELECT array_agg(json_data.key || '=eq.' || json_data.value)
FROM data CROSS JOIN json_each_text(data.row) AS json_data
WHERE json_data.key IN ('')
), array[]::text[]) ELSE array[]::text[] END AS header, '' AS body, nullif(current_setting('response.headers', true), '') AS response_headers, nullif(current_setting('response.status', true), '') AS response_status FROM (SELECT * FROM pgrst_source) _postgrest_t
We can't INSERT into "View columns that are not columns of their base relation".
The obvious workaround is to serve fruit_id as a straight column, just an integer. With some post and preprocessing at the nginx level we can hex encode it there (and hex decode incoming ids). I'm wondering if we can do better than that though. For large API operations, re-encoding the JSON will use a lot of memory and CPU time and it seems so unnecessary.
It would have been great to be able to use a custom CREATE CAST to take the incoming hexadecimal strings and turn them back into integers, something like this:
CREATE CAST (json AS hexid) WITH FUNCTION json_to_hexid AS ASSIGNMENT;
But alas custom casts are ignored on CREATE DOMAIN types. And we can't make a true custom column type because our cloud Postgres host (Google Cloud SQL) doesn't allow custom extensions.
It feels like some combination of INSTEAD OF triggers or rules could work. But when using query parameters to filter results using query parameters (e.g. select a fruit by id), I don't think there's an appropriate trigger to use. INSTEAD OF doesn't work for straight SELECT does it?
For example I've tested doing something like this to take care of INSERT and allow POST with PostgREST. It works:
CREATE OR REPLACE FUNCTION api_fruits_insert()
RETURNS trigger AS
$$
BEGIN
INSERT INTO fruits(fruit_id, name) VALUES (('x' || lpad(NEW.fruit_id, 16, '0'))::bit(64)::bigint::hexid, NEW.name);
RETURN NEW;
END
$$ LANGUAGE 'plpgsql';
CREATE TRIGGER api_fruits_insert
INSTEAD OF INSERT
ON api_fruits
FOR EACH ROW
EXECUTE PROCEDURE api_fruits_insert();
The trouble is in the WHERE clause. Let's PATCH api_fruits?fruit_id=in.(7b,caf3) with {"name": "pear"}. This works out of the box since the name column is updatable but look at the query:
WITH pgrst_source AS (WITH pgrst_payload AS (SELECT $1::json AS json_data), pgrst_body AS ( SELECT CASE WHEN json_typeof(json_data) = 'array' THEN json_data ELSE json_build_array(json_data) END AS val FROM pgrst_payload) UPDATE "api_x"."api_fruits" SET "name" = _."name" FROM (SELECT * FROM json_populate_recordset (null::"api_x"."api_fruits" , (SELECT val FROM pgrst_body) )) _ WHERE "api_x"."api_fruits"."fruit_id" = ANY ($2) RETURNING 1) SELECT '' AS total_result_set, pg_catalog.count(_postgrest_t) AS page_total, array[]::text[] AS header, '' AS body, nullif(current_setting('response.headers', true), '') AS response_headers, nullif(current_setting('response.status', true), '') AS response_status FROM (SELECT * FROM pgrst_source) _postgrest_t
DETAIL: parameters: $1 = '{
"name": "pear"
}', $2 = '{7b,caf3}'
So we have essentially UPDATE api_fruits SET name='berry' WHERE fruit_id IN ('7b', 'caf3');. Surprisingly this works but it's a full table scan so Postgres can evaluate to_hex(fruit_id) for each row looking for matches. The same happens if we try to GET a record by fruit_id. How would we rewrite the WHERE clauses?
It really feels like some combination of just the right Postgres and PostgREST features should be able to get us to a point where it's all happening in Postgres without nginx's help and without excessive complexity. Any ideas?

Can postgreSQL OnConflict combine with JSON obejcts?

I wanted to perform a conditional insert in PostgreSQL. Something like:
INSERT INTO {TABLE_NAME} (user_id, data) values ('{user_id}', '{data}')
WHERE not exists(select 1 from files where user_id='{user_id}' and data->'userType'='Type1')
Unfortunately, insert and where does not cooperate in PostGreSQL. What could be a suitable syntax for my query? I was considering ON CONFLICT, but couldn't find the syntax for using it with JSON object. (Data in the example)
Is it possible?
Rewrite the VALUES part to a SELECT, then you can use a WHERE condition:
INSERT INTO { TABLE_NAME } ( user_id, data )
SELECT
user_id,
data
FROM
( VALUES ( '{user_id}', '{data}' ) ) sub ( user_id, data )
WHERE
NOT EXISTS (
SELECT 1
FROM files
WHERE user_id = '{user_id}'
AND data -> 'userType' = 'Type1'
);
But, there is NO guarantee that the WHERE condition works! Another transaction that has not been committed yet, is invisible to this query. This could lead to data quality issues.
You can use INSERT ... SELECT ... WHERE ....
INSERT INTO elbat
(user_id,
data)
SELECT 'abc',
'xyz'
WHERE NOT EXISTS (SELECT *
FROM files
WHERE user_id = 'abc'
AND data->>'userType' = 'Type1')
And it looks like you're creating the query in a host language. Don't use string concatenation or interpolation for getting the values in it. That's error prone and makes your application vulnerable to SQL injection attacks. Look up how to use parameterized queries in your host language. Very likely for the table name parameters cannot be used. You need some other method of either whitelisting the names or properly quoting them.

How to format a number in Entity Framework LINQ (without trailing zeroes)?

In a SQL Server database I have a column of decimal datatype defined something like this:
CREATE TABLE MyTable
(
Id INT,
Number DECIMAL(9, 4)
)
I use Entity Framework and I would like to return column Number converted to a string with only the digits right of the decimal separator that are actually needed. A strict constraint is that a result must be an IQueryable.
So my query is:
IQueryable queryable = (
from myTable in MyDatabase.NyTable
select new
{
Id = myTable.Id,
Number = SqlFunctions.StringConvert(myTable.Number,9,4)
}
);
The problem with is that it always convert number to string with 4 decimals, even if they are 0.
Examples:
3 is converted to "3.0000"
1.2 is converted to "1.2000"
If I use other parameters for StringConvert i.e.
SqlFunctions.StringConvert(myTable.Number, 9, 2)
the results are also not OK:
0.375 gets rounded to 0.38.
StringConvert() function is translated into SQL Server function STR.
https://learn.microsoft.com/en-us/sql/t-sql/functions/str-transact-sql?view=sql-server-2017
This explains the weird results.
In the realm of Entity Framework and LINQ I was not able to find a working solution.
What I look for is something like C# function
String.Format("0.####", number)
but this cannot be used in a LINQ query.
In plain simple SQL I could write my query like this
SELECT
Id,
Number = CAST(CAST(Number AS REAL) AS VARCHAR(15))
FROM
MyTable
I have not managed to massage LINQ to produce query like that.
A workaround would be to forget doing this in LINQ, which is quite inflexible and messy thing, borderline on useless and just return type DECIMAL from database and do my formatting on a client side before displaying. But this is additional, unnecessary code and I would hate to di it that way if there perhaps is a simpler way via LINQ.
Is it possible to format numbers in LINQ queries?
I would absolutely return a decimal from he database and format it when needed. Possible directly after the query. But usually this is done at display time to take into account culture specific formatting from the the client.
var q =
(from myTable in MyDatabase.NyTable
select new
{
Id = myTable.Id,
Number = myTable.Number
})
.AsEnumerable()
.Select(x => new { Id = x.Id, Number = x.Number.ToString("G29") });

Track updated columns postgresql trigger?

I am using postgreSQL and I have tables on which I use triggers to get notified of changes on tables.
Now, I have a usecase where when an update on a table is done, I want to notify only the updated columns of my table. So if my table has 10 columns and only 5 get updated I need to notify only of 5 updated columns.
One approach would be use OLD and NEW on every column and compare. This would lead to a separate function for each table.
Is there any functionality in postgreSQL pertianing to such a case?
You can create triggers in pl/tcl or pl/perl.
Both produce all the information about the context of the table that triggers the function in a bunch of variables, and OLD and NEW are associative arrays, so you have a lot of freedom to do whatever you like. Eg:
CREATE OR REPLACE FUNCTION valid_id() RETURNS trigger AS $$
my ($new, $old) = ($_TD->{new}, $_TD->{old});
my (#allflds) = keys %$new;
my (#changed) = grep { $new->{$_} ne %$old->{$_} } #allflds;
my (%difold, %difnew);
%difold = map { _$ => $old->{_$} } #changed;
%difnew = map { _$ => $new->{_$} } #changed;
... notify the values in %difold and %difnew as you need ...
$$ LANGUAGE plperl;

Avoid COUNT to CAST to BIGINT

Using Open JPA 2.0, Database is DB2 9.7. For query like SELECT COUNT(1) FROM USER WHERE FNAME := fname, JPA is converting the query to SELECT COUNT(CAST(? AS BIGINT)) FROM TABLENAME.
How to avoid the CAST to BIGINT?
Code sample below:
query = entityManager.createNamedQuery("qry.checkuser");
query.setParameter("fname", fname);
Long count = (Long)query.getSingleResult();
Which one is the problem in the CAST?
I think you can't avoid it since is SQL generated by the JPA provider.
BTW, I allways use Number super class instead of specific subclass:
query = entityManager.createNamedQuery("qry.checkuser", Number.class);
query.setParameter("fname", fname);
Number count = query.getSingleResult();
// Do whatever is needed
if (count.longValue()...
This way there's no problem if the JPA provider returns a integer, long or BigXXXX.
What worked for me is:
SELECT COUNT(USER_ID) FROM USER WHERE FNAME := fname
Basically we need to use a non nullable column like Primary key column and with this change the CAST can be avoided which consumes additional CPU, a minor gain.