I am using postgreSQL and I have tables on which I use triggers to get notified of changes on tables.
Now, I have a usecase where when an update on a table is done, I want to notify only the updated columns of my table. So if my table has 10 columns and only 5 get updated I need to notify only of 5 updated columns.
One approach would be use OLD and NEW on every column and compare. This would lead to a separate function for each table.
Is there any functionality in postgreSQL pertianing to such a case?
You can create triggers in pl/tcl or pl/perl.
Both produce all the information about the context of the table that triggers the function in a bunch of variables, and OLD and NEW are associative arrays, so you have a lot of freedom to do whatever you like. Eg:
CREATE OR REPLACE FUNCTION valid_id() RETURNS trigger AS $$
my ($new, $old) = ($_TD->{new}, $_TD->{old});
my (#allflds) = keys %$new;
my (#changed) = grep { $new->{$_} ne %$old->{$_} } #allflds;
my (%difold, %difnew);
%difold = map { _$ => $old->{_$} } #changed;
%difnew = map { _$ => $new->{_$} } #changed;
... notify the values in %difold and %difnew as you need ...
$$ LANGUAGE plperl;
Related
I have used the List.Accumulate() to merge mutliple tables. This is the output I've got in this simple example:
Now, I need a solution to expand all these with a formula, because in real - world I need to merge multiple tables that keep increasing in number (think Eurostat tables, for instance), and modifying the code manually wastes much time in these situations.
I have been trying to solve it, but it seems to me that the complexity of syntax easily becomes the major limitation here. For instance, If I make a new step where I nest in another List.Accumulate() the Table.ExpandTableColumns(), I need to pass inside a column name of an inner table, as a text. Fine, but to drill it down actually, I first need to pass a current column name in [] in each iteration - for instance, Column 1 - and it triggers an error if I store column names to a list because these are between "". I also experimented with TransformColumns() but didn't work either.
Does anyone know how to solve this problem whatever the approach?
See https://blog.crossjoin.co.uk/2014/05/21/expanding-all-columns-in-a-table-in-power-query/
which boils down to this function
let Source = (TableToExpand as table, optional ColumnNumber as number) =>
//https://blog.crossjoin.co.uk/2014/05/21/expanding-all-columns-in-a-table-in-power-query/
let ActualColumnNumber = if (ColumnNumber=null) then 0 else ColumnNumber,
ColumnName = Table.ColumnNames(TableToExpand){ActualColumnNumber},
ColumnContents = Table.Column(TableToExpand, ColumnName),
ColumnsToExpand = List.Distinct(List.Combine(List.Transform(ColumnContents, each if _ is table then Table.ColumnNames(_) else {}))),
NewColumnNames = List.Transform(ColumnsToExpand, each ColumnName & "." & _),
CanExpandCurrentColumn = List.Count(ColumnsToExpand)>0,
ExpandedTable = if CanExpandCurrentColumn then Table.ExpandTableColumn(TableToExpand, ColumnName, ColumnsToExpand, NewColumnNames) else TableToExpand,
NextColumnNumber = if CanExpandCurrentColumn then ActualColumnNumber else ActualColumnNumber+1,
OutputTable = if NextColumnNumber>(Table.ColumnCount(ExpandedTable)-1) then ExpandedTable else ExpandAll(ExpandedTable, NextColumnNumber)
in OutputTable
in Source
alternatively, unpivot all the table columns to get one column, then expand that value column
ColumnsToExpand = List.Distinct(List.Combine(List.Transform(Table.Column(#"PriorStepNameHere", "ValueColumnNameHere"), each if _ is table then Table.ColumnNames(_) else {}))),
#"Expanded ColumnNameHere" = Table.ExpandTableColumn(#"PriorStepNameHere", "ValueColumnNameHere",ColumnsToExpand ,ColumnsToExpand ),
I'm transforming some old PostgreSQL code to jOOQ, and I'm currently struggling with SQL that has multiple WITH clauses, where each one depends on previous. It would be best to keep the SQL logic the way it was written and not to change it (e.g. multiple queries to DB).
As it seems, there is no way to do SELECT on something that is UPDATE ... RETURNING, for example
dsl.select(DSL.asterisk())
.from(dsl.update(...)
.returning(DSL.asterisk())
)
I've created some test tables, trying to create some sort of MVCE:
create table dashboard.test (id int primary key not null, data text); --test table
with updated_test AS (
UPDATE dashboard.test SET data = 'new data'
WHERE id = 1
returning data
),
test_user AS (
select du.* from dashboard.dashboard_user du, updated_test -- from previous WITH
where du.is_active AND du.data = updated_test.data
)
SELECT jsonb_build_object('test_user', to_jsonb(tu.*), 'updated_test', to_jsonb(ut.*))
FROM test_user tu, updated_test ut; -- from both WITH clauses
So far this is my jOOQ code (written in Kotlin):
dsl.with("updated_test").`as`(
dsl.update(Tables.TEST)
.set(Tables.TEST.DATA, DSL.value("new data"))
.returning(Tables.TEST.DATA) //ERROR is here: Required Select<*>, found UpdateResultStep<TestRecord>
).with("test_user").`as`(
dsl
.select(DSL.asterisk())
.from(
Tables.DASHBOARD_USER,
DSL.table(DSL.name("updated_test")) //or what to use here?
)
.where(Tables.DASHBOARD_USER.IS_ACTIVE.isTrue
.and(Tables.DASHBOARD_USER.DATA.eq(DSL.field(DSL.name("updated_test.data"), String::class.java)))
)
)
.select() //here goes my own logic for jsonBBuildObject (which is tested and works for other queries)
.from(
DSL.table(DSL.name("updated_test")), //what to use here
DSL.table(DSL.name("test_user")) //or here
)
Are there any workarounds for this? I'd like to avoid changing SQL if possible.
Also, in this project this trick is used very often to get JSON(B) from UPDATE clause (table has JSON(B) columns too):
with _updated AS (update dashboard.test SET data = 'something' WHERE id = 1 returning *)
select to_jsonb(_updated.*) from _updated;
and it will be a real step back for us if there is no workaround for this.
I'm using JOOQ version 3.13.3, and Postgres 12.0.
This is currently not supported in jOOQ, see:
https://github.com/jOOQ/jOOQ/issues/3185
https://github.com/jOOQ/jOOQ/issues/4474
The workaround is, as always, when some vendor specific syntax is unsupported, to resort to plain SQL templating
E.g.
// If you don't need to map data types
dsl.fetch("with t as ({0}) {1}", update, select);
// If you need to map data types
dsl.resultQuery("with t as ({0}) {1}", update, select).coerce(select.getSelect()).fetch();
How to auto increment sequence number once for every run of a stored procedure and how to use it in the where condition of an update statement?
I already assigned a sequence number to the next value in each run, but I'm not able to use it in the where condition.
CREATE OR REPLACE FUNCTION ops.mon_connect_easy()
RETURNS void
LANGUAGE plpgsql
AS $function$
declare
_inserted_rows bigint = 0;
sql_run bigint = 0;
--assigning the sequence number to the variable
select nextval('ops.mon_connecteasy_seq') into run_seq_num;
-- use for selection iteration_id. this is hwere I'm getting stuck
update t_contract c
set end_date = ce.correct_end_date, status='Active',
orig_end_date =ce.correct_end_date
from ops.t_mon_ConnectEasy ce
where c.contract_id = ce.contract_id
and run_seq_num = ??;
nextval() advances the sequence automatically before returning the resulting value. You don't need anything extra. Just use the function in your query directly:
update t_contract c
set end_date = ce.correct_end_date
, status = 'Active'
, orig_end_date = ce.correct_end_date
from ops.t_mon_ConnectEasy ce
where c.contract_id = ce.contract_id
and iteration_id = nextval('ops.mon_connecteasy_seq');
Be aware that concurrent transactions also might advance the sequence, creating virtual gaps in the sequential numbers.
And I have a nagging suspicion that this might not be the best way to achieve your undisclosed goals.
I have the following table:
CREATE TABLE scoped_data(
owner_id text,
scope text
key text,
data json,
PRIMARY KEY (owner_id, scope, key)
);
As part of each transaction we will potentially be inserting data for multiple scopes. Given this table has the potential to grow very quickly I would like not to store data if it is NULL or an empty JSON object.
An upsert felt like the idiomatic approach to this. The following is within the context of a PL/pgSQL function:
WITH upserts AS (
INSERT INTO scoped_data (owner_id, scope, key, data)
VALUES
(p_owner_id, 'broad', p_broad_key, p_broad_data),
(p_owner_id, 'narrow', p_narrow_key, p_narrow_data),
-- etc.
ON CONFLICT (owner_id, scope, key)
DO UPDATE SET data = scoped_data.data || COALESCE(EXCLUDED.data, '{}')
RETURNING scope, data
)
SELECT json_object_agg(u.scope, u.data)
FROM upserts u
INTO v_all_scoped_data;
I include the RETURNING as I would like the up-to-date version of each scope's data included in a variable for subsequent use, therefore I need the RETURNING to return something even if logically no data has been updated.
For example (all for key = 1 and scope = 'narrow'):
data = '{}' => v_scoped_data = {}, no data for key = 1 in scoped_data.
data = '{"some":"data"}' => v_scoped_data = { "narrow": { "some": "data" } }, data present in scoped_data.
data = '{}' => v_scoped_data = { "narrow": { "some": "data" }, data from 2. remains unaffected.
data = '{"more":"stuff"}' => v_scoped_data = { "narrow": { "some": "data", "more": "stuff" }. Updated data stored in table.
I initially added a trigger BEFORE INSERT ON scoped_data which did the following:
IF NULLIF(NEW.data, '{}') IS NULL THEN
RETURN NULL;
END IF;
RETURN NEW;
This worked fine for preventing the insertion of new records but the issue was that this trigger also prevented subsequent inserts to existing rows thereby no INSERT happened therefore there was no ON CONFLICT therefore nothing returned in the RETURNING.
A couple of approaches I've considered, both of which feel inelegant or like they should be unnecessary:
Add a CHECK constraint to scoped_data.data: CHECK(NULLIF(data, '{}') IS NOT NULL), allow the insert and catch the exception in the PL/pgSQL code.
DELETE in an AFTER INSERT trigger if the data field was NULL or empty.
Am I going about this in the right way? Am I trying to coerce this logic into an upsert when there is a better way? Might explicit INSERTs and UPDATEs be a more logical fit?
I am using Postgres 9.6.
I would go with the BEFORE trigger ON INSERT to prevent unnecessary inserts and updates.
To return the values even in the case that the operation is not performed, you can UNION ALL your query with a query on scoped_data that returns the original row, ORDER the results so that any new row is ordered first (introduce an artifical column to both queries) and use LIMIT 1 to get the correct result.
I tried looking for an example of using Arel::UpdateManager to form an update statement with a from clause (as in UPDATE t SET t.itty = "b" FROM .... WHERE ...), couldn.t find any. The way I've seen it, Arel::UpdateManager sets the main engine on initialization and allows to set the various fields and values to update. Is there actually a way to do this?
Another aside would be to find out how to express Postgres posix regex matching into ARel, but this might be impossible by now.
As far as I see the current version of arel gem is not support FROM keyword for the sql query. You can generate a query using the SET, and WHERE keywords only, like:
UPDATE t SET t.itty = "b" WHERE ...
and the code, which copies a value from field2 to field1 for the units table, will be like:
relation = Unit.all
um = Arel::UpdateManager.new(relation.engine)
um.table(relation.table)
um.ast.wheres = relation.wheres.to_a
um.set(Arel::Nodes::SqlLiteral.new('field1 = "field2"'))
ActiveRecord::Base.connection.execute(um.to_sql)
Exactly you can use the additional method to update a relation. So we create the Arel's UpdateManager, assigning to it the table, where clause, and values to set. Values shell be passed to the method as an argument. Then we need to add FROM keyword to the generated SQL request, we add it only if we have access to external table of the specified one by the UPDATE clause itself. And at the last we executes the query. So we get:
def update_relation!(relation, values)
um = Arel::UpdateManager.new(relation.engine)
um.table(relation.table)
um.ast.wheres = relation.wheres.to_a
um.set(values)
sql = um.to_sql
# appends FROM field to the query if needed
m = sql.match(/WHERE/)
tables = relation.arel.source.to_a.select {|v| v.class == Arel::Table }.map(&:name).uniq
tables.shift
sql.insert(m.begin(0), "FROM #{tables.join(",")} ") if m && !tables.empty?
# executes the query
ActiveRecord::Base.connection.execute(sql)
end
The you can issue the the relation update as:
values = Arel::Nodes::SqlLiteral.new('field1 = "field2", field2 = NULL')
relation = Unit.not_rejected.where(Unit.arel_table[:field2].not_eq(nil))
update_relation!(relation, values)