Using row data in pg_notify trigger as channel name? - postgresql

Is it possible to use data from the row a trigger is firing on, as the channel of a pg_notify, like this:
CREATE OR REPLACE FUNCTION notify_pricesinserted()
RETURNS trigger AS $$
DECLARE
BEGIN
PERFORM pg_notify(
NEW.my_label,
row_to_json(NEW)::text);
RETURN NEW;
END;
$$ LANGUAGE plpgsql;
CREATE TRIGGER notify_pricesinserted
AFTER INSERT ON prices
FOR EACH ROW
EXECUTE PROCEDURE notify_pricesinserted();
EDIT: I found out the reason it was not working is due to the case of my label. If I replace it with lower(NEW.my_label) and also do the same for the listener then it works.

The pg_notify() part would work without throwing an error. PostgreSQL places very few restrictions on what a channel name could be. But in practice it is probably useless because you would need to establish a LISTEN some_channel command prior to the pg_notify() statement to pick up the payload message somewhere outside of the trigger function and doing that on some dynamic value is difficult in most situations and probably terribly inefficient in all cases.
If - in your trigger - NEW.my_label has a small number of well-defined values, then you might work it out by establishing listening channels on all possible values, but you are probably better off defining a single channel identifier for your table, or perhaps for this specific trigger, and then construct the payload message in such a way that you can easily extract the appropriate information for some response. If you cannot predict the values of NEW.my_label then it is plain impossible.
In your specific case you could have a channel name 'prices' and then do something like:
pg_notify('prices', format('%s: %s, NEW.my_label, row_to_json(NEW)::text));
The session with LISTEN prices will receive:
Asynchronous notification "prices" with payload "some_label: {new_row_to_json}" received from server process with PID 8448.
That is a rather silly response (why the "Asynchronous notification "channel" with payload ..." instead of just the payload and the PID?) but you can easily extract the relevant parts and work with those. Since you would have to manipulate the string anyway it is not a big burden to strip away all the PG overhead in one go, on a single channel, making management of the trigger actions far easier.

Related

Deleting notifications from postgres message queue

I have a functionality where I have a table where for each insert I have to notify a channel with a JSON . I am using pg_notify in a trigger function to do that. Now I have to manage that message queue so that if its size reaches 80% it have to delete older messages till it reaches to 50%. I have searched online for it but I havent received any answers. Can any one help.
I am using below mentioned trigger for notifying.
CREATE OR REPLACE FUNCTION action_notify() RETURNS TRIGGER AS $BODY$
DECLARE
action_json TEXT;
BEGIN
RAISE DEBUG 'begin: action_notify';
action_json := '{"action": "'||NEW.action||'"
"action_params": "'||NEW.action_params||'"}';
PERFORM pg_notify(TG_TABLE_NAME, action_json);
RAISE DEBUG 'end: action_notify';
RETURN NULL;
END;
$BODY$ LANGUAGE plpgsql;
It will be great help if someone can guide me how to manage this message queue. I am not using any other message queue like rabbitmq ...just managing it from postgres..whats the best way to implement this.
Using PostgreSQL 9.3.
Short version: you can't do that with LISTEN and NOTIFY. It's a reliable-delivery queue, and there's no facility for purging old entries.
Instead of passing the message as a notification payload, you should probably insert it into a table, then just NOTIFY that the table has changed. You can manage the size of the table with periodic jobs to truncate old entries.

Reading postgres NOTICE message in C++ API

I am struggling to read my Postgres NOTICE messages in my C++ API. I can only read EXCEPTIONmessages using the function PQresultErrorMessage(PGresult), but not lower level messages.
PQresultErrorField(res, PG_DIAG_SEVERITY) returns null pointer.
How do I read NOTICE and other low level messages?
(Using PostgreSQL 9.2)
Set up a notice receiver or notice processor using PQsetNoticeReceiver / PQsetNoticeProcessor. Both set up callbacks that are invoked when asynchronous notifications are received. Note that this may happen before, during, or after processing of query data.
It's safe to assume that after all query results are returned (PQexec or whatever has returned) and you've called PQconsumeInput to make sure there's nothing else waiting, then all notices for the last command are received. The PQconsumeInput shouldn't really be necessary, it's just to be cautious.
See the documentation for libpq.

When and where to encode user input?

I am currently storing data submitted from users into my database already encoded like such:
<cfquery>
INSERT INTO dbo.MyTable (UserID, Comment)
VALUES
(
<cfqueryparam value="#FORM.UserID#" cfsqltype="cf_sql_integer"/>,
<cfqueryparam value="#EncodeForHTML(FORM.Comment)#" cfsqltype="cf_sql_nvarchar"/>
)
</cfquery>
Evidently this is not the right way to do it because now I have escaped characters in my DB table which are only useful for HTML output and difficult to perform searches on within SQL Server.
So how do I ensure that I apply the EncodeForHTML() on the input before it hits the server and then Canonicalize() the data received to be stored in the DB?
Mitigate potentially DB-harmful text when it heads towards the DB: pass it as a parameter, not hard-coded into the SQL statement, as you have kinda done in your example. You are still exposing yourself by not parameterising your ID value. As a rule, only SQL should go in your <cfquery>'s SQL string; any data values should be passed as parameters.
Similarly, mitigate risk your user-provided data might expose when you use the data. Not when it goes into storage, but when you actually use it. encodeForHtml() is only appropriate for stuff being written into HTML. It's no help if it's being passed on a URL, or used in JavaScript, etc. There are different mitigation approaches for those (urlEncodedFormat() and encodeForJavaScript() respectively). The point being you handle the mitigation on a use-by-use basis, not just generically.
And how to ensure this is done (you ask this)? Well... you write your code diligently and have a rigorous code review and QA process (with QA doing pen. tests).
You can store them as is, and use <cfqueryparam> for form.userid as well. On output, you use encodeforhtml().
If you prefer to have some data sanitizing done before storing, try AntiSamy (built-in in CF11) http://www.adobe.com/devnet/coldfusion/articles/security-improvements-cf11.edu.html#articlecontentAdobe_numberedheader_2

How a Java client app. can "catch" (via JDBC) the result produced by a trigger procedure query?

I'm trying to understand how a java (client) application that communicates, through JDBC, with a pgSQL database (server) can "catch" the result produced by a query that will be fired (using a trigger) whenever a record is inserted into a table.
So, to clarify, via JDBC I install a trigger procedure prepared to execute a query whenever a record is inserted into a given database table, and from this query's execution will result an output (wrapped in a resultSet, I suppose). And my problem is that I have no idea how the client will be aware of those results, that are asynchronously produced.
I wonder if JDBC supports any "callback" mechanism able to catch the results produced by a query that is fired through a trigger procedure under the "INSERT INTO table" condition. And if there is no such "callback" mechanism, what is the best approach to achieve this result?
Thank you in advance :)
Triggers can't return a resultset.
There's no way to send such a result to the JDBC driver.
There are a few dirty hacks you can use to get results from a trigger to the client, but they're all exactly that. Things like:
DECLARE a cursor for the resultset, then send the cursor name as a NOTIFY payload, so the app can FETCH ALL FROM <cursorname>;
Create a TEMPORARY table and report the name via NOTIFY
It is more typical to append anything the trigger needs to communicate to the app to a table that exists for that purpose and have the app SELECT from it after the operation that fired the trigger ran.
In most cases if you need to do this, you're probably using a trigger where a regular function is a better fit.

ActiveRecord find_or_initialize_by race conditions

I have a scenario where 2 db connections might both run Model.find_or_initialize_by(params) and raise an error: PG::UniqueViolation: ERROR: duplicate key value violates unique constraint
I'd like to update my code so it could gracefully recover from it. Something like:
record = nil
begin
record = Model.find_or_initialize_by(params)
rescue ActiveRecord::RecordNotUnique
record = Model.where(params).first
end
return record
The trouble is that there's not a nice/easy way to reproduce this on my local machine, so I'm not confident that my fix actually works.
So I thought I'd get a bit creative and try calling create 2 times (locally) in a row which should raise then PG::UniqueViolation: ERROR, then I could rescue from it and make sure everything is handled gracefully.
But I get this error: PG::InFailedSqlTransaction: ERROR: current transaction is aborted, commands ignored until end of transaction block
I get this error even when I wrap everything in individual transaction blocks
record = nil
Model.transaction do
record = Model.create(params)
end
begin
Model.transaction do
record = Model.create(params)
end
rescue ActiveRecord::RecordNotUnique
end
Model.transaction do
record = Model.where(params).first
end
return record
My questions:
What's the right way to gracefully handle the race condition I mentioned at the very beginning of this post?
How do I test this locally?
I imagine there's probably something simple that I'm missing here, but it's late and perhaps I'm not thinking too clearly.
I'm running postgres 9.3 and rails 4.
EDIT Turns out that find_or_initialize_by should have been find_or_create_by and the errors I was getting was from the actual save call that happened later on in execution. #VeryTiredWhenIWroteThis
Has this actually happenend?
Model.find_or_initialize_by(params)
should never raise an ´ActiveRecord::RecordNotUnique´ error as it is not saving anything to db. It just creates a new ActiveRecord.
However in the second snippet you are creating records.
create (without bang) does not throw exceptions caused by validations, but
ActiveRecord::RecordNotUnique is always thrown in case of a duplicate by both create and create!
If you're creating records you don't need transactions at all. As Postgres being ACID compliant guarantees that only one of the both operations succeeds and if it responds so it's changes will be durable. (a single statement query against postgres is also a transaction). So your above code is almost fine if you replace through find_or_create_by
begin
record = Model.find_or_create_by(params)
rescue ActiveRecord::RecordNotUnique
record = Model.where(params).first
end
You can test if the code behaves correctly by simply trying to create the same record twice in row. However this will not test ActiveRecord::RecordNotUnique is actually thrown correctly on race conditions.
It's also no the responsibility of your app to test and testing it is not easy. You would have to start rails in multithread mode on your machine, or test against a multi process staging rails instance. Webrick for example handles only one request at a time. You can use puma application server, however on MRI there is no true concurrency (GIL). Threads only share the GIL only on IO blocking. Because talking to Postgres is IO, i'd expect some concurrent requests, but to be 100% sure, the best testing scenario would be to deploy on passenger with multiple workers and then use jmeter to run concurrent request agains the server.