Common sense dictates that SQL query strings should never be assembled by hand. Thus, all database interfaces offer parameter substitution, and all users use it, without exceptions.*
I'm using PostgreSQL v10.5, nodejs v8.12.0, node-postgres 7.6.1.
Parameter substitution works as expected for SELECT statements:
> await db.query("select from users where id = 'mic'");
(success, 1 row returned)
> await db.query("select from users where id = $1", ["mic"]);
(success, 1 row returned)
But it doesn't work for LISTEN statements:
> await db.query("listen topicname");
(success)
> await db.query("listen $1", ["topicname"]);
(error: syntax error at or near "$1")
The name of the topic I want to listen to is dynamic. It is coming from semi-trustworthy sources, which should not be user-controllable. But why go against all established best practice and take any chances?
Unfortunately, from my tests I fear that PostgreSQL simply can't do parameter substitution for LISTEN queries.
Is there any solution or workaround for this?
*) This statement may only be true in some utopic future society.
I don't have enough reputation to comment on the answer, but the proposed solution doesn't work for me.
Using %L results in a quoted string, which causes the following error:
ERROR: syntax error at or near "'topic'"
The %I format should be used instead (SQL identifier, this is documented for table and column names, but it also works for the channel name,). You can also use the quote_ident function. See the documentation on creating dynamic queries here.
The following PL/pgSQL function works for us:
CREATE OR REPLACE FUNCTION listenForChannel(
channel_ TEXT
) RETURNS VOID AS $$
BEGIN
EXECUTE format('LISTEN %I', channel_);
END
$$ LANGUAGE PLPGSQL;
You are right that this cannot be done in PostgreSQL.
As a workaround, write a PL/pgSQL function that uses dynamic SQL like this:
EXECUTE format('LISTEN %L', topicname);
The format function escapes strings properly; in this case, the %L format that produces a properly quoted string Literal is the appropriate one.
Related
I am trying to encapsulate the functionality from this sample code here, inside a Table-Function.
I can run the sample alone without any problem.
But when I create a table function, just with a single call to OPEN_CURSOR , I receive SQL0577N
CREATE FUNCTION ROW_CHECKSUM
( IN sSchema VARCHAR(128) ,
IN sTable VARCHAR(128) ,
IN sColumnList VARCHAR(1024) ,
IN sWhere VARCHAR(1023),
IN iRows INTEGER
)
RETURNS TABLE (ROW_PK_VALUES VARCHAR(3000), CHECKSUM INTEGER )
LANGUAGE SQL
SPECIFIC ROW_CHECKSUM
--NO EXTERNAL ACTION
--MODIFIES SQL DATA
--NOT DETERMINISTIC
BEGIN
DECLARE iCheckSum INTEGER ;
DECLARE sKyes VARCHAR(1024) ;
DECLARE iCursor INTEGER;
DECLARE sQuery VARCHAR(32000) ;
SET sQuery = 'SELECT ' || sColumnList || ' FROM "' || sSchema || '"."' || sTable || '" WHERE ' || sWhere || ' FETCH FIRST ' || TO_CHAR(iRows) || ' ONLY' ;
CALL DBMS_SQL.OPEN_CURSOR(iCursor);
--CALL DBMS_SQL.PARSE(iCursor, sQuery, DBMS_SQL.native) ;
--PIPE (sKeys, iCheckSum) ;
--PIPE ('abcd', 1234) ;
RETURN ;
END
----
SQL0577N User defined routine "DB2ADMIN.ROW_CHECKSUM" (specific name "")
attempted to modify data but was not defined as MODIFIES SQL DATA. LINE
NUMBER=33. SQLSTATE=38002
it seems, OPEN_CURSOR demands to have the MODIFY SQL DATA specified.. ok.. let's go!
But, when I specify it, then I get the following error, instead:
SQL0628N Multiple or conflicting keywords involving the "MODIFIES SQL DATA"
clause are present. LINE NUMBER=33. SQLSTATE=42613
The error details for -628 error is too generic and does not help me to determine what's really going on here.
I need to perform dynamic SQL queries using DBMS_SQL module, and return the result set using PIPE , like this other sample here.
I have been reading spread documentations the entire day.. and so far was not able to determine exactly what rule I am violating.
Also, found some inconsistencies on documentation, which I don't understand:
This page, says:
SQL table functions cannot contain compiled compound statements.
While, the Rules from RETURN statement says the opposite, and matches with PIPE sample code:
In an SQL table function using a compound SQL (compiled) statement, an expression, NULL, or fullselectcannot be specified. Rows are returned from the function using the PIPE statement and the RETURN statement is required as the last statement to execute when the function exits (SQLSTATE 2F005).
Appreciate any help!
Look at the note about the MODIFIES SQL DATA in the CREATE FUNCTION statement description:
4 Valid only for compiled scalar function definition and an inlined
table function definition.
But you can't use PIPE in an inlined function.
So, you want to use different functionalities, which can't be used together.
The inconsistency you found in the documentation is not related to you problem.
I am struggling to figure out how to best handle the return of results or errors to my application from Postgres stored functions.
Consider the following contrived psudeocode example:
app.get_resource(_username text)
RETURNS <???>
BEGIN
IF ([ ..user exists.. ] = FALSE) THEN
RETURN 'ERR_USER_NOT_FOUND';
END IF;
IF ([ ..user has permission.. ] = FALSE) THEN
RETURN 'ERR_NO_PERMISSION';
END IF;
-- Return the full user object.
RETURN QUERY( SELECT 1
FROM app.resources
WHERE app.resources.owner = _username);
END
The function can fail with a specific error or succeed and return 0 or more resources.
At first I tried creating a custom type to always use as a standard return type in eachh function:
CREATE TYPE app.appresult AS (
success boolean,
error text,
result anyelement
);
Postgres does not allow this however:
[42P16] ERROR: column "result" has pseudo-type anyelement
I then discovered OUT parameters and attempted the following uses:
CREATE OR REPLACE FUNCTION app.get_resource(
IN _username text,
OUT _result app.appresult -- Custom type
-- {success bool, error text}
)
RETURNS SETOF record
AS
$$
BEGIN
IF 1 = 1 THEN -- just a test
_result.success = false;
_result.error = 'ERROR_ERROR';
RETURN NULL;
END IF;
RETURN QUERY(SELECT * FROM app.resources);
END;
$$
LANGUAGE 'plpgsql' VOLATILE;
Postgres doesn't like this either:
[42P13] ERROR: function result type must be app.appresult because of OUT parameters
Also tried a similar function but reversed: Returning a custom app.appresult object and setting the OUT param to "SETOF RECORD". This was also not allowed.
Lastly i looked into Postgres exception handling using
RAISE EXCEPTION 'ERR_MY_ERROR';
So in the example function, i'd just raise this error and return.
This resulted in the driver sending back the error as:
"ERROR: ERR_MY_ERROR\nCONTEXT: PL/pgSQL function app.test(text) line 6 at RAISE\n(P0001)"
This is easy enough to parse but doing things this way feels wrong.
What is the best way to solve this problem?
Is it possible to have a custom AppResult object that i could return?
Something like:
{ success bool, error text, result <whatever type> }
//Edit 1 //
I think I'm leaning more towards #Laurenz Albe solution.
My main goal is simple: Call a stored procedure which can return either an error or some data.
Using RAISE seems to accomplish this and the C++ driver allows easy checking for an error condition returned from a query.
if ([error code returned from the query] == 90100)
{
// 1. Parse out my overly verbose error from the raw driver
// error string.
// 2. Handle the error.
}
I'm also wondering about using custom SQLSTATE codes instead of parsing the driver string.
Throwing '__404' might mean that during the course of my SPs execution, it could not continue because some record needed was not found.
When calling the sql function from my app, i have a general idea of what it failing with a '__404' would mean and how to handle it. This avoids the additional step of parsing driver error string.
I can also see the potential of this being a bad idea.
Bedtime reading:
https://www.postgresql.org/docs/current/static/errcodes-appendix.html
This is slightly opinion based, but I think that throwing an error is the best and most elegant solution. That is what errors are for!
To distinguish various error messages, you could use SQLSTATEs that start with 6, 8 or 9 (these are not used), then you don't have to depend on the wording of the error message.
You can raise such an error with
RAISE EXCEPTION SQLSTATE '90001' USING MESSAGE = 'my own error';
We do something similar to what you're trying to do, but we use TEXT rather than ANYELEMENT, because (almost?) any type can be cast to TEXT and back. So our type looks something like:
(errors our_error_type[], result TEXT)
The function which returns this stores errors in the errors array (it's just some custom error type), and can store the result (cast to text) in the result field.
The calling function knows what type it expects, so it can first check the errors array to see if any errors were returned, and if not it can cast the result value to the expected return type.
As a general observation, I think exceptions are more elegant (possibly because I come from a c# background). The only problem is in plpgsql exception handling is (relatively) slow, so it depends on the context - if you're running something many times in a loop, I would prefer a solution that doesn't use exception handling; if it's a single call, and/or especially when you want it to abort, I prefer raising an exception. In practice we use both at various points throughout our call stacks.
And as Laurenz Albe pointed out, you're not meant to "parse" exceptions, so much as raise an exception with specific values in specific fields, which the function that catches the exception can then extract and act on directly.
As an example:
Setup:
CREATE TABLE my_table (id INTEGER, txt TEXT);
INSERT INTO my_table VALUES (1,'blah');
CREATE TYPE my_type AS (result TEXT);
CREATE OR REPLACE FUNCTION my_func()
RETURNS my_type AS
$BODY$
DECLARE
m my_type;
BEGIN
SELECT my_table::TEXT
INTO m.result
FROM my_table;
RETURN m;
END
$BODY$
LANGUAGE plpgsql STABLE;
Run:
SELECT (m.result::my_table).*
FROM my_func() AS m
Result:
| id | txt |
-------------
| 1 | blah |
I have a Python script that runs a pgSQL file through SQLAlchemy's connection.execute function. Here's the block of code in Python:
results = pg_conn.execute(sql_cmd, beg_date = datetime.date(2015,4,1), end_date = datetime.date(2015,4,30))
And here's one of the areas where the variable gets inputted in my SQL:
WHERE
( dv.date >= %(beg_date)s AND
dv.date <= %(end_date)s)
When I run this, I get a cryptic python error:
sqlalchemy.exc.ProgrammingError: (psycopg2.ProgrammingError) argument formats can't be mixed
…followed by a huge dump of the offending SQL query. I've run this exact code with the same variable convention before. Why isn't it working this time?
I encountered a similar issue as Nikhil. I have a query with LIKE clauses which worked until I modified it to include a bind variable, at which point I received the following error:
DatabaseError: Execution failed on sql '...': argument formats can't be mixed
The solution is not to give up on the LIKE clause. That would be pretty crazy if psycopg2 simply didn't permit LIKE clauses. Rather, we can escape the literal % with %%. For example, the following query:
SELECT *
FROM people
WHERE start_date > %(beg_date)s
AND name LIKE 'John%';
would need to be modified to:
SELECT *
FROM people
WHERE start_date > %(beg_date)s
AND name LIKE 'John%%';
More details in the pscopg2 docs: http://initd.org/psycopg/docs/usage.html#passing-parameters-to-sql-queries
As it turned out, I had used a SQL LIKE operator in the new SQL query, and the % operand was messing with Python's escaping capability. For instance:
dv.device LIKE 'iPhone%' or
dv.device LIKE '%Phone'
Another answer offered a way to un-escape and re-escape, which I felt would add unnecessary complexity to otherwise simple code. Instead, I used pgSQL's ability to handle regex to modify the SQL query itself. This changed the above portion of the query to:
dv.device ~ E'iPhone.*' or
dv.device ~ E'.*Phone$'
So for others: you may need to change your LIKE operators to regex '~' to get it to work. Just remember that it'll be WAY slower for large queries. (More info here.)
For me it's turn out I have % in sql comment
/* Any future change in the testing size will not require
a change here... even if we do a 100% test
*/
This works fine:
/* Any future change in the testing size will not require
a change here... even if we do a 100pct test
*/
I am trying to call a stored procedure with Time variable as in parameter. But whenever i try to call the procedure i m getting error as:
db2 'call PASS_FAIL_CHECKDATE('2014-01-21','13:42:25','CSS1',Null,'4500096651','10',Null)'
SQL0104N An unexpected token ":42" was found following "CKDATE(2014-01-21,
13". Expected tokens may include: "+". SQLSTATE=42601
My Procedures input parameter are :
PASS_fail_checkdate (in post_date date,in post_time time,in destplant varchar(4), in destloc varchar(4), in transnum varchar(10), in translineitemnum varchar(6), in inboundconsignment varchar(35))
I am not sure if my declaration for time variable is correct in procedure or if i am calling the time variable correctly in the procedure.
Please give me suggestions on the same.
As already offered, in words, try the following example as a revision to what was noted in the OP as tried already but failing; i.e. change to use double-quote vs the apostrophe, specified as the delimiter for the DB2 SQL statement string:
db2 "call PASS_FAIL_CHECKDATE('2014-01-21','13:42:25','CSS1',Null,'4500096651','10',Null)"
I had been using MySQL as database and had planned to move to postgresql. I had used aes_encrypt and aes_decrypt functions in MySQL extensively throughout my application. So whenever the encryption/decrytion fails, MySQL automatically returns 'null'.
I am unsure how to handle the same in postgresql. Tried using the pgp_sym_encrypt/pgp_sym_decrypt functions. If the encryption key is wrong, it throws error "Wrong key/corrupt data". I tried searching for some functions that could capture this error and return 'null' as in MySQL so that I need not modify my code. I had been searching but could not find one.
Has anybody used any error handling mechanism for individual queries? I had found that error handling can be done for procedures. But, I had to completely rewrite the entire application for that.
If you could share some details, it would be of great help. Thanks.
If you wish to avoid modifying your code and have the functions return NULL on error, you can do this by wrapping them in a PL/PgSQL function that uses a BEGIN ... EXCEPTION block to trap the error.
To do this, first I get the SQLSTATE for the error:
regress=# \set VERBOSITY verbose
regress=# SELECT pgp_sym_decrypt('fred','key');
ERROR: 39000: Wrong key or corrupt data
LOCATION: decrypt_internal, pgp-pgsql.c:607
I could use this directly in the error handler, but I prefer to use a symbolic name, so I look up the error name associated with 39000 in Appendix A - Error codes, finding that it's the generic function call error external_routine_invocation_exception. Not as specific as we would've liked, but it'll do.
Now a wrapper function is required. Something like this must be defined, with one function for each overloaded signature of pgp_sym_decrypt that you wish to support. For the (bytea,text) form that returns text, for example:
CREATE OR REPLACE FUNCTION pgp_sym_decrypt_null_on_err(data bytea, psw text) RETURNS text AS $$
BEGIN
RETURN pgp_sym_decrypt(data, psw);
EXCEPTION
WHEN external_routine_invocation_exception THEN
RAISE DEBUG USING
MESSAGE = format('Decryption failed: SQLSTATE %s, Msg: %s',
SQLSTATE,SQLERRM),
HINT = 'pgp_sym_encrypt(...) failed; check your key',
ERRCODE = 'external_routine_invocation_exception';
RETURN NULL;
END;
$$ LANGUAGE plpgsql;
I've chosen to preseve the original error in a DEBUG level message. Here's a comparison of the original and wrapper, with full message verbosity and debug level output.
Enable debug output to show the RAISE. Note that it also shows the *original query text of the pgp_decrypt_sym call, including parameters.
regress=# SET client_min_messages = DEBUG;
New wrapped function still reports the error if detailed logging is enabled, but returns NULL:
regress=# SELECT pgp_sym_decrypt_null_on_err('redsdfsfdsfd','bobsdf');
LOG: 00000: statement: SELECT pgp_sym_decrypt_null_on_err('redsdfsfdsfd','bobsdf');
LOCATION: exec_simple_query, postgres.c:860
DEBUG: 39000: Decryption failed: SQLSTATE 39000, Msg: Wrong key or corrupt data
HINT: pgp_sym_encrypt(...) failed; check your key
LOCATION: exec_stmt_raise, pl_exec.c:2806
pgp_sym_decrypt_null_on_err
-----------------------------
(1 row)
compared to the original, which fails:
regress=# SELECT pgp_sym_decrypt('redsdfsfdsfd','bobsdf');
LOG: 00000: statement: SELECT pgp_sym_decrypt('redsdfsfdsfd','bobsdf');
LOCATION: exec_simple_query, postgres.c:860
ERROR: 39000: Wrong key or corrupt data
LOCATION: decrypt_internal, pgp-pgsql.c:607
Note that both forms show the parameters the function was called with when it failed. The parameters won't be shown if you've used bind parameters ("prepared statements"), but you should still consider your logs to be security critical if you're using in-database encryption.
Personally, I think it's better to do crypto in the app, so the DB never has access to the keys.