Though I have been using SQL Server, Oracle from last decade, I have been asked to
do some research on PostgreSQL and after some initial investigation it is evident that I am now stuck on retrieving data from the PostgreSQL database using Function.
Using following piece of code to retrieve the data and getting error
('ERROR [26000] ERROR: prepared statement "mytabletest" does not exist;
'Error while executing the query)
Code Snippets
Dim oDBCommand As DbCommand = GetDBCommand(oConnectionType, "mytabletest", CommandType.StoredProcedure)
Dim dstResults As DataSet = GetDataSet(ConnectionTypes.ODBC, oDBCommand)
Public Function GetDataReader(dbType As ConnectionTypes, command As DbCommand) As DbDataReader
Try
Dim oConnection As DbConnection = GetDBConnection(dbType)
Dim oDBTransaction As DbTransaction = oConnection.BeginTransaction
command.Connection = oConnection
command.Transaction = oDBTransaction
'GETTING ERROR ON FOLLOWING LINE
'ERROR [26000] ERROR: prepared statement "mytabletest" does not exist;
'Error while executing the query
return command.ExecuteReader()
Catch ex As Exception
Throw ex
Finally
End Try
Return Nothing
End Function
Environement I am currently working on is following:-
32 Bit Machine.
Visual Studio 2010 + SP1
ODBC Prodiver: PostgreSQL Unicode 9.01.02.00
ADO.Net (System.Data.Odbc)
Please note that I am open to any suggestions i.e. if I am completely doing it wrong
OR partially etc. Please feel free to write.
In order to make it easier for you to create a same environment, please use following table/function definition.
--- Simple table to make things easier to understand. <br>
CREATE TABLE mytable
(
messagetypeid integer NOT NULL,
messagetype character varying(100) NOT NULL
)
-- Function to retrieve data. <br>
CREATE OR REPLACE FUNCTION mytabletest() <br>
RETURNS SETOF refcursor AS $$
DECLARE
ref1 refcursor;
BEGIN
OPEN ref1 FOR SELECT * FROM mytable;
RETURN NEXT ref1;
END;
$$ LANGUAGE plpgsql;
Please Note:
If I use <br>
Dim oDBCommand As DbCommand = GetDBCommand(oConnectionType, "SELECT * FROM mytable", CommandType.Text)
then system manages to retrieve information from the datbase without any issue, however, as I mentioned as soon we use "Function" it throws an exception.
During my failed efforts to search any solution from the internet someone mentioned that Table should be created with the lower case it so just for the sake of it I recreated with the lower case, however, problem persists.
I am unfamiliar with .net but I suspect you meant something more like:
GetDBCommand(oConnectionType, "SELECT myfunc()", CommandType.Text)
Or in the case of SETOF functions etc..
GetDBCommand(oConnectionType, "SELECT * FROM myfunc()", CommandType.Text)
PostgreSQL does not have 'stored procedures' per-ce. It does have functions and I believe that the client/server protocol has a method for preparing statements that can then be executed multiple times with different variables (to save on the cost of parsing the SQL), but this should be exposed via your client library.
Related
I started with PostgreSQL recently and I'm running into trouble calling a procedure from a PyQt5 client using the psycopg2 driver.
I set a simple procedure with 6 parameters, included a BEGIN and a COMMIT statements and run seamlessly from PgAdmin 4.
When tried to call the procedure from the PyQt5 client a got error code 2D000 with the following message:Invalid transaction termination
CONTEXT: PL/pgSQL function es_edit_text_report(integer,date,integer,integer,character varying,character varying) line 10 at COMMIT
I found several mentions to this error both in stackoverflow and in the web but I could not figure it out why am I getting this results. The closer explanation found may be by #Laurenz Albe on Invalid transaction termination but it looks to me that none of the mentioned conditions apply. Here we have a very simple procedure being called just once from a psycopg2 connection opened from a pool for this single purpose.
Following other responses I just commented the commit on the procedure and then I was able to run it from the client. This places some further questions:
I understand that the transaction is not operative any longer, consequently you would not be able to run complex procedures safely unless this issue is solve. I'll appreciate your thoughts about it.
Procedure Code:
CREATE OR REPLACE PROCEDURE es_edit_text_report(IN _id integer, IN _report_date,
IN _responsibleid integer,
IN _categoryid integer,
IN _description varchar,
IN _location varchar)
AS $$
BEGIN
UPDATE text_maintenance_news
SET report_date = _report_date,
reporterid = _responsibleid::smallint,
categoryid = _categoryid::smallint,
description = _description,
location = _location
WHERE id = _id;
COMMIT;
END; $$
LANGUAGE plpgsql
PyQt5 client code:
#pyqtSlot()
def save(self):
try:
with self.connPool.getconn() as conn:
with conn.cursor() as cur:
if self.mode == OPEN_NEW:
cur.execute('CALL es_load_text_report(%s, %s, %s, %s, %s)',(
self.reportDate.date().toString("yyyy-MM-dd"),
self.comboResponsible.getHiddenData(0),
self.comboCategory.getHiddenData(0),
self.txtDescription.text(),
self.txtLocation.text()))
conn.commit()
else:
cur.execute('CALL es_edit_text_report(%s, %s, %s, %s, %s, %s)',
(self.id,
self.reportDate.date().toString("yyyy-MM-dd"),
self.comboResponsible.getHiddenData(0),
self.comboCategory.getHiddenData(0),
self.txtDescription.text(),
self.txtLocation.text()))
conn.commit()
qry = QSqlQuery(self.db)
qry.exec("SELECT * FROM es_load_text_reports()")
if qry.lastError().type() != 0:
raise DataError('save: qry', qry.lastError().text())
self.tableNews.model().setQuery(qry)
self.clear()
except OSError as e:
QMessageBox.warning(self, 'Save : ', e.args, QMessageBox.Ok)
except DataError as e:
QMessageBox.warning(self, e.source, e.message, QMessageBox.Ok)
except DatabaseError as e:
QMessageBox.warning(self, 'Save', f'{e.pgcode} {e.pgerror}', QMessageBox.Ok)ter code here
Please notice I'm using a QSqlQuery to retrive the updated table data. The reason being is I'm still trying to figure out how to handle psycopg2 list of tuples return data.
UPDATE: So far the only workaround I was able to find is to remove the COMMIT line from the procedure. I'm under the impression that Postgres is transferring the transaction control to the driver. I wonder if this is the proper way to do it. I'll appreciate your comments about it.
With the psycopg2 driver, you have to set autocommit to True for statements that must run outside a transaction. From the docs:
A few commands (e.g. CREATE DATABASE, VACUUM, CALL on stored
procedures using transaction control…) require to be run outside any
transaction: in order to be able to run these commands from Psycopg,
the connection must be in autocommit mode: you can use the autocommit
property.
In your code add this line:
conn.autocommit = True
and then your CALL statement should run.
Here's my MRE
create procedure:
CREATE OR REPLACE PROCEDURE test_procedure(
par1 integer,
par2 integer)
LANGUAGE 'plpgsql'
AS $BODY$
BEGIN
COMMIT;
END;
$BODY$;
Python code:
import sys
import psycopg2
print(sys.version) # 3.7.1 and 3.8.2 tested
print(psycopg2.__version__) # 2.8.6 and 2.9.3 tested
conn = psycopg2.connect('host=myhost dbname=mydb user=myuser')
cur = conn.cursor()
conn.autocommit = True # this is the key line. without this, next line will fail
cur.execute('CALL aaa.test_procedure(%s,%s)', (1,1))
conn.commit()
I'm trying to create this trigger in my PostgresSql environment:
CREATE TRIGGER MYTRIGGER
BEFORE INSERT
ON MYTABLE
FOR EACH ROW
BEGIN
IF( LENGTH( :NEW.VAL ) > 10 )
THEN
RAISE_APPLICATION_ERROR( -20003,
'Cannot exceed 10 chars' );
END IF;
IF :NEW.FQN_ID IS NULL THEN
:NEW.FQN_ID :=
CASE :NEW.SUBTYPECODE
WHEN NULL THEN 'A:'
WHEN 0 THEN 'B:'
WHEN 1 THEN 'C:'
WHEN 2 THEN 'D:'
ELSE 'Z:' || :NEW.SUBTYPECODE || '::'
--END || :NEW.OBJECTID;
END || STRUCTURE_FQNID_SEQ.NEXTVAL;
END IF;
END;
But I get this error:
ERROR: syntax error at or near "BEGIN"
LINE 5: BEGIN
^
SQL state: 42601
Character: 79
I think I'm missing something but I can't get it.
Any suggestion would be greatly appreciated.
Thank you.
Here are my notes about triggers in several DBMSs: https://github.com/iwis/SQL-notes. I marked the differences between the DBMSs in orange. I think that the notes are quite complete, so you don't have to read about triggers in Postgres documentation.
I see the following changes that need to done in your example:
Change Oracle :NEW to Postgres NEW.
Instead of a BEGIN ... END block, write EXECUTE FUNCTION my_trigger_function();, where my_trigger_function is a function that needs to be created like in the example given by a_horse_with_no_name.
This function should return NEW in your case - the reason is described here.
If a more complicated code is fired by a trigger, then you also need to understand the differences between PL/SQL and PL/pgSQL languages. These languages are quite similar, though there are some differences. Your code is simple so the differences are small. It's probably enough to:
Write Postgres RETURNS in the function definition instead of Oracle RETURN.
Write $$ before BEGIN, and $$ LANGUAGE plpgsql; after END.
Write Postgres RAISE 'Cannot exceed 10 chars'; instead of Oracle RAISE_APPLICATION_ERROR(-20003, 'Cannot exceed 10 chars');.
I don't know if sequences work in the same way in PostgreSQL - I haven't read about them yet.
Let me know if my notes are understandable - I'm not sure about it because they are super compact so you need to decipher the markings used by me.
I am struggling to figure out how to best handle the return of results or errors to my application from Postgres stored functions.
Consider the following contrived psudeocode example:
app.get_resource(_username text)
RETURNS <???>
BEGIN
IF ([ ..user exists.. ] = FALSE) THEN
RETURN 'ERR_USER_NOT_FOUND';
END IF;
IF ([ ..user has permission.. ] = FALSE) THEN
RETURN 'ERR_NO_PERMISSION';
END IF;
-- Return the full user object.
RETURN QUERY( SELECT 1
FROM app.resources
WHERE app.resources.owner = _username);
END
The function can fail with a specific error or succeed and return 0 or more resources.
At first I tried creating a custom type to always use as a standard return type in eachh function:
CREATE TYPE app.appresult AS (
success boolean,
error text,
result anyelement
);
Postgres does not allow this however:
[42P16] ERROR: column "result" has pseudo-type anyelement
I then discovered OUT parameters and attempted the following uses:
CREATE OR REPLACE FUNCTION app.get_resource(
IN _username text,
OUT _result app.appresult -- Custom type
-- {success bool, error text}
)
RETURNS SETOF record
AS
$$
BEGIN
IF 1 = 1 THEN -- just a test
_result.success = false;
_result.error = 'ERROR_ERROR';
RETURN NULL;
END IF;
RETURN QUERY(SELECT * FROM app.resources);
END;
$$
LANGUAGE 'plpgsql' VOLATILE;
Postgres doesn't like this either:
[42P13] ERROR: function result type must be app.appresult because of OUT parameters
Also tried a similar function but reversed: Returning a custom app.appresult object and setting the OUT param to "SETOF RECORD". This was also not allowed.
Lastly i looked into Postgres exception handling using
RAISE EXCEPTION 'ERR_MY_ERROR';
So in the example function, i'd just raise this error and return.
This resulted in the driver sending back the error as:
"ERROR: ERR_MY_ERROR\nCONTEXT: PL/pgSQL function app.test(text) line 6 at RAISE\n(P0001)"
This is easy enough to parse but doing things this way feels wrong.
What is the best way to solve this problem?
Is it possible to have a custom AppResult object that i could return?
Something like:
{ success bool, error text, result <whatever type> }
//Edit 1 //
I think I'm leaning more towards #Laurenz Albe solution.
My main goal is simple: Call a stored procedure which can return either an error or some data.
Using RAISE seems to accomplish this and the C++ driver allows easy checking for an error condition returned from a query.
if ([error code returned from the query] == 90100)
{
// 1. Parse out my overly verbose error from the raw driver
// error string.
// 2. Handle the error.
}
I'm also wondering about using custom SQLSTATE codes instead of parsing the driver string.
Throwing '__404' might mean that during the course of my SPs execution, it could not continue because some record needed was not found.
When calling the sql function from my app, i have a general idea of what it failing with a '__404' would mean and how to handle it. This avoids the additional step of parsing driver error string.
I can also see the potential of this being a bad idea.
Bedtime reading:
https://www.postgresql.org/docs/current/static/errcodes-appendix.html
This is slightly opinion based, but I think that throwing an error is the best and most elegant solution. That is what errors are for!
To distinguish various error messages, you could use SQLSTATEs that start with 6, 8 or 9 (these are not used), then you don't have to depend on the wording of the error message.
You can raise such an error with
RAISE EXCEPTION SQLSTATE '90001' USING MESSAGE = 'my own error';
We do something similar to what you're trying to do, but we use TEXT rather than ANYELEMENT, because (almost?) any type can be cast to TEXT and back. So our type looks something like:
(errors our_error_type[], result TEXT)
The function which returns this stores errors in the errors array (it's just some custom error type), and can store the result (cast to text) in the result field.
The calling function knows what type it expects, so it can first check the errors array to see if any errors were returned, and if not it can cast the result value to the expected return type.
As a general observation, I think exceptions are more elegant (possibly because I come from a c# background). The only problem is in plpgsql exception handling is (relatively) slow, so it depends on the context - if you're running something many times in a loop, I would prefer a solution that doesn't use exception handling; if it's a single call, and/or especially when you want it to abort, I prefer raising an exception. In practice we use both at various points throughout our call stacks.
And as Laurenz Albe pointed out, you're not meant to "parse" exceptions, so much as raise an exception with specific values in specific fields, which the function that catches the exception can then extract and act on directly.
As an example:
Setup:
CREATE TABLE my_table (id INTEGER, txt TEXT);
INSERT INTO my_table VALUES (1,'blah');
CREATE TYPE my_type AS (result TEXT);
CREATE OR REPLACE FUNCTION my_func()
RETURNS my_type AS
$BODY$
DECLARE
m my_type;
BEGIN
SELECT my_table::TEXT
INTO m.result
FROM my_table;
RETURN m;
END
$BODY$
LANGUAGE plpgsql STABLE;
Run:
SELECT (m.result::my_table).*
FROM my_func() AS m
Result:
| id | txt |
-------------
| 1 | blah |
I just noticed that I could alter my stored procedure code with a misspelled user defined function in it.
I noticed that at 1st time I execute the SP.
Is there any way to get a compile error when an SP include an invalid user-defined function name in it?
At compile time? No.
You can, however, use some of SQL's dependency objects (if using MS SQL) to find problems just after deployment, or as part of your beta testing. Aaron Bertran has a pretty nice article rounding up the options, depending upon the version of SQL Server.
Here is an example using SQL Server 2008 sys object called sql_expression_dependencies
CREATE FUNCTION dbo.scalarTest
(
#input1 INT,
#input2 INT
)
RETURNS INT
AS
BEGIN
-- Declare the return variable here
DECLARE #ResultVar int
-- Add the T-SQL statements to compute the return value here
SELECT #ResultVar = #input1 * #input2
-- Return the result of the function
RETURN #ResultVar
END
GO
--Fn Works!
SELECT dbo.ScalarTest(2,2)
GO
CREATE PROCEDURE dbo.procTest
AS
BEGIN
SELECT TOP 1 dbo.scalarTest(3, 3) as procResult
FROM sys.objects
END
GO
--Sproc Works!
EXEC dbo.procTest
GO
--Remove a dependency needed by our sproc
DROP FUNCTION dbo.scalarTest
GO
--Does anything have a broken dependency? YES
SELECT OBJECT_NAME(referencing_id) AS referencing_entity_name,
referenced_entity_name, *
FROM sys.sql_expression_dependencies
WHERE referenced_id IS NULL --dependency is missing
GO
--Does it work? No
EXEC dbo.procTest
GO
I had been using MySQL as database and had planned to move to postgresql. I had used aes_encrypt and aes_decrypt functions in MySQL extensively throughout my application. So whenever the encryption/decrytion fails, MySQL automatically returns 'null'.
I am unsure how to handle the same in postgresql. Tried using the pgp_sym_encrypt/pgp_sym_decrypt functions. If the encryption key is wrong, it throws error "Wrong key/corrupt data". I tried searching for some functions that could capture this error and return 'null' as in MySQL so that I need not modify my code. I had been searching but could not find one.
Has anybody used any error handling mechanism for individual queries? I had found that error handling can be done for procedures. But, I had to completely rewrite the entire application for that.
If you could share some details, it would be of great help. Thanks.
If you wish to avoid modifying your code and have the functions return NULL on error, you can do this by wrapping them in a PL/PgSQL function that uses a BEGIN ... EXCEPTION block to trap the error.
To do this, first I get the SQLSTATE for the error:
regress=# \set VERBOSITY verbose
regress=# SELECT pgp_sym_decrypt('fred','key');
ERROR: 39000: Wrong key or corrupt data
LOCATION: decrypt_internal, pgp-pgsql.c:607
I could use this directly in the error handler, but I prefer to use a symbolic name, so I look up the error name associated with 39000 in Appendix A - Error codes, finding that it's the generic function call error external_routine_invocation_exception. Not as specific as we would've liked, but it'll do.
Now a wrapper function is required. Something like this must be defined, with one function for each overloaded signature of pgp_sym_decrypt that you wish to support. For the (bytea,text) form that returns text, for example:
CREATE OR REPLACE FUNCTION pgp_sym_decrypt_null_on_err(data bytea, psw text) RETURNS text AS $$
BEGIN
RETURN pgp_sym_decrypt(data, psw);
EXCEPTION
WHEN external_routine_invocation_exception THEN
RAISE DEBUG USING
MESSAGE = format('Decryption failed: SQLSTATE %s, Msg: %s',
SQLSTATE,SQLERRM),
HINT = 'pgp_sym_encrypt(...) failed; check your key',
ERRCODE = 'external_routine_invocation_exception';
RETURN NULL;
END;
$$ LANGUAGE plpgsql;
I've chosen to preseve the original error in a DEBUG level message. Here's a comparison of the original and wrapper, with full message verbosity and debug level output.
Enable debug output to show the RAISE. Note that it also shows the *original query text of the pgp_decrypt_sym call, including parameters.
regress=# SET client_min_messages = DEBUG;
New wrapped function still reports the error if detailed logging is enabled, but returns NULL:
regress=# SELECT pgp_sym_decrypt_null_on_err('redsdfsfdsfd','bobsdf');
LOG: 00000: statement: SELECT pgp_sym_decrypt_null_on_err('redsdfsfdsfd','bobsdf');
LOCATION: exec_simple_query, postgres.c:860
DEBUG: 39000: Decryption failed: SQLSTATE 39000, Msg: Wrong key or corrupt data
HINT: pgp_sym_encrypt(...) failed; check your key
LOCATION: exec_stmt_raise, pl_exec.c:2806
pgp_sym_decrypt_null_on_err
-----------------------------
(1 row)
compared to the original, which fails:
regress=# SELECT pgp_sym_decrypt('redsdfsfdsfd','bobsdf');
LOG: 00000: statement: SELECT pgp_sym_decrypt('redsdfsfdsfd','bobsdf');
LOCATION: exec_simple_query, postgres.c:860
ERROR: 39000: Wrong key or corrupt data
LOCATION: decrypt_internal, pgp-pgsql.c:607
Note that both forms show the parameters the function was called with when it failed. The parameters won't be shown if you've used bind parameters ("prepared statements"), but you should still consider your logs to be security critical if you're using in-database encryption.
Personally, I think it's better to do crypto in the app, so the DB never has access to the keys.