I can't seem to be able to create migration with simplest sql to create PostgreSQL procedure.
Error is:
Exception data:
Severity: ERROR
SqlState: 42601
MessageText: syntax error at end of input
Position: 186
File: scan.l
Line: 1184
Routine: scanner_yyerror
Sql code works fine when executed in pgAdmin.
protected override void Up(MigrationBuilder migrationBuilder)
{
var sp = #"CREATE PROCEDURE reset_primary_holder()
LANGUAGE SQL
BEGIN ATOMIC
SELECT * FROM id.users;
END;";
migrationBuilder.Sql(sp);
}
What am I doing wrong?
Npgsql (the ADO.NET driver, not the EF provider) by default parses SQL to find semicolons and rewrite statements, and the new PostgreSQL syntax breaks that parser; see https://github.com/npgsql/npgsql/issues/4445.
The general recommended fix is to add the following to the start of your program:
AppContext.SetSwitch("Npgsql.EnableSqlRewriting", false);
This disables statement parsing/rewriting altogether; it means you need to use positional parameter placeholders ($1, $2 instead of #p1, #p2) - see the docs.
However, EF is unfortunately not yet compatible with this mode, so you'll have to make do without this syntax...
In the end only viable solution was to to use different syntax:
protected override void Up(MigrationBuilder migrationBuilder)
{
var sp = #"CREATE PROCEDURE my_procedure()
AS $$
UPDATE id.users
SET ...;
Another statement;
$$ LANGUAGE sql;";
migrationBuilder.Sql(sp);
}
This syntax is not mentioned in documentation on stored procedures and since this is my first PG procedure / function it was not immediately obvious to me that there is alternative.
Thanks #Shay for pointing me in the right direction.
Related
I started with PostgreSQL recently and I'm running into trouble calling a procedure from a PyQt5 client using the psycopg2 driver.
I set a simple procedure with 6 parameters, included a BEGIN and a COMMIT statements and run seamlessly from PgAdmin 4.
When tried to call the procedure from the PyQt5 client a got error code 2D000 with the following message:Invalid transaction termination
CONTEXT: PL/pgSQL function es_edit_text_report(integer,date,integer,integer,character varying,character varying) line 10 at COMMIT
I found several mentions to this error both in stackoverflow and in the web but I could not figure it out why am I getting this results. The closer explanation found may be by #Laurenz Albe on Invalid transaction termination but it looks to me that none of the mentioned conditions apply. Here we have a very simple procedure being called just once from a psycopg2 connection opened from a pool for this single purpose.
Following other responses I just commented the commit on the procedure and then I was able to run it from the client. This places some further questions:
I understand that the transaction is not operative any longer, consequently you would not be able to run complex procedures safely unless this issue is solve. I'll appreciate your thoughts about it.
Procedure Code:
CREATE OR REPLACE PROCEDURE es_edit_text_report(IN _id integer, IN _report_date,
IN _responsibleid integer,
IN _categoryid integer,
IN _description varchar,
IN _location varchar)
AS $$
BEGIN
UPDATE text_maintenance_news
SET report_date = _report_date,
reporterid = _responsibleid::smallint,
categoryid = _categoryid::smallint,
description = _description,
location = _location
WHERE id = _id;
COMMIT;
END; $$
LANGUAGE plpgsql
PyQt5 client code:
#pyqtSlot()
def save(self):
try:
with self.connPool.getconn() as conn:
with conn.cursor() as cur:
if self.mode == OPEN_NEW:
cur.execute('CALL es_load_text_report(%s, %s, %s, %s, %s)',(
self.reportDate.date().toString("yyyy-MM-dd"),
self.comboResponsible.getHiddenData(0),
self.comboCategory.getHiddenData(0),
self.txtDescription.text(),
self.txtLocation.text()))
conn.commit()
else:
cur.execute('CALL es_edit_text_report(%s, %s, %s, %s, %s, %s)',
(self.id,
self.reportDate.date().toString("yyyy-MM-dd"),
self.comboResponsible.getHiddenData(0),
self.comboCategory.getHiddenData(0),
self.txtDescription.text(),
self.txtLocation.text()))
conn.commit()
qry = QSqlQuery(self.db)
qry.exec("SELECT * FROM es_load_text_reports()")
if qry.lastError().type() != 0:
raise DataError('save: qry', qry.lastError().text())
self.tableNews.model().setQuery(qry)
self.clear()
except OSError as e:
QMessageBox.warning(self, 'Save : ', e.args, QMessageBox.Ok)
except DataError as e:
QMessageBox.warning(self, e.source, e.message, QMessageBox.Ok)
except DatabaseError as e:
QMessageBox.warning(self, 'Save', f'{e.pgcode} {e.pgerror}', QMessageBox.Ok)ter code here
Please notice I'm using a QSqlQuery to retrive the updated table data. The reason being is I'm still trying to figure out how to handle psycopg2 list of tuples return data.
UPDATE: So far the only workaround I was able to find is to remove the COMMIT line from the procedure. I'm under the impression that Postgres is transferring the transaction control to the driver. I wonder if this is the proper way to do it. I'll appreciate your comments about it.
With the psycopg2 driver, you have to set autocommit to True for statements that must run outside a transaction. From the docs:
A few commands (e.g. CREATE DATABASE, VACUUM, CALL on stored
procedures using transaction control…) require to be run outside any
transaction: in order to be able to run these commands from Psycopg,
the connection must be in autocommit mode: you can use the autocommit
property.
In your code add this line:
conn.autocommit = True
and then your CALL statement should run.
Here's my MRE
create procedure:
CREATE OR REPLACE PROCEDURE test_procedure(
par1 integer,
par2 integer)
LANGUAGE 'plpgsql'
AS $BODY$
BEGIN
COMMIT;
END;
$BODY$;
Python code:
import sys
import psycopg2
print(sys.version) # 3.7.1 and 3.8.2 tested
print(psycopg2.__version__) # 2.8.6 and 2.9.3 tested
conn = psycopg2.connect('host=myhost dbname=mydb user=myuser')
cur = conn.cursor()
conn.autocommit = True # this is the key line. without this, next line will fail
cur.execute('CALL aaa.test_procedure(%s,%s)', (1,1))
conn.commit()
Common sense dictates that SQL query strings should never be assembled by hand. Thus, all database interfaces offer parameter substitution, and all users use it, without exceptions.*
I'm using PostgreSQL v10.5, nodejs v8.12.0, node-postgres 7.6.1.
Parameter substitution works as expected for SELECT statements:
> await db.query("select from users where id = 'mic'");
(success, 1 row returned)
> await db.query("select from users where id = $1", ["mic"]);
(success, 1 row returned)
But it doesn't work for LISTEN statements:
> await db.query("listen topicname");
(success)
> await db.query("listen $1", ["topicname"]);
(error: syntax error at or near "$1")
The name of the topic I want to listen to is dynamic. It is coming from semi-trustworthy sources, which should not be user-controllable. But why go against all established best practice and take any chances?
Unfortunately, from my tests I fear that PostgreSQL simply can't do parameter substitution for LISTEN queries.
Is there any solution or workaround for this?
*) This statement may only be true in some utopic future society.
I don't have enough reputation to comment on the answer, but the proposed solution doesn't work for me.
Using %L results in a quoted string, which causes the following error:
ERROR: syntax error at or near "'topic'"
The %I format should be used instead (SQL identifier, this is documented for table and column names, but it also works for the channel name,). You can also use the quote_ident function. See the documentation on creating dynamic queries here.
The following PL/pgSQL function works for us:
CREATE OR REPLACE FUNCTION listenForChannel(
channel_ TEXT
) RETURNS VOID AS $$
BEGIN
EXECUTE format('LISTEN %I', channel_);
END
$$ LANGUAGE PLPGSQL;
You are right that this cannot be done in PostgreSQL.
As a workaround, write a PL/pgSQL function that uses dynamic SQL like this:
EXECUTE format('LISTEN %L', topicname);
The format function escapes strings properly; in this case, the %L format that produces a properly quoted string Literal is the appropriate one.
I have written a function in PostgreSQL for insertion as follows:
CREATE OR REPLACE FUNCTION public.insert_blog("Url" character)
RETURNS void AS
$BODY$Begin
Insert Into "Blogs"("Url") Values("Url");
End$BODY$
LANGUAGE plpgsql VOLATILE
COST 100;
ALTER FUNCTION public.insert_blog(character)
OWNER TO postgres;
The above function adds an entry into the Blogs table (Url is a parameter).
I am trying to use this function in .Net Core (Npgsql.EntityFrameworkCore.PostgreSQL) as follows:
[HttpPost]
[ValidateAntiForgeryToken]
public IActionResult Create(Blog blog)
{
if (ModelState.IsValid)
{
//This works fine
var count = _context.Blogs.FromSql("Select insert_blog({0})", blog.Url).Count();
//This doesn't work -- it gives an error of "42601: syntax error at or near "insert_blog""
//var count = _context.Blogs.FromSql("insert_blog #Url={0}", blog.Url).Count();
return RedirectToAction("Index");
}
return View(blog);
}
Can someone tell me why the second command is not working? Also, even if the first command is working, is it the right way?
Since I have to write .FromSql(...).Count() in order for it to work, if I remove .Count() item doesn't get inserted. Can someone tell me why this is happening?
Is there any good article on using .FromSql() or "Using Postgres functions in entity framework core" (I'd guess that this is a new feature and that that's why I couldn't find much data on this)?
Can some one tell me why the second command is not working? Also even if the first command is working, is it the right way?
It's simply just not the way PostgreSQL syntax works. Select insert_blog({0}) is indeed the right way.
Since I have to write .FromSql(...).Count() in order for it to work. If I remove ".Count()" item doesn't get inserted. Can someone tell me why this is happening?
FromSql behaves just like Where and other functions on an IQueryable. Execution is postponed until the results are requested, because it will try to do everything in one database query.
To make sure your query actually gets executed, you need to call a method that returns something other than IQueryable such as .Count() or .ToList(). More info can be found here: https://learn.microsoft.com/en-us/ef/core/querying/overview#when-queries-are-executed
I just noticed that I could alter my stored procedure code with a misspelled user defined function in it.
I noticed that at 1st time I execute the SP.
Is there any way to get a compile error when an SP include an invalid user-defined function name in it?
At compile time? No.
You can, however, use some of SQL's dependency objects (if using MS SQL) to find problems just after deployment, or as part of your beta testing. Aaron Bertran has a pretty nice article rounding up the options, depending upon the version of SQL Server.
Here is an example using SQL Server 2008 sys object called sql_expression_dependencies
CREATE FUNCTION dbo.scalarTest
(
#input1 INT,
#input2 INT
)
RETURNS INT
AS
BEGIN
-- Declare the return variable here
DECLARE #ResultVar int
-- Add the T-SQL statements to compute the return value here
SELECT #ResultVar = #input1 * #input2
-- Return the result of the function
RETURN #ResultVar
END
GO
--Fn Works!
SELECT dbo.ScalarTest(2,2)
GO
CREATE PROCEDURE dbo.procTest
AS
BEGIN
SELECT TOP 1 dbo.scalarTest(3, 3) as procResult
FROM sys.objects
END
GO
--Sproc Works!
EXEC dbo.procTest
GO
--Remove a dependency needed by our sproc
DROP FUNCTION dbo.scalarTest
GO
--Does anything have a broken dependency? YES
SELECT OBJECT_NAME(referencing_id) AS referencing_entity_name,
referenced_entity_name, *
FROM sys.sql_expression_dependencies
WHERE referenced_id IS NULL --dependency is missing
GO
--Does it work? No
EXEC dbo.procTest
GO
Though I have been using SQL Server, Oracle from last decade, I have been asked to
do some research on PostgreSQL and after some initial investigation it is evident that I am now stuck on retrieving data from the PostgreSQL database using Function.
Using following piece of code to retrieve the data and getting error
('ERROR [26000] ERROR: prepared statement "mytabletest" does not exist;
'Error while executing the query)
Code Snippets
Dim oDBCommand As DbCommand = GetDBCommand(oConnectionType, "mytabletest", CommandType.StoredProcedure)
Dim dstResults As DataSet = GetDataSet(ConnectionTypes.ODBC, oDBCommand)
Public Function GetDataReader(dbType As ConnectionTypes, command As DbCommand) As DbDataReader
Try
Dim oConnection As DbConnection = GetDBConnection(dbType)
Dim oDBTransaction As DbTransaction = oConnection.BeginTransaction
command.Connection = oConnection
command.Transaction = oDBTransaction
'GETTING ERROR ON FOLLOWING LINE
'ERROR [26000] ERROR: prepared statement "mytabletest" does not exist;
'Error while executing the query
return command.ExecuteReader()
Catch ex As Exception
Throw ex
Finally
End Try
Return Nothing
End Function
Environement I am currently working on is following:-
32 Bit Machine.
Visual Studio 2010 + SP1
ODBC Prodiver: PostgreSQL Unicode 9.01.02.00
ADO.Net (System.Data.Odbc)
Please note that I am open to any suggestions i.e. if I am completely doing it wrong
OR partially etc. Please feel free to write.
In order to make it easier for you to create a same environment, please use following table/function definition.
--- Simple table to make things easier to understand. <br>
CREATE TABLE mytable
(
messagetypeid integer NOT NULL,
messagetype character varying(100) NOT NULL
)
-- Function to retrieve data. <br>
CREATE OR REPLACE FUNCTION mytabletest() <br>
RETURNS SETOF refcursor AS $$
DECLARE
ref1 refcursor;
BEGIN
OPEN ref1 FOR SELECT * FROM mytable;
RETURN NEXT ref1;
END;
$$ LANGUAGE plpgsql;
Please Note:
If I use <br>
Dim oDBCommand As DbCommand = GetDBCommand(oConnectionType, "SELECT * FROM mytable", CommandType.Text)
then system manages to retrieve information from the datbase without any issue, however, as I mentioned as soon we use "Function" it throws an exception.
During my failed efforts to search any solution from the internet someone mentioned that Table should be created with the lower case it so just for the sake of it I recreated with the lower case, however, problem persists.
I am unfamiliar with .net but I suspect you meant something more like:
GetDBCommand(oConnectionType, "SELECT myfunc()", CommandType.Text)
Or in the case of SETOF functions etc..
GetDBCommand(oConnectionType, "SELECT * FROM myfunc()", CommandType.Text)
PostgreSQL does not have 'stored procedures' per-ce. It does have functions and I believe that the client/server protocol has a method for preparing statements that can then be executed multiple times with different variables (to save on the cost of parsing the SQL), but this should be exposed via your client library.