I'm trying to insert into a remote database using DBLink, and I'm making the query through Psycopg2. This is my code that takes a list of column names called fields and a list of column values called values:
fields_str = ",".join(fields)
values_str = ("%s, " * len(values))[:-2]
query = """SELECT dblink_exec('{0}', 'INSERT INTO {1} ({2}) VALUES ({3});'""".format(self.foreign_db,
table,
fields_str,
values_str)
This doesn't work because DBLink expects two single quotes around every string value like in the example in their documentation: SELECT dblink_exec('myconn', 'insert into foo values(21,''z'',''{"a0","b0","c0"}'');');
The problem is that Psycopg2 sticks an "E" in front of all my strings to escape them. That means neither adding single quotes around all my values nor adding single quotes around all my "%s" parts in my query can fix this. The general problem seems to be that Psycopg2 cannot properly handle escaping strings within a DBLink query string.
psycopg2.ProgrammingError: syntax error at or near "f5907160"
LINE 1: ...ar,tag,oid,objecttype,updatedat,usedat) VALUES (E'f5907160-4...
Any way around this?
Related
I have a Lookup that retrieves a few records from a MS SQL table containing schema, table name and a whole where clause. These values are passed to a copy data (within a ForEach) In the copy data i use a Dynamic query statement like:
#concat('select a.*, current_date as crt_tms from ',item().shm_nam,'.',item().tab_nam,
item().where_clause )
This construction works fine without the where_clause or with a where clause with an integer. But it goes wrong with strings like:
'a where a.CODSYSBRN ='XXX' ;'
it's about the quote (')
How can i pass it through?
I know that the where clause as a fixed string in the dynamic query works when i use double quotes (to escape the single quote):
'a where a.CODSYSBRN =''XXX'' ;'
Point is i need the where clause to be completely dynamic because it differ per table
whatever i try i get this kind of error:
Syntax error or access violation;257 sql syntax error: incorrect syntax near "where a"
ps i also tested this, but with the same result:
select a.*, current_date as crt_tms from #{item().shm_nam}.#{item().tab_nam} a #{item().where_clause}
As you have mentioned you are getting whole where clause from the lookup table, the query must have included the column values in where clause for string and integer types separately.
Example lookup table:
In your copy activity, you can use Concat() function as you were already doing it, to combine static values & parameters.
#concat('select * from ',item().schma_name,'.',item().table_name,' ',item().where_clause)
For debugging purposes, I have added the expression in set variable activity, to see the value of the expression.
Iteration1:
Iteration2:
I have the following string:
SELECT hello as "helloHello", '' as "empty" FROM tbl_test
I want to do something like this:
INSERT INTO tbl_x (a, b, c) VALUES (x, y, string_from_above)
The strings are going to be dynamic (basically they're strings of sql statements), so I do not want to escape all the ' and ". If I open the postgres database and double click on the varchar column, I can copy and paste the string and it goes in exactly as it looks. I want to be able to programmatically do so. Is there a way using an insert command? I tried using pgFormat, but that didn't work. See attached pic with what it looks like in the table.
All PostgreSQL APIs worth their money have a way to escape strings for inclusion in a dynamic SQL statement.
For example, if you write in C, you can use libpq's PQescapeLiteral().
There are also SQL functions for this: quote_literal() and format(). They can be used in PL/pgSQL code.
Don't try to write your own code for this. Use prior art and avoid the risk of getting it wrong.
I am having issues with combining couple of different things together. I have tried a lot of things suggested on SO but nothing worked, hence posting my question.
So the requirements are
I need to build the dynamic query (to search jsonb column type) in Java
I need to use prepared statements of Java (to avoid sql injection)
I need to replace parameter which is inside single quotes
This query in CLI works perfectly
select * from items where cnt ## '$.**.text like_regex "#finance" flag "i"';
The bit that I want to parameterise is "#finance". So when I build this query
select * from items where cnt ## '$.**.text like_regex ? flag "i"';
When executing I get following error
"The column index is out of range: 1, number of columns: 0."
It's because ? is within single quote so jdbc driver is not able to identify as replaceable parameter perhaps.
The closest question/discussion I could find is in this post: JDBC Prepared statement parameter inside json.
I have tried it however this does not work for me for some reason. They query now I run is
select * from items where cnt ## ?::jsonb
but with this I get following error
org.postgresql.util.PSQLException: ERROR: operator does not exist: jsonb ## jsonb
Hint: No operator matches the given name and argument types. You might need to add explicit type casts.
Position: 106
I have tried various other ways like escaping single quotes etc but nothing really worked. I have very limited knowledge of PostgreSQL, so any help would be appreciated.
Postgres version: 13.1
Java version: 15
select * from items where cnt ## ?::jsonb
The thing on the right of the ## needs to be a jsonpath, not jsonb. So casting it to jsonb is clearly wrong. Try casting it to jsonpath instead.
I am trying to run an update command on postgresql 11.6 by below syntax
update "YP_SUPPLIERS" set "YP_SUPPLIERS.supplierName" = "update" where "YP_SUPPLIERS.supplierID" = da68e9d0-1100-43e2-0011-db8fbe654321;
I am getting this below error
ERROR: column "YP_SUPPLIERS.supplierID" does not exist
LINE 1: ... set "YP_SUPPLIERS.supplierName" = "update" where "YP_SUPPLI...
tired different combinations by only giving the column name , removing the quotes but nothing seems to be working.
Could any one suggest me a right way to do it.
You need to quote each element separately, and the table does not need to be repeated for the target column. String constants need to be enclosed in single quotes (') in SQL. Double quotes are only for identifiers.
update "YP_SUPPLIERS"
set "supplierName" = 'update' --<< single quotes for constant values
-- ^ no table name here
where "YP_SUPPLIERS"."supplierID" = 'da68e9d0-1100-43e2-0011-db8fbe654321';
-- ^ schema and table name must be quoted separately
For example, I'd like to run:
REGEXP_REPLACE("What's My Name?", "[^a-z0-9_\-]", "-");
and simply see what it returns, instead of doing a search against a DB Table. I tried to run it in the CLI and got
ERROR: syntax error at or near "REGEXP_REPLACE"
LINE 1: REGEXP_REPLACE("What's My Name?", "[^a-z0-9_\-]", "-")
(I'm trying to be generic- I'd like to be able to use this for other PSQL Aggregate Functions as well.)
Remember, this is SQL, so every output you get is a relation. Hence to calculate the result of a function, you need to run SELECT to retrieve the function's value.
Unfortunately, in many DBs, SELECT requires a table. In Oracle land, there's dual to work around this problem:
SELECT REGEXP_REPLACE('What''s My Name?', '[^a-z0-9_\-]', '-') FROM dual;
PostgreSQL, however, allows you to execute a SELECT query without having to specify a table:
SELECT REGEXP_REPLACE('What''s My Name?', '[^a-z0-9_\-]', '-');
Note that the string quote in SQL is ', not ". PostgreSQL uses double quotes to quote identifiers.
Side note: Not every function is an aggregate. An aggregate is a function that combines multiple values into a single output value. REGEXP_REPLACE() is just a normal string function.
Is there a way to run a single PSQL Aggregate Function without hitting a database table?
Yes, in PostgreSQL you can write a SELECT statement without a FROM part at all.