I'm using mysql.connector and Python 3.7.
For SQL queries, I use this method:
cursor.execute("INSERT INTO table VALUES (%s, %s, %s)", (var1, var2, var3))
The thing is... when I want to print or debug the entire sql query, I can't get the values instead of the %s (I know the whole point is that you can't).
How do you guys do to debug this, then?
Thanks.
The syntax for printing formatted strings is the following:
name = "John"
print("Hello, %s!" % name)
You can apply that to your SQL statement and print it like this:
print("INSERT INTO table VALUES (%s, %s, %s)" % (var1, var2, var3))
Related
I have this table in postgres
CREATE TABLE target (
a json
b integer
c text []
id integer
CONSTRAINT id_fkey FOREIGN KEY (id)
REFERENCES public.other_table(id) MATCH SIMPLE
ON UPDATE NO ACTION
ON DELETE NO ACTION,
)
Which I would like to insert data to from psycopg2 using
import psycopg2
import psycopg2.extras as extras
# data is of the form dict, integer, list(string), string <- used to get fkey id
data = [[extras.Json([{'a':1,'b':2}, {'d':3,'e':2}]), 1, ['hello', 'world'], 'ident1'],
[extras.Json([{'a':4,'b':3}, {'d':1,'e':9}]), 5, ['hello2', 'world2'], 'ident2']]
# convert data to list of tuples containing objects
x = [tuple(u) for u in data]
# insert data to the database
query = ('WITH ins (a, b, c, ident) AS '
'(VALUES %s) '
'INSERT INTO target (a, b, c, id) '
'SELECT '
'ins.a '
'ins.b '
'ins.c '
'other_table.id'
'FROM '
'ins '
'LEFT JOIN other_table ON ins.ident = other_table.ident;')
cursor = conn.cursor()
extras.execute_values(cursor, query, x)
When I run this I get the error: column "a" is of type json but expression is of type text. I tried to solve this by adding a type cast in the SELECT statement but then I got the same error for c and then for b.
Originally I thought the problem lies in the WITH statement but based on the answers to my previous question this seems to not be the case Postgres `WITH ins AS ...` casting everything as text
It seems that execute_values is sending all the values as text with ' '.
Main Question: How can I get execute_values to send the values based on their python data type rather than just as text?
Sub questions:
How can I confirm that execute_values is in fact sending the values as text with quotation marks?
What is the purpose of the template argument of execute_values https://www.psycopg.org/docs/extras.html and could that be of help?
The issue, as Adrian Klaver points out in their comment, and also seen in this answer, is that the typing is lost in the CTE.
We can show this with an example in the psql shell:
CREATE TABLE test (col1 json);
WITH cte (c) AS (VALUES ('{"a": 1}'))
INSERT INTO test (col) SELECT c FROM cte;
resulting in
ERROR: column "col" is of type json but expression is of type text
whereas this version, with the type specified, succeeds:
WITH cte(c) AS (VALUES ('{"a": 1}'::json))
INSERT INTO test (col) SELECT c FROM cte;
We can mimic this in execute_valuesby providing the typing information in the template argument:
extras.execute_values(cursor, query, data, template='(%s::json, %s, %s, %s)')
In python 3+, I want to insert values from a dictionary (or pandas dataframe) into a database. I have opted for psycopg2 with a postgres database.
The problems is that I cannot figure out the proper way to do this. I can easily concatenate a SQL string to execute, but the psycopg2 documentation explicitly warns against this. Ideally I wanted to do something like this:
cur.execute("INSERT INTO table VALUES (%s);", dict_data)
and hoped that the execute could figure out that the keys of the dict matches the columns in the table. This did not work. From the examples of the psycopg2 documentation I got to this approach
cur.execute("INSERT INTO table (" + ", ".join(dict_data.keys()) + ") VALUES (" + ", ".join(["%s" for pair in dict_data]) + ");", dict_data)
from which I get a
TypeError: 'dict' object does not support indexing
What is the most phytonic way of inserting a dictionary into a table with matching column names?
Two solutions:
d = {'k1': 'v1', 'k2': 'v2'}
insert = 'insert into table (%s) values %s'
l = [(c, v) for c, v in d.items()]
columns = ','.join([t[0] for t in l])
values = tuple([t[1] for t in l])
cursor = conn.cursor()
print cursor.mogrify(insert, ([AsIs(columns)] + [values]))
keys = d.keys()
columns = ','.join(keys)
values = ','.join(['%({})s'.format(k) for k in keys])
insert = 'insert into table ({0}) values ({1})'.format(columns, values)
print cursor.mogrify(insert, d)
Output:
insert into table (k2,k1) values ('v2', 'v1')
insert into table (k2,k1) values ('v2','v1')
I sometimes run into this issue, especially with respect to JSON data, which I naturally want to deal with as a dict. Very similar. . .But maybe a little more readable?
def do_insert(rec: dict):
cols = rec.keys()
cols_str = ','.join(cols)
vals = [ rec[k] for k in cols ]
vals_str = ','.join( ['%s' for i in range(len(vals))] )
sql_str = """INSERT INTO some_table ({}) VALUES ({})""".format(cols_str, vals_str)
cur.execute(sql_str, vals)
I typically call this type of thing from inside an iterator, and usually wrapped in a try/except. Either the cursor (cur) is already defined in an outer scope or one can amend the function signature and pass a cursor instance in. I rarely insert just a single row. . .And like the other solutions, this allows for missing cols/values provided the underlying schema allows for it too. As long as the dict underlying the keys view is not modified as the insert is taking place, there's no need to specify keys by name as the values will be ordered as they are in the keys view.
[Suggested answer/workaround - better answers are appreciated!]
After some trial/error I got the following to work:
sql = "INSERT INTO table (" + ", ".join(dict_data.keys()) + ") VALUES (" + ", ".join(["%("+k+")s" for k in dict_data]) + ");"
This gives the sql string
"INSERT INTO table (k1, k2, ... , kn) VALUES (%(k1)s, %(k2)s, ... , %(kn)s);"
which may be executed by
with psycopg2.connect(database='deepenergy') as con:
with con.cursor() as cur:
cur.execute(sql, dict_data)
Post/cons?
using %(name)s placeholders may solve the problem:
dict_data = {'key1':val1, 'key2':val2}
cur.execute("""INSERT INTO table (field1, field2)
VALUES (%(key1)s, %(key2)s);""",
dict_data)
you can find the usage in psycopg2 doc Passing parameters to SQL queries
Here is another solution inserting a dictionary directly
Product Model (has the following database columns)
name
description
price
image
digital - (defaults to False)
quantity
created_at - (defaults to current date)
Solution:
data = {
"name": "product_name",
"description": "product_description",
"price": 1,
"image": "https",
"quantity": 2,
}
cur = conn.cursor()
cur.execute(
"INSERT INTO products (name,description,price,image,quantity) "
"VALUES(%(name)s, %(description)s, %(price)s, %(image)s, %(quantity)s)", data
)
conn.commit()
conn.close()
Note: The columns to be inserted is specified on the execute statement .. INTO products (column names to be filled) VALUES ..., data <- the dictionary (should be the same **ORDER** of keys)
I want to execute following raw SQL with Korma:
k/exec-raw
["INSERT INTO events ?, VALUES ? ON CONFLICT (id) DO UPDATE SET title = EXCLUDED.title;" [keys values]]
with params equal to:
keys (str "(" (keys->str res) ")")
values (str "(" (serialize (merge res) ", ") ")" )
The both evaluate to correct strings and work in repl.
But in runtime i have following error in psql console:
ERROR: syntax error at or near "$1" at character 20
STATEMENT: INSERT INTO events $1, VALUES $2 ON CONFLICT (id) DO UPDATE SET title = EXCLUDED.title
Can't figure out what's the problem. Anybody have ever done insert with Korma?
PostgreSQL 9.5 + Korma 0.4.2
Only values can be used as parameters in a prepared statement. So to make it work:
INSERT INTO events (column) VALUES (?)
If you want to work it like you do you have to prepare the sql string yourself and make sure you don't have an sql injection.
Please see the manual
I'm currently trying to insert data into a database from a text boxes, $enter / $enter2 being where the text is being written.
The database consists of three columns ID, name and nametwo
ID is auto incrementing and works fine
Both statements work fine on their own, but because they are being issued separately the first leaves nametwo blank and the second leaves name blank.
I've tried combining both but haven't had much luck, hope someone can help.
$dbh->do("INSERT INTO $table(name) VALUES ('".$enter."')");
$dbh->do("INSERT INTO $table(nametwo) VALUES ('".$enter2."')");
To paraphrase what others have said:
my $sth = $dbh->prepare("INSERT INTO $table(name,nametwo) values (?,?)");
$sth->execute($enter, $enter2);
So you don't have to worry about quoting.
You should read database manual.
The query should be:
$dbh->do("INSERT INTO $table(name,nametwo) VALUES ('".$enter."', '".$enter2."')");
The SQL syntax is
INSERT INTO MyTable (
name_one,
name_two
) VALUES (
"value_one",
"value_two"
)
Your way of generating SQL statements is very fragile. For example, it will fail if the table name is Values or the value is Jester's.
Solution 1:
$dbh->do("
INSERT INTO ".$dbh->quote_identifier($table_name)."
name_one,
name_two
) VALUES (
".$dbh->quote($value_one).",
".$dbh->quote($value_two)."
)
");
Solution 2: Placeholders
$dbh->do(
" INSERT INTO ".$dbh->quote_identifier($table_name)."
name_one,
name_two
) VALUES (
?, ?
)
",
undef,
$value_one,
$value_two,
);
I want to insert values into Sqlite3 table using Perl DBI. I was able to insert hard coded values without any problem. When I tried to use perl variables, then I get an error "DBD::SQLite::db do failed: no such column:"
This works:
$dbh->do("insert into Gene values (12, 'AAAAAA', 66, 86, 76)");
But this code
$dbh->do("INSERT INTO Gene values (NULL, $sequence, $siteNumber, $begin, $length)");
throws the error
DBD::SQLite::db do failed: no such column
You should always prepare and execute your SQL statements and use placeholders for variable values as a matter of course.
Try this code
my $sth = $dbh->prepare('INSERT INTO Gene VALUES (?, ?, ?, ?, ?)');
$sth->execute(undef, $sequence, $siteNumber, $begin, $length);
Your problem will have been at $sequence, which required quoting to be a valid SQL string literal in the insert statement. As a result it thought you were trying to refer to a column called AAAAAA, and not the string 'AAAAAA' that you intended. Perl's string interpolation doesn't add the quotes for you.
Placeholders and $dbh->prepapre are by far the most robust solution here though (for example, what to do if your string contains the quote character ' ? $dbh->prepare already has that coded correctly).