I am trying to use postgresql in Python.The query is against a numeric field value in the where condition. The result set is not fetching and giving error ("psycopg2.ProgrammingError: no results to fetch").There are records in the database with agent_id (integer field) > 1.
import psycopg2
# Try to connect
try:
conn=psycopg2.connect("dbname='postgres' host ='localhost')
except:
print "Error connect to the database."
cur = conn.cursor()
agentid = 10000
try:
sql = 'SELECT * from agent where agent_id > %s::integer;'
data = agentid
cur.execute(sql,data)
except:
print "Select error"
rows = cur.fetchall()
print "\nRows: \n"`
for row in rows:``
print " ", row[9]
Perhaps try these things in your code:
conn=psycopg2.connect("dbname=postgres host=localhost user=user_here password=password_here port=port_num_here")
sql = 'SELECT * from agent where agent_id > %s;'
data = (agentid,) # A single element tuple.
then use
cur.execute(sql,data)
Also, I am confused here to what you want to do with this code
for row in rows:``
print " ", row[9]
Do you want to print each row in rows or just the 8th index of rows, from
rows = cur.fetchall()
If you wanted that index, you could
print rows[9]
Related
I have this python3 code :
conn = psycopg2.connect( ... )
curr = conn.cursor()
curr.execute(code)
rows = curr.fetchall()
where 'code' has the select query statement
After executing this, 'rows' list will have lists of only the selected row values. How do I run 'curr.execute' in such a way that I also get the respective col headers too?
Meaning if I have say
Select col1, col2 from table Where some_condition;
I want my 'rows' list to have something like [['col1', 'col2'], [some_val_for_col1, some_val_for_col2] ....]. Any other ways of getting these col headers are also fine, but the select query in 'code' shouldn't change.
you have to execute 2 commands
curr.execute("Select * FROM people LIMIT 0")
colnames = [desc[0] for desc in curs.description]
curr.execute(code)
you can follow steps mentioned in https://kb.objectrocket.com/postgresql/get-the-column-names-from-a-postgresql-table-with-the-psycopg2-python-adapter-756
I would expect to always receive a resultset with one row on a SELECT COUNT, but results.next() always returns false. This is on HSQLDB 2.5.1.
The code below prints:
number of columns: 1. First column C1 with type INTEGER
No COUNT results
statement = connection.createStatement();
// check if table empty
statement.executeQuery("SELECT COUNT(*) FROM mytable");
ResultSet results = statement.getResultSet();
System.out.println("number of columns: " + results.getMetaData().getColumnCount() + ". First column " +results.getMetaData().getColumnName(1) + " with type " +results.getMetaData().getColumnTypeName(1) );
int numberOfRows = 0;
boolean hasResults = results.next();
if (hasResults){
numberOfRows = results.getInt(1);
System.out.println("Table size " + numberOfRows );
}else{
System.out.println("No COUNT results" );
}
statement.close();
Executing the same SQL statement in my SQL console works fine:
C1
104
Other JDBC actions on this database work fine as well.
Is there something obvious I'm missing?
The getResultSet method is applicable to execute, but not executeQuery which returns a ResultSet. That is the one you need to refer to, at the moment you are losing it as you are not assigning it to anything.
See https://docs.oracle.com/javase/7/docs/api/java/sql/Statement.html#executeQuery(java.lang.String) and https://docs.oracle.com/javase/7/docs/api/java/sql/Statement.html#getResultSet()
ResultSet results = statement.executeQuery("SELECT COUNT(*) FROM mytable");
I have a huge data in elastic search and i want to write a script to create a table corresponding to a particular index and transfer all the data to postgres.
Nevermind, I got my answer. What i did was to
create a connection with postgres and elastic search
create table in postgresql
store data in chunks of 10k in a list of dictionary.
transfer data from that list of dictionary in the postgresql and then empty the list for next iteration.
import psycopg2
from elasticsearch import Elasticsearch
from elasticsearch_dsl import Search
from collections import defaultdict
dict = defaultdict(list)
t_host = "localhost"
t_port = "9200"
t_dbname_ES = "companydatabase" #index
t_user = "elastic"
t_pw = "changeme"
client_ES = Elasticsearch([t_host],http_auth=(t_user, t_pw),port=t_port)
t_host = "localhost"
t_port = "5999"
t_dbname = "postgres"
t_user = "postgres"
t_pw = "postgres"
db_conn = psycopg2.connect(host=t_host, port=t_port, dbname=t_dbname, user=t_user, password=t_pw)
db_cursor = db_conn.cursor()
column_name_list = ["Address","Age","DateOfJoining","Designation","FirstName","Gender","Interests","LastName","MaritalStatus","Salary"]
column_type_list = ["text not null","integer","date","text","text","text","text","text","text","text","text","text","integer"]
table_name = 'sample_table2' #table name to insert data into
column_names = ', '.join(column_name_list)
column_types = ", ".join(column_type_list)
#table creation
create_table_query = "CREATE TABLE {} (".format(table_name)
for i in range(len(column_name_list)):
create_table_query += column_name_list[i]
create_table_query += " "
create_table_query += column_type_list[i]
if i != len(column_name_list) - 1:
create_table_query += ", "
create_table_query += ");"
try:
db_cursor.execute(create_table_query)
db_conn.commit()
except psycopg2.Error as e:
t_message = "Database error: " + e
#data insertion
s = Search(index=t_dbname_ES).using(client_ES).query("match_all")
total_documents = s.count() #total count of records in the index
count=0
for hit in s.scan(): #looping over all records one at a time
count+=1
total_documents -=1
for i in range(len(column_name_list)): #appending the data fethed from document in a list of dictionary.
dict[column_name_list[i]].append(hit[column_name_list[i]])
if count==10000 or total_documents==0: #appending data in postgres 10k records at a time
insert_query = "INSERT INTO "+table_name+" (" + column_names + ")"+" VALUES"
for i in range(min(10000,count)):
insert_query += "("
for j in range(len(column_name_list)):
if j!=0:
insert_query+=', '+ "'"+str(dict[column_name_list[j]][i])+"'"
else:
insert_query+="'"+str(dict[column_name_list[j]][i])+"'"
insert_query += "),"
insert_query= insert_query[:-1]
insert_query += ";"
for i in range(len(column_name_list)): #making the list empty for next iteration of 10k records
dict[column_name_list[i]]=[]
try:
db_cursor.execute(insert_query)
db_conn.commit()
count=0
except psycopg2.Error as e:
t_message = "Database error: " + e
db_cursor.close()
db_conn.close()
In python 3+, I want to insert values from a dictionary (or pandas dataframe) into a database. I have opted for psycopg2 with a postgres database.
The problems is that I cannot figure out the proper way to do this. I can easily concatenate a SQL string to execute, but the psycopg2 documentation explicitly warns against this. Ideally I wanted to do something like this:
cur.execute("INSERT INTO table VALUES (%s);", dict_data)
and hoped that the execute could figure out that the keys of the dict matches the columns in the table. This did not work. From the examples of the psycopg2 documentation I got to this approach
cur.execute("INSERT INTO table (" + ", ".join(dict_data.keys()) + ") VALUES (" + ", ".join(["%s" for pair in dict_data]) + ");", dict_data)
from which I get a
TypeError: 'dict' object does not support indexing
What is the most phytonic way of inserting a dictionary into a table with matching column names?
Two solutions:
d = {'k1': 'v1', 'k2': 'v2'}
insert = 'insert into table (%s) values %s'
l = [(c, v) for c, v in d.items()]
columns = ','.join([t[0] for t in l])
values = tuple([t[1] for t in l])
cursor = conn.cursor()
print cursor.mogrify(insert, ([AsIs(columns)] + [values]))
keys = d.keys()
columns = ','.join(keys)
values = ','.join(['%({})s'.format(k) for k in keys])
insert = 'insert into table ({0}) values ({1})'.format(columns, values)
print cursor.mogrify(insert, d)
Output:
insert into table (k2,k1) values ('v2', 'v1')
insert into table (k2,k1) values ('v2','v1')
I sometimes run into this issue, especially with respect to JSON data, which I naturally want to deal with as a dict. Very similar. . .But maybe a little more readable?
def do_insert(rec: dict):
cols = rec.keys()
cols_str = ','.join(cols)
vals = [ rec[k] for k in cols ]
vals_str = ','.join( ['%s' for i in range(len(vals))] )
sql_str = """INSERT INTO some_table ({}) VALUES ({})""".format(cols_str, vals_str)
cur.execute(sql_str, vals)
I typically call this type of thing from inside an iterator, and usually wrapped in a try/except. Either the cursor (cur) is already defined in an outer scope or one can amend the function signature and pass a cursor instance in. I rarely insert just a single row. . .And like the other solutions, this allows for missing cols/values provided the underlying schema allows for it too. As long as the dict underlying the keys view is not modified as the insert is taking place, there's no need to specify keys by name as the values will be ordered as they are in the keys view.
[Suggested answer/workaround - better answers are appreciated!]
After some trial/error I got the following to work:
sql = "INSERT INTO table (" + ", ".join(dict_data.keys()) + ") VALUES (" + ", ".join(["%("+k+")s" for k in dict_data]) + ");"
This gives the sql string
"INSERT INTO table (k1, k2, ... , kn) VALUES (%(k1)s, %(k2)s, ... , %(kn)s);"
which may be executed by
with psycopg2.connect(database='deepenergy') as con:
with con.cursor() as cur:
cur.execute(sql, dict_data)
Post/cons?
using %(name)s placeholders may solve the problem:
dict_data = {'key1':val1, 'key2':val2}
cur.execute("""INSERT INTO table (field1, field2)
VALUES (%(key1)s, %(key2)s);""",
dict_data)
you can find the usage in psycopg2 doc Passing parameters to SQL queries
Here is another solution inserting a dictionary directly
Product Model (has the following database columns)
name
description
price
image
digital - (defaults to False)
quantity
created_at - (defaults to current date)
Solution:
data = {
"name": "product_name",
"description": "product_description",
"price": 1,
"image": "https",
"quantity": 2,
}
cur = conn.cursor()
cur.execute(
"INSERT INTO products (name,description,price,image,quantity) "
"VALUES(%(name)s, %(description)s, %(price)s, %(image)s, %(quantity)s)", data
)
conn.commit()
conn.close()
Note: The columns to be inserted is specified on the execute statement .. INTO products (column names to be filled) VALUES ..., data <- the dictionary (should be the same **ORDER** of keys)
(or How to iterate thru information schema Using perl DBI (DBD::PG) and placeholders?)
Windows 7, ActiveState Perl 5.20.2, PostgreSQL 9.4.1 .
Cases A, B and C below were successful when using a placeholder for a COLUMN VALUE. In order
no placeholder used
passed a literal
passed a variable (populated with same literal)
It would be great to raise it up a level to DB Objects.. (tables, views etc)
Here's the output with the error for Case D:
Z:\CTAM\data_threat_mapping\DB Stats\perl scripts>test_placeholder.pl
A Row Count: 1
B Row Count: 1
C Row Count: 1
DBD::Pg::st execute failed: ERROR: syntax error at or near "$1"
LINE 1: SELECT COUNT(*) FROM $1 WHERE status = 'Draft';
^ at Z:\CTAM\data_threat_mapping\DB Stats\perl
scripts\test_placeholder.pl line 34.
Much obliged for any direction!
#!/usr/bin/perl -w
use strict;
use diagnostics;
use DBI;
my $num_rows = 0;
# connect
my $dbh = DBI->connect("DBI:Pg:dbname=CTAM;host=localhost",
"postgres", "xxxxx",
{ 'RaiseError' => 1, pg_server_prepare => 1 });
#---------------------
# A - success
my $sthA = $dbh->prepare(
"SELECT COUNT(*) FROM cwe_compound_element WHERE status = 'Draft';"
);
$sthA->execute(); # no placeholders
#---------------------
# B - success
my $sthB = $dbh->prepare (
"SELECT COUNT(*) FROM cwe_compound_element WHERE status = ?;"
);
$sthB->execute('Draft'); # pass 'Draft' to placeholder
#---------------------
# C - success
my $status_value = 'Draft';
my $sthC = $dbh->prepare(
"SELECT COUNT(*) FROM cwe_compound_element WHERE status = ?;"
);
$sthC->execute($status_value); # pass variable (column value) to placeholder
#---------------------
# D - failure
my $sthD = $dbh->prepare(
"SELECT COUNT(*) FROM ? WHERE status = 'Draft';"
);
$sthD->execute('cwe_compound_element'); # pass tablename to placeholder
I've tried single/double/sans quotes (q, qq)...
If
SELECT * FROM Foo WHERE field = ?
means
SELECT * FROM Foo WHERE field = 'val'
then
SELECT * FROM ?
means
SELECT * FROM 'Table'
and that's obviously wrong. Placeholders can only be used in an expression. Fix:
my $sthD = $dbh->prepare("
SELECT COUNT(*)
FROM ".$dbh->quote_identifier($table)."
WHERE status = 'Draft'
");
$sthD->execute();