I want to create a table from Pytnon:
import psycopg2 as pg
from psycopg2 import sql
conn = pg.connect("dbname=test user=test")
table_name = "testDB"
column_name = "mykey"
column_type = "bigint"
cu = conn.cursor()
cu.execute(sql.SQL("CREATE TABLE {t} ({c} {y})").format(
t=sql.Identifier(table_name),
c=sql.Identifier(column_name),
y=sql.Literal(column_type)))
Alas, this emits CREATE TABLE "testDB" ("mykey" 'bigint') which fails with a
psycopg2.ProgrammingError: syntax error at or near "'bigint'"
Of course, I can do something like
cu.execute(sql.SQL("CREATE TABLE {t} ({c} %s)" % (column_name)).format(
t=sql.Identifier(table_name),
c=sql.Identifier(column_name)))
but I suspect there is a more elegant (and secure!) solution.
PS. See also How to make psycopg2 emit nested quotes?
There is an example in the documentation how to build a query text with a placeholder. Use psycopg2.extensions.AsIs(object) for column_type:
query = sql.SQL("CREATE TABLE {t} ({c} %s)").format(
t=sql.Identifier(table_name),
c=sql.Identifier(column_name)).as_string(cu)
cu.execute(query, [AsIs(column_type)])
Related
As per "relation does not exist" in pg_table_size, I need to emit nested single and double quotes:
import psycopg2 as pg
from psycopg2 import sql
conn = pg.connect("dbname=test user=test")
table_name = "testDB"
cu = conn.cursor()
cu.execute(sql.SQL("SELECT pg_table_size(%s)"), (table_name,))
emits SELECT pg_table_size('testDB') which raises
psycopg2.ProgrammingError: relation "testdb" does not exist
while
cu.execute(sql.SQL("SELECT pg_table_size({t})").format(t=sql.Identifier(table_name)))
emits SELECT pg_table_size("testDB") which raises
psycopg2.ProgrammingError: column "testDB" does not exist
Obviously,
cu.execute(sql.SQL("SELECT pg_table_size(%s)"),('"testDB"',))
works fine, but I want to find the "official" way to emit SELECT pg_table_size('"testDB"').
Experimentally, the following works:
cu.execute(sql.SQL("SELECT pg_table_size(%s)"),
(sql.Identifier(table_name).as_string(conn), ))
is this TRT?
You can use the Postgres function quote_ident(string text):
cu.execute("SELECT pg_table_size(quote_ident(%s))", (table_name, ))
I think your last example is a good alternative for the above solution.
I am working with the script below.
If I change the script so I avoid the bytea datatype, I can easily copy data from my postgres table into a python variable.
But if the data is in a bytea postgres column, I encounter a strange object called memory which confuses me.
Here is the script which I run against anaconda python 3.5.2:
# bytea.py
import sqlalchemy
# I should create a conn
db_s = 'postgres://dan:dan#127.0.0.1/dan'
conn = sqlalchemy.create_engine(db_s).connect()
sql_s = "drop table if exists dropme"
conn.execute(sql_s)
sql_s = "create table dropme(c1 bytea)"
conn.execute(sql_s)
sql_s = "insert into dropme(c1)values( cast('hello' AS bytea) );"
conn.execute(sql_s)
sql_s = "select c1 from dropme limit 1"
result = conn.execute(sql_s)
print(result)
# <sqlalchemy.engine.result.ResultProxy object at 0x7fcbccdade80>
for row in result:
print(row['c1'])
# <memory at 0x7f4c125a6c48>
How to get the data which is inside of memory at 0x7f4c125a6c48 ?
You can cast it use python bytes()
for row in result:
print(bytes(row['c1']))
Just started using PostgreSQL 9.5 and have ran into my first problem with jsonb column. I have been trying to find an answer to this for a while but failing badly. Can someone help?
I have a json array in python containing json objects like this:
[{"name":"foo", "age":"18"}, {"name":"bar", "age":"18"}]
I'm trying to insert this into a jsonb column like this:
COPY person(person_jsonb) FROM '/path/to/my/json/file.json';
But only 1 row gets inserted. I hope to have each json object in the array as a new row like this:
1. {"name":"foo", "age":"18"}
2. {"name":"bar", "age":"18"}
Also tried:
INSERT INTO person(person_jsonb)
VALUES (%s)
,(json.dumps(data['person'])
Still only one row gets inserted. Can someone please help??
EDIT: Added python code as requested
import psycopg2, sys, json
con = None
orders_file_path = '/path/to/my/json/person.json'
try:
with open(orders_file_path) as data_file:
data = json.load(data_file)
con = psycopg2.connect(...)
cur = con.cursor()
person = data['person']
cur.execute("""
INSERT INTO orders(orders_jsonb)
VALUES (%s)
""", (json.dumps(person), ))
con.commit()
except psycopg2.DatabaseError, e:
if con:
con.rollback()
finally:
if con:
con.close()
person.json file:
{"person":[{"name":"foo", "age":"18"}, {"name":"bar", "age":"18"}]}
Assuming the simplest schema:
CREATE TABLE test(data jsonb);
Option 1: parse the JSON in Python
You need to insert each row in PostgreSQL apart, you could parse the JSON on Python side and split the upper level array, then use cursor.executemany to execute the INSERT with each json data already split:
import json
import psycopg2
con = psycopg2.connect('...')
cur = con.cursor()
data = json.loads('[{"name":"foo", "age":"18"}, {"name":"bar", "age":"18"}]')
with con.cursor() as cur:
cur.executemany('INSERT INTO test(data) VALUES(%s)', [(json.dumps(d),) for d in data])
con.commit()
con.close()
Option 2: parse the JSON in PostgreSQL
Another option is to push the JSON processing into PostgreSQL side using json_array_elements:
import psycopg2
con = psycopg2.connect('...')
cur = con.cursor()
data = '[{"name":"foo", "age":"18"}, {"name":"bar", "age":"18"}]'
with con.cursor() as cur:
cur.execute('INSERT INTO test(data) SELECT * FROM json_array_elements(%s)', (data,))
con.commit()
con.close()
I have imported one shapefile named tc_bf25 using qgis, and the following is my python script typed in pyscripter,
import sys
import psycopg2
conn = psycopg2.connect("dbname = 'routing_template' user = 'postgres' host = 'localhost' password = '****'")
cur = conn.cursor()
query = """
ALTER TABLE tc_bf25 ADD COLUMN source integer;
ALTER TABLE tc_bf25 ADD COLUMN target integer;
SELECT assign_vertex_id('tc_bf25', 0.0001, 'the_geom', 'gid')
;"""
cur.execute(query)
query = """
CREATE OR REPLACE VIEW tc_bf25_ext AS
SELECT *, startpoint(the_geom), endpoint(the_geom)
FROM tc_bf25
;"""
cur.execute(query)
query = """
CREATE TABLE node1 AS
SELECT row_number() OVER (ORDER BY foo.p)::integer AS id,
foo.p AS the_geom
FROM (
SELECT DISTINCT tc_bf25_ext.startpoint AS p FROM tc_bf25_ext
UNION
SELECT DISTINCT tc_bf25_ext.endpoint AS p FROM tc_bf25_ext
) foo
GROUP BY foo.p
;"""
cur.execute(query)
query = """
CREATE TABLE network1 AS
SELECT a.*, b.id as start_id, c.id as end_id
FROM tc_bf25_ext AS a
JOIN node AS b ON a.startpoint = b.the_geom
JOIN node AS c ON a.endpoint = c.the_geom
;"""
cur.execute(query)
query = """
ALTER TABLE network1 ADD COLUMN shape_leng double precision;
UPDATE network1 SET shape_leng = length(the_geom)
;"""
cur.execute(query)
I got the error at the second cur.execute(query),
But I go to pgAdmin to check result, even though no error occurs, the first cur.execute(query) didn't add new columns in my table.
What mistake did I make？ And how to fix it？
I am working with postgresql 8.4, python 2.7.6 under Windows 8.1 x64.
When using psycopg2, autocommit is set to False by default. The first two statements both refer to table tc_bf25, but the first statement makes an uncommitted change to the table. So try running conn.commit() between statements to see if this resolves the issue
You should run each statement individually. Do not combine multiple statements into a semicolon separated series and run them all at one. It makes error handling and fetching of results much harder.
If you still have the problem once you've made that change, show the exact statement you're having the problem with.
Just to add to #Talvalin you can enable auto-commit by adding
psycopg2.connect("dbname='mydb',user='postgres',host ='localhost',password = '****'")
conn.autocommit = True
after you connect to your database using psycopg2
I am using PostgreSQL 9.1.4 with hstore and the PostgreSQL JDBC driver (9.1-901.jdbc4).
I am trying to use the contains operators (?, ?&, ?|) in a PreparedStatement, however the ? character is parsed as a variable placeholder. Can this character be escaped to send the correct operator in the query?
An example:
PreparedStatement stmt = conn.prepareStatement("SELECT a, b FROM table1 WHERE c ? 'foo' AND d = ?");
stmt.setInt(1, dValue);
stmt.executeQuery();
In this form the following example would raise an exception:
org.postgresql.util.PSQLException: No value specified for parameter 2.
Update:
After investigating the query parser in the pgjdbc driver this snippet seems to indicate that it is not possible to escape the ? character. The questions that remain are:
Is there anything in the JDBC spec which allows a ? to be escaped and be anything other than a parameter placeholder?
Is there any better work around for this issue than just using plain Statements with variables manually inserted into the query string?
Effectively, it looks like the java SQL parser is not hstore compliant.
But since the syntax c ? 'foo' is equivalent to exist(c, 'foo'), you can easily workaround this problem. Have a look at the following page to see what the verbose operators for hstore are.
Postgres hstore documentation
There is a discussion about this issue on pgsql-hackers mailing list: http://grokbase.com/t/postgresql/pgsql-hackers/1325c6ys9n/alias-hstores-to-so-that-it-works-with-jdbc
For now I like most this workaround which also supports indexes:
CREATE FUNCTION exist_inline (hstore, text) RETURNS bool AS $$ SELECT $1 ? $2; $$ LANGUAGE sql;
You can use this query to find the function backing an operator in PostgreSQL like this. In your example:
SELECT
oprname,
oprcode || '(' || format_type(oprleft, NULL::integer) || ', '
|| format_type(oprright, NULL::integer) || ')' AS function
FROM pg_operator
WHERE oprname LIKE '?%'
AND (SELECT oid FROM pg_type WHERE typname = 'hstore') IN (oprleft, oprright);
This produces:
|oprname|function |
|-------|--------------------------|
|? |exist(hstore, text) |
|?| |exists_any(hstore, text[])|
|?& |exists_all(hstore, text[])|
See also a related question about using JSON operators containing ?. Note that the function usage may not profit from the same indexing capability when using a GIN index on your HSTORE column.
If you'd like to add multiple key-value pairs using PreparedStatement then you can do:
PreparedStatement ps = c.prepareStatement(
"insert into xyz(id, data) values(?, hstore(?, ?))");
ps.setLong(1, 23456L);
ps.setArray(2, c.createArrayOf("text", new String[]{"name", "city"}));
ps.setArray(3, c.createArrayOf("text", new String[]{"Duke", "Valley"}));
This will insert: 23456, 'name=>Duke, city=>Valley'