I created a temporary table with sqlalchemy (with an underlying postgres database) that is going to be joined with a database table. However, in some cases when a value is empty '' then postgres throws the error:
failed to find conversion function from unknown to text
SqlAlchemy assembles everything to the following context
[SQL: 'WITH temp_table AS \n(SELECT %(param_1)s AS id, %(param_2)s AS email, %(param_3)s AS phone)\n SELECT campaigns_contact.id, campaigns_contact.email, campaigns_contact.phone \nFROM campaigns_contact JOIN temp_table ON temp_table.id = campaigns_contact.id AND temp_table.email = campaigns_contact.email AND temp_table.phone = campaigns_contact.phone'] [parameters: {'param_1': 83, 'param_2': '', 'param_3': '+1234567890'}]
I assemble the temporary table as follows
stmts = []
for row in import_data:
row_values = [literal(row[value]).label(value) for value in values]
stmts.append(select(row_values))
subquery = union_all(*stmts)
subquery = subquery.cte(name="temp_table")
The problem seems to be the part here
...%(param_2)s AS email...
which after replacing the param_2 results in
...'' AS email...
which will cause the error mentioned above.
One way to solve the issue is to perform a cast
...''::text AS email...
However, I don't know how to perform ::text cast with sqlalchemy!?
Related
I have this table in postgres
CREATE TABLE target (
a json
b integer
c text []
id integer
CONSTRAINT id_fkey FOREIGN KEY (id)
REFERENCES public.other_table(id) MATCH SIMPLE
ON UPDATE NO ACTION
ON DELETE NO ACTION,
)
Which I would like to insert data to from psycopg2 using
import psycopg2
import psycopg2.extras as extras
# data is of the form dict, integer, list(string), string <- used to get fkey id
data = [[extras.Json([{'a':1,'b':2}, {'d':3,'e':2}]), 1, ['hello', 'world'], 'ident1'],
[extras.Json([{'a':4,'b':3}, {'d':1,'e':9}]), 5, ['hello2', 'world2'], 'ident2']]
# convert data to list of tuples containing objects
x = [tuple(u) for u in data]
# insert data to the database
query = ('WITH ins (a, b, c, ident) AS '
'(VALUES %s) '
'INSERT INTO target (a, b, c, id) '
'SELECT '
'ins.a '
'ins.b '
'ins.c '
'other_table.id'
'FROM '
'ins '
'LEFT JOIN other_table ON ins.ident = other_table.ident;')
cursor = conn.cursor()
extras.execute_values(cursor, query, x)
When I run this I get the error: column "a" is of type json but expression is of type text. I tried to solve this by adding a type cast in the SELECT statement but then I got the same error for c and then for b.
Originally I thought the problem lies in the WITH statement but based on the answers to my previous question this seems to not be the case Postgres `WITH ins AS ...` casting everything as text
It seems that execute_values is sending all the values as text with ' '.
Main Question: How can I get execute_values to send the values based on their python data type rather than just as text?
Sub questions:
How can I confirm that execute_values is in fact sending the values as text with quotation marks?
What is the purpose of the template argument of execute_values https://www.psycopg.org/docs/extras.html and could that be of help?
The issue, as Adrian Klaver points out in their comment, and also seen in this answer, is that the typing is lost in the CTE.
We can show this with an example in the psql shell:
CREATE TABLE test (col1 json);
WITH cte (c) AS (VALUES ('{"a": 1}'))
INSERT INTO test (col) SELECT c FROM cte;
resulting in
ERROR: column "col" is of type json but expression is of type text
whereas this version, with the type specified, succeeds:
WITH cte(c) AS (VALUES ('{"a": 1}'::json))
INSERT INTO test (col) SELECT c FROM cte;
We can mimic this in execute_valuesby providing the typing information in the template argument:
extras.execute_values(cursor, query, data, template='(%s::json, %s, %s, %s)')
I'm using the following Knex statement to copy data from two tables into another table:
const insert = knex.from(knex.raw('?? (??, ??, ??, ??, ??, ??, ??)', ['carrier', 'docket_number', 'dot_number', 'legal_name', 'dba_name', 'nbr_power_unit', 'rating', 'carrier_operation']))
.insert(knex('carrier_temp as ct').leftJoin('carrier_census_temp as cen', 'ct.dot_number', 'cen.DOT_NUMBER')
.select(['ct.docket_number as docket_number',
'ct.dot_number as dot_number',
'ct.legal_name as legal_name',
'ct.dba_name as dba_name',
'cen.tot_pwr as nbr_power_unit',
'cen.RATING as rating',
knex.raw('CASE WHEN cen.crrinter = \'A\' THEN \'INTERSTATE\' ELSE \'INTRASTATE\' END as "carrier_operation"')])).toString()
const conflict = knex.raw('ON CONFLICT (docket_number) DO NOTHING;').toString()
const q = insert + conflict
await knex.raw(q).debug()
The generated sql is:
INSERT INTO "carrier"
(
"docket_number",
"dot_number",
"legal_name",
"dba_name",
"nbr_power_unit",
"rating",
"carrier_operation"
)
SELECT "ct"."docket_number" AS "docket_number",
"ct"."dot_number" AS "dot_number",
"ct"."legal_name" AS "legal_name",
"ct"."dba_name" AS "dba_name",
"cen"."tot_pwr" AS "nbr_power_unit",
"cen"."RATING" AS "rating",
CASE
WHEN cen.crrinter = 'A' THEN 'INTERSTATE'
ELSE 'INTRASTATE'
END AS "carrier_operation"
FROM "carrier_temp" AS "ct"
left join "carrier_census_temp" AS "cen"
ON "ct"."dot_number" = "cen"."DOT_NUMBER" ON conflict (docket_number) DO nothing;
The carrier table receives all the columns that are referenced from carrier_temp or ct correctly, but the columns pulled from carrier_census_temp or cen are all ending up as NULL (nbr_power_unit, rating, and carrier_operation). That is, except for the case statement, which is setting every row's carrier_operation to INTRASTATE. If I instead equate NULL in the case statement, it still sets every row to INTRASTATE. Does anyone have any idea why this is?
See comments on question. I totally missed something wrong in my data that led to nothing ever being joined, which just leads everything to always be NULL.
I have a table in which I need to change the values of a couple of columns in multiple rows.
The table with values to be changed is like:
The code I have tried containing updated values, with no success, is:
UPDATE <table_name>
SET (IDENTIFIER_1, IDENTIFIER_2)
VALUES (1635, 1755),
(2024, 2199),
(1868, 1692),
(3577, 4825)
WHERE ID
IN ('1',
'23',
'54',
'21');
To be honest, I am not sure if this is even supported in db2 SQL. The error is:
[Error Code: -104, SQL State: 42601] DB2 SQL Error: SQLCODE=-104, SQLSTATE=42601, SQLERRMC=update *
I should also advise that I am a db2 newbie.
You can always use Merge
MERGE INTO TABLE1
USING (
VALUES (1,1635, 1755),
(23,2024, 2199),
(54,1868, 1692) ) dummytable(ID_T, INF1,INF2)
on table1.id_table = dummytable.id_t
when matched
then UPDATE set TABLE1.IDENTIFIER_1 = dummytable.INF1
, TABLE1.IDENTIFIER_2 = dummytable.INF2
else ignore
I have imported one shapefile named tc_bf25 using qgis, and the following is my python script typed in pyscripter,
import sys
import psycopg2
conn = psycopg2.connect("dbname = 'routing_template' user = 'postgres' host = 'localhost' password = '****'")
cur = conn.cursor()
query = """
ALTER TABLE tc_bf25 ADD COLUMN source integer;
ALTER TABLE tc_bf25 ADD COLUMN target integer;
SELECT assign_vertex_id('tc_bf25', 0.0001, 'the_geom', 'gid')
;"""
cur.execute(query)
query = """
CREATE OR REPLACE VIEW tc_bf25_ext AS
SELECT *, startpoint(the_geom), endpoint(the_geom)
FROM tc_bf25
;"""
cur.execute(query)
query = """
CREATE TABLE node1 AS
SELECT row_number() OVER (ORDER BY foo.p)::integer AS id,
foo.p AS the_geom
FROM (
SELECT DISTINCT tc_bf25_ext.startpoint AS p FROM tc_bf25_ext
UNION
SELECT DISTINCT tc_bf25_ext.endpoint AS p FROM tc_bf25_ext
) foo
GROUP BY foo.p
;"""
cur.execute(query)
query = """
CREATE TABLE network1 AS
SELECT a.*, b.id as start_id, c.id as end_id
FROM tc_bf25_ext AS a
JOIN node AS b ON a.startpoint = b.the_geom
JOIN node AS c ON a.endpoint = c.the_geom
;"""
cur.execute(query)
query = """
ALTER TABLE network1 ADD COLUMN shape_leng double precision;
UPDATE network1 SET shape_leng = length(the_geom)
;"""
cur.execute(query)
I got the error at the second cur.execute(query),
But I go to pgAdmin to check result, even though no error occurs, the first cur.execute(query) didn't add new columns in my table.
What mistake did I make？ And how to fix it？
I am working with postgresql 8.4, python 2.7.6 under Windows 8.1 x64.
When using psycopg2, autocommit is set to False by default. The first two statements both refer to table tc_bf25, but the first statement makes an uncommitted change to the table. So try running conn.commit() between statements to see if this resolves the issue
You should run each statement individually. Do not combine multiple statements into a semicolon separated series and run them all at one. It makes error handling and fetching of results much harder.
If you still have the problem once you've made that change, show the exact statement you're having the problem with.
Just to add to #Talvalin you can enable auto-commit by adding
psycopg2.connect("dbname='mydb',user='postgres',host ='localhost',password = '****'")
conn.autocommit = True
after you connect to your database using psycopg2
I have a database in postgreSQL. I want to read some data from there, but I get an error (column anganridref does not exist) when I execute my command.
Here is my NpgsqlCommand:
cmd.CommandText = "select * from angebot,angebotstatus,anrede where anrid=anganridref and anstaid=anganstaidref";
and my 3 tables
the names of my columns are rights. So I don't understand why that error comes. Someone can explain me why it does crash? Its not the problem of large and lowercase.
You are not prefixing your column names in the where clause:
select *
from angebot,
angebotstatus,
anrede
where anrid = anganridref <-- missing tablenames for the columns
and anstaid = anganstaidre
It's also recommended to use an explicit JOIN instead of the old SQL 89 implicit join syntax:
select *
from angebot
join angebotstatus on angebot.aaaa = angebotstatus.bbbb
join anrede on angebot.aaaa = anrede.bbbb