I have this query :
INSERT INTO lytnobjects.devices (id,idedge,uniqueref,constructeur,ipaddress,macaddress,
hostname,devicetype,isfirewall,isvisible,iscorporate,
ishub,osname,osversion,datecreation,lasttrafic,
hourtrafic,daytrafic,monthtrafic)
VALUES ('e1e455e98b6ed0037a58d0c1f5dc245a',3183,'TODO','TODO','192.168.143.49',
'b0:0c:d1:bb:36:1c','HPBB361C','Other',False,False,False,False,'','',
'2021-10-29T00:58:53.709','2021-01-01T00:00:00','0/0','0/0','0/0')
When I execute the query using python 3.9 and psycopg2_binary (PostgreSQL), I get an error :
unterminated quoted string at or near "'HPBB361C"
conn is the opened connection to the database (AWS RDS PostgreSQL)
sql is a string with the query above
def SQLExec(conn,sql):
try: cur = conn.cursor()
cur.execute(sql)
except (Exception, psycopg2.DatabaseError) as error:
print("***** ERROR:",error)
cur.close()
If I execute the same request from pgAdmin, I get no error !
There is no missing quote as you can see in the query, and no reason to point an error at this place!
So, I have a string (sql) with the query ("INSERT ...")
I call execute from psycopg2, and get an error: unterminated quoted string at or near "'HPBB361C"
I copy/paste the same string into pgAdmin, and the query is executed with no error
The same string (query)
Any idea why I get an error from my python app?
I am looking for an answer since many hours, but find no explanation, and I don't know how to fix the problem (which doesn't exist for me)
Your help is very appreciated
Thank you
I finaly found the answer!
I build the sql query (string) using some variables coming from various sources, like Amazon S3 for instance.
I assumed that the variable was really a string, with nothing "bizarre" in it... But in fact, sometimes, the "string" was ended with a "\x00" char, that is not displayed, so the string looks just normal :-/
When I execute my query (string) with psycopg2, it receives the extra \x00 char, which ends the string at this place! This is why it says there is a missing quote
I put a trace in the code to display the .encode() version of my string, and it revealed the \x00 at the end. So now I "clean" all string variables used in my queries, just with myvariable.replace("\x00","")
And it works now. There is probably a more conventional way to fix this...
I hope it may help somebody sometime! ;-)
Related
I'm trying to use Query Variables in Grafana, the panel query source is PostgreSQL for QuestDB.
I have added the variable without any issue, but I'm unable to use the variable in Panel query since the variable values contains the spaces (SENSOR01 ON_OFF), also I'm unable to figure-out how to add single quote escape.
Following are the scenarios I tried:
Scenario1: this indicates due to space in the Variable value, on_off considered as separate word
where sensor_name = $sensor
db query error: pq: unexpected token: on_off
.
.
Scenario2: tried to add single quotes explicitly for the variable value, but there is generic error from source DB (QuestDB)
where sensor_name = concat('''', $sensor, '''')
db query error: pq: dangling expression
When tried Scenario2 approach directly in query of Variable, getting the same error
..
Scenario3: Hard-coded the variable value with space and with single quotes, but this giving me error with first part of the variable, looks like the hard-coded single quotes not passed here!
Error (Scenario3):
Is there any way/workaround to tackle this issue?
Could you just add the quotes directly in the query?
where sensor_name = '$sensor'
I have a similar grafana panel querying a questDB database using a variable and it works for me. This is my query:
select device_type, avg(duration_ms) as avg_duration_ms, avg(speed) as avg_speed, avg(measure1) as avg_m1, avg(measure2) as avg_m2 from ilp_test
WHERE
$__timeFilter(timestamp) and device_type = '$deviceType'
A rather hacky workaround would be to do:
where sensor_name = concat(cast(cast('&' as int) + 1 as char), $sensor, cast(cast('&' as int) + 1 as char))
This should work, but I'm pretty sure there is a better solution. Let me find it and get back to you.
Update. We may support Postgres syntax (which is '' escaping for a single quote char) in one of upcoming versions. For now, you'd have to use the above workaround.
I get this error when trying out this command in the BIRT Classic Models sample database in Data Studio
select xmlelement(name "custno", customers.customernumber) from customers
Syntax error: Encountered "\"custno\"" at line 1, column 24.
I do not know how to correct it.
Thanks.
I'm not familiar with db2, but according to this your statement looks quite alrigth (although I'd place an alias to name this field...)
But this
Syntax error: Encountered "\"custno\"" at line 1, column 24.
seems to be a quite clear hint, that your error is connected to the NAME of the element.
I'm pretty sure, that this statement was created on string level.
Did you try to escape the "-characters with \"?
The SQL reaching the engine might look like
select xmlelement(name \"custno\", customers.customernumber) from customers
or
select xmlelement(name "\"custno"\", customers.customernumber) from customers
... which is wrong of course...
But to be honest: just guessing...
Having a problem with PostgreSQL dialect in PyCharm. I have the below SELECT query:
"SELECT * FROM table WHERE ST_DWithin(table.geog_column,
ST_GeographyFromText(\'SRID=4326;POINT(%s %s)\'), %s)"
The query performs as expected in a query editor but Pycharm complains <expression> expected, got '%'. I have set the dialect detection to PostgreSQL.
I believe there is an issue with the parameter binding but not able to figure out what the issue is. Any help would be appreciated.
EDIT: I somehow missed the clear warnings on psycopg2 documentation about using python string interpolation and concatenation.
The right way of doing it is to use SQLAlchemy to construct raw SQL queries:
from sqlalchemy import text
sql = text("SELECT * FROM table WHERE ST_DWithin(table.geog_column,
ST_GeographyFromText(\'SRID=4326;POINT(:long :lat)\'), :distance)")
data = {'long': longitude, 'lat': latitude, 'distance': distance}
result = conn.execute(sql, data)
The below approach is WRONG and is susceptible to SQL injections. I have left it here for reference only.
I just found the mistake and for anyone else who is caffeine starved, you need to add the %s within single quotes. Elementary but can easily be missed.
"SELECT * FROM table WHERE ST_DWithin(table.geog_column,
ST_GeographyFromText(\'SRID=4326;POINT('%s' '%s')\'), *'%s'*)"
The quotes fixed the issue for me but I am not entirely sure if this is the right approach and hence leaving it here to get some input.
I'm currently working on an elixir project that parses XML from an API and inserts data into postgres using postgrex.
Most inserts work fine, however for the odd insert I get this error. I've seen a lot of other people facing this error, but I'm not to sure how to solve it in Elixir.
23:52:32.402 [error] Process #PID<0.224.0> raised an exception
** (KeyError) key :constraint not found in: %{code: :character_not_in_repertoire, file: "wchar.c", line: "2011", message: "invalid byte sequence for encoding \"UTF8\": 0xe3 0x83 0x22", pg_code: "22021", routine: "report_invalid_encoding", severity: "ERROR"}
(pipeline_processor) lib/worker.ex:133: PipelineProcessor.Worker.recursive_db_insert/1
(pipeline_processor) lib/worker.ex:47: PipelineProcessor.Worker.process_article/1
(pipeline_processor) lib/worker.ex:17: PipelineProcessor.Worker.request_article/0
I'm aware that the error is actually due to accessing an invalid property of the map. However I'm trying to solve the issue that postgrex is giving.
My postgrex insert code:
sql_string = "INSERT INTO articles (title, source, content) VALUES ($1, $2, $3) RETURNING id"
{:ok, pid} = Postgrex.Connection.start_link(Application.get_env(:pipeline_processor, :db_details))
response = Postgrex.Connection.query(
pid,
sql_string,
[article.title, article.source, article.content]
)
Postgrex.Connection.stop(pid)
Is there anyway in Elixir to scrub out invalid bytes so that these inserts can succeed? Or for some way to have postgres handle it?
Thanks
As you already guessed postgres is complaining that you are inserting invalid UTF8 into a text type. I would initially try to fix the bad encodings if you cannot do that you can use a combination of String.codepoints/1 and String.valid_character?/1 to either scrub or escape the invalid bytes.
I'm using Pdo_Mssql adapter against a Sybase database and working around issues encountered. One pesky issue remaining is Zend_Db's instance on quoting BIT field values. When running the following for an insert:
$row = $this->createRow();
...
$row->MyBitField = $data['MyBitField'];
...
$row->save();
FreeTDS log output shows:
dbutil.c:87:msgno 257: "Implicit conversion from datatype 'VARCHAR' to 'BIT' is not allowed. Use the CONVERT function to run this query.
I've tried casting values as int and bool, but this seems to be a table metadata problem, not a data type problem with input.
Fortunately, Zend_Db_Expr works nicely. The following works, but I'd like to be database server agnostic.
$row->MyBitField = new Zend_Db_Expr("CONVERT(BIT, {$data['MyBitField']})");
I've verified that the describeTable() is returning BIT for the field. Any ideas on how to get ZF to stop quoting MS SQL/Sybase BIT fields?
You can simply try this (works for mysql bit type):
$row->MyBitField = new Zend_Db_Expr($data['MyBitField']);