Is there a place that says what that documents the Postgres parse tree? For example from Query Parsing it gives an example of:
SELECT * FROM foo where bar = 42 ORDER BY id DESC LIMIT 23;
(
{SELECT
:distinctClause <>
:intoClause <>
:targetList (
{RESTARGET
:name <>
:indirection <>
:val
...
But I can't find a place that gives more information, for example, what is indirection in the RESTARGET clause? It does link to the Postgres internals docs # https://www.postgresql.org/docs/current/overview.html, but I couldn't find anything about the Postgres parse tree in detail from that. Is anything documented on that?
Note: for the question I'm only concerned with the SELECT statement.
Most of the relevant structs are defined in parsenodes.h, e.g. ResTarget and its attributes are defined here:
https://github.com/postgres/postgres/blob/master/src/include/nodes/parsenodes.h#L461
You may also find pg_query useful, a library I wrote to parse queries using the Postgres parser: https://github.com/pganalyze/pg_query (the output format is either JSON or Protobuf, and maps to the parsenodes.h struct names 1:1)
Related
I am trying to use Postgresql Full Text Search. I read that the stop words (words ignored for indexing) are implemented via dictionary. But I would like to give the user a limited control over the stop words (insert new ones), so I grouped then in a table.
From the example below:
select strip(to_tsvector('simple', texto)) from longtxts where id = 23;
I can get the vector:
{'alta' 'aluno' 'cada' 'do' 'em' 'leia' 'livro' 'pedir' 'que' 'trecho' 'um' 'voz'}
And now I would like to remove the elements from the stopwords table:
select array(select palavra_proibida from stopwords);
That returns the array:
{a,as,ao,aos,com,default,e,eu,o,os,da,das,de,do,dos,em,lhe,na,nao,nas,no,nos,ou,por,para,pra,que,sem,se,um,uma}
Then, following documentation:
ts_delete(vector tsvector, lexemes text[]) tsvector remove any occurrence of lexemes in lexemes from vector ts_delete('fat:2,4 cat:3 rat:5A'::tsvector, ARRAY['fat','rat'])
I tried a lot. For example:
select ts_delete((select strip(to_tsvector('simple', texto)) from longtxts where id = 23), array[(select palavra_proibida from stopwords)]);
But I always receive the error:
ERROR: function ts_delete(tsvector, character varying[]) does not exist
LINE 1: select ts_delete((select strip(to_tsvector('simple', texto))...
^
HINT: No function matches the given name and argument types. You might need to add explicit type casts.
Could anyone help me? Thanks in advance!
ts_delete was introduced in PostgreSQL 9.6. Based on the error message, you're using an earlier version. You may try select version(); to be sure.
When you land on the PostgreSQL online documentation with a web search, it may correspond to any version. The version is in the URL and there's a "This page in another version" set of links at the top of each page to help switching to the equivalent doc for a different version.
I have a Python script that runs a pgSQL file through SQLAlchemy's connection.execute function. Here's the block of code in Python:
results = pg_conn.execute(sql_cmd, beg_date = datetime.date(2015,4,1), end_date = datetime.date(2015,4,30))
And here's one of the areas where the variable gets inputted in my SQL:
WHERE
( dv.date >= %(beg_date)s AND
dv.date <= %(end_date)s)
When I run this, I get a cryptic python error:
sqlalchemy.exc.ProgrammingError: (psycopg2.ProgrammingError) argument formats can't be mixed
…followed by a huge dump of the offending SQL query. I've run this exact code with the same variable convention before. Why isn't it working this time?
I encountered a similar issue as Nikhil. I have a query with LIKE clauses which worked until I modified it to include a bind variable, at which point I received the following error:
DatabaseError: Execution failed on sql '...': argument formats can't be mixed
The solution is not to give up on the LIKE clause. That would be pretty crazy if psycopg2 simply didn't permit LIKE clauses. Rather, we can escape the literal % with %%. For example, the following query:
SELECT *
FROM people
WHERE start_date > %(beg_date)s
AND name LIKE 'John%';
would need to be modified to:
SELECT *
FROM people
WHERE start_date > %(beg_date)s
AND name LIKE 'John%%';
More details in the pscopg2 docs: http://initd.org/psycopg/docs/usage.html#passing-parameters-to-sql-queries
As it turned out, I had used a SQL LIKE operator in the new SQL query, and the % operand was messing with Python's escaping capability. For instance:
dv.device LIKE 'iPhone%' or
dv.device LIKE '%Phone'
Another answer offered a way to un-escape and re-escape, which I felt would add unnecessary complexity to otherwise simple code. Instead, I used pgSQL's ability to handle regex to modify the SQL query itself. This changed the above portion of the query to:
dv.device ~ E'iPhone.*' or
dv.device ~ E'.*Phone$'
So for others: you may need to change your LIKE operators to regex '~' to get it to work. Just remember that it'll be WAY slower for large queries. (More info here.)
For me it's turn out I have % in sql comment
/* Any future change in the testing size will not require
a change here... even if we do a 100% test
*/
This works fine:
/* Any future change in the testing size will not require
a change here... even if we do a 100pct test
*/
EDIT: I have found a relevant answer already on stack overflow here:
XQuery [value()]: 'value()' requires a singleton (or empty sequence), found operand of type 'xdt:untypedAtomic *'
I have not dealt with XML in T-SQL before, and I am modifying an existing legacy stored proc, and picking most if it up through trial and error.
however I have hit a problem where trial and error is proving fruitless, and very slow. Think it's time to appeal to stack overflow gurus!
Here is some XML
<?xml version=\"1.0\"?>
<Notification xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" xmlns:xsd=\"http://www.w3.org/2001/XMLSchema\">
<NotificationId>0</NotificationId>
<UserNotifications>
<UserNotification>
<UserNotificationId>0</UserNotificationId>
<NotificationId>0</NotificationId>
<UserId>13514</UserId>
<MessageTypeId>1</MessageTypeId>
</UserNotification>
<UserNotification>
<UserNotificationId>0</UserNotificationId>
<NotificationId>0</NotificationId>
<UserId>13514</UserId>
<MessageTypeId>2</MessageTypeId>
</UserNotification>
</UserNotifications>
</Notification>
The Stored Proc in question accepts the above XML as a parameter:
CREATE PROCEDURE [dbo].[Notification_Insert]
#ParametersXml XML
AS
BEGIN
The XML contains child "UserNotification" elements. I would like to select the UserId, MessageTypeId of each UserNotification, into a table like this
UserId | MessageTypeId
13514 | 1
13514 | 2
Obviously the size of the collection is not fixed.
My current attempt (which doesn't work - is along these lines:
DECLARE #UserDetails TABLE ( UserId INT, MessageTypeId INT);
INSERT INTO #UserDetails (UserId, MessageTypeId)
SELECT Tab.Col.value('#UserId','INT'),
Tab.Col.value('#MessageTypeId','INT')
FROM #ParametersXml.nodes('/Notification/UserNotifications[not(#xsi:nil = "true")][1]/UserNotification') AS Tab(Col)
But this never inserts anything..
I have been playing around with this for a while now and not had any joy :(
I would suggest going through the links below. I found them short and quick to go through:
http://blog.sqlauthority.com/2009/02/12/sql-server-simple-example-of-creating-xml-file-using-t-sql/
http://blog.sqlauthority.com/2009/02/13/sql-server-simple-example-of-reading-xml-file-using-t-sql/
I found the solution to this problem through further searching stack overflow.
The query I need (thanks to XQuery [value()]: 'value()' requires a singleton (or empty sequence), found operand of type 'xdt:untypedAtomic *')
INSERT INTO #UserDetails (UserId, MessageTypeId)
SELECT UserNotification.value('UserId[1]','INT'),
UserNotification.value('MessageTypeId[1]','INT')
FROM #ParametersXml.nodes('//Notification/UserNotifications') AS x(Coll)
cross apply #ParametersXml.nodes('//Notification/UserNotifications/UserNotification') as un(UserNotification)
I have the following table structure:
CREATE TABLE SpecialTable
(
Key UNIQUEIDENTIFIER,
XMLField VARCHAR(MAX)
)
In the first tuple:
Key = "28384841-17283848-7836-18292939"
XMLField =
"<RootNode>
<ForeignKey>92383829-27374848-1298-19283789</ForeignKey>
<ForeignKey>47585853-27374848-4759-19283939</ForeignKey>
<ForeignKey>37383829-27374848-3747-19283930</ForeignKey>
</RootNode>"
In another tuple, I see:
Key = "89984841-17283848-7836-18292939"
XMLField =
"<RootNode>
<ForeignKey>92383829-27374848-1298-19283789</ForeignKey>
<ForeignKey>37383829-27374848-3747-19283930</ForeignKey>
</RootNode>"
In a further tuple, I see:
Key = "11114841-17283848-7836-18292939"
XMLField =
"<RootNode>
<ForeignKey>37383829-27374848-3747-19283930</ForeignKey>
</RootNode>"
What I need to do is to get the following dataset out:
Key ForeignKey
28384841-17283848-7836-18292939 92383829-27374848-1298-19283789
28384841-17283848-7836-18292939 47585853-27374848-4759-19283939
28384841-17283848-7836-18292939 37383829-27374848-3747-19283930
89984841-17283848-7836-18292939 92383829-27374848-1298-19283789
89984841-17283848-7836-18292939 37383829-27374848-3747-19283930
11114841-17283848-7836-18292939 37383829-27374848-3747-19283930
I must say that this is a simplified data set and that the data was more complex than this and I have got to a point where I cannot get any further.
I have tried this:
SELECT sp.Key,
x.XmlCol.Query('.')
FROM SpecialTable AS sp
CROSS APPLY sp.XMLField.nodes('/RootNode') x(XmlCol)
However, it seems just to show the Key and the whole of the XML of the XMLField.
Also, I tried this:
SELECT sp.Key,
x.XmlCol.Query('ForeignKey[text]')
FROM SpecialTable AS sp
CROSS APPLY sp.XMLField.nodes('/RootNode') x(XmlCol)
And I get only the first value in the first ForeignKey node and not the others.
What am I doing wrong?
Kindest regards,
QuietLeni
First of all - if your data looks like XML, quacks like XML, smells like XML - then it IS XML and you should use the XML datatype to store it!
Also: be aware that Key is a very generic term, and also a T-SQL reserved keyword, so it makes for a really bad column name - use something more meaningful that doesn't clash with a keyword!
Once you've done that, you should be able to use this code to get your desired results:
SELECT
[Key],
ForeignKey = xc.value('.', 'varchar(50)')
FROM
dbo.SpecialTable
CROSS APPLY
XMLField.nodes('/RootNode/ForeignKey') AS XT(XC)
This will only work if you column XMLField is of XML datatype !! (which it really should be anyway)
I am using PostgreSQL 9.4 and the awesome JSONB field type. I am trying to query against a field in a document. The following works in the psql CLI
SELECT id FROM program WHERE document -> 'dept' ? 'CS'
When I try to run the same query via my Scala app, I'm getting the error below. I'm using Play framework and Anorm, so the query looks like this
SQL(s"SELECT id FROM program WHERE document -> 'dept' ? {dept}")
.on('dept -> "CS")
....
SQLException: : No value specified for parameter 5.
(SimpleParameterList.java:223)
(in my actual queries there are more parameters)
I can get around this by casting my parameter to type jsonb and using the #> operator to check containment.
SQL(s"SELECT id FROM program WHERE document -> 'dept' #> {dept}::jsonb")
.on('dept -> "CS")
....
I'm not too keen on the work around. I don't know if there are performance penalties for the cast, but it's extra typing, and non-obvious.
Is there anything else I can do?
As a workaround to avoid the ? operator, you could create a new operator doing exactly the same.
This is the code of the original operator:
CREATE OPERATOR ?(
PROCEDURE = jsonb_exists,
LEFTARG = jsonb,
RIGHTARG = text,
RESTRICT = contsel,
JOIN = contjoinsel);
SELECT '{"a":1, "b":2}'::jsonb ? 'b'; -- true
Use a different name, without any conflicts, like #-# and create a new one:
CREATE OPERATOR #-#(
PROCEDURE = jsonb_exists,
LEFTARG = jsonb,
RIGHTARG = text,
RESTRICT = contsel,
JOIN = contjoinsel);
SELECT '{"a":1, "b":2}'::jsonb #-# 'b'; -- true
Use this new operator in your code and it should work.
Check pgAdmin -> pg_catalog -> Operators for all the operators that use a ? in the name.
In JDBC (and standard SQL) the question mark is reserved as a parameter placeholder. Other uses are not allowed.
See Does the JDBC spec prevent '?' from being used as an operator (outside of quotes)? and the discussion on jdbc-spec-discuss.
The current PostgreSQL JDBC driver will transform all occurrences (outside text or comments) of a question mark to a PostgreSQL specific parameter placeholder. I am not sure if the PostgreSQL JDBC project has done anything (like introducing an escape as discussed in the links above) to address this yet. A quick look at the code and documentation suggests they didn't, but I didn't dig too deep.
Addendum: As shown in the answer by bobmarksie, current versions of the PostgreSQL JDBC driver now support escaping the question mark by doubling it (ie: use ?? instead of ?).
I had the same issue a couple of days ago and after some investigation I found this.
https://jdbc.postgresql.org/documentation/head/statement.html
In JDBC, the question mark (?) is the placeholder for the positional parameters of a PreparedStatement. There are, however, a number of PostgreSQL operators that contain a question mark. To keep such question marks in a SQL statement from being interpreted as positional parameters, use two question marks (??) as escape sequence. You can also use this escape sequence in a Statement, but that is not required. Specifically only in a Statement a single (?) can be used as an operator.
Using 2 question marks seemed to work well for me - I was using the following driver (illustrated using maven dependency) ...
<dependency>
<groupId>org.postgresql</groupId>
<artifactId>postgresql</artifactId>
<version>9.4-1201-jdbc41</version>
</dependency>
... and MyBatis for creating the SQL queries and it seemed to work well. Seemed easier / cleaner than creating an PostgreSQL operator.
SQL went from e.g.
select * from user_docs where userTags ?| array['sport','property']
... to ...
select * from user_docs where userTags ??| array['sport','property']
Hopefully this works with your scenario!
As bob said just use ?? instead of ?
SQL(s"SELECT id FROM program WHERE document -> 'dept' ?? {dept}")
.on('dept -> "CS")