I am using Groovy Sql in Grails with named parameters to get results from a Postgres DB. My statement is generated dynamically, i.e. concatenated to become the final statement, with the params being added to a map as I go along.
sqlWhere += " AND bar = :namedParam1"
paramsMap.namedParam1 = "blah"
For readability, I am using the groovy string syntax which allows me to write my sql statement over multiple lines, like this:
sql = """
SELECT *
FROM foo
WHERE 1=1
${sqlWhere}
"""
The expression is evaluated as a string containing the linebreaks as \n:
SELECT *\n ...
This is not a problem when I pass params like this
results = sql.rows(sqlString, paramsMap)
but it does become one if paramsMap is empty (which happens since AND bar = :namedParam1 is not always concatenated into the query). I then get an error
org.postgresql.util.PSQLException: No hstore extension installed
which does not really seem to relate to the true nature of the problem. I have for now fixed this with an if...else
if (sqlQuery.params.size() > 0) {
results = sql.rows(sqlString, paramsMap)
} else {
results = sql.rows(sqlString.replace('\n',' '))
}
But this seems a bit weird (especially since it does not work if I use the replace in the if-branch as well).
My question is: why do I really get this error message and is there a better way to prevent it from occuring?
It's certainly a bug in groovy.sql.SQL implementation. The method rows() can't deal with an empty map passed as params. As a workaround, you can test for it and pass an empty list instead.
def paramsMap = [:]
...
if (paramsMap.isEmpty())
paramsMap= []
Issue created at https://issues.apache.org/jira/browse/GROOVY-8082
Related
Since we upgraded from Hibernate Search 5.11 to Hibernate Search 6 we are having problems with Scrollable results
When we get a chunk of hits from the SearchScroll object each hit is stored in an Arrays.ArrayList
What we expected is that each chunk hits would be an ArrayList of say for example of type long
What we get is an ArrayList where where each hit is an Arrays.ArrayList with the Long value
Current code
SearchScroll scroll = searchSession
.search(scope)
.select(projectionArray)
.where(searchPredicate)
.sort(getSort(resultType))
.scroll(20);
Old code with Hibernate Search 5
FullTextQuery fullTextQuery = fullTextSession
.createFullTextQuery(query, resultType)
.setSort(getSort(resultType));
fullTextQuery.setProjection(fields);
ScrollableResults scrollableResults = fullTextQuery.scroll();
Any suggestions welcome
At worst we can loop through the results and convert the Arrays.ArrayList item to a long but cannot find a way to make that work either
The acual search results are correct just coming back in a different format that what we expect
Changing the code to
SearchScroll<Long> scroll = searchSession
.search(scope)
.select(projectionArray)
.where(searchPredicate)
.sort(getSort(resultType))
.scroll(20);
Makes no difference which seems to match the example in the docs
try ( SearchScroll<Book> scroll = searchSession.search(
Book.class )
.where( f -> f.matchAll() )
.scroll( 20 ) ) {
for ( SearchScrollResult<Book> chunk = scroll.next();
chunk.hasHits(); chunk = scroll.next() ) {
for ( Book hit : chunk.hits() ) {
// ... do something with the hits ...
}
totalHitCount = chunk.total().hitCount();
entityManager.flush();
entityManager.clear();
}
}
Not sure if the projection is what is causing the problem
Tested further if I remove the projection I get the results as an ArrayList of the object as expected so obviously I am doing something wrong with the use of projections in Hibernate Search 6
Without projection everything is good
With projection the results are Arrays.ArrayList
If I understand correctly, you are surprised that you get a List for each hit instead of just a Long.
First, I would recommend that you don't ignore raw type warnings.
I'll wager that your projectionArray is defined this way:
SearchProjection[] projectionArray = new SearchProjection[1];
That's wrong because you're using a "raw" type for SearchProjection, which basically disables all kinds of type-checking for all the code afterwards.
The correct way of defining that array is as follows:
SearchProjection<?>[] projectionArray = new SearchProjection<?>[1];
If you do that, then you'll get a compile-time error with the following code, telling you something like "cannot convert SearchScroll<List<?>> to SearchScroll<Long>":
SearchScroll<Long> scroll = searchSession
.search(scope)
.select(projectionArray)
.where(searchPredicate)
.sort(getSort(resultType))
.scroll(20);
Now, the reason you're getting a SearchScroll<List<?>> is you're passing an array of projections to .select(), so you're calling this method from SearchQuerySelectStep:
SearchQueryWhereStep<?, List<?>, LOS, ?> select(SearchProjection<?>... projections);
This method takes an array of projections as an argument, and (ultimately) returns a query whose hits are lists, with the results of requested projections in the same order as your array of projections.
You want to call that method instead:
<P> SearchQueryWhereStep<?, P, LOS, ?> select(SearchProjection<P> projection);
That method takes a single projection as an argument, and (ultimately) returns a query whose hits are directly the result of the requested projection.
To call that method, pass a single projection instead of an array of projections; then you will get the Long values you expect instead of Lists:
SearchProjection<Long> projection = ...;
SearchScroll<Long> scroll = searchSession
.search(scope)
.select(projection)
.where(searchPredicate)
.sort(getSort(resultType))
.scroll(20);
I'm trying to use jOOQ to create a function akin to arrayRemove but that allows removing several elements at once from a PostgreSQL column of type uuid[].
So my first attempt was:
private Field<UUID[]> arrayRemoveAll(final Field<UUID[]> field, final Set<UUID> elements) {
return select(field("array_agg(tab.col)", UUID[].class))
.from(unnest(field).as("tab", "col"))
.where(field("tab.col", UUID.class).notIn(elements))
.asField();
}
Which succeeds at removing every requested element, but has the problem of returning null instead of an empty array if I attempt to remove every element.
So I added a coalesce to my code to make it return an empty array:
private Field<UUID[]> arrayRemoveAll(final Field<UUID[]> field, final Set<UUID> elements) {
final Field<UUID[]> newArray = select(field("array_agg(tab.col)", UUID[].class))
.from(unnest(field).as("tab", "col"))
.where(field("tab.col", UUID.class).notIn(elements))
.asField();
return coalesce(newArray, field("{}", UUID[].class));
}
But running this code threw this exception:
org.jooq.exception.DataAccessException: SQL [<<confidential SQL removed>>]
Caused by: org.postgresql.util.PSQLException: ERROR: syntax error at or near ")"
This is the part of the SQL exception it is complaining about (notice the trailing comma and missing 2nd parameter in the coalesce):
coalesce((select array_agg(tab.col)
from unnest("my_schema"."my_table"."my_field") as "tab"("col")
where tab.col not in (?, ?)), )
Is this a bug in jOOQ?
I found that I had a mix of field and val in the code above, changing field("{}", UUID[].class) to val(new UUID[0]) solves the problem.
Also check Lukas Eder's answer about how to solve the issue using field.
So the final code, with generics, looks like this:
private <T> Field<T[]> arrayRemoveAll(final Field<T[]> field, final Set<T> elements, final T[] emptyArray) {
final Field<T[]> newArray = select(field("array_agg(tab.col)"))
.from(unnest(field).as("tab", "col"))
.where(field("tab.col").notIn(elements))
.asField();
return coalesce(newArray, val(emptyArray));
}
And you can use it in your statements like this:
using(configuration)
.update(MY_TABLE)
.set(MY_TABLE.MY_COLUMN,
arrayRemoveAll(MY_TABLE.MY_COLUMN, someElements, new UUID[0]))
.where(MY_TABLE.ID.eq(...))
.execute();
Your field("{}") does not generate the {} string in the SQL, but is considered a part of jOOQ's plain SQL templating language, which unfortunately doesn't allow for escaping those braces:
https://www.jooq.org/doc/latest/manual/sql-building/plain-sql-templating
Luckily, PostgreSQL supports a more formal, standards-compliant way to create an empty array literal:
field("array[]::uuid[]", UUID.class)
I am implementing a function to get an estimate of the count as described in the PostgreSQL documentation here https://wiki.postgresql.org/wiki/Count_estimate
I'm using the function:
public static Field<Integer> countEstimate(final QueryPart query) {
final String sql = String.format("count_estimate(%s)", escape(query.toString()));
return field(sql(sql), PostgresDataType.INT);
}
Which looks fine until I pass it an IN clause array field in the query. When this happens jOOQ strips the array curly braces from within my SQL. e.g. Calling it with this java code:
final UUID[] ids = new UUID[]{UUID.randomUUID()};
return db.select(countEstimate(db.select(TABLE.ID)
.from(TABLE)
.where(overlaps(ids, TABLE.FILTER_IDS))));
Results in both the variable sql and DSL.sql(sql) in the above function rendering:
count_estimate(E'select "schema"."table"."id"
from "schema"."table"
where (
((\'{"75910f3b-83e6-41ed-bf57-085c225e0131"}\') && ("schema"."table"."filter_ids"))
)')
But field(sql(sql), PostgresDataType.INT) renders this:
count_estimate(E'select "schema"."table"."id"
from "schema"."table"
where (
((\'"75910f3b-83e6-41ed-bf57-085c225e0131"\') && ("schema"."table"."filter_ids"))
)')
Is there any way to work around this and to tell jOOQ to leave my query alone?
(jOOQ 3.8.3, PG 9.5.5, PG driver 9.4-1203-jdbc4)
It turns out it only strips '{}' style arrays. Replacing the code that turns the UUID[] into sql from
DSL.val(ids)
with
DSL.array(Arrays.stream(ids)
.map(UUID::toString)
.collect(Collectors.toList())
.toArray(new String[0]))
.cast(PostgresDataType.UUID.getArrayDataType()
results in it rendering cast(array[\'75910f3b-83e6-41ed-bf57-085c225e0131\'] as uuid[]) prevents it being stripped
This is the query I am trying to run in PostgreSQL:
SELECT * FROM message WHERE id IN (
SELECT unnest(message_ids) "mid"
FROM session_messages WHERE session_id = '?' ORDER BY "mid" ASC
);
However, I am not able do something:
create.selectFrom(Tables.MESSAGE).where(Tables.MESSAGE.ID.in(
create.select(DSL.unnest(..))
Because DSL.unnest is a Table<?>, which makes sense since it is trying to take a List-like object (mostly a literal) and convert it to table.
I have a feeling that I need to find a way to wrap the function around my field name, but I have no clue as to how to proceed.
NOTE. The field message_ids is of type bigint[].
EDIT
So, this is how I am doing it now, and it works exactly as expected, but I am not sure if this is the best way to do it:
Field<Long> unnestMessageIdField = DSL.field(
"unnest(" + SESSION_MESSAGES.MESSAGE_IDS.getName() + ")",
Long.class)
.as("mid");
Field<Long> messageIdField = DSL.field("mid", Long.class);
MESSAGE.ID.in(
ctx.select(messageIdField).from(
ctx.select(unnestMessageIdField)
.from(Tables.CHAT_SESSION_MESSAGES)
.where(Tables.CHAT_SESSION_MESSAGES.SESSION_ID.eq(sessionId))
)
.where(condition)
)
EDIT2
After going through the code on https://github.com/jOOQ/jOOQ/blob/master/jOOQ/src/main/java/org/jooq/impl/DSL.java I guess the right way to do this would be:
DSL.function("unnest", SQLDataTypes.BIGINT.getArrayType(), SESSION_MESSAGES.MESSAGE_IDS)
EDIT3
Since as always lukas is here for my jOOQ woes, I am going to capitalize on this :)
Trying to generalize this function, in a signature of sort
public <T> Field<T> unnest(Field<T[]> arrayField) {
return DSL.function("unnest", <??>, arrayField);
}
I don't know how I can fetch the datatype. There seems to be a way to get DataType<T[]> from DataType<T> using DataType::getArrayDataType(), but the reverse is not possible. There is this class I found ArrayDataType, but it seems to be package-private, so I cannot use it (and even if I could, it does not expose the field elementType).
Old PostgreSQL versions had this funky idea that it is OK to produce a table from within the SELECT clause, and expand it into the "outer" table, as if it were declared in the FROM clause. That is a very obscure PostgreSQL legacy, and this example is a good chance to get rid of it, and use LATERAL instead. Your query is equivalent to this one:
SELECT *
FROM message
WHERE id IN (
SELECT "mid"
FROM session_messages
CROSS JOIN LATERAL unnest(message_ids) AS t("mid")
WHERE session_id = '?'
);
This can be translated to jOOQ much more easily as:
DSL.using(configuration)
.select()
.from(MESSAGE)
.where(MESSAGE.ID).in(
select(field(name("mid"), MESSAGE.ID.getDataType()))
.from(SESSION_MESSAGES)
.crossJoin(lateral(unnest(SESSION_MESSAGES.MESSAGE_IDS)).as("t", "mid"))
.where(SESSION_MESSAGES.SESSION_ID.eq("'?'"))
)
The Edit3 in the question is quite close to a decent solution for this problem.
We can create a custom generic unnest method for jOOQ which accepts Field and use it in jOOQ query normally.
Helper method:
public static <T> Field<T> unnest(Field<T[]> field) {
var type = (Class<T>) field.getType().getComponentType();
return DSL.function("unnest", type, field);
}
Usage:
public void query(SessionId sessionId) {
var field = unnest(SESSION_MESSAGES.MESSAGE_IDS, UUID.class);
dsl.select().from(MESSAGE).where(
MESSAGE.ID.in(
dsl.select(field).from(SESSION_MESSAGES)
.where(SESSION_MESSAGES.SESSION_ID.eq(sessionId.id))
.orderBy(field)
)
);
}
I am using Ichikawa CakePHP MongoDB plugin. I have a problem in using unset in it. I have tried the command in shell:
db.patents.update({}, {$unset : {"lv.2" : 1 }},{'multi':true})
db.patents.update({},{$pull:{pid:"2"}},{'multi':true})
These are working fine.
But when I am converting them to CakePHP command as follows:
$this->Detail->updateAll(array('$unset'=>array('lv.2'=>1,array('multi'=>true))));
Then it doesn't work and gives error:
MongoCollection::update(): expects parameter 1 to be an array or object, boolean given
Can anyone help me to figure out the problem.
Thanks.
There are no conditions
The error message means that the query being generated is the equivalent of:
db.details.update(true
This can be confirmed by checking the query log (easy if you're using debug kit).
How is that happening
The second parameter for model updateAll is missing, which means it will have the default:
public function updateAll($fields, $conditions = true) {
^
return $this->getDataSource()->update($this, $fields, null, $conditions);
}
Therefore in the mongodb datasource class - the conditions passed are true:
public function updateAll(&$Model, $fields = null, $conditions = null) {
^
As a consequence, the resultant update statement has true as the first parameter, not an array.
Correct syntax
The correct syntax for such a query is:
$this->Detail->updateAll(
array('$unset'=>array('lv.2'=>1))
array() # <- do not omit this
);
Note that it's not necessary to specify 'multi'=>true as the datasource does that for you, especially not in the fields argument.