Remove several elements from Postgres array with jOOQ - postgresql

I'm trying to use jOOQ to create a function akin to arrayRemove but that allows removing several elements at once from a PostgreSQL column of type uuid[].
So my first attempt was:
private Field<UUID[]> arrayRemoveAll(final Field<UUID[]> field, final Set<UUID> elements) {
return select(field("array_agg(tab.col)", UUID[].class))
.from(unnest(field).as("tab", "col"))
.where(field("tab.col", UUID.class).notIn(elements))
.asField();
}
Which succeeds at removing every requested element, but has the problem of returning null instead of an empty array if I attempt to remove every element.
So I added a coalesce to my code to make it return an empty array:
private Field<UUID[]> arrayRemoveAll(final Field<UUID[]> field, final Set<UUID> elements) {
final Field<UUID[]> newArray = select(field("array_agg(tab.col)", UUID[].class))
.from(unnest(field).as("tab", "col"))
.where(field("tab.col", UUID.class).notIn(elements))
.asField();
return coalesce(newArray, field("{}", UUID[].class));
}
But running this code threw this exception:
org.jooq.exception.DataAccessException: SQL [<<confidential SQL removed>>]
Caused by: org.postgresql.util.PSQLException: ERROR: syntax error at or near ")"
This is the part of the SQL exception it is complaining about (notice the trailing comma and missing 2nd parameter in the coalesce):
coalesce((select array_agg(tab.col)
from unnest("my_schema"."my_table"."my_field") as "tab"("col")
where tab.col not in (?, ?)), )
Is this a bug in jOOQ?

I found that I had a mix of field and val in the code above, changing field("{}", UUID[].class) to val(new UUID[0]) solves the problem.
Also check Lukas Eder's answer about how to solve the issue using field.
So the final code, with generics, looks like this:
private <T> Field<T[]> arrayRemoveAll(final Field<T[]> field, final Set<T> elements, final T[] emptyArray) {
final Field<T[]> newArray = select(field("array_agg(tab.col)"))
.from(unnest(field).as("tab", "col"))
.where(field("tab.col").notIn(elements))
.asField();
return coalesce(newArray, val(emptyArray));
}
And you can use it in your statements like this:
using(configuration)
.update(MY_TABLE)
.set(MY_TABLE.MY_COLUMN,
arrayRemoveAll(MY_TABLE.MY_COLUMN, someElements, new UUID[0]))
.where(MY_TABLE.ID.eq(...))
.execute();

Your field("{}") does not generate the {} string in the SQL, but is considered a part of jOOQ's plain SQL templating language, which unfortunately doesn't allow for escaping those braces:
https://www.jooq.org/doc/latest/manual/sql-building/plain-sql-templating
Luckily, PostgreSQL supports a more formal, standards-compliant way to create an empty array literal:
field("array[]::uuid[]", UUID.class)

Related

Is there any difference between initializing with empty values or null value?

Is there any difference between these two part of code? (I am specially using Flutter/Dart but also interested to know about this in any other popular languages like C/C++, Java, JS, Python, etc.)
Code1:
String a = null; (in Flutter: a = null as String; )
List<T> = null; (in Flutter: List<T> = null as List<T>;)
Code 2:
String a = '';
List<T> = [];
With Dart nullsafety your first examples are invalid (assuming you meant to have a variable name on your List). If a variable can contain null the type needs a '?' suffix:
String? a = null;
List<T>? b = null;
But with or without nullsafety or '?', use of the variables in your second example won't result in runtime errors. For example a.trim(): if a is null a runtime error will occur because you're trying to call (null).trim(). In your second example no error because there is a String object to access, even though that String object is empty of characters. Same for your List: b.forEach((e) {}) when b==null a runtime error occurs because there's no Object to find forEach() - i.e. the runtime doesn't know what to do with (null).forEach(). In your second example the forEach() doesn't execute the function because List b is empty, but there's no runtime error because there's an Object to call forEach() on.
In C: NULL for pointers is usually synonymous with 0. The result of trying to access memory address NULL is "undefined" in C, because it could be valid as 0x00 in some instances like embedded systems or low-level system code, but will usually result in your process being terminated (crash).

Passing an aggregate select expression to Dynamic Linq's GroupBy

I have simplified the following example from my code and hoping there's no obvious compilation errors because of it. Lets say I have the following entities (not what i actually have, please assume I have no EF or schema issues, this is just for example):
public class Company
{
public string GroupProperty {get;set;}
public virtual ICollection<PricingForm> PricingForms {get;set;}
}
public class PricingForm
{
public decimal Cost {get;set;}
}
And I want to query like so:
IQueryable DynamicGrouping<T>(IQueryable<T> query)
{
Expression<Func<Company, decimal?>> exp = c => c.PricingForms.Sum(fr => fr.Cost);
string selector = "new (it.Key as Key, #0(it) as Value)";
IQueryable grouping = query.GroupBy("it.GroupProperty", "it").Select(selector, exp);
return grouping;
}
I get the following error when calling the groupby/select line:
System.Linq.Dynamic.ParseException: 'Argument list incompatible with lambda expression'
What type is "it" when grouped? I have tried using other expressions that assume it is an IGrouping<string, Company>, or a IQueryable<Company>, same error. I've tried just selecting "Cost" and moving the Sum() aggregate into the selector string (i.e. Sum(#0(it)) as Value) and always seem to get the same error.
I eventually tried something along the lines of:
Expression<Func<IEnumerable<Company>, decimal?>> exp = l => l.SelectMany(c => c.PricingForms).Sum(fr => fr.Cost);
However this one, I get farther but when attempting to iterate through the results I got a different error.
The LINQ expression node type 'Invoke' is not supported in LINQ to Entities.
So, with this dynamic grouping and injecting my own select expression, what should I assume the datatype of 'it' is? Will this even work?
The type of it is IGrouping<TKey, TElement>, where TKey is dynamic based on the keySelector result type, and TElement is the element type of the input IQueryable. Luckily IGrouping<TKey, TElement> inherits (is a) IEnumerable<TElement>, so as soon as you know the input element type, you can safely base selector on IEnumerable<TElement>.
In other words, the last attempt based on Expression<Func<IEnumerable<Company>, decimal?>> is correct.
The new error you are getting is because #0(it) generates Expression.Invoke call which is not supported by EF. The easiest way to fix that is to use LINQKit Expand method:
Expression<Func<Company, decimal?>> exp = c => c.PricingForms.Sum(fr => fr.Cost);
string selector = "new (it.Key as Key, #0(it) as Value)";
IQueryable grouping = query.GroupBy("it.GroupProperty", "it").Select(selector, exp);
// This would fix the EF invocation expression error
grouping = grouping.Provider.CreateQuery(grouping.Expression.Expand());
return grouping;

Stop jOOQ stripping characters from SQL field

I am implementing a function to get an estimate of the count as described in the PostgreSQL documentation here https://wiki.postgresql.org/wiki/Count_estimate
I'm using the function:
public static Field<Integer> countEstimate(final QueryPart query) {
final String sql = String.format("count_estimate(%s)", escape(query.toString()));
return field(sql(sql), PostgresDataType.INT);
}
Which looks fine until I pass it an IN clause array field in the query. When this happens jOOQ strips the array curly braces from within my SQL. e.g. Calling it with this java code:
final UUID[] ids = new UUID[]{UUID.randomUUID()};
return db.select(countEstimate(db.select(TABLE.ID)
.from(TABLE)
.where(overlaps(ids, TABLE.FILTER_IDS))));
Results in both the variable sql and DSL.sql(sql) in the above function rendering:
count_estimate(E'select "schema"."table"."id"
from "schema"."table"
where (
((\'{"75910f3b-83e6-41ed-bf57-085c225e0131"}\') && ("schema"."table"."filter_ids"))
)')
But field(sql(sql), PostgresDataType.INT) renders this:
count_estimate(E'select "schema"."table"."id"
from "schema"."table"
where (
((\'"75910f3b-83e6-41ed-bf57-085c225e0131"\') && ("schema"."table"."filter_ids"))
)')
Is there any way to work around this and to tell jOOQ to leave my query alone?
(jOOQ 3.8.3, PG 9.5.5, PG driver 9.4-1203-jdbc4)
It turns out it only strips '{}' style arrays. Replacing the code that turns the UUID[] into sql from
DSL.val(ids)
with
DSL.array(Arrays.stream(ids)
.map(UUID::toString)
.collect(Collectors.toList())
.toArray(new String[0]))
.cast(PostgresDataType.UUID.getArrayDataType()
results in it rendering cast(array[\'75910f3b-83e6-41ed-bf57-085c225e0131\'] as uuid[]) prevents it being stripped

Groovy sql.rows returns org.postgresql.util.PSQLException: No hstore extension installed

I am using Groovy Sql in Grails with named parameters to get results from a Postgres DB. My statement is generated dynamically, i.e. concatenated to become the final statement, with the params being added to a map as I go along.
sqlWhere += " AND bar = :namedParam1"
paramsMap.namedParam1 = "blah"
For readability, I am using the groovy string syntax which allows me to write my sql statement over multiple lines, like this:
sql = """
SELECT *
FROM foo
WHERE 1=1
${sqlWhere}
"""
The expression is evaluated as a string containing the linebreaks as \n:
SELECT *\n ...
This is not a problem when I pass params like this
results = sql.rows(sqlString, paramsMap)
but it does become one if paramsMap is empty (which happens since AND bar = :namedParam1 is not always concatenated into the query). I then get an error
org.postgresql.util.PSQLException: No hstore extension installed
which does not really seem to relate to the true nature of the problem. I have for now fixed this with an if...else
if (sqlQuery.params.size() > 0) {
results = sql.rows(sqlString, paramsMap)
} else {
results = sql.rows(sqlString.replace('\n',' '))
}
But this seems a bit weird (especially since it does not work if I use the replace in the if-branch as well).
My question is: why do I really get this error message and is there a better way to prevent it from occuring?
It's certainly a bug in groovy.sql.SQL implementation. The method rows() can't deal with an empty map passed as params. As a workaround, you can test for it and pass an empty list instead.
def paramsMap = [:]
...
if (paramsMap.isEmpty())
paramsMap= []
Issue created at https://issues.apache.org/jira/browse/GROOVY-8082

Using unnest as a field rather than a table in jOOQ

This is the query I am trying to run in PostgreSQL:
SELECT * FROM message WHERE id IN (
SELECT unnest(message_ids) "mid"
FROM session_messages WHERE session_id = '?' ORDER BY "mid" ASC
);
However, I am not able do something:
create.selectFrom(Tables.MESSAGE).where(Tables.MESSAGE.ID.in(
create.select(DSL.unnest(..))
Because DSL.unnest is a Table<?>, which makes sense since it is trying to take a List-like object (mostly a literal) and convert it to table.
I have a feeling that I need to find a way to wrap the function around my field name, but I have no clue as to how to proceed.
NOTE. The field message_ids is of type bigint[].
EDIT
So, this is how I am doing it now, and it works exactly as expected, but I am not sure if this is the best way to do it:
Field<Long> unnestMessageIdField = DSL.field(
"unnest(" + SESSION_MESSAGES.MESSAGE_IDS.getName() + ")",
Long.class)
.as("mid");
Field<Long> messageIdField = DSL.field("mid", Long.class);
MESSAGE.ID.in(
ctx.select(messageIdField).from(
ctx.select(unnestMessageIdField)
.from(Tables.CHAT_SESSION_MESSAGES)
.where(Tables.CHAT_SESSION_MESSAGES.SESSION_ID.eq(sessionId))
)
.where(condition)
)
EDIT2
After going through the code on https://github.com/jOOQ/jOOQ/blob/master/jOOQ/src/main/java/org/jooq/impl/DSL.java I guess the right way to do this would be:
DSL.function("unnest", SQLDataTypes.BIGINT.getArrayType(), SESSION_MESSAGES.MESSAGE_IDS)
EDIT3
Since as always lukas is here for my jOOQ woes, I am going to capitalize on this :)
Trying to generalize this function, in a signature of sort
public <T> Field<T> unnest(Field<T[]> arrayField) {
return DSL.function("unnest", <??>, arrayField);
}
I don't know how I can fetch the datatype. There seems to be a way to get DataType<T[]> from DataType<T> using DataType::getArrayDataType(), but the reverse is not possible. There is this class I found ArrayDataType, but it seems to be package-private, so I cannot use it (and even if I could, it does not expose the field elementType).
Old PostgreSQL versions had this funky idea that it is OK to produce a table from within the SELECT clause, and expand it into the "outer" table, as if it were declared in the FROM clause. That is a very obscure PostgreSQL legacy, and this example is a good chance to get rid of it, and use LATERAL instead. Your query is equivalent to this one:
SELECT *
FROM message
WHERE id IN (
SELECT "mid"
FROM session_messages
CROSS JOIN LATERAL unnest(message_ids) AS t("mid")
WHERE session_id = '?'
);
This can be translated to jOOQ much more easily as:
DSL.using(configuration)
.select()
.from(MESSAGE)
.where(MESSAGE.ID).in(
select(field(name("mid"), MESSAGE.ID.getDataType()))
.from(SESSION_MESSAGES)
.crossJoin(lateral(unnest(SESSION_MESSAGES.MESSAGE_IDS)).as("t", "mid"))
.where(SESSION_MESSAGES.SESSION_ID.eq("'?'"))
)
The Edit3 in the question is quite close to a decent solution for this problem.
We can create a custom generic unnest method for jOOQ which accepts Field and use it in jOOQ query normally.
Helper method:
public static <T> Field<T> unnest(Field<T[]> field) {
var type = (Class<T>) field.getType().getComponentType();
return DSL.function("unnest", type, field);
}
Usage:
public void query(SessionId sessionId) {
var field = unnest(SESSION_MESSAGES.MESSAGE_IDS, UUID.class);
dsl.select().from(MESSAGE).where(
MESSAGE.ID.in(
dsl.select(field).from(SESSION_MESSAGES)
.where(SESSION_MESSAGES.SESSION_ID.eq(sessionId.id))
.orderBy(field)
)
);
}