I am implementing a function to get an estimate of the count as described in the PostgreSQL documentation here https://wiki.postgresql.org/wiki/Count_estimate
I'm using the function:
public static Field<Integer> countEstimate(final QueryPart query) {
final String sql = String.format("count_estimate(%s)", escape(query.toString()));
return field(sql(sql), PostgresDataType.INT);
}
Which looks fine until I pass it an IN clause array field in the query. When this happens jOOQ strips the array curly braces from within my SQL. e.g. Calling it with this java code:
final UUID[] ids = new UUID[]{UUID.randomUUID()};
return db.select(countEstimate(db.select(TABLE.ID)
.from(TABLE)
.where(overlaps(ids, TABLE.FILTER_IDS))));
Results in both the variable sql and DSL.sql(sql) in the above function rendering:
count_estimate(E'select "schema"."table"."id"
from "schema"."table"
where (
((\'{"75910f3b-83e6-41ed-bf57-085c225e0131"}\') && ("schema"."table"."filter_ids"))
)')
But field(sql(sql), PostgresDataType.INT) renders this:
count_estimate(E'select "schema"."table"."id"
from "schema"."table"
where (
((\'"75910f3b-83e6-41ed-bf57-085c225e0131"\') && ("schema"."table"."filter_ids"))
)')
Is there any way to work around this and to tell jOOQ to leave my query alone?
(jOOQ 3.8.3, PG 9.5.5, PG driver 9.4-1203-jdbc4)
It turns out it only strips '{}' style arrays. Replacing the code that turns the UUID[] into sql from
DSL.val(ids)
with
DSL.array(Arrays.stream(ids)
.map(UUID::toString)
.collect(Collectors.toList())
.toArray(new String[0]))
.cast(PostgresDataType.UUID.getArrayDataType()
results in it rendering cast(array[\'75910f3b-83e6-41ed-bf57-085c225e0131\'] as uuid[]) prevents it being stripped
Related
I'm trying to use jOOQ to create a function akin to arrayRemove but that allows removing several elements at once from a PostgreSQL column of type uuid[].
So my first attempt was:
private Field<UUID[]> arrayRemoveAll(final Field<UUID[]> field, final Set<UUID> elements) {
return select(field("array_agg(tab.col)", UUID[].class))
.from(unnest(field).as("tab", "col"))
.where(field("tab.col", UUID.class).notIn(elements))
.asField();
}
Which succeeds at removing every requested element, but has the problem of returning null instead of an empty array if I attempt to remove every element.
So I added a coalesce to my code to make it return an empty array:
private Field<UUID[]> arrayRemoveAll(final Field<UUID[]> field, final Set<UUID> elements) {
final Field<UUID[]> newArray = select(field("array_agg(tab.col)", UUID[].class))
.from(unnest(field).as("tab", "col"))
.where(field("tab.col", UUID.class).notIn(elements))
.asField();
return coalesce(newArray, field("{}", UUID[].class));
}
But running this code threw this exception:
org.jooq.exception.DataAccessException: SQL [<<confidential SQL removed>>]
Caused by: org.postgresql.util.PSQLException: ERROR: syntax error at or near ")"
This is the part of the SQL exception it is complaining about (notice the trailing comma and missing 2nd parameter in the coalesce):
coalesce((select array_agg(tab.col)
from unnest("my_schema"."my_table"."my_field") as "tab"("col")
where tab.col not in (?, ?)), )
Is this a bug in jOOQ?
I found that I had a mix of field and val in the code above, changing field("{}", UUID[].class) to val(new UUID[0]) solves the problem.
Also check Lukas Eder's answer about how to solve the issue using field.
So the final code, with generics, looks like this:
private <T> Field<T[]> arrayRemoveAll(final Field<T[]> field, final Set<T> elements, final T[] emptyArray) {
final Field<T[]> newArray = select(field("array_agg(tab.col)"))
.from(unnest(field).as("tab", "col"))
.where(field("tab.col").notIn(elements))
.asField();
return coalesce(newArray, val(emptyArray));
}
And you can use it in your statements like this:
using(configuration)
.update(MY_TABLE)
.set(MY_TABLE.MY_COLUMN,
arrayRemoveAll(MY_TABLE.MY_COLUMN, someElements, new UUID[0]))
.where(MY_TABLE.ID.eq(...))
.execute();
Your field("{}") does not generate the {} string in the SQL, but is considered a part of jOOQ's plain SQL templating language, which unfortunately doesn't allow for escaping those braces:
https://www.jooq.org/doc/latest/manual/sql-building/plain-sql-templating
Luckily, PostgreSQL supports a more formal, standards-compliant way to create an empty array literal:
field("array[]::uuid[]", UUID.class)
It is great to get prompt replies on the npgsql queries. Thanks to its owner! I am trying to port a sproc that took in array valued parameters to postgres with similar abstraction/semantics. I am wondering how one would shape the pgsql sproc and use npgsql api to call that. I have another layer of application abstraction on top of our dal abstraction. We use data adapters to go to sql server, assoc arrays to oracle and trying to figure out what we could map this to for postgres using npgsql. We have some room in shaping the sproc but still keep the number of input params the same. we could certainly build this sproc much different but we still need it behind the same app api which supplies some set of typed arrays as shown below
public static void Flush2OraWithAssocArrayInsnetworkdatabatch(string dbKey ,int?[] ENDPOINTID,DateTime?[] INSERTEDDATETIME,int?[] RECORDTYPEID,long?[] RECORDVALUE,int?[] PACKETSIZE)
{
Database db = Helper.GetDatabase(dbKey);
using (DbConnection con = db.CreateConnection()){
con.Open();
using (DbCommand cmd = con.CreateCommand()){
cmd.CommandText = "Insnetworkdatabatch";
Helper.InitializeCommand(cmd, 300, "Insnetworkdatabatch");
BuildInsnetworkdatabatchOracleAssocArrayCommandParameters(cmd ,ENDPOINTID,INSERTEDDATETIME,RECORDTYPEID,RECORDVALUE,PACKETSIZE);
try {
Helper.ExecuteNonQuery(cmd, cmd.CommandText);
con.Close();
} catch (DALException ) {
throw;
}
}
}
}
I have a oracle sproc written as follows
create or replace PROCEDURE InsNetworkDataBatch2
(
-- Add the parameters for the stored procedure here
v_endPointID IN arrays.t_number ,
v_insertedDateTime IN arrays.t_date ,
v_recordTypeID IN arrays.t_number ,
v_recordValue IN arrays.t_number ,
v_packetSize IN arrays.t_number )
AS
BEGIN
DECLARE
BEGIN
FORALL i IN v_endpointID.FIRST..v_endpointID.LAST SAVE EXCEPTIONS
INSERT
INTO STGNETWORKSTATS
(
INSERTEDDATE,
ENDPOINTID,
RECORDTYPEID,
RECORDVALUE,
PACKETSIZE
)
VALUES
(
v_insertedDateTime(i),
v_endPointID(i),
v_recordTypeID(i),
v_recordValue(i),
v_packetSize(i)
);
END;
END;
-- END PL/SQL BLOCK (do not remove this line) ----------------------------------
Here is the assoc array package in oracle
create or replace PACKAGE Arrays AS
type t_number is table of number index by binary_integer;
type t_date is table of date index by binary_integer;
END Arrays;
Here is how we build the oracle parm and wondering what its equivalency if at all possible in postgres and trying to see how npgsql will support it
public override void CreateAssociativeArrayParameter(DbCommand cmd, string parameterName, object parameterValue, string dbType, ParameterDirection direction)
{
OracleDbType oracleDbType = dbSpecificTypesMap[dbType];
OracleParameter param = new OracleParameter(parameterName, oracleDbType, direction);
param.CollectionType = OracleCollectionType.PLSQLAssociativeArray;
param.Value = parameterValue;
cmd.Parameters.Add(param);
}
I don't know anything about Oracle arrays or associative arrays. However, PostgreSQL has a rich support for complex types. PostgreSQL arrays are a good way to store an array of values in a column, and PostgreSQL even provides indexing and database-side functions to work with arrays.
If you're looking for a dictionary type (associative array?), take a look at hstore or json.
EDITED: If your associative array has a fixed schema (i.e. the fields don't change), you can also consider PostgreSQL composite.
Here is an attempt with Postgres stored procedure. This is now working. I got around some casting issues thrown from inside the npgsql which was a result of my .net type not being compatible with the sproc parameter data type in postgres.
Here is how i am trying to add the param value
create or replace FUNCTION InsNetworkDataBatch
(
-- Add the parameters for the stored procedure here
v_endPointID IN int[] ,
v_insertedDateTime IN timestamp[] ,
v_recordTypeID IN int[] ,
v_recordValue IN bigint[] ,
v_packetSize IN int[] ) RETURNS void
LANGUAGE 'plpgsql'
AS $$
BEGIN
DECLARE
BEGIN
FOR i IN array_lower(v_endPointID, 1) .. array_upper(v_endPointID, 1)
loop
INSERT INTO STGNETWORKSTATS
(
INSERTEDDATE,
ENDPOINTID,
RECORDTYPEID,
RECORDVALUE,
PACKETSIZE
)
VALUES
(
v_insertedDateTime[i],
v_endPointID[i],
v_recordTypeID[i],
v_recordValue[i],
v_packetSize[i]
);
end loop;
END;
END;
$$
Here is how i am trying to bind the app to the command params
public override void CreateAssociativeArrayParameter(DbCommand cmd, string parameterName, object parameterValue, string dbType, ParameterDirection direction)
{
NpgsqlDbType npgsqlDbType;
if (dbSpecificTypesMap.ContainsKey(dbType))
{
npgsqlDbType = dbSpecificTypesMap[dbType];
}
else
{
throw new ApplicationException($"The db type {dbType} could not be parsed into the target NpgsqlDbType. Please check the underlying type of the parameter");
}
NpgsqlParameter param = new NpgsqlParameter(parameterName.ToLower(), NpgsqlDbType.Array | npgsqlDbType);
param.Value = parameterValue;
cmd.Parameters.Add(param);
}
I am using Groovy Sql in Grails with named parameters to get results from a Postgres DB. My statement is generated dynamically, i.e. concatenated to become the final statement, with the params being added to a map as I go along.
sqlWhere += " AND bar = :namedParam1"
paramsMap.namedParam1 = "blah"
For readability, I am using the groovy string syntax which allows me to write my sql statement over multiple lines, like this:
sql = """
SELECT *
FROM foo
WHERE 1=1
${sqlWhere}
"""
The expression is evaluated as a string containing the linebreaks as \n:
SELECT *\n ...
This is not a problem when I pass params like this
results = sql.rows(sqlString, paramsMap)
but it does become one if paramsMap is empty (which happens since AND bar = :namedParam1 is not always concatenated into the query). I then get an error
org.postgresql.util.PSQLException: No hstore extension installed
which does not really seem to relate to the true nature of the problem. I have for now fixed this with an if...else
if (sqlQuery.params.size() > 0) {
results = sql.rows(sqlString, paramsMap)
} else {
results = sql.rows(sqlString.replace('\n',' '))
}
But this seems a bit weird (especially since it does not work if I use the replace in the if-branch as well).
My question is: why do I really get this error message and is there a better way to prevent it from occuring?
It's certainly a bug in groovy.sql.SQL implementation. The method rows() can't deal with an empty map passed as params. As a workaround, you can test for it and pass an empty list instead.
def paramsMap = [:]
...
if (paramsMap.isEmpty())
paramsMap= []
Issue created at https://issues.apache.org/jira/browse/GROOVY-8082
This is the query I am trying to run in PostgreSQL:
SELECT * FROM message WHERE id IN (
SELECT unnest(message_ids) "mid"
FROM session_messages WHERE session_id = '?' ORDER BY "mid" ASC
);
However, I am not able do something:
create.selectFrom(Tables.MESSAGE).where(Tables.MESSAGE.ID.in(
create.select(DSL.unnest(..))
Because DSL.unnest is a Table<?>, which makes sense since it is trying to take a List-like object (mostly a literal) and convert it to table.
I have a feeling that I need to find a way to wrap the function around my field name, but I have no clue as to how to proceed.
NOTE. The field message_ids is of type bigint[].
EDIT
So, this is how I am doing it now, and it works exactly as expected, but I am not sure if this is the best way to do it:
Field<Long> unnestMessageIdField = DSL.field(
"unnest(" + SESSION_MESSAGES.MESSAGE_IDS.getName() + ")",
Long.class)
.as("mid");
Field<Long> messageIdField = DSL.field("mid", Long.class);
MESSAGE.ID.in(
ctx.select(messageIdField).from(
ctx.select(unnestMessageIdField)
.from(Tables.CHAT_SESSION_MESSAGES)
.where(Tables.CHAT_SESSION_MESSAGES.SESSION_ID.eq(sessionId))
)
.where(condition)
)
EDIT2
After going through the code on https://github.com/jOOQ/jOOQ/blob/master/jOOQ/src/main/java/org/jooq/impl/DSL.java I guess the right way to do this would be:
DSL.function("unnest", SQLDataTypes.BIGINT.getArrayType(), SESSION_MESSAGES.MESSAGE_IDS)
EDIT3
Since as always lukas is here for my jOOQ woes, I am going to capitalize on this :)
Trying to generalize this function, in a signature of sort
public <T> Field<T> unnest(Field<T[]> arrayField) {
return DSL.function("unnest", <??>, arrayField);
}
I don't know how I can fetch the datatype. There seems to be a way to get DataType<T[]> from DataType<T> using DataType::getArrayDataType(), but the reverse is not possible. There is this class I found ArrayDataType, but it seems to be package-private, so I cannot use it (and even if I could, it does not expose the field elementType).
Old PostgreSQL versions had this funky idea that it is OK to produce a table from within the SELECT clause, and expand it into the "outer" table, as if it were declared in the FROM clause. That is a very obscure PostgreSQL legacy, and this example is a good chance to get rid of it, and use LATERAL instead. Your query is equivalent to this one:
SELECT *
FROM message
WHERE id IN (
SELECT "mid"
FROM session_messages
CROSS JOIN LATERAL unnest(message_ids) AS t("mid")
WHERE session_id = '?'
);
This can be translated to jOOQ much more easily as:
DSL.using(configuration)
.select()
.from(MESSAGE)
.where(MESSAGE.ID).in(
select(field(name("mid"), MESSAGE.ID.getDataType()))
.from(SESSION_MESSAGES)
.crossJoin(lateral(unnest(SESSION_MESSAGES.MESSAGE_IDS)).as("t", "mid"))
.where(SESSION_MESSAGES.SESSION_ID.eq("'?'"))
)
The Edit3 in the question is quite close to a decent solution for this problem.
We can create a custom generic unnest method for jOOQ which accepts Field and use it in jOOQ query normally.
Helper method:
public static <T> Field<T> unnest(Field<T[]> field) {
var type = (Class<T>) field.getType().getComponentType();
return DSL.function("unnest", type, field);
}
Usage:
public void query(SessionId sessionId) {
var field = unnest(SESSION_MESSAGES.MESSAGE_IDS, UUID.class);
dsl.select().from(MESSAGE).where(
MESSAGE.ID.in(
dsl.select(field).from(SESSION_MESSAGES)
.where(SESSION_MESSAGES.SESSION_ID.eq(sessionId.id))
.orderBy(field)
)
);
}
I have this scenario:
A SQL Server table myTable with field1, xmlField (nvarchar(50) and xml sql server data type)
Linq to entities
Now I'd like to get a query like this:
SELECT Field1, XmlField
FROM MyTable
WHERE CAST(XmlField AS nvarchar(4000)) = '<myXml />'
Obviously this is a correct query in SQL Server but I can't find a solution to write this in L2E.
Please notify that this code doesn't work:
var query = from row in context.MyTables
where (string)row.XmlField == "<myXml />"
select row
and other cast methods too.
This just because in L2E the "ToString" does't work correctly.
Now my idea is this one: an extension method:
var query = from row in context.MyTables
select row
query = query.CompareXml("XmlField", "<myXml />")
and this is the extended method:
public static IQueryable<TSource> CompareXml<TSource>(this IQueryable<TSource> source, string xmlFieldName, string xmlToCompare)
{
ConstantExpression xmlValue = Expression.Constant(xmlToCompare);
ParameterExpression parameter = Expression.Parameter(typeof(TSource), source.ElementType.Name);
PropertyInfo propertyInfo = typeof(TSource).GetProperty(xmlFieldName);
MemberExpression memberAccess = Expression.MakeMemberAccess(parameter, propertyInfo);
var stringMember = Expression.Convert(memberAccess, typeof(string));
BinaryExpression clauseExpression = Expression.Equal(xmlValue, stringMember);
return source.Where(Expression.Lambda<Func<TSource, bool>>(clauseExpression, parameter));
}
and again this doesn't work too.
Now I'd like to understand how I can force a "Convert" using Cast so I can compare Xml and nvarchar.
Thanks in advance
Massimiliano
Unfortunately EF still doesn't properly support XML columns. I'm afraid pretty much the only choice I know of is to create a view that does the cast and map that to a different entity. This will probably make the code awkward but also offers additional possible scenarios; for example, with a lot of SQL code, you could map elements in the XML columns to actual columns in the view, allowing you to make queries on specific parts of the XML.
On the bright side, at least inserting values in an XML column works pretty much as expected.