Can I cast the result of plainto_tsquery to a string using npgsql / ef core to allow searching by part of a word or phrase? - entity-framework-core

My end goal is to be able to do a prefix search in npgsql using ef core, very similar to Npgsql Entity Framework Core PostgreSQL full text search by part of word. And Match a phrase starting with a prefix with full text search explains what the query will look like, something like:
select * from mytable
where
mycolumn ## to_tsquery(CONCAT(CAST(plainto_tsquery('the query') as text), ':*'))
In dotnet, I can get close to this, but not exactly what I want.
Using "Matches" converts to plainto_tsquery:
var results= MyTable
.Where(c => c.MyColumn.Matches("the query"))
.ToList();
which results in:
select * from mytable
where
mycolumn ## plainto_tsquery('query')
Using EF.Functions.ToTsQuery converts to to_tsquery (closer):
var results= MyTable
.Where(c => EF.Functions.ToTsQuery("the query")))
.ToList();
which results in:
select * from mytable
where
mycolumn ## to_tsquery('the query')
And now I want to use to_tsquery, passing in the results of plainto_tsquery and appending the ":*" to make it a prefix, but I can't figure out how to convert plainto_tsquery results to a string. I am trying the following:
var results= MyTable
.Where(c => c.SearchVector.Matches(
EF.Functions.ToTsQuery(
EF.Functions.PlainToTsQuery("the query") + ":*")))
.ToList();
But this fails with:
System.InvalidCastException: Unable to cast object of type 'System.String' to type 'NpgsqlTypes.NpgsqlTsQuery'.
I think I just need to be able to cast the results of EF.Functions.PlainToTsQuery("the query") to a string, but I am not sure how to do that or if possible.
Appreciate any help.
Thanks,
Eric

I was not able to figure out how to cast the results of EF.Functions.PlainToTsQuery("the query") to text, but I was able to solve my issue by creating a stored function which creates the tsquery. And then calling the stored function via ef core.
The stored function is something like the following:
CREATE OR REPLACE FUNCTION public.get_ts_query_for_prefix_search(
query_text text)
RETURNS tsquery
LANGUAGE 'sql'
AS $BODY$
SELECT
CASE plainto_tsquery('simple', query_text) = ''
WHEN true THEN ''
ELSE to_tsquery('simple', CAST(plainto_tsquery('simple', query_text) AS TEXT) || ':*')
END
$BODY$;
And the ef core where clause will be something like the following:
var matches = MyTable
.Where(w => w.MyColumn.Matches(GetTsQueryForPartialSearch(queryText));

Related

PostgreSQL execute SELECT * FROM (t as a); return ERROR: syntax error at or near ")"

Why these SQLs can't work in PostgreSQL?
SELECT * FROM (t as a);
or
SELECT * FROM (t);
ERROR: syntax error at or near ")"
Those SQLs work well in MySQL.
Well, it's invalid SQL.
If you want to give the table an alias you need to use from t as a.
The form from (....) requires a valid query inside the parentheses and t as a (or just t) is not a valid query. Additionally: a derived table (which is what (...) defines) requires an alias. So at least it has to be from (...) as bla
To get all rows and all columns from a table the SQL standard provides the shortcut table some_table which is supported by Postgres.
So the following would be valid:
select * from (table t) as a
Typically, table alias' are applied where there is a join. For example:
SELECT alias1.login_name, alias2.name
FROM tablename1 AS alias1
JOIN tablename2 AS alias2
ON alias1.id = alias2.role_id;
To apply an alias to a single table:
SELECT * FROM tablename AS alias;
..will do it for you, no need for parentheses.
You can confirm/test by example below if value is an integer:
SELECT * FROM tablename AS alias
WHERE alias.colum = value;
..or if value is a string:
SELECT * FROM tablename AS alias
WHERE alias.colum = 'value';
postgreSQL is strongly-typed, so the first example above will work for Int and Bool values, however if you have Date or other values, you may need to apply type casting. See this link for helpful info*: www.postgresqltutorial.com/postgresql-tutorial/postgresql-cast.
**Remember: For psql strings, always use single quotes for values, use double quotes for table names and column names.

xmlcast is returing null value in Db2

I am always getting a null value from using XMLCAST function in Db2. But when using a normal function call it is returning value in xml format... as below query proper xml data is coming. function Getdatavalue is returning xml datatype. using Linux 8.2 and db2 11.5
db2 "select sample.Getdatavalue(1,4504)) from sysibm.sysdummy1"
in the above result we have some double quotes in output xml.
but once we have used xmlcast function as below, it is returning the hexa value of null. i am using MD5 hashing mechanism.
db2 "select HEX(HASH(xmlcast(sample.Getdatavalue(1,4504) as varchar(32672)),0)) as hash_val from sysibm.sysdummy1";
even once i tried without Hash it was returning null result for below query. it looks xmlcast is not working.
db2 "select xmlcast(sample.Getdatavalue(1,4504) as varchar(32672)) as hash_val from sysibm.sysdummy1";
is there any way to convert xml datatype to clob or varchar.
body of Getdatavalue as:
CREATE FUNCTION sample.Getdatavalue (v1 smallint, v2 integer)
RETURNS xml
LANGUAGE SQL
BEGIN ATOMIC
DECLARE v_Data xml;
set v_Data = (SELECT XMLELEMENT( NAME "rt_ky_cmpnnt",
XMLAGG(XMLELEMENT(NAME "rt", XMLAttributes(rt.rt_field_add_val AS
"rt_add_val",
rt.rt_name AS
"rt_name",
rt.rt_name AS
"rt_name",
rt.rt_val1 AS "rt_val1",
rt_lim AS "rt_lim",
rt.rt_val2 AS "rt__val2",
rt.rt_whole_val AS "rt_whole_val" ))order by
rf.rt_num asc)OPTION null ON NULL)
FROM sample.cmpnnt_val rt inner join sample.field_val rf
on rf.abc=rt.abc
and rf.pqr=rt.pqr
and rt.xyz=rf.xyz
where rt.abc = v1
AND rt.bcd = v2 );
RETURN v_Data;
END!
can anyone please explain how to handle xmlcast.
To retrieve a string from an XML value, use the function XMLSERIALIZE.
The XMLSERIALIZE function returns a serialized XML value of the specified data type generated from the XML-expression argument.

With PostgREST, convert a column to and from an external encoding in the API

We are using PostgREST to automatically generate a REST API for a Postgres database. Our primary keys have an external representation that's different from how we store them internally. For simplicity's sake lets pretend the ids are stored as integers but we represent them as hexadecimal strings outwardly.
It's simple enough to get PostgREST to convert to the external representation for read operations:
CREATE DOMAIN hexid AS bigint;
CREATE TABLE fruits (
fruit_id hexid PRIMARY KEY,
name text
);
CREATE OR REPLACE VIEW api_fruits AS
SELECT to_hex(fruit_id) as fruit_id, name FROM fruits;
INSERT INTO fruits(fruit_id, name) VALUES('51955', 'avocado');
PostgREST generates the expected representation when we GET api_fruits:
[
{
"fruit_id": "caf3",
"name": "avocado"
}
]
But that's about as far as we get with this solution. It's a one way transformation so we won't be able to POST/PATCH records this way. The way PostgREST works is to transform such requests into equivalent INSERT and UPDATE statements. But this view with its custom formatting is not updatable. This is what would happen if we tried:
ERROR: cannot insert into column "fruit_id" of view "api_fruits"
DETAIL: View columns that are not columns of their base relation are not updatable.
STATEMENT: WITH pgrst_source AS (WITH pgrst_payload AS (SELECT $1::json AS json_data), pgrst_body AS ( SELECT CASE WHEN json_typeof(json_data) = 'array' THEN json_data ELSE json_build_array(json_data) END AS val FROM pgrst_payload) INSERT INTO "api_x"."api_fruits"("fruit_id", "name") SELECT "fruit_id", "name" FROM json_populate_recordset (null::"api_x"."api_fruits", (SELECT val FROM pgrst_body)) _ RETURNING "api_x"."api_fruits".*) SELECT '' AS total_result_set, pg_catalog.count(_postgrest_t) AS page_total, CASE WHEN pg_catalog.count(_postgrest_t) = 1 THEN coalesce((
WITH data AS (SELECT row_to_json(_) AS row FROM pgrst_source AS _ LIMIT 1)
SELECT array_agg(json_data.key || '=eq.' || json_data.value)
FROM data CROSS JOIN json_each_text(data.row) AS json_data
WHERE json_data.key IN ('')
), array[]::text[]) ELSE array[]::text[] END AS header, '' AS body, nullif(current_setting('response.headers', true), '') AS response_headers, nullif(current_setting('response.status', true), '') AS response_status FROM (SELECT * FROM pgrst_source) _postgrest_t
We can't INSERT into "View columns that are not columns of their base relation".
The obvious workaround is to serve fruit_id as a straight column, just an integer. With some post and preprocessing at the nginx level we can hex encode it there (and hex decode incoming ids). I'm wondering if we can do better than that though. For large API operations, re-encoding the JSON will use a lot of memory and CPU time and it seems so unnecessary.
It would have been great to be able to use a custom CREATE CAST to take the incoming hexadecimal strings and turn them back into integers, something like this:
CREATE CAST (json AS hexid) WITH FUNCTION json_to_hexid AS ASSIGNMENT;
But alas custom casts are ignored on CREATE DOMAIN types. And we can't make a true custom column type because our cloud Postgres host (Google Cloud SQL) doesn't allow custom extensions.
It feels like some combination of INSTEAD OF triggers or rules could work. But when using query parameters to filter results using query parameters (e.g. select a fruit by id), I don't think there's an appropriate trigger to use. INSTEAD OF doesn't work for straight SELECT does it?
For example I've tested doing something like this to take care of INSERT and allow POST with PostgREST. It works:
CREATE OR REPLACE FUNCTION api_fruits_insert()
RETURNS trigger AS
$$
BEGIN
INSERT INTO fruits(fruit_id, name) VALUES (('x' || lpad(NEW.fruit_id, 16, '0'))::bit(64)::bigint::hexid, NEW.name);
RETURN NEW;
END
$$ LANGUAGE 'plpgsql';
CREATE TRIGGER api_fruits_insert
INSTEAD OF INSERT
ON api_fruits
FOR EACH ROW
EXECUTE PROCEDURE api_fruits_insert();
The trouble is in the WHERE clause. Let's PATCH api_fruits?fruit_id=in.(7b,caf3) with {"name": "pear"}. This works out of the box since the name column is updatable but look at the query:
WITH pgrst_source AS (WITH pgrst_payload AS (SELECT $1::json AS json_data), pgrst_body AS ( SELECT CASE WHEN json_typeof(json_data) = 'array' THEN json_data ELSE json_build_array(json_data) END AS val FROM pgrst_payload) UPDATE "api_x"."api_fruits" SET "name" = _."name" FROM (SELECT * FROM json_populate_recordset (null::"api_x"."api_fruits" , (SELECT val FROM pgrst_body) )) _ WHERE "api_x"."api_fruits"."fruit_id" = ANY ($2) RETURNING 1) SELECT '' AS total_result_set, pg_catalog.count(_postgrest_t) AS page_total, array[]::text[] AS header, '' AS body, nullif(current_setting('response.headers', true), '') AS response_headers, nullif(current_setting('response.status', true), '') AS response_status FROM (SELECT * FROM pgrst_source) _postgrest_t
DETAIL: parameters: $1 = '{
"name": "pear"
}', $2 = '{7b,caf3}'
So we have essentially UPDATE api_fruits SET name='berry' WHERE fruit_id IN ('7b', 'caf3');. Surprisingly this works but it's a full table scan so Postgres can evaluate to_hex(fruit_id) for each row looking for matches. The same happens if we try to GET a record by fruit_id. How would we rewrite the WHERE clauses?
It really feels like some combination of just the right Postgres and PostgREST features should be able to get us to a point where it's all happening in Postgres without nginx's help and without excessive complexity. Any ideas?

Best way to search in postgres by a group of keyword

right now I have a keyword array like:
['key1', 'key2', 'key3'.......] , the keyword can be number or character.
If I want to search in my table (postgres database), and find out all record contain any of keyword in that array, how can I do that?
For example:
I got a table which has a column called name and a column called description
I need find all record that either name or description contains any keywords in that array.
thanks
Maybe this example will be useful:
CREATE TABLE TEST(
FIELD_KEY TEXT);
INSERT INTO TEST VALUES('this is hello');
INSERT INTO TEST VALUES('hello');
INSERT INTO TEST VALUES('this');
INSERT INTO TEST VALUES('other message');
SELECT *
FROM TEST
WHERE FIELD_KEY LIKE ANY (array['%this%', '%hel%']);
This will return:
this is hello
hello
this
Here other example:
SELECT *
FROM TEST
WHERE FIELD_KEY ~* 'this|HEL';
~* is case insensitive, ~ is case sensitive
You can try this example here.
select *
from t
where
array[name] <# my_array
or array[description] <# my_array
Couple the like operator with the any subquery expression:
select *
from t
where name like any (values ('%John%'), ('%Mary%'))
Or the array syntax:
where name like any (array['%John%', '%Mary%'])

Escaping formula for Grails derived properties

Grails offers derived properties to generate a field from a SQL expression using the formula mapping parameter:
static mapping = {
myfield formula: "field1 + field2"
}
I'm trying to use the formula parameter with a PostgreSQL database to make a concatenated field. The syntax is a little strange since PostgreSQL 8.4 doesn't yet support concat_ws:
static mapping = {
myfield formula: "array_to_string(array[field1, field2],' ')"
}
The produced SQL shown with loggingSql = true in the DataSource config has the table prefix inserted into some strange places:
select table0_.field1 as field1_19_0_,
table0_.field2 as field2_19_0_,=
array_to_string(table0_.array[field1, table0_.field2], ' ') as formula0_0_
from test_table table0_ where table0_.id=?
The table prefix errantly appears before array but not before field1 in the derived formula. Is there a way to escape the prefix or correct this behavior more explicitly?
This is just an issue with parsing the formula syntax. GORM tries to insert the table prefix for unquoted expressions not followed by parens, so the ARRAY[] notation trips it up.
My solution was to define the concat_ws function:
CREATE OR REPLACE FUNCTION concat_ws(separator text, variadic str text[])
RETURNS text as $$
SELECT array_to_string($2, $1);
$$ LANGUAGE sql;
The GORM formula parameter can now avoid the ARRAY[] syntax, and works as expected.
myfield formula: "concat_ws(' ', field1, field2)"
I had a very similar problem and solved it by adding single-quotes around the things that GORM was trying to prefix:
static mapping =
{
dayOfYear formula: " EXTRACT('DOY' FROM observed) "
}
GORM then produced this, which worked:
select
EXTRACT('DOY' FROM observed) as y1_
This may not work in all cases, but I hope it helps somebody.