Wonder if someone can help me with an array aggregator issue
I’ve got a query that does a join using a joining table then it filters down all values that are inside a given array and filters out values that are in another array.
The code looks like this:
Product
|> join(:inner, [j], jt in "job_tech", on: j.id == jt.product_id)
|> join(:inner, [j, jt], t in Tech, on: jt.ingredient_id == t.id)
|> group_by([j], j.id)
|> having_good_ingredients(good_ingredients)
|> not_having_bad_ingredients(bad_ingredients)
With having_good_ingredients look like this:
def having_good_ingredients(query, good_ingredients) do
if Enum.count(good_ingredients) > 0 do
query
|> having(fragment("array_agg(t2.name) && (?)::varchar[]", ^good_ingredients))
else
query
end
end
This works but it’ll grab all values that satisfy any of the values in the good_stacks array where I want them to only satisfy if all of the stacks work
i.e. if I’ve got [A, C] in my array, I want to return values that have A AND C, not just A and not just C.
Anyone have any ideas?
I believe you want to use the #> operator, instead of the overlap && operator:
having(fragment("array_agg(t2.name) #> (?)::varchar[]", ^good_ingredients))
Reference: https://www.postgresql.org/docs/current/functions-array.html#ARRAY-OPERATORS-TABLE
Related
In an Ecto query I'm running against Postgres (13.6), I need to interpolate a list of ids into a fragment - this is generally not something I have a problem with, but in this case, the list of ids is being received as a list of strings that need to be cast to integers (or, more specifically, BIGINT). The query that I think that I need is as follows, which the troublesome bit being ANY(ARRAY?::BIGINT[]):
ModelA
|> where(
[ma],
fragment(
"EXISTS (SELECT * FROM model_b mb WHERE mb.a_id = ? AND mb.c_id = ANY(ARRAY?::BIGINT[]))",
a.id,
^c_ids
)
)
where c_ids would be a list like ["1449441579", "2345556834"]
However, when I run this, I get the error
(Postgrex.Error) ERROR 42703 (undefined_column) column "array$4" does not exist
referring to the generated SQL
ANY(ARRAY$4::BIGINT[])
Of course, I could convert the array of c_ids to integers beforehand in my app code, but I'd like to see if I can get it to cast in the query itself.
Writing the fragment in straight SQL works out just fine:
SELECT * FROM model_b mb WHERE mb.a_id = 1 AND mb.c_id = ANY(ARRAY['1449441579', '2345556834']::BIGINT[]);
What is the idiomatic way to get this kind of array casting to work in an Ecto fragment? Many thanks.
Just to codify my comment, I would do the integer conversion before the query. You can use the dynamic macro to support IN queries:
import Ecto.Query
alias YourApp.Repo
alias YourApp.SomeSchema, as: ModelA
strs = ["1", "2", "3"]
ids = Enum.map(strs, fn x -> String.to_integer(x) end)
conditions = dynamic([tbl], tbl.id in ^ids)
Repo.all(
from ma in ModelA,
where: ^conditions
)
|> IO.inspect()
I did find a post that led me to a potential solution here - https://elixirforum.com/t/interpolating-lists-into-ecto-query-fragment-1/16690/7 - however, I'm not convinced that this is optimal for me.
By using Enum.join and converting my initial array of strings into a string and then allowing Postgres to cast that string to an array of bigints, it worked out:
ModelA
|> where(
[ma],
fragment(
"EXISTS (SELECT * FROM model_b mb WHERE mb.a_id = ? AND mb.c_id = ANY(ARRAY?::BIGINT[]))",
a.id,
^Enum.join(c_ids, ",")
)
)
So I'm posting it here for reference and because it does "work", but I feel sort of silly doing this, because I specifically was trying to avoid processing the list before passing it to PG... so yeah I'd still really like to hear about the "right" way to do this...
(Ecto.Query.CompileError) any(l.signers) is not a valid query expression.
(from l in Listing, where: "xxxxxx#gmail.com" == any(l.signers))
|> Repo.all()
Ecto.Query.API.in/2 is supposed to be used to cover postgresql ANY selector.
where: ^mail_addr in l.signers
It is assumed that l.signers is an enumerable.
select id,address,owners from listings where 'xxxxxx#gmail.com' = ANY(signers);
This query is working in PostgreSQL.
I have a table in postgresql and i want to update the value of column "points" for all the records with a random number. In other languages we could loop over all the db records but how can i do it with ecto? I tried this:
Repo.all(from u in User, update: User.changeset(u, %{points: :rand.uniform(100)}))
but it outputs the following error:
== Compilation error in file lib/hello_remote/user.ex ==
** (Ecto.Query.CompileError) malformed update `User.changeset(u, %{points: :rand.uniform(100)})` in query expression, expected a keyword list with set/push/pop as keys with field-value pairs as values
expanding macro: Ecto.Query.update/3
lib/hello_remote/user.ex:30: HelloRemote.User.update_points/0
expanding macro: Ecto.Query.from/2
lib/hello_remote/user.ex:30: HelloRemote.User.update_points/0
(elixir 1.10.4) lib/kernel/parallel_compiler.ex:304: anonymous fn/4 in Kernel.ParallelCompiler.spawn_workers/7
I've also tried this:
from(u in User)
|> Repo.update_all(set: [points: Enum.random(0..100)])
but it updates all the records with the same value
You can use fragment/1 with update_all/3, calling a PostgreSQL function to calculate the random values, for example:
update(User, set: [points: fragment("floor(random()*100)")])
|> Repo.update_all([])
I don't think it is possible with the update_all function. The function was created to update many rows with the same value.
You can create a loop and update each record separately, it's not nice but a working solution.
User
|> Repo.all()
|> Enum.each(fn user -> update_user(user) end)
def update_user(user) do
user
|> Ecto.Changeset.cast(%{"points" => Enum.random(0..100)}, [:points])
|> Repo.update!()
end
If you would like to do it one call you can construct a raw SQL query and use that.
I have an EctoEnum.Postgres:
# #see: https://en.wikipedia.org/wiki/ISO_4217
defmodule PricingEngine.Pricing.CurrencyEnum do
#options [
:AED,
:AFN,
# snip...
:ZWL
]
use EctoEnum.Postgres,
type: :currency,
enums: #options
def values, do: #options
end
This enum has been included in our Postgres database
We also have a structure:
defmodule PricingEngine.Pricing.Currency do
use Ecto.Schema
import Ecto.Changeset
schema "currencies" do
field(:currency, PricingEngine.Pricing.CurrencyEnum)
timestamps()
end
#doc false
def changeset(currency, attrs) do
currency
|> cast(attrs, [:currency])
|> validate_required([:currency])
|> unique_constraint(:currency)
end
end
We can currently successfully use the following functions to figure out which currencies are active/used:
def active_currency_isos do
Repo.all(select(Currency, [record], record.currency))
end
defdelegate all_currency_isos,
to: CurrencyEnum,
as: :values
def inactive_currency_iso do
Pricing.all_currency_isos() -- Pricing.active_currency_isos()
end
This works, but I'm led to believe this could be more efficient if we just asked the database for this information.
Any idea(s) how to do this?
If you want to get a list of all the used enums you should just do a distinct on the currency field. This uses the Postgres DISTINCT ON operator:
from(c in Currency,
distinct: c.currency,
select: c.currency
)
This will query the table, unique by the currency column, and return only the currency column values. You should get an array of all of the enums that exist in the table.
There are some efficiency concerns with doing it this way which could be mitigated by materialized views, lookup tables, in-memory cache etc. However, if your data set isn't extremely large, you should be able to use this for a while.
Edit:
Per the response, I will show how to get the unused enums.
There are 2 ways to do this.
Pure SQL
This query will get all of the used ones and do a difference from the entire set of available enums. The operator we use to do this is EXCEPT and you can get a list of all available enums with enum_range. I will use unnest to turn the array of enumerated types into individual rows:
SELECT unnest(enum_range(NULL::currency)) AS unused_enums
EXCEPT (
SELECT DISTINCT ON (c.name) c.name
FROM currencies c
)
You can execute this raw SQL in Ecto by doing this:
Ecto.Adapters.SQL.query!(MyApp.Repo, "SELECT unnest(...", [])
From this you'll get a Postgresx.Result that you'll have to get the values out of:
result
|> Map.get(:rows, [])
|> List.flatten()
|> Enum.map(&String.to_existing_atom/1)
I'm not really sure of a way to code this query up in pure Ecto, but let me know if you figure it out.
In Code
You can do the first query that I posted before with distinct then do a difference in the code.
query = from(c in Currency,
distinct: c.currency,
select: c.currency
)
CurrencyEnum.__enums__() -- Repo.all(query)
Either way is probably negligible in terms of performance so it's up to you.
I have two ecto queries, say query1 and query2, now I am performing full join between those two ecto queries, something like this
query1 |> join(:full, [a], b in subquery(query2), a.id == b.id)
Everything is working fine, but some of the records are returned as nil, something like this
[%user{}, %user{}, %user{}, nil, %user{}, %user{}, nil, %user{}, %user{}, nil]
I think the same issue has also been discussed in this ecto thread.
Is there some workaround for this ecto join issue.
That Ecto thread is a separate issue. :)
You are using a full join, which means the left and right side of a join will be returned even when there is no match. If you use an :inner join, it should work as expected:
query1 |> join(:inner, [a], b in subquery(query2), a.id == b.id)
If you need to keep a full join, maybe you want to return both a and b in your example?
query1
|> join(:inner, [a], b in subquery(query2), a.id == b.id)
|> select([a, b], {a, b})
This way it is guaranteed one of them is not nil.