In Elixir with Postgres, how can I have the database return the enum values which are NOT in use? - postgresql

I have an EctoEnum.Postgres:
# #see: https://en.wikipedia.org/wiki/ISO_4217
defmodule PricingEngine.Pricing.CurrencyEnum do
#options [
:AED,
:AFN,
# snip...
:ZWL
]
use EctoEnum.Postgres,
type: :currency,
enums: #options
def values, do: #options
end
This enum has been included in our Postgres database
We also have a structure:
defmodule PricingEngine.Pricing.Currency do
use Ecto.Schema
import Ecto.Changeset
schema "currencies" do
field(:currency, PricingEngine.Pricing.CurrencyEnum)
timestamps()
end
#doc false
def changeset(currency, attrs) do
currency
|> cast(attrs, [:currency])
|> validate_required([:currency])
|> unique_constraint(:currency)
end
end
We can currently successfully use the following functions to figure out which currencies are active/used:
def active_currency_isos do
Repo.all(select(Currency, [record], record.currency))
end
defdelegate all_currency_isos,
to: CurrencyEnum,
as: :values
def inactive_currency_iso do
Pricing.all_currency_isos() -- Pricing.active_currency_isos()
end
This works, but I'm led to believe this could be more efficient if we just asked the database for this information.
Any idea(s) how to do this?

If you want to get a list of all the used enums you should just do a distinct on the currency field. This uses the Postgres DISTINCT ON operator:
from(c in Currency,
distinct: c.currency,
select: c.currency
)
This will query the table, unique by the currency column, and return only the currency column values. You should get an array of all of the enums that exist in the table.
There are some efficiency concerns with doing it this way which could be mitigated by materialized views, lookup tables, in-memory cache etc. However, if your data set isn't extremely large, you should be able to use this for a while.
Edit:
Per the response, I will show how to get the unused enums.
There are 2 ways to do this.
Pure SQL
This query will get all of the used ones and do a difference from the entire set of available enums. The operator we use to do this is EXCEPT and you can get a list of all available enums with enum_range. I will use unnest to turn the array of enumerated types into individual rows:
SELECT unnest(enum_range(NULL::currency)) AS unused_enums
EXCEPT (
SELECT DISTINCT ON (c.name) c.name
FROM currencies c
)
You can execute this raw SQL in Ecto by doing this:
Ecto.Adapters.SQL.query!(MyApp.Repo, "SELECT unnest(...", [])
From this you'll get a Postgresx.Result that you'll have to get the values out of:
result
|> Map.get(:rows, [])
|> List.flatten()
|> Enum.map(&String.to_existing_atom/1)
I'm not really sure of a way to code this query up in pure Ecto, but let me know if you figure it out.
In Code
You can do the first query that I posted before with distinct then do a difference in the code.
query = from(c in Currency,
distinct: c.currency,
select: c.currency
)
CurrencyEnum.__enums__() -- Repo.all(query)
Either way is probably negligible in terms of performance so it's up to you.

Related

Ecto - casting an array of strings to integers in a fragment

In an Ecto query I'm running against Postgres (13.6), I need to interpolate a list of ids into a fragment - this is generally not something I have a problem with, but in this case, the list of ids is being received as a list of strings that need to be cast to integers (or, more specifically, BIGINT). The query that I think that I need is as follows, which the troublesome bit being ANY(ARRAY?::BIGINT[]):
ModelA
|> where(
[ma],
fragment(
"EXISTS (SELECT * FROM model_b mb WHERE mb.a_id = ? AND mb.c_id = ANY(ARRAY?::BIGINT[]))",
a.id,
^c_ids
)
)
where c_ids would be a list like ["1449441579", "2345556834"]
However, when I run this, I get the error
(Postgrex.Error) ERROR 42703 (undefined_column) column "array$4" does not exist
referring to the generated SQL
ANY(ARRAY$4::BIGINT[])
Of course, I could convert the array of c_ids to integers beforehand in my app code, but I'd like to see if I can get it to cast in the query itself.
Writing the fragment in straight SQL works out just fine:
SELECT * FROM model_b mb WHERE mb.a_id = 1 AND mb.c_id = ANY(ARRAY['1449441579', '2345556834']::BIGINT[]);
What is the idiomatic way to get this kind of array casting to work in an Ecto fragment? Many thanks.
Just to codify my comment, I would do the integer conversion before the query. You can use the dynamic macro to support IN queries:
import Ecto.Query
alias YourApp.Repo
alias YourApp.SomeSchema, as: ModelA
strs = ["1", "2", "3"]
ids = Enum.map(strs, fn x -> String.to_integer(x) end)
conditions = dynamic([tbl], tbl.id in ^ids)
Repo.all(
from ma in ModelA,
where: ^conditions
)
|> IO.inspect()
I did find a post that led me to a potential solution here - https://elixirforum.com/t/interpolating-lists-into-ecto-query-fragment-1/16690/7 - however, I'm not convinced that this is optimal for me.
By using Enum.join and converting my initial array of strings into a string and then allowing Postgres to cast that string to an array of bigints, it worked out:
ModelA
|> where(
[ma],
fragment(
"EXISTS (SELECT * FROM model_b mb WHERE mb.a_id = ? AND mb.c_id = ANY(ARRAY?::BIGINT[]))",
a.id,
^Enum.join(c_ids, ",")
)
)
So I'm posting it here for reference and because it does "work", but I feel sort of silly doing this, because I specifically was trying to avoid processing the list before passing it to PG... so yeah I'd still really like to hear about the "right" way to do this...

Ecto: How to update all records with a different random number

I have a table in postgresql and i want to update the value of column "points" for all the records with a random number. In other languages we could loop over all the db records but how can i do it with ecto? I tried this:
Repo.all(from u in User, update: User.changeset(u, %{points: :rand.uniform(100)}))
but it outputs the following error:
== Compilation error in file lib/hello_remote/user.ex ==
** (Ecto.Query.CompileError) malformed update `User.changeset(u, %{points: :rand.uniform(100)})` in query expression, expected a keyword list with set/push/pop as keys with field-value pairs as values
expanding macro: Ecto.Query.update/3
lib/hello_remote/user.ex:30: HelloRemote.User.update_points/0
expanding macro: Ecto.Query.from/2
lib/hello_remote/user.ex:30: HelloRemote.User.update_points/0
(elixir 1.10.4) lib/kernel/parallel_compiler.ex:304: anonymous fn/4 in Kernel.ParallelCompiler.spawn_workers/7
I've also tried this:
from(u in User)
|> Repo.update_all(set: [points: Enum.random(0..100)])
but it updates all the records with the same value
You can use fragment/1 with update_all/3, calling a PostgreSQL function to calculate the random values, for example:
update(User, set: [points: fragment("floor(random()*100)")])
|> Repo.update_all([])
I don't think it is possible with the update_all function. The function was created to update many rows with the same value.
You can create a loop and update each record separately, it's not nice but a working solution.
User
|> Repo.all()
|> Enum.each(fn user -> update_user(user) end)
def update_user(user) do
user
|> Ecto.Changeset.cast(%{"points" => Enum.random(0..100)}, [:points])
|> Repo.update!()
end
If you would like to do it one call you can construct a raw SQL query and use that.

Using Postgres UPDATE ... FROM in Ecto without raw SQL

Based on an Elixir thread from last year, I was able to write a raw SQL query to bulk update records with the values from an unrelated table. However, I would like to be able to generate this query using Ecto.
In the example below, assume there are two tables, cats and dogs, and the cats table has a foreign key (dog_id). I want to link a dog to a cat.
The code below is how I'm doing this manually with Elixir and raw SQL:
cat_ids = [1,2,3] # pretend these are uuids
dog_ids = [4,5,6] # ... uuids
values =
cat_ids
|> Enum.zip(dog_ids)
|> Enum.map(fn {cat_id, dog_id} ->
"('#{cat_id}'::uuid, '#{dog_id}'::uuid)"
end)
|> Enum.join(", ")
sql = """
UPDATE cats as a
SET dog_id = c.dog_id
from (values #{values}) as c(cat_id, dog_id)
where c.cat_id = a.id;
"""
Repo.query(sql)
Is there a way to move this to Repo.update_all or some use of fragments so I'm not manually building the query?
Sure, you can use Ecto syntax, but it ain't much different in my opinion, tho you have to use a Schema, for example in my application I have an user authentication and that's how we update the token:
def update_token(user_id, token) do
Repo.transaction(fn ->
from(t in UserAuthentication, where: t.user_id == ^to_string(user_id))
|> Repo.update_all(set: [token: token])
end
and the UserAuthentication schema looks more or less like:
defmodule MyApp.UserAuthentication do
use Ecto.Schema
import Ecto.Changeset
schema "user_authentication" do
field(:user_id, :integer)
field(:token, :string)
timestamps()
end
def changeset(%__MODULE__{} = user, attrs) do
user
|> cast(attrs, [:user_id, :token])
|> validate_required([:user_id, :token])
end
end
This is good for data validation and works in any DB you attach.

Elixir Ecto - PostgreSQL jsonb Functions

I am in the process of converting a Ruby on Rails API over to Elixir and Phoenix. In my Postgres database, I have a table with a jsonb column type. One of the keys in the json is an array of colors. For example:
{"id": 12312312, "colors": ["Red", "Blue", "White"]}
What I am trying to do from Ecto is query my table for all records that contain the colors Red or Blue. Essentially, recreate this query:
select * from mytable where data->'colors' ?| array['Red', 'Blue']
I'm having some difficulties constructing this query with Ecto. Here is what I have:
Note: "value" will be a pipe delimited list of colors
def with_colors(query, value) do
colors = value
|> String.split("|")
|> Enum.map(fn(x) -> "'#{x}'" end)
|> Enum.join(", ")
# colors should look like "'Red', 'Blue'"
from c in query,
where: fragment("data->'colors' \\?| array[?]", ^colors))
end
This is currently not working as expected. I am having issues with the replacement question mark, as it seems to wrap additional quotes around my field. What is the proper way to do this use fragment? Or maybe there is a better way?
I'm going to run into this problem again because I'm also going to have to recreate this query:
select * from mytable where data->'colors' #> '["Red", "Blue"]'
I have found a solution to my problem.
def with_colors(query, value) do
colors = value
|> String.split("|")
from c in query,
where: fragment("data->'colors' \\?| ?", ^colors))
end

Prepare dynamic case statement using PostgreSQL 9.3

I have the following case statement to prepare as a dynamic as shown below:
Example:
I have the case statement:
case cola
when cola between '2001-01-01' and '2001-01-05' then 'G1'
when cola between '2001-01-10' and '2001-01-15' then 'G2'
when cola between '2001-01-20' and '2001-01-25' then 'G3'
when cola between '2001-02-01' and '2001-02-05' then 'G4'
when cola between '2001-02-10' and '2001-02-15' then 'G5'
else ''
end
Note: Now I want to create dynamic case statement because of the values dates and name passing as a parameter and it may change.
Declare
dates varchar = '2001-01-01to2001-01-05,2001-01-10to2001-01-15,
2001-01-20to2001-01-25,2001-02-01to2001-02-05,
2001-02-10to2001-02-15';
names varchar = 'G1,G2,G3,G4,G5';
The values in the variables may change as per the requirements, it will be dynamic. So the case statement should be dynamic without using loop.
You may not need any function for this, just join to a mapping data-set:
with cola_map(low, high, value) as (
values(date '2001-01-01', date '2001-01-05', 'G1'),
('2001-01-10', '2001-01-15', 'G2'),
('2001-01-20', '2001-01-25', 'G3'),
('2001-02-01', '2001-02-05', 'G4'),
('2001-02-10', '2001-02-15', 'G5')
-- you can include as many rows, as you want
)
select table_name.*,
coalesce(cola_map.value, '') -- else branch from case expression
from table_name
left join cola_map on table_name.cola between cola_map.low and cola_map.high
If your date ranges could collide, you can use DISTINCT ON or GROUP BY to avoid row duplication.
Note: you can use a simple sub-select too, I used a CTE, because it's more readable.
Edit: passing these data (as a single parameter) can be achieved by passing a multi-dimensional array (or an array of row-values, but that requires you to have a distinct, predefined composite type).
Passing arrays as parameters can depend on the actual client (& driver) you use, but in general, you can use the array's input representation:
-- sql
with cola_map(low, high, value) as (
select d[1]::date, d[2]::date, d[3]
from unnest(?::text[][]) d
)
select table_name.*,
coalesce(cola_map.value, '') -- else branch from case expression
from table_name
left join cola_map on table_name.cola between cola_map.low and cola_map.high
// client pseudo code
query = db.prepare(sql);
query.bind(1, "{{2001-01-10,2001-01-15,G2},{2001-01-20,2001-01-25,G3}}");
query.execute();
Passing each chunk of data separately is also possible with some clients (or with some abstractions), but this is highly depends on your driver/orm/etc. you use.