Using Postgres UPDATE ... FROM in Ecto without raw SQL - postgresql

Based on an Elixir thread from last year, I was able to write a raw SQL query to bulk update records with the values from an unrelated table. However, I would like to be able to generate this query using Ecto.
In the example below, assume there are two tables, cats and dogs, and the cats table has a foreign key (dog_id). I want to link a dog to a cat.
The code below is how I'm doing this manually with Elixir and raw SQL:
cat_ids = [1,2,3] # pretend these are uuids
dog_ids = [4,5,6] # ... uuids
values =
cat_ids
|> Enum.zip(dog_ids)
|> Enum.map(fn {cat_id, dog_id} ->
"('#{cat_id}'::uuid, '#{dog_id}'::uuid)"
end)
|> Enum.join(", ")
sql = """
UPDATE cats as a
SET dog_id = c.dog_id
from (values #{values}) as c(cat_id, dog_id)
where c.cat_id = a.id;
"""
Repo.query(sql)
Is there a way to move this to Repo.update_all or some use of fragments so I'm not manually building the query?

Sure, you can use Ecto syntax, but it ain't much different in my opinion, tho you have to use a Schema, for example in my application I have an user authentication and that's how we update the token:
def update_token(user_id, token) do
Repo.transaction(fn ->
from(t in UserAuthentication, where: t.user_id == ^to_string(user_id))
|> Repo.update_all(set: [token: token])
end
and the UserAuthentication schema looks more or less like:
defmodule MyApp.UserAuthentication do
use Ecto.Schema
import Ecto.Changeset
schema "user_authentication" do
field(:user_id, :integer)
field(:token, :string)
timestamps()
end
def changeset(%__MODULE__{} = user, attrs) do
user
|> cast(attrs, [:user_id, :token])
|> validate_required([:user_id, :token])
end
end
This is good for data validation and works in any DB you attach.

Related

Expressing Postgresql VALUES command in SQLAlchemy ORM?

How to express the query
VALUES ('alice'), ('bob') EXCEPT ALL SELECT name FROM users;
(i.e. "list all names in VALUES that are not in table 'users'") in SQLAlchemy ORM? In other words, what should the statement 'X' below be like?
def check_for_existence_of_all_users_in_list(list):
logger.debug(f"checking that each user in {list} is in the database")
query = X(list)
(There is sqlalchemy.values which could be used like this:
query = sa.values(sa.column('name', sa.String)).data(['alice', 'bob']) # .???
but it appears that it can only be used as argument to INSERT or UPDATE.)
I am using SQLAlchemy 1.4.4.
This should work for you:
user_names = ['alice', 'bob']
q = values(column('name', String), name="temp_names").data([(_,) for _ in user_names])
query = select(q).except_all(select(users.c.name)) # 'users' is Table instance

Ecto: How to update all records with a different random number

I have a table in postgresql and i want to update the value of column "points" for all the records with a random number. In other languages we could loop over all the db records but how can i do it with ecto? I tried this:
Repo.all(from u in User, update: User.changeset(u, %{points: :rand.uniform(100)}))
but it outputs the following error:
== Compilation error in file lib/hello_remote/user.ex ==
** (Ecto.Query.CompileError) malformed update `User.changeset(u, %{points: :rand.uniform(100)})` in query expression, expected a keyword list with set/push/pop as keys with field-value pairs as values
expanding macro: Ecto.Query.update/3
lib/hello_remote/user.ex:30: HelloRemote.User.update_points/0
expanding macro: Ecto.Query.from/2
lib/hello_remote/user.ex:30: HelloRemote.User.update_points/0
(elixir 1.10.4) lib/kernel/parallel_compiler.ex:304: anonymous fn/4 in Kernel.ParallelCompiler.spawn_workers/7
I've also tried this:
from(u in User)
|> Repo.update_all(set: [points: Enum.random(0..100)])
but it updates all the records with the same value
You can use fragment/1 with update_all/3, calling a PostgreSQL function to calculate the random values, for example:
update(User, set: [points: fragment("floor(random()*100)")])
|> Repo.update_all([])
I don't think it is possible with the update_all function. The function was created to update many rows with the same value.
You can create a loop and update each record separately, it's not nice but a working solution.
User
|> Repo.all()
|> Enum.each(fn user -> update_user(user) end)
def update_user(user) do
user
|> Ecto.Changeset.cast(%{"points" => Enum.random(0..100)}, [:points])
|> Repo.update!()
end
If you would like to do it one call you can construct a raw SQL query and use that.

Slick:Insert into a Table from Raw SQL Select

Insert into a Table from Raw SQL Select
val rawSql: DBIO[Vector[(String, String)]] = sql"SELECT id, name FROM SomeTable".as[(String, String)]
val myTable :TableQuery[MyClass] // with columns id (String), name(String) and some other columns
Is there a way to use forceInsert functions to insert data from select into the tables?
If not, Is there a way to generate a sql string by using forceInsertStatements?
Something like:
db.run {
myTable.map{ t => (t.id, t.name)}.forceInsert????(rawSql)
}
P.S. I don't want to make two I/O calls because my RAW SQL might be returning thousands of records.
Thanks for the help.
If you can represent your rawSql query as a Slick query instead...
val query = someTable.map(row => (row.id, row.name))
...for example, then forceInsertQuery will do what you need. An example might be:
val action =
myTable.map(row => (row.someId, row.someName))
.forceInsertQuery(
someTable.map(query)
)
However, I presume you're using raw SQL for a good reason. In that case, I don't believe you can use forceInsert (without a round-trip to the database) because the raw SQL is already an action (not a query).
But, as you're using raw SQL, why not do the whole thing in raw SQL? Something like:
val rawEverything =
sqlu" insert into mytable (someId, someName) select id, name from sometable "
...or similar.

In Elixir with Postgres, how can I have the database return the enum values which are NOT in use?

I have an EctoEnum.Postgres:
# #see: https://en.wikipedia.org/wiki/ISO_4217
defmodule PricingEngine.Pricing.CurrencyEnum do
#options [
:AED,
:AFN,
# snip...
:ZWL
]
use EctoEnum.Postgres,
type: :currency,
enums: #options
def values, do: #options
end
This enum has been included in our Postgres database
We also have a structure:
defmodule PricingEngine.Pricing.Currency do
use Ecto.Schema
import Ecto.Changeset
schema "currencies" do
field(:currency, PricingEngine.Pricing.CurrencyEnum)
timestamps()
end
#doc false
def changeset(currency, attrs) do
currency
|> cast(attrs, [:currency])
|> validate_required([:currency])
|> unique_constraint(:currency)
end
end
We can currently successfully use the following functions to figure out which currencies are active/used:
def active_currency_isos do
Repo.all(select(Currency, [record], record.currency))
end
defdelegate all_currency_isos,
to: CurrencyEnum,
as: :values
def inactive_currency_iso do
Pricing.all_currency_isos() -- Pricing.active_currency_isos()
end
This works, but I'm led to believe this could be more efficient if we just asked the database for this information.
Any idea(s) how to do this?
If you want to get a list of all the used enums you should just do a distinct on the currency field. This uses the Postgres DISTINCT ON operator:
from(c in Currency,
distinct: c.currency,
select: c.currency
)
This will query the table, unique by the currency column, and return only the currency column values. You should get an array of all of the enums that exist in the table.
There are some efficiency concerns with doing it this way which could be mitigated by materialized views, lookup tables, in-memory cache etc. However, if your data set isn't extremely large, you should be able to use this for a while.
Edit:
Per the response, I will show how to get the unused enums.
There are 2 ways to do this.
Pure SQL
This query will get all of the used ones and do a difference from the entire set of available enums. The operator we use to do this is EXCEPT and you can get a list of all available enums with enum_range. I will use unnest to turn the array of enumerated types into individual rows:
SELECT unnest(enum_range(NULL::currency)) AS unused_enums
EXCEPT (
SELECT DISTINCT ON (c.name) c.name
FROM currencies c
)
You can execute this raw SQL in Ecto by doing this:
Ecto.Adapters.SQL.query!(MyApp.Repo, "SELECT unnest(...", [])
From this you'll get a Postgresx.Result that you'll have to get the values out of:
result
|> Map.get(:rows, [])
|> List.flatten()
|> Enum.map(&String.to_existing_atom/1)
I'm not really sure of a way to code this query up in pure Ecto, but let me know if you figure it out.
In Code
You can do the first query that I posted before with distinct then do a difference in the code.
query = from(c in Currency,
distinct: c.currency,
select: c.currency
)
CurrencyEnum.__enums__() -- Repo.all(query)
Either way is probably negligible in terms of performance so it's up to you.

Elixir Ecto - PostgreSQL jsonb Functions

I am in the process of converting a Ruby on Rails API over to Elixir and Phoenix. In my Postgres database, I have a table with a jsonb column type. One of the keys in the json is an array of colors. For example:
{"id": 12312312, "colors": ["Red", "Blue", "White"]}
What I am trying to do from Ecto is query my table for all records that contain the colors Red or Blue. Essentially, recreate this query:
select * from mytable where data->'colors' ?| array['Red', 'Blue']
I'm having some difficulties constructing this query with Ecto. Here is what I have:
Note: "value" will be a pipe delimited list of colors
def with_colors(query, value) do
colors = value
|> String.split("|")
|> Enum.map(fn(x) -> "'#{x}'" end)
|> Enum.join(", ")
# colors should look like "'Red', 'Blue'"
from c in query,
where: fragment("data->'colors' \\?| array[?]", ^colors))
end
This is currently not working as expected. I am having issues with the replacement question mark, as it seems to wrap additional quotes around my field. What is the proper way to do this use fragment? Or maybe there is a better way?
I'm going to run into this problem again because I'm also going to have to recreate this query:
select * from mytable where data->'colors' #> '["Red", "Blue"]'
I have found a solution to my problem.
def with_colors(query, value) do
colors = value
|> String.split("|")
from c in query,
where: fragment("data->'colors' \\?| ?", ^colors))
end