I'm trying to find out how to get JSON object results of selected rows, and not to show duplicated rows.
My current query:
SELECT DISTINCT ON (vp.id) jsonb_agg(jsonb_build_object('affiliate',a.*)) as affiliates, jsonb_agg(jsonb_build_object('vendor',vp.*)) as vendors FROM
affiliates a
INNER JOIN related_affiliates ra ON a.id = ra.affiliate_id
INNER JOIN related_vendors rv ON ra.product_id = rv.product_id
INNER JOIN vendor_partners vp ON rv.vendor_partner_id = vp.id
WHERE ra.product_id = 79 AND a.is_active = true
GROUP BY vp.id
The results that I receive from this is:
[
affiliates: {
affiliate: affiliate1
affiliate: affiliate2
},
vendors: {
vendor: vendor1,
vendor: vendor1,
}
As you can see in the second record, vendor is still vendor1 because there are no more results, so I'd like to also know if there's a way to remove duplicates.
Thanks.
First point : the result you display here above doesn't conform the json type : the keys are not double-quoted, the string values are not double-quoted, having dupplicated keys in the same json object ('{"affiliate": "affiliate1", "affiliate": "affiliate2"}' :: json) is not be accepted with the jsonb type (but it is with the json type).
Second point : you can try to add the DISTINCT key word directly in the jsonb_agg function :
jsonb_agg(DISTINCT jsonb_build_object('vendor',vp.*))
and remove the DISTINCT ON (vp.id) clause.
You can also add an ORDER BY clause directly in any aggregate function. For more information, see the manual.
You could aggregate first, then join on the results of the aggregates:
SELECT a.affiliates, v.vendors
FROM (
select af.id, jsonb_agg(jsonb_build_object('affiliate',af.*)) as affiliates
from affiliates af
group by af.id
) a
JOIN related_affiliates ra ON a.id = ra.affiliate_id
JOIN related_vendors rv ON ra.product_id = rv.product_id
JOIN (
select vp.id, jsonb_agg(jsonb_build_object('vendor',vp.*)) as vendors
from vendor_partners vp
group by vp.id
) v ON rv.vendor_partner_id = v.id
WHERE ra.product_id = 79
AND a.is_active = true
Here are three tables: product, model, and product_model that maps products and models in N:M relationship.
product product_model model
id name product_id model_id id name
------------ ------------------- ----------
p1 Product 1 p1 m1 m1 Model 1
p2 Product 2 p2 m1 m2 Model 2
... p2 m2
What I want to do: Find all products that support Model 2(eg. product 2). Then, for each product, show the list of model_ids that the product supports(product 2 => [ m1,m2 ])
This was my first try. I needed N more queries to search model_ids for each product.
# 1 query for searching products
my #products = $schema->resultset('Product')->search(
{ 'product_models.model_id' => 'm2' },
{ 'join' => 'product_model' },
)
# N queries for searching product_models for each product
foreach my $product ( #products ) {
my #model_ids = map { $_->model_id } $product->product_models;
# #model_ids = ( 'm1', 'm2' ) for p2
}
I looked for a way to get the result using only one query. Replacing join with prefetch didn't work.
my #products = $schema->resultset('Product')->search(
{ 'product_models.model_id' => 'm2' },
{ 'prefetch' => 'product_model' }, # here
)
# no additional queries, but...
foreach my $product ( #products ) {
my #model_ids = map { $_->model_id } $product->product_models;
# now, #model_ids contains only ( `m2` )
}
Next, I tried "prefetch same table twice":
my #products = $schema->resultset('Product')->search(
{ 'product_models.model_id' => 'm2' },
{ 'prefetch' => [ 'product_models', 'product_models' ] },
);
foreach my $product ( #products ) {
my #model_ids = map { $_->model_id } $product->product_models;
}
It seemed that I succeeded. Only one query was executed and I got all model IDs from it.
However I wasn't so sure that this is the right(?) way. Is this a correct approach?
For example, if I used join instead of prefetching, Product 2 appears in the loop twice. I understand that, because the joined table is like:
id name p_m.p_id p_m.m_id p_m_2.p_id p_m_2.m_id
p2 Product 2 p2 m2 p2 m1
p2 Product 2 p2 m2 p2 m2 -- Product 2, one more time
Why does Product 2 appear only once when I use prefetch?
The resulting queries are almost same, except the difference of SELECT fields:
SELECT "me"."id", "me"."name",
"product_models"."product_id", "product_models"."model_id", -- only in prefetch
"product_models_2"."product_id", "product_models_2"."model_id" --
FROM "product" "me"
LEFT JOIN "product_model" "product_models"
ON "product_models"."product_id" = "me"."id"
LEFT JOIN "product_model" "product_models_2"
ON "product_models_2"."product_id" = "me"."id"
WHERE "product_models"."model_id" = 'm2'
If you have the correct relationships in your schema, this is possible with a single query. But it's tricky. Let's assume your database looks like this:
CREATE TABLE product
(`id` VARCHAR(2) PRIMARY KEY, `name` VARCHAR(9))
;
INSERT INTO product
(`id`, `name`) VALUES
('p1', 'Product 1'),
('p2', 'Product 2')
;
CREATE TABLE product_model (
`product_id` VARCHAR(2),
`model_id` VARCHAR(2),
PRIMARY KEY (product_id, model_id),
FOREIGN KEY(product_id) REFERENCES product(id),
FOREIGN KEY(model_id) REFERENCES model(id)
)
;
INSERT INTO product_model
(`product_id`, `model_id`) VALUES
('p1', 'm1'),
('p2', 'm1'),
('p2', 'm2')
;
CREATE TABLE model
(`id` VARCHAR(2) PRIMARY KEY, `name` VARCHAR(7))
;
INSERT INTO model
(`id`, `name`) VALUES
('m1', 'Model 1'),
('m2', 'Model 2')
;
This is essentially your DB from the question. I added primary keys and foreign keys. You probably have those anyway.
We can now create a schema from that. I've written a simple program that uses DBIx::Class::Schema::Loader to do that. It creates an SQLite database on the fly. (If no-one has put this on CPAN, I will).
The SQL from above will go in the __DATA__ section.
use strict;
use warnings;
use DBIx::Class::Schema::Loader qw/ make_schema_at /;
# create db
unlink 'foo.db';
open my $fh, '|-', 'sqlite3 foo.db' or die $!;
print $fh do { local $/; <DATA> };
close $fh;
$ENV{SCHEMA_LOADER_BACKCOMPAT} = 1;
# create schema
my $dsn = 'dbi:SQLite:foo.db';
make_schema_at(
'DB',
{
# debug => 1,
},
[ $dsn, 'sqlite', '', ],
);
$ENV{DBIC_TRACE} = 1;
# connect schema
my $schema = DB->connect($dsn);
# query goes here
__DATA__
# SQL from above
Now that we have that, we can concentrate on the query. At first this will look scary, but I'll try to explain.
my $rs = $schema->resultset('Product')->search(
{ 'product_models.model_id' => 'm2' },
{
'prefetch' => {
product_models => {
product_id => {
product_models => 'model_id'
}
}
}
},
);
while ( my $product = $rs->next ) {
foreach my $product_model ( $product->product_models->all ) {
my #models;
foreach my $supported_model ( $product_model->product_id->product_models->all ) {
push #models, $supported_model->model_id->id;
}
printf "%s: %s\n", $product->id, join ', ', #models;
}
}
The prefetch means join on this relation, and keep the data around for later. So to get all models for your product, we have to write
# 1 2
{ prefetch => { product_models => 'product_id' } }
Where product_models is the N:M table, and product_id is the name of the relation to the Models table. The arrow => 1 is for the first join from Product to ProductModel. The 2 is for ProductModel back to every product that has the model m2. See the drawing of the ER model for an illustration.
Now we want to have all the ProductModels that this Product has. That's arrow 3.
# 1 2 3
{ prefetch => { product_models => { product_id => 'product_models' } } }
And finally, to get the Models for that N:M relation, we have to use the model_id relationshop with arrow 4.
{
'prefetch' => { # 1
product_models => { # 2
product_id => { # 3
product_models => 'model_id' # 4
}
}
}
},
Looking at the ER model drawing should make that clear. Remember that each of those joins is a LEFT OUTER join by default, so it will always fetch all the rows, without loosing anything. DBIC just takes care of that for you.
Now to access all of that, we need to iterate. DBIC gives us some tools to do that.
while ( my $product = $rs->next ) {
# 1
foreach my $product_model ( $product->product_models->all ) {
my #models;
# 2 3
foreach my $supported_model ( $product_model->product_id->product_models->all ) {
# 4
push #models, $supported_model->model_id->id;
}
printf "%s: %s\n", $product->id, join ', ', #models;
}
}
First we grab all the ProductModel entries (1). For each of those, we take the Product (2). There is always only one Product in every line, because that way we have a 1:N relation, so we can directly access it. This Product in turn has a ProductModel relation. That's 3. Because this is the N side, we need to take all of them and iterate. We then push the id of every Model (4) into our list of models for this product. After that, it's just printing.
Here's another way to look at it:
We could eliminate that last model_id in the prefetch, but then we'd have to use get_column('model_id') to get the ID. It would save us a join.
Now if we turn on DBIC_TRACE=1, we get this SQL statement:
SELECT me.id, me.name, product_models.product_id, product_models.model_id, product_id.id, product_id.name, product_models_2.product_id, product_models_2.model_id, model_id.id, model_id.name
FROM product me
LEFT JOIN product_model product_models ON product_models.product_id = me.id
LEFT JOIN product product_id ON product_id.id = product_models.product_id
LEFT JOIN product_model product_models_2 ON product_models_2.product_id = product_id.id
LEFT JOIN model model_id ON model_id.id = product_models_2.model_id
WHERE (product_models.model_id = 'm2')
ORDER BY me.id
If we run this against our DB, we have these rows:
p2|Product 2|p2|m2|p2|Product 2|p2|m1|m1|Model 1
p2|Product 2|p2|m2|p2|Product 2|p2|m2|m2|Model 2
Of course that's pretty useless if we do it manually, but DBIC's magic really helps us, because all the weird joining and combining is completely abstracted away, and we only need one single query to get all the data.
For example take these two tables:
Company
CompanyID | Name | Address | ...
Employee
EmployeeID | Name | Function | CompanyID | ...
where a Company has several Employees.
When we want to retrieve the Company and Employee data for a certain Employee, this simple SQL statement will do the job:
SELECT e.name as employeename, c.name as companyname
FROM Company c
INNER JOIN Employee e
ON c.CompanyID = e.CompanyID
where e.EmployeeID=3
Now, the question is how to translate this SQL statement into a 'lambda' construct. We have modelled the tables as objects in the MS Entity Framework where we also defined the relationship between the tables (.edmx file).
Also important to mention is that we use the 'Repository' pattern.
The closest I can get is something like this:
List<Company> tmp = _companyRepository.GetAll().Where
(
c.Employee.Any
(
e => e.FKEngineerID == engineerId && e.DbId == jobId
)
).ToList();
Any help is very much appreciated!
This should do, assuming your repositories are returning IQueryables of the types
var list = (from c in _companyRepository.GetAll()
join e in _employeeRepository.GetAll() on c.CompanyId equals e.CompanyId
where e.FKEngineerID == engineerId && e.DbId == jobId
select new
{
EmployeeName = e.name,
CompanyName = c.name
}).ToList();
Since you are constraining the query to a single employee (e.Employee=3) why don't you start with employees.
Also, your sql query return a custom set of columns, one column from the employee table and the other column from company table. To reflect that, you need a custom projection at the ef side.
var result = _employeeRepository.GetAll()
.Where( e => e.DbId == 3 ) // this corresponds to your e.EmployeeID = 3
.Select( e => new {
employeename = e.EmployeeName,
companyname = e.Company.CompanyName
} ) // this does the custom projection
.FirstOrDefault(); // since you want a single entry
if ( result != null ) {
// result is a value of anonymous type with
// two properties, employeename and companyname
}
Update
Created a runnable demo for this problem.
https://github.com/narayanjr/anorm_test
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
I am unable to access fields on an aliased table. I keep getting error messages saying the field is not an option and the available fields are the base field name, or the `table_name'.field_name. But not the aliased field name. This makes it impossible to JOIN the same table twice and access all the fields.
var vendor_client_parser_1 = SqlParser.long("vid") ~ SqlParser.str("vname") ~ SqlParser.long("cid") ~ SqlParser.str("cname") map
{
case vid ~ vn ~ cid ~ cn => println(vid + "," + vn + "," + cid + "," + cn + ",")
}
var vendor_client_parser_2 = SqlParser.long("v.business_id") ~ SqlParser.str("v.name") ~ SqlParser.long("c.business_id") ~ SqlParser.str("c.name") map
{
case vid ~ vn ~ cid ~ cn => println(vid + "," + vn + "," + cid + "," + cn + ",")
}
var vendor_client_parser_3 = SqlParser.long(1) ~ SqlParser.str(2) ~ SqlParser.long(3) ~ SqlParser.str(4) map
{
case vid ~ vn ~ cid ~ cn => println(vid + "," + vn + "," + cid + "," + cn + ",")
}
DB.withConnection
{
implicit c =>
var results =
SQL"""
SELECT v.business_id AS vid, v.name AS vname, c.business_id AS cid, c.name AS cname
FROM #$BUSINESS_CONNECTION_TABLE
JOIN #$BUSINESS_TABLE AS v ON (vendor_id = v.business_id)
JOIN #$BUSINESS_TABLE AS c ON (client_id = c.business_id)
LIMIT 20
""".as(vendor_client_parser.*)
}
Expected result:
1, Vendor A, 10, Vendor K
2, Vendor B, 11, Vendor L
2, Vendor B, 1, Vendor A
12, Vendor M, 3, Vendor C
Result from vendor_client_parser_1:
10, Vendor K, 10, Vendor K
11, Vendor L, 11, Vendor L
1, Vendor A, 1, Vendor A
3, Vendor C, 3, Vendor C
Result from vendor_client_parser_2:
Execution exception[[AnormException: 'v.business_id' not found, available columns: business.business_id, business_id, business.name, name, business.business_id, business_id, business.name, name]]
Result from vendor_client_parser_3: (Same as expected)
1, Vendor A, 10, Vendor K
2, Vendor B, 11, Vendor L
2, Vendor B, 1, Vendor A
12, Vendor M, 3, Vendor C
vendor_client_parser_3 works but it relies on using index instead of names. I dis like using indexes because If I mess up an index I might still get a valid response back and not notice. If I mess up a name the column wont exist and I will know something is wrong.
Is there something I am missing? Is there any way to achieve the results I need without having to rely on using the index?
Play Scala 2.4.1
Anorm 2.5.0
Update:
If I do not alias the columns and use vendor_client_parser_2 I get the same result as when the columns are alias.
Modified Query:
SQL"""
SELECT v.business_id, v.name, c.business_id, c.name
FROM #$BUSINESS_CONNECTION_TABLE
JOIN #$BUSINESS_TABLE AS v ON (vendor_id = v.business_id)
JOIN #$BUSINESS_TABLE AS c ON (client_id = c.business_id)
LIMIT 20
""".as(vendor_client_parser_2.*)
Result with vendor_client_parser_2.*:
Execution exception[[AnormException: 'v.business_id' not found, available columns: business.business_id, business_id, business.name, name, business.business_id, business_id, business.name, name]]
I also tested it with a single table aliased and it refuses to see the aliased table name
Single Table Test:
SQL"""
SELECT v.business_id, v.name, business_id, name
FROM #$BUSINESS_TABLE AS v
LIMIT 20
""".as(test_parser.*)
test_parser:
var test_parser = SqlParser.long("v.business_id") ~ SqlParser.str("v.name") ~ SqlParser.long("business_id") ~ SqlParser.str("name") map
{
case vid ~ vn ~ cid ~ cn => println(vid + "," + vn + "," + cid + "," + cn + ",")
}
Result:
[AnormException: 'v.business_id' not found, available columns: business.business_id, business_id, business.name, name, business.business_id, business_id, business.name, name]
I then tested if aliased columns are accessible by both their original and aliased names.
Test Aliased columns:
SQL"""
SELECT business_id AS vid, name AS vname
FROM #$BUSINESS_TABLE
LIMIT 20
""".as(test_parser_2.*)
test_parser_2:
var test_parser_2 = SqlParser.long("business_id") ~ SqlParser.str("name") ~ SqlParser.long("vid") ~ SqlParser.str("vname") map
{
case vid ~ vn ~ cid ~ cn => println(vid + "," + vn + "," + cid + "," + cn + ",")
}
This test did not error out and it properly pulled in values as business_id and vid. As well as name and vname.
I forced it to error out so it would give me a list of column names. And it does seem like Anorm doesn't offer the non-aliased names as suggestions but they do work in this case.
[AnormException: 'forceError' not found, available columns: business.business_id, vid, business.name, vname]
I also tried not using SqlParser.
var businesses = SQL"""
SELECT v.business_id AS vid, v.name AS vname, c.business_id AS cid, c.name AS cname
FROM #$BUSINESS_CONNECTION_TABLE
JOIN #$BUSINESS_TABLE AS v ON (vendor_id = v.business_id)
JOIN #$BUSINESS_TABLE AS c ON (client_id = c.business_id)
LIMIT 20
""".fold(List[(Long, String, Long, String)]())
{
(list, row) =>
list :+ (row[Long]("v.business_id"), row[String]("v.name"), row[Long]("c.business_id"), row[String]("c.name")) //attempt_1
//list :+ (row[Long]("vid"), row[String]("vname"), row[Long]("cid"), row[String]("cname")) //attempt_2
}
If I use attempt_1 I get this error which as you suggested shouldn't work.
Left('v.business_id' not found, available columns: business.business_id, vid, business.name, vname, business.business_id, cid, business.name, cname)))
If I use attempt_2, I get the same results as vendor_client_parser_1
10, Vendor K, 10, Vendor K
11, Vendor L, 11, Vendor L
1, Vendor A, 1, Vendor A
3, Vendor C, 3, Vendor C
If I do not alias the columns and use this same method
SQL"""
SELECT v.business_id, v.name, c.business_id, c.name
FROM #$BUSINESS_CONNECTION_TABLE
JOIN #$BUSINESS_TABLE AS v ON (vendor_id = v.business_id)
JOIN #$BUSINESS_TABLE AS c ON (client_id = c.business_id)
LIMIT 20
""".fold(List[(Long, String, Long, String)]())
{
(list, row) =>
list :+ (row[Long]("v.business_id"), row[String]("v.name"), row[Long]("c.business_id"), row[String]("c.name")) //Attempt_3
}
Using this query without aliasing the columns causes this error,
Left('v.business_id' not found, available columns: business.business_id, business_id, business.name, name, business.business_id, business_id, business.name, name)))
I then tested a simple aliased tabled using this method
SQL"""
SELECT v.business_id, v.name
FROM #$BUSINESS_TABLE AS v
LIMIT 20
""".fold(List[(Long, String)]())
{
(list, row) =>
list :+ (row[Long]("v.business_id"), row[String]("v.name")) //simple_attempt_1
}
I get the same error
Left(List(java.lang.RuntimeException: Left('v.business_id' not found, available columns: business.business_id, business_id, business.name, name)))
So as far as I can tell it isn't possible to access fields that are part of an aliased table if the same table is used twice using the field names instead of index.
Update 2:
I tried reversing the order of fields in the SQL so it was c.business_id AS cid, c.name AS cname, v.business_id AS vid, v.name AS vname and rerunning vendor_client_parser_1. It gave me the inverse results
Result from vendor_client_parser_1 with mysql fields switched:
1, Vendor A, 1, Vendor A
2, Vendor B, 2, Vendor B
2, Vendor B, 2, Vendor B
12, Vendor M, 12, Vendor M
When I force an error and it shows me possible fields I get these,
Fields in original order:
Left('forceError' not found, available columns: business.business_id, vid, business.name, vname, business.business_id, cid, business.name, cname)
Fields in switched order:
Left('forceError' not found, available columns: business.business_id, cid, business.name, cname, business.business_id, vid, business.name, vname)
This makes me think this scenario is happening.
In case several columns are found with same name in query result, for example columns named code in both Country and CountryLanguage tables, there can be ambiguity. By default a mapping like following one will use the last column:
https://www.playframework.com/documentation/2.4.1/ScalaAnorm
If you look at the suggested fields business.business_id and business.name occur twice because the table is referenced twice. It seems like Anorm is associating the last occurrence of business.business_id and business.name with both aliases.
Update
Created a runnable demo for this problem.
https://github.com/narayanjr/anorm_test
I have not tried it in 2.4, but you could possibly use a parser method to generate the column name for you, rather than a variable. My approach would be something like the following:
case class BusinessConnection(vendor_id: Int, client_id: Int)
case class Business(id: Int, name: String)
case class VendorClient(vendor: Business, client: Business)
object VendorClient {
val businessConnectionP =
get[Int]("vendor_id") ~
get[Int]("client_id") map {
case vendor_id ~ client_id =>
BusinessConnection(vendor_id, client_id)
}
def businessP(alias: String) =
getAliased[Int](alias + ".id") ~
getAliased[String](alias + ".name") map {
case id ~ name =>
Business(id, name)
}
val vendorClientP =
businessP("v") ~ businessP("c") map {
case business ~ client =>
VendorClient(business, client)
}
val sqlSelector = "v.id, v.name, c.id, c.name"
def all() = DB.withConnection { implicit c =>
SQL(s"""
select $sqlSelector
from businessConnection
join business as v on vendor_id=v.id
join business as c on client_id=c.id
""").as(vendorClientP *)
}
}