I'm trying to do the following query in linq, however I'm getting an exception error, though my query looks fine to me. So here is the story:
Diagram
I have a many to many relationship between the users and the organizations. A user can be a part of many organizations, and an organization can have many users.
What Im trying to query
So given a user id, i want to query all the team members (users) i have in all the organizations i belong to. So
Input: User X id (guid), and this user belongs to Organization A, and Organization B
Output:
User A, Organization A
User B, Organization A
User C, Organization B
The Actual Query
I though this would do just that
var user = db.Users.Include(q => q.UserOrganization).SingleOrDefault( q => q.Id == id.ToString());
var members = (from us in db.Users.Include(q => q.UserOrganization)
let orgs = user.UserOrganization.Select(z => z.OrganizationId)
where us.UserOrganization.Any(q => orgs.Contains(q.OrganizationId) )
select new UserResource{
id = Guid.Parse(us.Id),
email = us.Email
}
).ToArray();
My query fails on the where clause, with the error:
Processing of the LINQ expression 'AsQueryable<long>((Unhandled parameter: __Select_0))' by 'NavigationExpandingExpressionVisitor' failed. This may indicate either a bug or a limitation in EF Core
Not sure what to change in the query. Please help.
PS: I wrote the query initially in MySql as follows:
SELECT UU.`Id`, UU.`Email`, UUO.`OrganizationId`
FROM aspnetusers AS UU
LEFT JOIN userorganization AS UUO ON UUO.`UserId` = `UU`.Id
WHERE UUO.`OrganizationId` IN
(
SELECT UO.`OrganizationId` FROM aspnetusers AS U
LEFT JOIN userorganization AS UO ON UO.UserId = U.Id
WHERE u.Id = '6caa67e7-69f3-49a3-ad61-10b07d379f10'
)
AND UU.Id != '6caa67e7-69f3-49a3-ad61-10b07d379f10'
The "SingleOrDefault" always executes the Query. User is not an IQueryable.
So the let orgs = user.UserOrganization.Select(z => z.OrganizationId) cannot be translated to SQL, do your var orgs = user.UserOrganization.Select(z => z.OrganizationId) before the Query, in Plain C#. This cannot be used in SQL-Queries.
With orgs being an IList<int> it will work.
But it should be prefered to find a solution that can be solved with one query only. Here you have two.
The SingleOrDefault might be not useful, you go better without, than you have a simple IQueryable. And The "Any" can most often be realized with a simple (Inner) Join, returning only values, if you have a match between to tables. That is the Same as Where - Any - Contains
Related
context
I have Users collection and Recipe collection, ManyToMany relation between them
I'm new in this framework, wondering how can I do the following query:
count users with at least one recipe
count users without any recipes
I have found loadRelationCountAndMap is very useful in counting how many recipes a user has, but I can't seem to filter the total response according to this property.
I have tried this:
const users_without_recipes = await getRepository(User)
.createQueryBuilder('user')
.addSelect(['user.createdAt', 'user.email'])
.loadRelationCountAndMap('user.recipes_count', 'user.recipes')
.where('user.recipes_count = :count', {count: 0})
.getManyAndCount();
also tried to use postgres array_count but not sure how to integrate it with the typeORM framework
and help is very appreciated
You can do this with subqueries I think.
Something like this in SQL:
SELECT *
// ... other stuff
WHERE user.id IN (
SELECT u.id
FROM user u JOIN recipie r USING(id)
GROUP BY u.id
HAVING COUNT(*) > 10
)
Or in TypeORM:
// ...
.where(qb => {
const subQuery = qb.subQuery()
.select("u.id")
.from(User, "u")
// join users and recipies
.groupBy("u.id")
.having("COUNT(*) > :count", { count: 10 })
.getQuery();
return "user.id IN " + subQuery;
});
//...
I have stumbled upon this exact problem myself. I have fond a couple of sources that could help you find the solution. Your underlying problem is more general. If you turn on logging, you will probably see an error message something like:
Error column "user"."recipes_count" does not exist
The problem is that you are trying to use an alias. This question deals with this problem:
Using an Alias in a WHERE clause
I hope you are more successful then me, but I decided to use a workaround after a log trial and error. I am sure there is a better way. If you manage to find it, please let me know.
(If you want to get back the list of user entities, without the user.recipes_count property, include the last .map as well.)
const users_without_recipes = await getRepository(User)
.createQueryBuilder('user')
.loadRelationCountAndMap('user.recipes_count', 'user.recipes')
.getMany();
return users_without_recipes
.filter((user) => user['user.recipes_count'] === 0)
.map(({recipes_count, ...otherProperties}) => otherProperties);
Entity relation: Transaction(#ManyToOne - eager by default) -> Account
String sql = "SELECT new com.test.Pojo(t.account, SUM(t.value)) FROM Transaction t GROUP BY t.account";
List list = entityManager.createQuery(sql).getResultList();
By default JPA using Hibernate implementation will generate 1 + n queries. The n queries are for lazy loading of the account entities.
How can I make this query eager and load everything with a single query? The sql equivalent would be something like
SELECT account.*, SUM(t.value) FROM transactions JOIN accounts on transactions.account_id = accounts.id GROUP BY account.id
, a syntax that works well on PostgreSQL. From my findings Hibernate is generating a query that justifies the lazy loading.
SELECT account.id, SUM(t.value) FROM transactions JOIN accounts on transactions.account_id = accounts.id GROUP BY account.id
Try marking the #ManyToOne field as lazy:
#ManyToOne(fetch = FetchType.LAZY)
private Account account;
And change your query using a JOIN FETCH of the account field to generate only one query with all you need, like this:
String sql = "SELECT new com.test.Pojo(acc, SUM(t.value)) "
+ "FROM Transaction t JOIN FETCH t.account acc GROUP BY acc";
UPDATE:
Sorry, you're right, the fetch attribute of #ManyToOne is not required because in Hibernate that is the default value. The JOIN FETCH isn't working, it's causing a QueryException: "Query specified join fetching, but the owner of the fetched association was not present".
I have tried with some other approaches, the most simple one that avoids doing n + 1 queries is to remove the creation of the Pojo object from your query and process the result list, manually creating the objects:
String hql = "SELECT acc, SUM(t.value)"
+ " FROM " + Transaction.class.getName() + " t"
+ " JOIN t.account acc"
+ " GROUP BY acc";
Query query = getEntityManager().createQuery(hql);
List<Pojo> pojoList = new ArrayList<>();
List<Object[]> list = query.getResultList();
for (Object[] result : list)
pojoList.add(new Pojo((Account)result[0], (BigDecimal)result[1]));
Well PostgreSQL (And any other SQL database too) will block you from using mentioned query: you have to group by all columns of account table, not by id. That is why Hibernate generates the query, grouping by ID of the account - That is what is intended to be, and then fetching the other parts. Because it cannot predict in general way, what else will be needed to be joined and grouped(!!!), and in general this could produce situation, when multiple entities with the same ID are fetched (just create a proper query and take a look at execution plan, this will be especially significant when you have OneToMany fields in your Account entity, or any other ManyToOne part of the Account entity) that is why Hibernate behaves this way.
Also, having accounts with mentioned IDs in First level cache, will force Hibernate to pick them up from that. Or IF they are rarely modified entities, you can put them in Second level cache, and hibernate will not make query to database, but rather pick them from Second level cache.
If you need to get those from database in single hint, but not use all the goodness of Hibernate, just go to pure JPA Approach based on Native queries, like this:
#NamedNativeQuery(
name = "Pojo.groupedInfo",
query = "SELECT account.*, SUM(t.value) as sum FROM transactions JOIN accounts on transactions.account_id = accounts.id GROUP BY account.id, account.etc ...",
resultClass = Pojo.class,
resultSetMapping = "Pojo.groupedInfo")
#SqlResultSetMapping(
name = "Pojo.groupedInfo",
classes = {
#ConstructorResult(
targetClass = Pojo.class,
columns = {
#ColumnResult(name = "sum", type = BigDecimal.class),
/*
* Mappings for Account part of entity.
*/
}
)
}
)
public class Pojo implements Serializable {
private BigDecimal sum;
/* .... */
public Pojo(BigDecimal sum, ...) {}
/* .... */
}
For sure this will work for you well, unless you will use the Account, fetched by this query in other entities. This will make Hibernate "mad" - the "entity", but not fetched by Hibernate...
Interesting, the described behaviour is as if t instances are returned from the actual query and t.account association in the first argument of Pojo constructor is actually navigated on t instances when marshalling results of the query (when creating Pojo instances from the result rows of the query). I am not sure if this is a bug or intended feature for constructor expressions.
But the following form of the query should work (no t.account navigation in the constructor expression, and no join fetch without the owner of the fetched association because it does not make sense to eagerly initialize something that is not actually returned from the query):
SELECT new com.test.Pojo(acc, SUM(t.value))
FROM Transaction t JOIN t.account acc
GROUP BY acc
EDIT
Very good observation by Ilya Dyoshin about the group by clause; I completely oversaw it here. To stay in the HQL world, you could simply preload all accounts with transactions before executing the query with grouping:
SELECT acc FROM Account acc
WHERE acc.id in (SELECT t.account.id FROM Transaction t)
For instance, I have a query:
SELECT * FROM
persons
LEFT JOIN vehicles
ON persons.Id = vehicles.OwnerId
I would like execute this query on an EF data context and have array of pairs "person-vehicle". how do I do it?
Another example:
SELECT persons.*, COUNT(vehicles.*) as cnt FROM
persons
JOIN vehicles
ON persons.Id = vehicles.OwnerId
GROUP BY vehicles.Id
Here I want to have a dictionary of a person as a key and number of vehicles he owns as a value.
I know that these quesies are simple enough and it's better to avoid raw sql in these cases. But I want to know possibilities of raw query handling, because real life queries can be much more complex.
You probably want to do some reading ion LINQ to Entities. https://msdn.microsoft.com/en-us/library/vstudio/bb386964(v=vs.100).aspx
The first one is pretty basic:
var persons = context.Persons
.Include(p => p.Vehicles)
.ToList();
The second one is a little more advanced:
var persons = context.Persons
.Select(p => new { Person p, VehicleCount = p.Vehicles.Count() }
.ToList();
You could also do a group by which is described in the link.
This is my real world example.
I have 4 tables:
Person
Plan
Coverage
CoveredMembers
Each person can have many plans, each of those plans can have many coverages. Each of those coverages can have many CoveredMembers.
I need a query that will apply a filter on Plan.PlanType == 1 and CoveredMembers.TermDate == null.
This query should bring back any person who has a medical type plan that is not terminated.
This SQL statement would do just that:
SELECT Person.*, Plans.*, Coverages.*, CoveredMembers.*
FROM Person P
INNER JOIN Plan PL ON P.PersonID = PL.PersonID
INNER JOIN Coverage C on PL.PlanID = C.PlanID
INNER JOIN CoveredMember CM on C.CoverageID = CM.CoverageID
WHERE CM.TermDate = NULL AND PL.PlanType = 1
I have figured out how to do this using anonymous types, but I sometimes need to update the data and save back to the database - and anonymous types are read only.
I was given a solution that did work using JOIN but it only brought back the persons (albeit filtered the way I needed). I can then loop through each person:
foreach (var person in persons) {
foreach (var plan in person.Plans{
//do stuff
}
}
But wouldn't that make a db call for each iteration of the loop? I have 500 persons with 3 unterminated medical plans each, so it would call the db 1500 times?
This is why I want to bring the whole data tree from Persons to CoveredMembers back in one shot. Is this not possible?
I believe this is accomplished in two parts:
Your query to determine the people you wish to have returned based on your criteria as discussed in this question previously: Entity framework. Need help filtering results
Properly setting the navigation properties for entities you want brought together to be eagerly loaded: http://msdn.microsoft.com/en-us/data/jj574232.aspx
For example if your Person entity looks like:
public class Person {
public List<Plan> Plans {get; set;}
...
}
When returning data from the dbcontext you can also use explicit eager loading with the include option:
var people = context.People
.Include(p => p.Plans)
.ToList();
....
If these are nested - coverage is part of plan, etc (which it looks like, it goes something like):
var people = context.People
.Include(p => p.Plans.Select(pl=>pl.Coverage).Select(c=>c.CoveredMembers)))
.ToList();
....
I am making some assumptions about your data model here, and my code above probably needs a little tweaking.
EDIT:
I might need someone else to weigh in here, but I don't think you can add the where clause into an include like that (my example above leads you that way a bit by putting the include on the context object, instead return an IQueryable with your conditions set as solved in your first post (without a ToList() called on it) and then use the code you wrote above without the Where clauses:
From first post (you supplied different criteria in this one, but same concept)
var q = from q1 in dbContext.Parent
join q2 in dbContext.Children
on q1.key equals q2.fkey
join q3 in ........
where q4.col1 == 3000
select q1;
Then:
List<Person> people = q.Include(p => p.Plans
.Select(pl => pl.Coverages)
.Select(c => c.CoveredMembers).ToList();
Again, doing this without being able to troubleshoot - I am sure it would take me a few attempts to iron this one out too.
I have a table AccountSecurity which is a many-to-many table that relates Account entities and Securities. When I write the query below it returns all Securities that satisfy the where clause. However each Security instance in the list no longer has the reference to the AccountSecurity it came from. So when I do list[0].AccountSecurity it is empty. Is there anyway to include that information? I know I can rewrite the query to return AccountSecurities instead and use .Include("Security") on that, but I wonder if it can be done another way.
var list = (from acctSec in base.context.AccountSecurities
where acctSec.AccountId == accountId
select acctSec.Security).ToList();
UPDATE
Of course if I do two queries the graph gets populated properly, there has to be a way to do this in one shot.
var securities = (from acctSec in base.context.AccountSecurities
where acctSec.AccountId == accountId
select acctSec.Security).ToList();
//this query populates the AccountSecurities references within Security instances returned by query above
var xref = (from acctSec in base.context.AccountSecurities
where acctSec.AccountId == accountId
select acctSec).ToList();
var list = (from sec in base.context.Securities
.Include("AccountSecurity")
where sec.AccountSecurities.Any(as => as.AccountId == accountId)
select sec).ToList();
Try this:
var list = (from acctSec in base.context.AccountSecurities.Include("Security")
where acctSec.AccountId == accountId
select acctSec).ToList();
Then simply use the Security property as needed, and since it's read at the same time AccountSecurities is (single SQL with join), it will be very efficient.