I am working on a new project and we are using Entity Framework and the dev lead would like to use lambda queries whenever possible. One thing we are having a hard time figuring out is how to select two columns specifically. Also how to select distinct. We have a table that has multiple entries for a vendor but we want to just get a list of vendors and load to a dictionary object. It fails because as written it is trying to add a key value that has already been added. Take the following query.
Dictionary<int, string> dict = new Dictionary<int, string>();
dict = GetWamVendorInfo().AsEnumerable()
.Where(x => x.vendor_name != null && x.vendor_id != null)
//.Select(x => x.vendor_id).Distinct()
.Take(2)
.ToDictionary(o => int.Parse(o.vendor_id.ToString()), o => o.vendor_name);
What I would like to do is select just vendor_id and vendor_name so we can get just the distinct records.
Any help would be greatly appreciated.
Thanks,
Rhonda
Use an anonymous type:
// earlier bit of query
.Select(x => new { VendorId = x.vendor_id, VendorName = x.vendor_name } )
.Distinct()
.ToDictionary(o => o.VendorId, o => o.VendorName);
I've removed the call to Take(2) as it wasn't clear why you'd want it - and also removed the parsing of VendorId, which I would have expected to already be an integer type.
Note that you should almost certainly remove the AsEnumerable call from your query - currently you'll be fetching all the vendors and filtering with LINQ to Objects. There's also no point creating an empty dictionary and then ignoring it entirely. I suspect your complete query should be:
var vendors = GetWamVendorInfo()
.Select(x => new { VendorId = x.vendor_id,
VendorName = x.vendor_name } )
.Distinct()
.ToDictionary(o => o.VendorId,
o => o.VendorName);
As an aside, you should ask your dev lead why he wants to use lambda expressions (presumably as opposed to query expressions) everywhere. Different situations end up with more readable code using different syntax options - it's worth being flexible on this front.
Just use an anonymous object:
var vendors = GetWamVendorInfo().AsEnumerable()
.Where(x => x.vendor_name != null && x.vendor_id != null)
.Select(new {x.vendor_id, x.vendor_name})
.Take(2)
That's it. You can now work with vendors[0].vendor_id, vendors[0].vendor_name, and so on.
Related
I have a table called Products and I need to find the products with unique title for a particular category. Earlier we used to do with this query in entity framework core 2.2 :
currentContext.Products
.GroupBy(x => x.Title)
.Select(x => x.FirstOrDefault()))
.Select(x => new ProductViewModel
{
Id = x.Id,
Title = x.Title,
CategoryId= x.CategoryId
}).ToList();
But after upgrading to Entity Framework Core 5.0, we get an error for Groupby Shaker exception:
The LINQ expression 'GroupByShaperExpression:KeySelector: t.title, ElementSelector:EntityShaperExpression: EntityType: Project ValueBufferExpression: ProjectionBindingExpression: EmptyProjectionMember IsNullable: False .FirstOrDefault()' could not be translated. Either rewrite the query in a form that can be translated, or switch to client evaluation explicitly by inserting a call to 'AsEnumerable', 'AsAsyncEnumerable', 'ToList', or 'ToListAsync'.
I know there are multiple way to client projection but I am searching for most efficient way to search.
Most likely that LINQ query couldn't be translated in EF Core 2.2 either, because of some limitations that the GroupBy operator has.
From the docs:
Since no database structure can represent an IGrouping, GroupBy operators have no translation in most cases. When an aggregate operator is applied to each group, which returns a scalar, it can be translated to SQL GROUP BY in relational databases. The SQL GROUP BY is restrictive too. It requires you to group only by scalar values. The projection can only contain grouping key columns or any aggregate applied over a column.
What happened in EF Core 2.x is that whenever it couldn't translate an expression, it would automatically switch to client evaluation and give just a warning.
This is listed as the breaking change with highest impact when migrating to EF Core >= 3.x :
Old behavior
Before 3.0, when EF Core couldn't convert an expression that was part of a query to either SQL or a parameter, it automatically evaluated the expression on the client. By default, client evaluation of potentially expensive expressions only triggered a warning.
New behavior
Starting with 3.0, EF Core only allows expressions in the top-level projection (the last Select() call in the query) to be evaluated on the client. When expressions in any other part of the query can't be converted to either SQL or a parameter, an exception is thrown.
So if the performance of that expression was good enough when using EF Core 2.x, it will be as good as before if you decide to explicitly switch to client evaluation when using EF Core 5.x. That's because both are client evaluated, before and now, with the only difference being that you have to be explicit about it now. So the easy way out, if the performance was acceptable previously, would be to just client evaluate the last part of the query using .AsEnumerable() or .ToList().
If client evaluation performance is not acceptable (which will imply that it wasn't before the migration either) then you have to rewrite the query. There are a couple of answers by Ivan Stoev that might get you inspired.
I am a little confused by the description of what you want to achieve: I need to find the products with unique title for a particular category and the code you posted, since I believe it's not doing what you explained. In any case, I will provide possible solutions for both interpretations.
This is my attempt of writing a query to find the products with unique title for a particular category.
var uniqueProductTitlesForCategoryQueryable = currentContext.Products
.Where(x => x.CategoryId == categoryId)
.GroupBy(x => x.Title)
.Where(x => x.Count() == 1)
.Select(x => x.Key); // Key being the title
var productsWithUniqueTitleForCategory = currentContext.Products
.Where(x => x.CategoryId == categoryId)
.Where(x => uniqueProductTitlesForCategoryQueryable .Contains(x.Title))
.Select(x => new ProductViewModel
{
Id = x.Id,
Title = x.Title,
CategoryId= x.CategoryId
}).ToList();
And this is my attempt of rewriting the query you posted:
currentContext.Products
.Select(product => product.Title)
.Distinct()
.SelectMany(uniqueTitle => currentContext.Products.Where(product => product.Title == uniqueTitle ).Take(1))
.Select(product => new ProductViewModel
{
Id = product.Id,
Title = product.Title,
CategoryId= product.CategoryId
})
.ToList();
I am getting the distinct titles in the Product table and per each distinct title I get the first Product that matches it (that should be equivalent as GroupBy(x => x.Title)+ FirstOrDefault AFAIK). You could add some sorting before the Take(1) if needed.
You can use Join for this query as below :
currentContext.Products
.GroupBy(x => x.Title)
.Select(x => new ProductViewModel()
{
Title = x.Key,
Id = x.Min(b => b.Id)
})
.Join(currentContext.Products, a => a.Id, b => b.Id,
(a, b) => new ProductViewModel()
{
Id = a.Id,
Title = a.Title,
CategoryId = b.CategoryId
}).ToList();
If you watch or log translated SQL query, it would be as below:
SELECT [t].[Title], [t].[c] AS [Id], [p0].[CategoryId] AS [CategoryId]
FROM (
SELECT [p].[Title], MIN([p].[Id]) AS [c]
FROM [Product].[Products] AS [p]
GROUP BY [p].[Title]
) AS [t]
INNER JOIN [Product].[Products] AS [p0] ON [t].[c] = [p0].[Id]
As you can see, the entire query is translated into one SQL query and it is highly efficient because GroupBy operation is being performed in database and no additional record is fetched by the client.
As mentioned by Ivan Stoev, EFC 2.x just silently loads full table to the client side and then apply needed logic for extracting needed result. It is resource consuming way and thanks that EFC team uncovered such potential harmful queries.
Most effective way is already known - raw SQL and window functions. SO is full of answers like this.
SELECT
s.Id,
s.Title,
s.CategoryId
FROM
(SELECT
ROW_NUMBER() OVER (PARTITION BY p.Title ORDER BY p.Id) AS RN,
p.*
FROM Products p) s
WHERE s.RN = 1
Not sure that EFC team will invent universal algorithm for generating such SQL in nearest future, but for special edge cases it is doable and maybe it is their plan to do that for EFC 6.0
Anyway if performance and LINQ is priority for such question, I suggest to try our adaptation of linq2db ORM for EF Core projects: linq2db.EntityFrameworkCore
And you can get desired result without leaving LINQ:
urrentContext.Products
.Select(x => new
{
Product = x,
RN = Sql.Ext.RowNumber().Over()
.PartitionBy(x.Title)
.OrderBy(x.Id)
.ToValue()
})
.Where(x => x.RN == 1)
.Select(x => x.Product)
.Select(x => new ProductViewModel
{
Id = x.Id,
Title = x.Title,
CategoryId = x.CategoryId
})
.ToLinqToDB()
.ToList();
Short answer is you deal with breaking changes in EF Core versions.
You should consider the total API and behavior changes for migration from 2.2 to 5.0 as I provided bellow:
Breaking changes included in EF Core 3.x
Breaking changes in EF Core 5.0
You may face other problems to write valid expressions using the newer version. In my opinion, upgrading to a newer version is not important itself. This is important to know how to work with a specific version.
You should use .GroupBy() AFTER materialization. Unfortunately, EF core doesn't support GROUP BY. In version 3 they introduced strict queries which means you can not execute IQeuriables that can't be converted to SQL unless you disable this configuration (which is not recommended). Also, I'm not sure what are you trying to get with GroupBy() and how it will influence your final result. Anyway, I suggest you upgrade your query like this:
currentContext.Products
.Select(x=> new {
x.Id,
x.Title,
x.Category
})
.ToList()
.GroupBy(x=> x.Title)
.Select(x => new Wrapper
{
ProductsTitle = x.Key,
Products = x.Select(p=> new ProductViewModel{
Id = p.Id,
Title = p.Title,
CategoryId= p.CategoryId
}).ToList()
}).ToList();
I am trying to build a query using asp.net core c#
https://www.reflectionit.nl/blog/2017/paging-in-asp-net-core-mvc-and-entityframework-core
I trying to do a filtering however I need the data from another table which have my unique id
var result = _context.UserRoles.Where(y => y.RoleId.Contains(selectedRoles.Id)); // Retrieve the the userid i have from another table with the selected roleid
var query = _context.Users.Where(x => //I have already tried contains, where join );
If there is a site where i can learn this query please recommend. "Join()" does not work as I am doing paging
a least two solutions (please note that I do not check the identity classes members, so the following is the "spirit" of the solution (you miss the select clauses) ):
var result = _context.UserRoles.
Where(y => selectedRoles.Contains(y.RoleId)).
Select(y => y.User);
or
var result = _context.UserRoles.
Where(y => selectedRoles.Contains(y.RoleId)).
Select(y => y.UserId);
query = _context.Users.
Where(x => result.Contains(x.Id));
That said, assuming that there is no UserRoles table exposed in Identity (v2), you probably want:
userManager.Users.
Where(u => u.Roles.Any(r => selectecRoles.Contains(r.RoleId)));
Up to you to instanciate the userManager.
I have the following EF code
Func<CxForumArticle, bool> whereClause = a => a.CreatedBy == authorId;
IEnumerable<CxForumArticle> articlesCol = ctx.Articles
.Where(whereClause)
.Where(a => a.PublishingStatus == EnPublishStatus.PUBLISHED)
.OrderByDescending(a => a.ModifiedOn).Skip(offset).Take(pageSize);
It produces the following SQL
SELECT
[Extent1].[ArticleId] AS [ArticleId],
[Extent1].[Alias] AS [Alias],
[Extent1].[MigratedId] AS [MigratedId],
[Extent1].[Title] AS [Title],
[Extent1].[Teaser] AS [Teaser],
[Extent1].[ClobId] AS [ClobId],
[Extent1].[UnifiedContentId] AS [UnifiedContentId],
[Extent1].[EditorComments] AS [EditorComments],
[Extent1].[CreatedOn] AS [CreatedOn],
[Extent1].[CreatedBy] AS [CreatedBy],
[Extent1].[ModifiedOn] AS [ModifiedOn],
[Extent1].[ModifiedBy] AS [ModifiedBy],
[Extent1].[PublishingStatus] AS [PublishingStatus]
FROM [dbo].[ForumArticle] AS [Extent1]
As you see, there is no ordering and paging in this SQL. So EF orders and pages data in memory.
This doesn't seem to be a good thing to do.
I read an article, claiming that I have to use expression in OrderBy clause. I did that
Func<CxForumArticle, bool> whereClause = a => a.CreatedBy == authorId;
Expression<Func<CxForumArticle, DateTime>> orderByFunc = a => a.ModifiedOn;
IEnumerable<CxForumArticle> articlesCol = ctx.Articles
.Where(whereClause)
.Where(a => a.PublishingStatus == EnPublishStatus.PUBLISHED)
.OrderByDescending(orderByFunc.Compile()).Skip(offset).Take(pageSize)
;
But I got the same result. Any ideas how can I force EF to sort and page the data in DB?
I know this is pretty old, but you actually need to have the whereClause use an expression as well. Entity Framework only knows how to convert expressions to SQL. If you use a delegate instead of an expression, it is going to query the entire table and then then filter using LINQ to Objects.
I have been trying to figure out how to optimize the following query for the past few days and just not having much luck. Right now my test db is returning about 300 records with very little nested data, but it's taking 4-5 seconds to run and the SQL being generated by LINQ is awfully long (too long to include here). Any suggestions would be very much appreciated.
To sum up this query, I'm trying to return a somewhat flattened "snapshot" of a client list with current status. A Party contains one or more Clients who have Roles (ASPNET Role Provider), Journal is returning the last 1 journal entry of all the clients in a Party, same goes for Task, and LastLoginDate, hence the OrderBy and FirstOrDefault functions.
Guid userID = 'some user ID'
var parties = Parties.Where(p => p.BrokerID == userID).Select(p => new
{
ID = p.ID,
Title = p.Title,
Goal = p.Goal,
Groups = p.Groups,
IsBuyer = p.Clients.Any(c => c.RolesInUser.Any(r => r.Role.LoweredName == "buyer")),
IsSeller = p.Clients.Any(c => c.RolesInUser.Any(r => r.Role.LoweredName == "seller")),
Journal = p.Clients.SelectMany(c => c.Journals).OrderByDescending(j => j.OccuredOn).Select(j=> new
{
ID = j.ID,
Title = j.Title,
OccurredOn = j.OccuredOn,
SubCatTitle = j.JournalSubcategory.Title
}).FirstOrDefault(),
LastLoginDate = p.Clients.OrderByDescending(c=>c.LastLoginDate).Select(c=>c.LastLoginDate).FirstOrDefault(),
MarketingPlanCount = p.Clients.SelectMany(c => c.MarketingPlans).Count(),
Task = p.Tasks.Where(t=>t.DueDate != null && t.DueDate > DateTime.Now).OrderBy(t=>t.DueDate).Select(t=> new
{
ID = t.TaskID,
DueDate = t.DueDate,
Title = t.Title
}).FirstOrDefault(),
Clients = p.Clients.Select(c => new
{
ID = c.ID,
FirstName = c.FirstName,
MiddleName = c.MiddleName,
LastName = c.LastName,
Email = c.Email,
LastLogin = c.LastLoginDate
})
}).OrderBy(p => p.Title).ToList()
I think posting the SQL could give us some clues, as small things like the order of OrderBy coming before or after the projection could make a big difference.
But regardless, try extracting the Clients in a seperate query, this will simplify your query probably. And then include other tables like Journal and Tasks before projecting and see how this affects your query:
//am not sure what the exact query would be, and project it using ToList()
var clients = GetClientsForParty();
var parties = Parties.Include("Journal").Include("Tasks")
.Where(p=>p.BrokerID == userID).Select( p => {
....
//then use the in-memory clients
IsBuyer = clients.Any(c => c.RolesInUser.Any(r => r.Role.LoweredName == "buyer")),
...
}
)
In all cases, install EF profiler and have a look at how your query is affected. EF can be quiet surprising. Something like putting OrderBy before the projection, the same for all these FirstOrDefault or SingleOrDefault, they can all have a big effect.
And go back to the basics, if you are searching on LoweredRoleName, then make sure it is indexed so that the query is fast (even though that could be useless since EF could end up not making use of the covering index since it is querying so many other columns).
Also, since this is query is to view data (you will not alter data), don't forget to turn off Entity tracking, that will give you some performance boost as well.
And last, don't forget that you could always write your SQL query directly and project to your a ViewModel rather than anonymous type (which I see as a good practice anyhow) so create a class called PartyViewModel that includes the flatten view you are after, and use it with your hand-crafted SQL
//use your optimized SQL query that you write or even call a stored procedure
db.Database.SQLQuery("select * from .... join .... on");
I am writing a blog post about these issues around EF. The post is still not finished, but all in all, just be patient, use some of these tricks and observe their effect (and measure it) and you will reach what you want.
I need to filter a dealer based on id and the uncomplete checkins
Initially, it returned the dealer based only on id:
// TODO: limit checkins to those that are not complete
return this.ObjectContext.Dealers
.Include("Groups")
.Include("Groups.Items")
.Include("Groups.Items.Observations")
.Include("Groups.Items.Recommendations")
.Include("Checkins")
.Include("Checkins.Inspections")
.Include("Checkins.Inspections.InspectionItems")
.Where(d => d.DealerId == id)
.FirstOrDefault();
As you can see the requirement is to limit the checkins.
Here's what I did:
var query = from d in this.ObjectContext.Dealers
.Include("Groups")
.Include("Groups.Items")
.Include("Groups.Items.Observations")
.Include("Groups.Items.Recommendations")
.Include("Checkins.Inspections")
.Include("Checkins.Inspections.InspectionItems")
.Where(d => d.DealerId == id)
select new
{
Dealer = d,
Groups = from g in d.Groups
select new
{
Items = from i in g.Items
select new
{
Group = i.Group,
Observations = i.Observations,
Recommendations = i.Recommendations
}
},
Checkins = from c in d.Checkins
where c.Complete == true
select new
{
Inspections = from i in c.Inspections
select new
{
InspectionItems = i.InspectionItems
}
}
};
var dealer = query.ToArray().Select(o => o.Dealer).First();
return dealer;
It works.
However, I am not convinced I am doing the right thing.
What is the best way to accomplish what I did? A stored procedure maybe?
I am not sure I even have to use Include clause anymore
Thank you.
If you want to load filtered relations with single query you indeed have to execute such projection but you don't need those calls to Include. Once you are building projections includes are not use - you have returned data under your control.
Stored procedure will help you only if you fall back to plain ADO.NET because stored procedures executed through Entity framework are not able to fill related entities (only flattened structures).
Automatic fixupu mentioned by #Andreas requires multiple database queries and as I know it works only if lazy loading is disabled because proxied object somehow doesn't have information about fixup and it still has its internal flags for each relation as not loaded so when you access them for the first time they still execute additional query.
Maybe you can make use of the relation fixup mechanism in the EF ObjectContexts. When you do multiple queries in the same context for entities, that are related by associations, these are resolved.
Assuming your association between Dealers and Checkins is 1:n with navigation properties on each side, you could do like:
var dealer = yourContext.Dealers
.Where(p => p.DealerId == id)
.FirstOrDefault();
if(dealer != null)
{
yourContext.Checkins
.Where(c => c.Complete && c.DealerId == dealer.DealerId)
.ToList();
I have not tested this by now, but since EF recognises that the Checkins, it inserts into the context by the second query belong to the dealer from the first query, corresponding references are created.
#Andreas H:
Awesome, thank you a lot.
I had to adjust your suggestion like this and it worked:
var dealer = this.ObjectContext.Dealers
.Include("Groups")
.Include("Groups.Items")
.Include("Groups.Items.Observations")
.Include("Groups.Items.Recommendations")
.Where(p => p.DealerId == id).
FirstOrDefault();
if (dealer != null)
{
this.ObjectContext.Checkins
.Include("Inspections")
.Include("Inspections.InspectionItems")
.Where(c => !c.Complete && c.Dealer.DealerId == dealer.DealerId)
.ToList();
}
return dealer;
I still have to use the Include otherwise it won't return the referenced entities.
Note also that Dealer.Groups are unrelated to the Dealer.Checkins.
So if there's no checkins satisfying the condition, Groups still need to be returned.
It's interesting to note that at first, I put the two include for checkins to the dealer
var dealer = this.ObjectContext.Dealers
.Include("Groups")
.Include("Groups.Items")
.Include("Groups.Items.Observations")
.Include("Groups.Items.Recommendations")
.Include("Checkins.Inspections")
.Include("Checkins.Inspections.InspectionItems")
.Where(p => p.DealerId == id).
FirstOrDefault();
if (dealer != null)
{
this.ObjectContext.Checkins
.Where(c => c.Complete && c.DealerId == id)
.ToList();
}
return dealer;
but it returned all the Checkins including those which are not complete.
I don't understand exactly why the latter doesn't work but the former does, how are the entities are resolved. I somehow can intuit that the former returns all data.
Your accepted solution will generate multiple database queries. As Ladislav Mrnka said a projection is the only way to pull your result with one query. The maintance of your code indeed hard. Maybe you could use an IQueryable-Extension that builds the projection dynamically and keep your code clean:
var query = this.ObjectContext.Dealers.SelectIncluding( new List<Expression<Func<T,object>>>>(){
x => x.Groups,
x => x.Groups.Select(y => y.Items),
x => x.Groups.Select(y => y.Items.Select(z => z.Observations)),
x => x.Groups.Select(y => y.Items.Select(z => z.Recommendations)),
x => x.Checkins.Where(y => y.Complete==true),
x => x.Checkins.Select(y => y.Inspections),
x => x.Checkins.Select(y => y.Inspections.Select(z => z.InspectionItems))
});
var dealer = query.First();
return dealer;
You can find the extension at thiscode/DynamicSelectExtensions on github