I'm looking for a way to merge multiple expression trees in order to build selectors for an Entity Framework query. The query knows which columns to select based on user-provided parameters. For example, a basic query returns ID/Name columns of an entity. If a parameter is explicitly set to also retrieve the Description column, then the query will return ID/Name/Description.
So, what I need it the code for the MergeExpressions method in the following code.
Expression<Func<T, TDto>> selector1 = x => new TDto
{
Id = x.Id,
Name = x.Name
}
Expression<Func<T, TDto>> selector2 = x => new TDto
{
Description = x.Description
}
var selector = selector1;
if (includeDescription)
selector = MergeExpressions(selector1, selector2);
var results = repo.All().Select(selector).ToList();
Thank you.
Not sure for general case, but merging MemberInitExpression bodied lambdas like in your sample is relatively easy. All you need is to create another MemberInitExpression with combined Bindings:
static Expression<Func<TInput, TOutput>> MergeExpressions<TInput, TOutput>(Expression<Func<TInput, TOutput>> first, Expression<Func<TInput, TOutput>> second)
{
Debug.Assert(first != null && first.Body.NodeType == ExpressionType.MemberInit);
Debug.Assert(second != null && second.Body.NodeType == ExpressionType.MemberInit);
var firstBody = (MemberInitExpression)first.Body;
var secondBody = (MemberInitExpression)second.Body.ReplaceParameter(second.Parameters[0], first.Parameters[0]);
var body = firstBody.Update(firstBody.NewExpression, firstBody.Bindings.Union(secondBody.Bindings));
return first.Update(body, first.Parameters);
}
Note that the lambda expressions must be bound to one and the same parameters, so the above code uses the following parameter replacer helper to rebind second lambda body to the first lambda parameter:
public static partial class ExpressionUtils
{
public static Expression ReplaceParameter(this Expression expression, ParameterExpression source, Expression target)
{
return new ParameterReplacer { Source = source, Target = target }.Visit(expression);
}
class ParameterReplacer : ExpressionVisitor
{
public ParameterExpression Source;
public Expression Target;
protected override Expression VisitParameter(ParameterExpression node)
{
return node == Source ? Target : base.VisitParameter(node);
}
}
}
Check out PredicateBuilder.
Example:
Expression<Func<Customer, bool>> expr1 = (Customer c) => c.CompanyName.StartsWith("A");
Expression<Func<Customer, bool>> expr2 = (Customer c) => c.CompanyName.Contains("B");
var expr3 = PredicateBuilder.And(expr1, expr2);
var query = context.Customers.Where(expr3);
or
var expr3 = expr1.And(expr2);
var query = context.Customers.Where(expr3);
I do this kind of thing with extension methods. Its syntactically a bit nicer than using expression trees everywhere. I call this composable repositories.
I also wrote a tool (LinqExpander) to combine the expression trees of different extension methods togeather, which is especially useful for doing projection (selects) from your database. This is only nessacary when you are doing things with sub-entities. (see my post here: Composable Repositories - Nesting extensions)
usage would be something along the lines of:
var dtos = context.Table
.ThingsIWant() //filter the set
.ToDtos() //project from database model to something else (your Selector)
.ToArray();//enumerate the set
ToDtos might look something like:
public static IQueryable<DtoType> ToDtos(this IQueryable<DatabaseType> things)
{
return things.Select(x=> new DtoType{ Thing = x.Thing ... });
}
You want to merge two selects togeather (im assuming to avoid an underfetch but this seems a bit wierd). I would do this by using a projection like this:
context.Table
.AsExpandable()
.Select(x=>new {
Dto1 = x.ToDto1(),
Dto2 = x.ToDto2()
})
.ToArray();
if you really wanted it to return a single entity like this you could probably do something like:
context.Table
.AsExpandable()
.Select(x=> ToDto1(x).ToDto2(x));
but I havent ever tried this.
As this uses a sub projection you will need the .AsExpandable extensions.
How can I include a related entity, but only select the top 1?
public EntityFramework.Member Get(string userName)
{
var query = from member in context.Members
.Include(member => member.Renewals)
where member.UserName == userName
select member;
return query.SingleOrDefault();
}
According to MSDN:
"Note that it is not currently possible to filter which related entities are loaded. Include will always bring in all related entities."
http://msdn.microsoft.com/en-us/data/jj574232
There is also a uservoice item for this functionality:
http://data.uservoice.com/forums/72025-entity-framework-feature-suggestions/suggestions/1015345-allow-filtering-for-include-extension-method
The approach to use an anonymous object works, even though it's not clean as you wish it would be:
public Member GetMember(string username)
{
var result = (from m in db.Members
where m.Username == username
select new
{
Member = m,
FirstRenewal = m.Renewals.FirstOrDefault()
}).AsEnumerable().Select(r => r.Member).FirstOrDefault();
return result;
}
The FirstRenewal property is used just to make EF6 load the first renewal into the Member object. As a result the Member returned from the GetMember() method contains only the first renewal.
This code generates a single Query to the DB, so maybe it's good enough for You.
Is there any way how to Group By multiple columns dynamically?
Eg. group x by new { x.Column1, x.Column2 }
but the x.Column1 etc. I want to set dynamically (from UI)
The way to achieve this dynamically on db site is quite complicated as we cannot dynamically create anonymous types. To replace them I would suggest to create a class:
public class CustomTuple<T1, T2>
{
public T1 Item1 { get; set; }
public T2 Item2 { get; set; }
}
We cannot use Tuple here as it does not have default constructor. In CustomTuple class place as much parameters T and as much properties as you would need at max. If you will define in that class 5 properties but for the query you will use only 3 you just set only 3 properties to proper values and the remaining 2 properties you keep null - the query will still work. Alternatively you may dynamically at run time generate proper class with CodeDOM. Then comes query logic:
Type[] parameterTypes = new Type[] { typeof(int), typeof(object) };
Type tupleType = typeof(CustomTuple<,>).MakeGenericType(parameterTypes);
ParameterExpression x = Expression.Parameter(typeof(Entity));
NewExpression body = Expression.New(tupleType.GetConstructor(new Type[0]), new Expression[0]);
MemberBinding binding1 = Expression.Bind(
typeof(CustomTuple<,>).MakeGenericType(parameterTypes).GetProperty("Item1"),
Expression.Property(x, "Value"));
MemberInitExpression memberInitExpression =
Expression.MemberInit(
body,
binding1);
Expression<Func<Entity, object>> exp = Expression.Lambda<Func<Entity, object>>(memberInitExpression, x);
using (MyDbContext context = new MyDbContext())
{
var list = context.Entities.GroupBy(exp).ToList();
}
The above code groups Entities by Value property. parameterTypes may be dynamically build during program execution - this is list of types of properties anonymous type for key selection in group by would have. Basing on that we create proper CustomTuple type. Then we dynamically create at run time binding1 elements - one per each property to be set for grouping key. In the example above I create only one. With use of the NewExpression and MemberBinding expression we may build initialization expression with MemberInit method. Finally you build lambda expression from that and execute it against db.
I generally use a generic repository to boilerplate my EF queries so I have to write limited code and also use caching. The source code for the repository can be found here.
The backbone query within the code is this one below. FromCache<T>() is an IEnumerable<T> extension method that utilizes the HttpContext.Cache to store the query using a stringified representation of the lambda expression as a key.
public IQueryable<T> Any<T>(Expression<Func<T, bool>> expression = null)
where T : class, new()
{
// Check for a filtering expression and pull all if not.
if (expression == null)
{
return this.context.Set<T>()
.AsNoTracking()
.FromCache<T>(null)
.AsQueryable();
}
return this.context.Set<T>()
.AsNoTracking<T>()
.Where<T>(expression)
.FromCache<T>(expression)
.AsQueryable<T>();
}
Whilst this all works it is subject to the N+1 problem for related tables since If I were to write a query like so:
var posts = this.ReadOnlySession.Any<Post>(p => p.IsDeleted == false)
.Include(p => p.Author);
The Include() will have no effect on my query since it has already been run in order to be cached.
Now I know that I can force Entity Framework to use eager loading within my model by removing the virtual prefix on my navigation properties but that to me feels like the wrong place to do it as you cannot predict the types of queries you will be making. To me it feels like something I would be doing in a controller class. What I am wondering is whether I can pass a list of includes into my Any<T>() method that I could then iterate though when I make the call?
ofDid you mean something like...
IQueryable<T> AnyWithInclude<T,I>(Expression<Func<T,bool>> predicate,
Expression<Func<T,I>> includeInfo)
{
return DbSet<T>.where(predicate).include(includeInfo);
}
the call
Context.YourDbSetReference.AnyWithInclude(t => t.Id==someId, i => i.someNavProp);
In response to extra question on as collection.
I realised late, there was an overload on Property. You can just pass a string
This might work but call is not easy. Well I find it hard.
IQueryable<T> GetListWithInclude<I>(Expression<Func<T, bool>> predicate,
params Expression<Func<T, I>>[] IncludeCollection);
so i tried
public virtual IQueryable<T> GetListWithInclude(Expression<Func<T, bool>> predicate,
List<string> includeCollection)
{ var result = EntityDbSet.Where(predicate);
foreach (var incl in includeCollection)
{
result = result.Include(incl);
}
return result;
}
and called with
var ic = new List<string>();
ic.Add("Membership");
var res = context.DbSte<T>.GetListWithInclude( t=>t.UserName =="fred", ic);
worked as before.
In the interest of clarity I'm adding the solution I came up with based upon #soadyp's answer.
public IQueryable<T> Any<T>(Expression<Func<T, bool>> expression = null,
params Expression<Func<T, object>>[] includeCollection)
where T : class, new()
{
IQueryable<T> query = this.context.Set<T>().AsNoTracking().AsQueryable<T>();
if (includeCollection.Any())
{
query = includeCollection.Aggregate(query,
(current, include) => current.Include(include));
}
// Check for a filtering expression and pull all if not.
if (expression != null)
{
query = query.Where<T>(expression);
}
return query.FromCache<T>(expression, includeCollection)
.AsQueryable<T>();
}
Usage:
// The second, third, fourth etc parameters are the strongly typed includes.
var posts = this.ReadOnlySession.Any<Post>(p => p.IsDeleted == false,
p => p.Author);
I have a tree structure in the DB with TreeNodes table. the table has nodeId, parentId and parameterId. in the EF, The structure is like TreeNode.Children where each child is a TreeNode...
I also have a Tree table with contain id,name and rootNodeId.
At the end of the day I would like to load the tree into a TreeView but I can't figure how to load it all at once.
I tried:
var trees = from t in context.TreeSet.Include("Root").Include("Root.Children").Include("Root.Children.Parameter")
.Include("Root.Children.Children")
where t.ID == id
select t;
This will get me the the first 2 generations but not more.
How do I load the entire tree with all generations and the additional data?
I had this problem recently and stumbled across this question after I figured a simple way to achieve results. I provided an edit to Craig's answer providing a 4th method, but the powers-that-be decided it should be another answer. That's fine with me :)
My original question / answer can be found here.
This works so long as your items in the table all know which tree they belong to (which in your case it looks like they do: t.ID). That said, it's not clear what entities you really have in play, but even if you've got more than one, you must have a FK in the entity Children if that's not a TreeSet
Basically, just don't use Include():
var query = from t in context.TreeSet
where t.ID == id
select t;
// if TreeSet.Children is a different entity:
var query = from c in context.TreeSetChildren
// guessing the FK property TreeSetID
where c.TreeSetID == id
select c;
This will bring back ALL the items for the tree and put them all in the root of the collection. At this point, your result set will look like this:
-- Item1
-- Item2
-- Item3
-- Item4
-- Item5
-- Item2
-- Item3
-- Item5
Since you probably want your entities coming out of EF only hierarchically, this isn't what you want, right?
.. then, exclude descendants present at the root level:
Fortunately, because you have navigation properties in your model, the child entity collections will still be populated as you can see by the illustration of the result set above. By manually iterating over the result set with a foreach() loop, and adding those root items to a new List<TreeSet>(), you will now have a list with root elements and all descendants properly nested.
If your trees get large and performance is a concern, you can sort your return set ASCENDING by ParentID (it's Nullable, right?) so that all the root items are first. Iterate and add as before, but break from the loop once you get to one that is not null.
var subset = query
// execute the query against the DB
.ToList()
// filter out non-root-items
.Where(x => !x.ParentId.HasValue);
And now subset will look like this:
-- Item1
-- Item2
-- Item3
-- Item4
-- Item5
About Craig's solutions:
You really don't want to use lazy loading for this!! A design built around the necessity for n+1 querying will be a major performance sucker. ********* (Well, to be fair, if you're going to allow a user to selectively drill down the tree, then it could be appropriate. Just don't use lazy loading for getting them all up-front!!)I've never tried the nested set stuff, and I wouldn't suggest hacking EF configuration to make this work either, given there is a far easier solution. Another reasonable suggestion is creating a database view that provides the self-linking, then map that view to an intermediary join/link/m2m table. Personally, I found this solution to be more complicated than necessary, but it probably has its uses.
When you use Include(), you are asking the Entity Framework to translate your query into SQL. So think: How would you write an SQL statement which returns a tree of an arbitrary depth?
Answer: Unless you are using specific hierarchy features of your database server (which are not SQL standard, but supported by some servers, such as SQL Server 2008, though not by its Entity Framework provider), you wouldn't. The usual way to handle trees of arbitrary depth in SQL is to use the nested sets model rather than the parent ID model.
Therefore, there are three ways which you can use to solve this problem:
Use the nested sets model. This requires changing your metadata.
Use SQL Server's hierarchy features, and hack the Entity Framework into understanding them (tricky, but this technique might work). Again, you'll need to change your metadata.i
Use explicit loading or EF 4's lazy loading instead of eager loading. This will result in many database queries instead of one.
I wanted to post up my answer since the others didn't help me.
My database is a little different, basically my table has an ID and a ParentID. The table is recursive. The following code gets all children and nests them into a final list.
public IEnumerable<Models.MCMessageCenterThread> GetAllMessageCenterThreads(int msgCtrId)
{
var z = Db.MCMessageThreads.Where(t => t.ID == msgCtrId)
.Select(t => new MCMessageCenterThread
{
Id = t.ID,
ParentId = t.ParentID ?? 0,
Title = t.Title,
Body = t.Body
}).ToList();
foreach (var t in z)
{
t.Children = GetChildrenByParentId(t.Id);
}
return z;
}
private IEnumerable<MCMessageCenterThread> GetChildrenByParentId(int parentId)
{
var children = new List<MCMessageCenterThread>();
var threads = Db.MCMessageThreads.Where(x => x.ParentID == parentId);
foreach (var t in threads)
{
var thread = new MCMessageCenterThread
{
Id = t.ID,
ParentId = t.ParentID ?? 0,
Title = t.Title,
Body = t.Body,
Children = GetChildrenByParentId(t.ID)
};
children.Add(thread);
}
return children;
}
For completeness, here's my model:
public class MCMessageCenterThread
{
public int Id { get; set; }
public int ParentId { get; set; }
public string Title { get; set; }
public string Body { get; set; }
public IEnumerable<MCMessageCenterThread> Children { get; set; }
}
I wrote something recently that does N+1 selects to load the whole tree, where N is the number of levels of your deepest path in the source object.
This is what I did, given the following self-referencing class
public class SomeEntity
{
public int Id { get; set; }
public int? ParentId { get; set; }
public string Name { get; set;
}
I wrote the following DbSet helper
using System;
using System.Collections.Generic;
using System.Linq;
using System.Linq.Expressions;
using System.Threading.Tasks;
namespace Microsoft.EntityFrameworkCore
{
public static class DbSetExtensions
{
public static async Task<TEntity[]> FindRecursiveAsync<TEntity, TKey>(
this DbSet<TEntity> source,
Expression<Func<TEntity, bool>> rootSelector,
Func<TEntity, TKey> getEntityKey,
Func<TEntity, TKey> getChildKeyToParent)
where TEntity: class
{
// Keeps a track of already processed, so as not to invoke
// an infinte recursion
var alreadyProcessed = new HashSet<TKey>();
TEntity[] result = await source.Where(rootSelector).ToArrayAsync();
TEntity[] currentRoots = result;
while (currentRoots.Length > 0)
{
TKey[] currentParentKeys = currentRoots.Select(getEntityKey).Except(alreadyProcessed).ToArray();
alreadyProcessed.AddRange(currentParentKeys);
Expression<Func<TEntity, bool>> childPredicate = x => currentParentKeys.Contains(getChildKeyToParent(x));
currentRoots = await source.Where(childPredicate).ToArrayAsync();
}
return result;
}
}
}
Whenever you need to load a whole tree you simply call this method, passing in three things
The selection criteria for your root objects
How to get the property for the primary key of the object (SomeEntity.Id)
How to get the child's property that refers to its parent (SomeEntity.ParentId)
For example
SomeEntity[] myEntities = await DataContext.SomeEntity.FindRecursiveAsync(
rootSelector: x => x.Id = 42,
getEntityKey: x => x.Id,
getChildKeyToParent: x => x.ParentId).ToArrayAsync();
);
Alternatively, if you can add a RootId column to the table then for each non-root entry you can set this column to the ID of the root of the tree. Then you can fetch everything with a single select
DataContext.SomeEntity.Where(x => x.Id == rootId || x.RootId == rootId)
For an example of loading in child objects, I'll give the example of a Comment object that holds a comment. Each comment has a possible child comment.
private static void LoadComments(<yourObject> q, Context yourContext)
{
if(null == q | null == yourContext)
{
return;
}
yourContext.Entry(q).Reference(x=> x.Comment).Load();
Comment curComment = q.Comment;
while(null != curComment)
{
curComment = LoadChildComment(curComment, yourContext);
}
}
private static Comment LoadChildComment(Comment c, Context yourContext)
{
if(null == c | null == yourContext)
{
return null;
}
yourContext.Entry(c).Reference(x=>x.ChildComment).Load();
return c.ChildComment;
}
Now if you were having something that has collections of itself you would need to use Collection instead of Reference and do the same sort of diving down. At least that's the approach I took in this scenario as we were dealing with Entity and SQLite.
This is an old question, but the other answers either had n+1 database hits or their models were conducive to bottom-up (trunk to leaves) approaches. In this scenario, a tag list is loaded as a tree, and a tag can have multiple parents. The approach I use only has two database hits: the first to get the tags for the selected articles, then another that eager loads a join table. Thus, this uses a top-down (leaves to trunk) approach; if your join table is large or if the result cannot really be cached for reuse, then eager loading the whole thing starts to show the tradeoffs with this approach.
To begin, I initialize two HashSets: one to hold the root nodes (the resultset), and another to keep a reference to each node that has been "hit."
var roots = new HashSet<AncestralTagDto>(); //no parents
var allTags = new HashSet<AncestralTagDto>();
Next, I grab all of the leaves that the client requested, placing them into an object that holds a collection of children (but that collection will remain empty after this step).
var startingTags = await _dataContext.ArticlesTags
.Include(p => p.Tag.Parents)
.Where(t => t.Article.CategoryId == categoryId)
.GroupBy(t => t.Tag)
.ToListAsync()
.ContinueWith(resultTask =>
resultTask.Result.Select(
grouping => new AncestralTagDto(
grouping.Key.Id,
grouping.Key.Name)));
Now, let's grab the tag self-join table, and load it all into memory:
var tagRelations = await _dataContext.TagsTags.Include(p => p.ParentTag).ToListAsync();
Now, for each tag in startingTags, add that tag to the allTags collection, then travel down the tree to get the ancestors recursively:
foreach (var tag in startingTags)
{
allTags.Add(tag);
GetParents(tag);
}
return roots;
Lastly, here's the nested recursive method that builds the tree:
void GetParents(AncestralTagDto tag)
{
var parents = tagRelations.Where(c => c.ChildTagId == tag.Id).Select(p => p.ParentTag);
if (parents.Any()) //then it's not a root tag; keep climbing down
{
foreach (var parent in parents)
{
//have we already seen this parent tag before? If not, instantiate the dto.
var parentDto = allTags.SingleOrDefault(i => i.Id == parent.Id);
if (parentDto is null)
{
parentDto = new AncestralTagDto(parent.Id, parent.Name);
allTags.Add(parentDto);
}
parentDto.Children.Add(tag);
GetParents(parentDto);
}
}
else //the tag is a root tag, and should be in the root collection. If it's not in there, add it.
{
//this block could be simplified to just roots.Add(tag), but it's left this way for other logic.
var existingRoot = roots.SingleOrDefault(i => i.Equals(tag));
if (existingRoot is null)
roots.Add(tag);
}
}
Under the covers, I am relying on the properties of a HashSet to prevent duplicates. To that end, it's important that the intermediate object that you use (I used AncestralTagDto here, and its Children collection is also a HashSet), override the Equals and GetHashCode methods as appropriate for your use-case.