I am trying to migrate a Web API from .NET Core 2.2 to .NET Core 3.0 and I have stumbled across the following:
public Dictionary<int, Tag> GetTagMap(IList<int> tagIds = null)
{
var tags = context.Tag.AsNoTracking();
if (tagIds != null)
tags = tags.Where(t => tagIds.Contains(t.TagId));
return tags
.ToList() // explicit client evaluation in 3.0
.ToDictionary(t => t.TagId, t => t);
}
This used to generate a SQL statement similar to this one:
SELECT TagId, Name FROM Tag WHERE TagId IN (1, 2, 3)
which worked very well for correctly indexed column and a small number of IN values.
Now I receive the following error suggesting that List<>.Contains translation is not supported anymore:
System.InvalidOperationException: 'The LINQ expression 'Where(
source: DbSet, predicate: (t) => (Unhandled parameter:
__tagIds_0).Contains(t.TagId))' could not be translated. Either rewrite the query in a form that can be translated, or switch to
client evaluation explicitly by inserting a call to either
AsEnumerable(), AsAsyncEnumerable(), ToList(), or ToListAsync(). See
Client vs. Server Evaluation - EF Core for more information.'
This suggests the LINQ queries are no longer evaluated on the client breaking change, but AFAIK Contains was not evaluated on the client.
It's a 3.0 bug, tracked by #17342: Contains on generic IList/HashSet/ImmutableHashSet will throw exception.
Already fixed in 3.1. The workaround (if you can't wait) is to force the usage of Enumerable.Contains, for instance
t => tagIds.AsEnumerable().Contains(t.TagId)
or changing the type of the variable.
I understood what was happening and why my code was not behaving as expected (server side evaluation) and why the explicit client side evaluation is actually a very good thing:
IList<T>.Contains was actually evaluated on the client, but List<T>.Contains is evaluated on the server side. Simply replacing IList with List made the code work without explicit client evaluation:
public Dictionary<int, Tag> GetTagMap(List<int> tagIds = null)
{
var tags = context.Tag.AsNoTracking();
if (tagIds != null)
tags = tags.Where(t => tagIds.Contains(t.TagId));
return tags
.ToList() // explicit client evaluation in 3.0
.ToDictionary(t => t.TagId, t => t);
}
Related
I'm working in Entity Framework Core 2.0, and I'm trying to do something like this. I want a list of all the classes taught by the oldest teacher. I tried to make the two requests into one query, but I couldn't make it work. Is there a concise way to make this one query?
private async Task<Teacher> GetOldestTeacher(int schoolId)
{
using (var db = new SchoolContext())
{
return await db.Teacher
.Where(t => t.SchoolId == schoolId)
.OrderByDescending(t => t.DateOfBirth)
.FirstAsync();
}
}
public async Task<IEnumerable<Class>> GetOldestTeachersClasses(int schoolId)
{
var oldestTeacher = await GetOldestTeacher(schoolId);
using (var db = new SchoolContext())
{
return await db.Class
.Where(c => c.TeacherId == oldestTeacher.Id && c.SchoolId == schoolId)
.ToListAsync();
}
}
This isn't exactly my code, but it's close enough to what I'm shooting for. This works, but I'm looking to make it more efficient. Any help would be appreciated.
Assuming there exists a navigation property called Classes in the Teacher class, you could do this:
public async Task<IEnumerable<Class>> GetOldestTeachersClasses(int schoolId)
{
using (var db = new SchoolContext())
{
return await db.Teacher
.Where(t => t.SchoolId == schoolId)
.OrderByDescending(t => t.DateOfBirth)
.Take(1)
.SelectMany(t => t.Classes)
.ToListAsync();
}
}
If you use First, you inevitably ask EF to immediately load the object into memory. Because of that, if you use that returned object to do the further parts of the query, you would actually be doing at least 2 roundtrips to the database and performing 2 queries instead of one despite in your C# code there being only one expression.
The trick here is that you filter the teachers just as you've already done, and instead of loading the first one using First(), you Take() the topmost one, according to the ordering requirements. The key difference is that Take will not actually load the object immediately - instead, it enables you to further specify the query which will be translated to one proper SQL query.
The reason for using SelectMany() is that semantically, there still can be more than one teachers as the Take() call actually returns an IQueryable<Teacher> object. But the semantics of your filter criteria ensures that in reality, there will only be 0 or 1 teacher, so collecting his/her classes using SelectMany will result in only his/her taught classes. You could also do the same by doing a Join as DevilSuichiro remarked in the comments.
However, make sure you verify the generated SQL. I personally don't yet have experience with EF Core 2.0 only with 1.x, which sometimes failed to translate rather trivial queries to proper SQL statements and instead performed a lot of work in my app. The last time I checked the 2.0 roadmap it was promised to significantly improve the translator, but I've written this code just from the top of my head so a proper verification is required.
I am trying to get an entity with EF by having an initial sql as input.
I tried the context.Entities.SQLQuery method but this returns a DBSet when I require an IQueriable.
I learned that I cannot transform DBSet to IQueryable because the first is already a result of data while the second is the container for the results of a "query" (executed yet or not). Correct me if i'm wrong :)
So I thought that when I write the following lambda I get the resulting query:
db.MyTable.Where(x => x.id == "123")
Becomes:
SELECT * FROM myTable WHERE id = '123'
With this I thought if I can set directly my query without needing to set my lambda...
Is that an option?
Or an alternative?
Thanks!
It's a bit unclear what you mean:
I am trying to get an entity with EF having an initial sql as input.
I'd interpret this, as that you have an SQL statement as input of something and you want to get an entity framework entity that would have this statement (whatever having a statement means)? Not understandable!
I learned that I cannot transform DBSet to IQueryable because the
first (the DbSet) is already a result of data while the second (the IQueryable) is the container for the results of a "query"
NOT!
Every DbSet<T> implements IQueryable<T>, meaning that if you have an object of class DbSet<t>, this object implements all functionality of IQueryable<T>. Just using this IQueryable does not execute the query. The query will only be executed once the first element of the sequence if requested.
using (var dbContext = new MyDbcontext())
{
var result = dbContext.MyItems
.Where(item => ...)
.Select(item => new
{
X = item.Property1,
Y = item.Property2,
...
};
Until here, the first element of the sequence is not asked, the query is not performed yet. No communication with the database was needed (except to create the dbContext object)
Only if you use execution functions like ToList(),Count(), First(), Max(), etc, the query is performed.
You can check this, because you get exceptoin if you do these kind of functions after the using block:
Wrong
IQueryable largeItems;
using (var dbContext = new MyDbcontext())
{
largeItems = dbContext.MyItems
.Where(item => item.Size > 1000);
// query not executed yet
}
int nrOfLargeItems = largeItems.Count();
// exception, query executed after dbContext is disposed
correct
int nrOfLargeItems;
using (var dbContext = new MyDbcontext())
{
var largeItems = dbContext.MyItems
.Where(item => item.Size > 1000);
// query not executed yet
nrOfLargeItems = largeItems.Count();
// the query is performed
}
Conclusion: users of a DbSet<T> inside a dbContext can use the DbSet<T> as if it was an IQueryable<T>, the query will not be executed until you perform any function that needs the first element of the query.
This includes complex functions like Join, GroupBy, OrderBy, etc. You can recognize these functions because MSDN add the following to the remarks section
This method is implemented by using deferred execution. The immediate return value is an object that stores all the information that is required to perform the action. The query represented by this method is not executed until the object is enumerated either by calling its GetEnumerator method directly or by using foreach.
I am trying to get Envelope's back from a query. Envelope is defined as follows.
[<CLIMutable>]
type Envelope<'T> = {
Id : Guid
StreamId: Guid
Created : DateTimeOffset
Item : 'T }
MyLibAAS.DataStore.MyLibAASDbContext is a EF DbContext written in c#. When I extend it in f# as follows, I get the error: Only parameterless constructors and initializers are supported in LINQ to Entities.
type MyLibAAS.DataStore.MyLibAASDbContext with
member this.GetEvents streamId =
query {
for event in this.Events do
where (event.StreamId = streamId)
select {
Id = event.Id;
StreamId = streamId;
Created = event.Timestamp;
Item = (JsonConvert.DeserializeObject<QuestionnaireEvent> event.Payload)
}
}
If I return the event and map it to Envelope after the fact, it works fine.
type MyLibAAS.DataStore.MyLibAASDbContext with
member this.GetEvents streamId =
query {
for event in this.Events do
where (event.StreamId = streamId)
select event
} |> Seq.map (fun event ->
{
Id = event.Id
StreamId = streamId
Created = event.Timestamp
Item = (JsonConvert.DeserializeObject<QuestionnaireEvent> event.Payload)
}
)
Why does this make a difference? The Envelope type is not even a EF type.
How F# records work
F# records get compiled into .NET classes with read-only properties and a constructor that takes values of all fields as parameters (plus a few interfaces).
For example, your record would be expressed in C# roughly as follows:
public class Envelope<T> : IComparable<Envelope<T>>, IEquatable<Envelope<T>>, ...
{
public Guid Id { get; private set; }
public Guid StreamId { get; private set; }
public DateTimeOffset Created { get; private set; }
public T Item { get; private set; }
public Envelope( Guid id, Guid streamId, DateTimeOffset created, T item ) {
this.Id = id;
this.StreamId = streamId;
this.Created = created;
this.Item = item;
}
// Plus implementations of IComparable, IEquatable, etc.
}
When you want to create an F# record, the F# compiler emits a call to this constructor, supplying values for all fields.
For example, the select part of your query would look in C# like this:
select new Envelope<QuestionnaireEvent>(
event.Id, streamId, event.Timestamp,
JsonConvert.DeserializeObject<QuestionnaireEvent>(event.Payload) )
Entity Framework limitations
It so happens that Entity Framework does not allow calling non-default constructors in queries. There is a good reason: if it did allow it, you could, in principle, construct a query like this:
from e in ...
let env = new Envelope<E>( e.Id, ... )
where env.Id > 0
select env
Entity Framework wouldn't know how to compile this query, because it doesn't know that the value of e.Id passed to the constructor becomes the value of the property env.Id. This is always true for F# records, but not for other .NET classes.
Entity Framework could, in principle, recognize that Envelope is an F# record and apply the knowledge of the connection between constructor arguments and record properties. But it doesn't. Unfortunately, the designers of Entity Framework did not think of F# as a valid use case.
(fun fact: C# anonymous types work the same way, and EF does make an exception for them)
How to fix this
In order to make this work, you need to declare Envelope as a type with default constructor. The only way to do this is to make it a class, not a record:
type Envelope<'T>() =
member val Id : Guid = Guid.Empty with get, set
member val StreamId : Guid = Guid.Empty with get, set
member val Created : DateTimeOffset = DateTimeOffset.MinValue with get, set
member val Item : 'T = Unchecked.defaultof<'T> with get, set
And then create it using property initialization syntax:
select Envelope<_>( Id = event.Id, StreamId = streamId, ... )
Why does moving the select to a Seq.map work
The Seq.map call is not part of the query expression. It does not end up as part of the IQueryable, so it does not end up compiled to SQL by Entity Framework. Instead, EF compiles just what's inside query and returns you the resulting sequence, after fetching it from SQL Server. And only after that you apply Seq.map to that sequence.
The code inside Seq.map is executed on CLR, not compiled to SQL, so it can call anything it wants, including non-default constructors.
This "fix" comes with a cost though: instead of just the fields you need, the whole Event entity gets fetched from DB and materialized. If this entity is heavy, this may have a performance impact.
Another thing to watch out for
Even if you fix the problem by making Envelope a type with default constructor (as suggested above), you'll still hit the next problem: the method JsonConvert.DeserializeObject can't be compiled to SQL, so Entity Framework will complain about it, too. The way you should do it is fetch all fields to the CLR side, then apply whatever non-SQL-compilable transformations you need.
Using LINQ to Entities, everything that happens in the query computational expression is actually executed within the database engine, which may reside on a remote server. Everything outside of it is executed in the running application on the client.
So, in your first snippet, Entity Framework refuses to execute Envelope<'T>'s constructor because, in order to do so, it would need to translate that into SQL commands for the server. This is plainly not something it can guarantee, because the constructor could potentially contain any sort of side effects and .NET framework code - it could request user input, read files from the client's hard disk, whatever.
What EF can do, in your second snippet, is sending its own POCO event objects back to the client, which is then tasked with Seq.mapping them to your fancy Envelopes, which it can do because it's running on your client machine with access to the full .NET framework.
Addendum: So why are parameterless constructors ok? What if I were to call MsgBox() in a parameterless constructor? I think that parameterless constructors work by having the client construct the objects (without knowing the query results), sending them to the server in serialised form, and having the server just fill the object's properties with the query results.
I haven't actually tested that. But F# record types have no parameterless constructors anyway, so the point is moot in your case.
I'm using entity framework 6 and have exception "LINQ to Entities does not recognize the method 'System.String AsNonUnicode(System.String)' method". Here is what I do.
I have table Customers with email column defined as varchar(254) (non-Unicode).
In my mapping I have following line:
Property(t => t.Email).IsUnicode(false);
When I put my query like this:
var q = context.Customers.Where(t => t.Email == "a#gmail.com").Select(t => t.FirstName);
everything is fine, the constant is non-Unicode. However, if I use this code:
string email = "a#gmail.com";
var q = context.Customers.Where(t => t.Email == email);
the variable being generated is Unicode one, which results in not using index.
So, I tried using EntityFunctions.AsNonUnicode extension, which seemed to be designed for this:
string email = "a#gmail.com";
var q = context.Customers.Where(t => t.Email == EntityFunctions.AsNonUnicode(email));
However, I got an exception LINQ to Entities does not recognize the method 'System.String AsNonUnicode(System.String)' method exception when trying to actually query data.
What am I doing wrong?
Make sure you are not using EntityFnctions from System.Data.Entity.dll when using EF6. EF5 and EF6 are not binary compatible and ideally you should remove a reference to System.Data.Entity.dll if you are using EF6 to avoid confusing errors like this. In this particular case make sure you are using EntityFunctions from System.Data.Entity.Core.Objects namespace which is where EntityFunction type lives in EF6
I am currently writing unit tests for my repository implementation in an MVC4 application. In order to mock the data context, I started by adopting some ideas from this post, but I have now discovered some limitations that make me question whether it is even possible to properly mock IQueryable.
In particular, I have seen some situations where the tests pass but the code fails in production and I have not been able to find any way to mock the behavior that causes this failure.
For example, the following snippet is used to select Post entities that fall within a predefined list of categories:
var posts = repository.GetEntities<Post>(); // Returns IQueryable<Post>
var categories = GetCategoriesInGroup("Post"); // Returns a fixed list of type Category
var filtered = posts.Where(p => categories.Any(c => c.Name == p.Category)).ToList();
In my test environment, I have tried mocking posts using the fake DbSet implementation mentioned above, and also by creating a List of Post instances and converting it to IQueryable using the AsQueryable() extension method. Both of these approaches work under test conditions, but the code actually fails in production, with the following exception:
System.NotSupportedException : Unable to create a constant value of type 'Category'. Only primitive types or enumeration types are supported in this context.
Although LINQ issues like this are easy enough to fix, the real challenge is finding them, given that they do not reveal themselves in the test environment.
Am I being unrealistic in expecting that I can mock the behavior of Entity Framework's implementation of IQueryable?
Thanks for your ideas,
Tim.
I think it is very very hard, if impossible, to mock Entity Framework behaviour. First and foremost because it would require profound knowledge of all peculiarities and edge cases where linq-to-entites differs from linq-to-objects. As you say: the real challenge is finding them. Let me point out three main areas without claiming to be even nearly exhaustive:
Cases where Linq-to-Objects succeeds and Linq-to-Entities fails:
.Select(x => x.Property1.ToString(). LINQ to Entities does not recognize the method 'System.String ToString()' method... This applies to nearly all methods in native .Net classes and of course to own methods. Only a few .Net methods will be translated into SQL. See CLR Method to Canonical Function Mapping. As of EF 6.1, ToString is supported by the way. But only the parameterless overload.
Skip() without preceding OrderBy.
Except and Intersect: can produce monstrous queries that throw Some part of your SQL statement is nested too deeply. Rewrite the query or break it up into smaller queries.
Select(x => x.Date1 - x.Date2): DbArithmeticExpression arguments must have a numeric common type.
(your case) .Where(p => p.Category == category): Only primitive types or enumeration types are supported in this context.
Nodes.Where(n => n.ParentNodes.First().Id == 1): The method 'First' can only be used as a final query operation.
context.Nodes.Last(): LINQ to Entities does not recognize the method '...Last...'. This applies to many other IQueryable extension methods. See Supported and Unsupported LINQ Methods.
(See Slauma's comment below): .Select(x => new A { Property1 = (x.BoolProperty ? new B { BProp1 = x.Prop1, BProp2 = x.Prop2 } : new B { BProp1 = x.Prop1 }) }): The type 'B' appears in two structurally incompatible initializations within a single LINQ to Entities query... from here.
context.Entities.Cast<IEntity>(): Unable to cast the type 'Entity' to type 'IEntity'. LINQ to Entities only supports casting EDM primitive or enumeration types.
.Select(p => p.Category?.Name). Using null propagation in an expression throws CS8072 An expression tree lambda may not contain a null propagating operator. This may get fixed one day.
This question: Why does this combination of Select, Where and GroupBy cause an exception? made me aware of the fact that there are even entire query constructions that are not supported by EF, while L2O wouldn't have any trouble with them.
Cases where Linq-to-Objects fails and Linq-to-Entities succeeds:
.Select(p => p.Category.Name): when p.Category is null L2E returns null, but L2O throws Object reference not set to an instance of an object. This can't be fixed by using null propagation (see above).
Nodes.Max(n => n.ParentId.Value) with some null values for n.ParentId. L2E returns a max value, L2O throws Nullable object must have a value.
Using EntityFunctions (DbFunctions as of EF 6) or SqlFunctions.
Cases where both succeed/fail but behave differently:
Nodes.Include("ParentNodes"): L2O has no implementation of include. It will run and return nodes (if Nodes is IQueryable), but without parent nodes.
Nodes.Select(n => n.ParentNodes.Max(p => p.Id)) with some empty ParentNodes collections: both fail but with different exceptions.
Nodes.Where(n => n.Name.Contains("par")): L2O is case sensitive, L2E depends on the database collation (often not case sensitive).
node.ParentNode = parentNode: with a bidirectional relationship, in L2E this will also add the node to the nodes collection of the parent (relationship fixup). Not in L2O. (See Unit testing a two way EF relationship).
Work-around for failing null propagation: .Select(p => p.Category == null ? string.Empty : p.Category.Name): the result is the same, but the generated SQL query also contains the null check and may be harder to optimize.
Nodes.AsNoTracking().Select(n => n.ParentNode. This one is very tricky!. With AsNoTracking EF creates new ParentNode objects for each Node, so there can be duplicates. Without AsNoTracking EF reuses existing ParentNodes, because now the entity state manager and entity keys are involved. AsNoTracking() can be called in L2O, but it doesn't do anything, so there will never be a difference with or without it.
And what about mocking lazy/eager loading and the effect of context life cycle on lazy loading exceptions? Or the effect of some query constructs on performance (like constructs that trigger N+1 SQL queries). Or exceptions due to duplicate or missing entity keys? Or relationship fixup?
My opinion: nobody is going to fake that. The most alarming area is where L2O succeeds and L2E fails. Now what's the value of green unit tests? It has been said before that EF can only reliably be tested in integration tests (e.g. here) and I tend to agree.
However, that does not mean that we should forget about unit tests in projects with EF as data layer. There are ways to do it, but, I think, not without integration tests.
I have written a few Unit Tests with Entity Framework 6.1.3 using Moq and used it to override IQueryable. Note that all DbSet that should be tested needs to be marked as virtual. Example from Microsoft themselves:
Query:
using Microsoft.VisualStudio.TestTools.UnitTesting;
using Moq;
using System.Collections.Generic;
using System.Data.Entity;
using System.Linq;
namespace TestingDemo
{
[TestClass]
public class QueryTests
{
[TestMethod]
public void GetAllBlogs_orders_by_name()
{
var data = new List<Blog>
{
new Blog { Name = "BBB" },
new Blog { Name = "ZZZ" },
new Blog { Name = "AAA" },
}.AsQueryable();
var mockSet = new Mock<DbSet<Blog>>();
mockSet.As<IQueryable<Blog>>().Setup(m => m.Provider).Returns(data.Provider);
mockSet.As<IQueryable<Blog>>().Setup(m => m.Expression).Returns(data.Expression);
mockSet.As<IQueryable<Blog>>().Setup(m => m.ElementType).Returns(data.ElementType);
mockSet.As<IQueryable<Blog>>().Setup(m => m.GetEnumerator()).Returns(0 => data.GetEnumerator());
var mockContext = new Mock<BloggingContext>();
mockContext.Setup(c => c.Blogs).Returns(mockSet.Object);
var service = new BlogService(mockContext.Object);
var blogs = service.GetAllBlogs();
Assert.AreEqual(3, blogs.Count);
Assert.AreEqual("AAA", blogs[0].Name);
Assert.AreEqual("BBB", blogs[1].Name);
Assert.AreEqual("ZZZ", blogs[2].Name);
}
}
}
Insert:
using Microsoft.VisualStudio.TestTools.UnitTesting;
using Moq;
using System.Data.Entity;
namespace TestingDemo
{
[TestClass]
public class NonQueryTests
{
[TestMethod]
public void CreateBlog_saves_a_blog_via_context()
{
var mockSet = new Mock<DbSet<Blog>>();
var mockContext = new Mock<BloggingContext>();
mockContext.Setup(m => m.Blogs).Returns(mockSet.Object);
var service = new BlogService(mockContext.Object);
service.AddBlog("ADO.NET Blog", "http://blogs.msdn.com/adonet");
mockSet.Verify(m => m.Add(It.IsAny<Blog>()), Times.Once());
mockContext.Verify(m => m.SaveChanges(), Times.Once());
}
}
}
Example service:
using System.Collections.Generic;
using System.Data.Entity;
using System.Linq;
using System.Threading.Tasks;
namespace TestingDemo
{
public class BlogService
{
private BloggingContext _context;
public BlogService(BloggingContext context)
{
_context = context;
}
public Blog AddBlog(string name, string url)
{
var blog = _context.Blogs.Add(new Blog { Name = name, Url = url });
_context.SaveChanges();
return blog;
}
public List<Blog> GetAllBlogs()
{
var query = from b in _context.Blogs
orderby b.Name
select b;
return query.ToList();
}
public async Task<List<Blog>> GetAllBlogsAsync()
{
var query = from b in _context.Blogs
orderby b.Name
select b;
return await query.ToListAsync();
}
}
}
Source: https://learn.microsoft.com/en-us/ef/ef6/fundamentals/testing/mocking