EF Cannot be translated - entity-framework

i have problem with translated query, ToList(), AsEnumerable etc.
I need construct or create query which is shared.
Branches -> Customer -> some collection -> some collection
Customer -> some collection -> some collection.
Do you help me how is the best thingh how to do it and share the query.
i access to repository via graphql use projection etc.
public IQueryable<CustomerTableGraphQL> BranchTableReportTest(DateTime actualTime, long userId)
{
var r =
(
from b in _dbContext.Branches
let t = Customers(b.Id).ToList()
select new CustomerTableGraphQL
{
Id = b.Id,
Name = b.Name,
Children =
(
from c in t
select new CustomerTableGraphQL
{
Id = c.Id,
Name = c.Name
}
)
.AsEnumerable()
}
);
return r;
}
public IQueryable<Customer> Customers(long branchId) =>
_dbContext.Customers.Where(x => x.BranchId.Value == branchId).ToList().AsQueryable();
Some example how to doit and share iquearable between query

Using ToList / AsEnumerable etc. entirely defeats the potential benefits of using IQueryable. If your code needs to do this rather than return an IQueryable<TEntity> then you should be returning IEnumerable<TResult> where TResult is whatever entity or DTO/ViewModel you want to return.
An example of an IQueryable<TEntity> repository pattern would be something like this:
public IQueryable<Customer> GetCustomersByBranch(long branchId) =>
_dbContext.Customers.Where(x => x.BranchId.Value == branchId);
Normally I wouldn't really even have a repository method for that, I'd just use:
public IQueryable<Customer> GetCustomers() =>
_dbContext.Customers.AsQueryable();
... as the "per branch" is simple enough for the consumer to request without adding methods for every possible filter criteria. The AsQueryable in this case is only needed because I want to ensure the result matches the IQueryable type casting. When your expression has a Where clause then this is automatically interpreted as being an IQueryable result.
So a caller calling the Repository's "GetCustomers()" method would look like:
// get customer details for our branch.
var customers = _Repository.GetCustomers()
.Where(x => x.BranchId == branchId)
.OrderBy(x => x.LastName)
.ThenBy(x => x.FirstName)
.Select(x => new CustomerSummaryViewModel
{
CustomerId = x.Id,
FirstName = x.FirstName,
LastName = x.LastName,
// ...
}).Skip(pageNumber * pageSize)
.Take(pageSize)
.ToList();
In this example the repository exposes a base query to fetch data, but without executing/materializing anything. The consumer of that call is then free to:
Filter the data by branch,
Sort the data,
Project the data down to a desired view model
Paginate the results
... before the query is actually run. This pulls just that page of data needed to populate the VM after filters and sorts as part of the query. That Repository method can serve many different calls without needing parameters, code, or dedicated methods to do all of that.
Repositories returning IQueryable that just expose DbSets aren't really that useful. The only purpose they might provide is making unit testing a bit easier as Mocking the repository is simpler than mocking a DbContext & DbSets. Where the Repository pattern does start to help is in enforcing standardized rules/filters on data. Examples like soft delete flags or multi-tenant systems where rows might belong to different clients so a user should only ever search/pull across one tenant's data. This also extends to details like authorization checks before data is returned. Some of this can be managed by things like global query filters but wherever there are common rules to enforce about what data is able to be retrieved, the Repository can serve as a boundary to ensure those rules are applied consistently. For example with a soft-delete check:
public IQueryable<Customer> GetCustomers(bool includeInactive = false)
{
var query = _context.Customers.AsQueryable();
if (!includeInactive)
query = query.Where(x => x.IsActive);
return query;
}
A repository can be given a dependency for locating the current logged in user and retrieving their roles, tenant information, etc. then use that to ensure that:
a user is logged in.
The only data retrieved is available to that user.
An appropriate exception is raised if specific data is requested that this user should never be able to access.
An IQueryable repository does require a Unit of Work scope pattern to work efficiently within an application. IQueryable queries do no execute until something like a ToList or Single, Any, Count, etc. are called. This means that the caller of the repository ultimately needs to be managing the scope of the DbContext that the repository is using, and this sometimes rubs developers the wrong way because they feel the Repository should be a layer of abstraction between the callers (Services, Controllers, etc.) and the data access "layer". (EF) To have that abstraction means adding a lot of complexity that ultimately has to conform to the rules of EF (or even more complexity to avoid that) or significantly hamper performance. In cases where there is a clear need or benefit to tightly standardizing a common API-like approach for a Repository that all systems will conform to, then an IQueryable pattern is not recommended over a general IEnumerable typed result. The benefit of IQueryable is flexibility and performance. Consumers decide and optimize for how the data coming from the Repository is consumed. This flexibility extends to cover both synchronous and asynchronous use cases.

EF Core will translate only inlined query code. This query will work:
public IQueryable<CustomerTableGraphQL> BranchTableReportTest(DateTime actualTime, long userId)
{
var r =
(
from b in _dbContext.Branches
select new CustomerTableGraphQL
{
Id = b.Id,
Name = b.Name,
Children =
(
from c in _dbContext.Customers
where c.BranchId == b.Id
select new CustomerTableGraphQL
{
Id = c.Id,
Name = c.Name
}
)
.AsEnumerable()
}
);
return r;
}
If you plan to reuse query parts, you have to deal with LINQKit and its ExpandableAttribute (will show sample on request)

Related

Entity Framework Multiple Include at runtime

I have a service that passes in parameters for how much I want to include for navigation properties. Based upon the boolean args it concatenates an entity list to include each required foreign entity.
At runtime I want to include either no navigation entities or many.
What I can't do is daisy chain with .Include().Include as I don't know which and how many to include based around passed in args.
I want to achieve this, but I don't seem to be able to pass in a comma separated entity list. Any ideas?
var res = db.Entity.Include(entityListCommaSeparated).Where(_=>_.ID == ID).FirstOrDefault();
This looks like a repository pattern, and generally gets messy if you want to try and "hide" EF / the DbContext from calling code.
A couple options you can consider:
Down the complexity rabit hole: use a params Expression<Func<TEntity, object>>[] includes in your applicable repository methods, and then be prepared to also pass OrderBy expressions, as well as pagination values when you want to return multiple entities.
THrough the simplicity mirror: Embrace IQueryable as a return type and let the consumers handle Includes, OrderBy's, Counts/Any/Skip/Take/First/ToList, and .Select() as they need.
Option 1:
public Order GetById(int id, params Expression<Func<Order, object>>[] includes)
{
var query = db.Orders.Where(x => x.ID == id);
// This part can be moved into an extension method or a base repository method.
if(includes.Any)
includes.Aggregate(query, (current, include) =>
{
current.Include(include);
}
// Don't use .FirstOrDefault() If you intend for 1 record to be returned, use .Single(). If it really is optional to find, .SingleOrDefault()
return query.Single();
}
//ToDo
public IEnumerable<Order> GetOrders(/* criteria?, includes?, order by?, (ascending/descending) pagination? */)
{ }
// or
public IEnumerable<Order> GetOrdersByCustomer(/* includes?, order by?, (ascending/descending) pagination? */)
{ }
// plus..
public IEnumerable<Order> GetOrdersByDate(/* includes?, order by?, (ascending/descending) pagination? */)
{ }
public bool CustomerHasOrders(int customerId)
{ }
public bool OrderExists(int id)
{ }
public int OrdersOnDate(DateTime date)
{ }
// etc. etc. etc.
Keep in mind this doesn't handle custom order by clauses, and the same will be needed for methods that are returning lists of entities. Your repository is also going to need to expose methods for .Any() (DoesExist) because everyone loves checking for #null on every return. :) Also .Count().
Option 2:
public IQueryable<Order> GetById(int id)
{
return db.Orders.Where(x => x.ID == id);
}
public IQueryable<Order> GetOrders()
{
return db.Orders.AsQueryable();
}
Callers can grok Linq and .Include() what they want before calling .Single(), or do a .Any().. They may not need the entire entity graph so they can .Select() from the entity and related entities without .Include() to compose and execute a more efficient query to populate a ViewModel / DTO. GetById might be used in a number of places so we can reduce duplication and support it in the repository. We don't need all of the filter scenarios etc, callers can call GetOrders and then filter as they see fit.
Why bother with a repository if it just returns DBSets?
Centralize low-level data filtering. For instance if you use Soft Deletes (IsActive) or are running multi-tenant, or explicit authorization. These common rules can be centralized at the repository level rather than having to remembered everywhere a DbSet is touched.
Testing is simpler. While you can mock a DbContext, or point it at an in-memory database, mocking a repository returning IQueryable is simpler. (Just populate a List<TEntity> and return .AsQueryable().
Repositories handle Create and Delete. Create to serve as a factory to ensure that all required data and relationships are established for a viable entity. Delete to handle soft-delete scenarios, cascades/audits etc. beyond what the DB handles behind the scenes.

In Entity Framework Core, with one query, how do I select objects by the order of a parent object?

I'm working in Entity Framework Core 2.0, and I'm trying to do something like this. I want a list of all the classes taught by the oldest teacher. I tried to make the two requests into one query, but I couldn't make it work. Is there a concise way to make this one query?
private async Task<Teacher> GetOldestTeacher(int schoolId)
{
using (var db = new SchoolContext())
{
return await db.Teacher
.Where(t => t.SchoolId == schoolId)
.OrderByDescending(t => t.DateOfBirth)
.FirstAsync();
}
}
public async Task<IEnumerable<Class>> GetOldestTeachersClasses(int schoolId)
{
var oldestTeacher = await GetOldestTeacher(schoolId);
using (var db = new SchoolContext())
{
return await db.Class
.Where(c => c.TeacherId == oldestTeacher.Id && c.SchoolId == schoolId)
.ToListAsync();
}
}
This isn't exactly my code, but it's close enough to what I'm shooting for. This works, but I'm looking to make it more efficient. Any help would be appreciated.
Assuming there exists a navigation property called Classes in the Teacher class, you could do this:
public async Task<IEnumerable<Class>> GetOldestTeachersClasses(int schoolId)
{
using (var db = new SchoolContext())
{
return await db.Teacher
.Where(t => t.SchoolId == schoolId)
.OrderByDescending(t => t.DateOfBirth)
.Take(1)
.SelectMany(t => t.Classes)
.ToListAsync();
}
}
If you use First, you inevitably ask EF to immediately load the object into memory. Because of that, if you use that returned object to do the further parts of the query, you would actually be doing at least 2 roundtrips to the database and performing 2 queries instead of one despite in your C# code there being only one expression.
The trick here is that you filter the teachers just as you've already done, and instead of loading the first one using First(), you Take() the topmost one, according to the ordering requirements. The key difference is that Take will not actually load the object immediately - instead, it enables you to further specify the query which will be translated to one proper SQL query.
The reason for using SelectMany() is that semantically, there still can be more than one teachers as the Take() call actually returns an IQueryable<Teacher> object. But the semantics of your filter criteria ensures that in reality, there will only be 0 or 1 teacher, so collecting his/her classes using SelectMany will result in only his/her taught classes. You could also do the same by doing a Join as DevilSuichiro remarked in the comments.
However, make sure you verify the generated SQL. I personally don't yet have experience with EF Core 2.0 only with 1.x, which sometimes failed to translate rather trivial queries to proper SQL statements and instead performed a lot of work in my app. The last time I checked the 2.0 roadmap it was promised to significantly improve the translator, but I've written this code just from the top of my head so a proper verification is required.

f#: Only parameterless constructors and initializers are supported in LINQ to Entities

I am trying to get Envelope's back from a query. Envelope is defined as follows.
[<CLIMutable>]
type Envelope<'T> = {
Id : Guid
StreamId: Guid
Created : DateTimeOffset
Item : 'T }
MyLibAAS.DataStore.MyLibAASDbContext is a EF DbContext written in c#. When I extend it in f# as follows, I get the error: Only parameterless constructors and initializers are supported in LINQ to Entities.
type MyLibAAS.DataStore.MyLibAASDbContext with
member this.GetEvents streamId =
query {
for event in this.Events do
where (event.StreamId = streamId)
select {
Id = event.Id;
StreamId = streamId;
Created = event.Timestamp;
Item = (JsonConvert.DeserializeObject<QuestionnaireEvent> event.Payload)
}
}
If I return the event and map it to Envelope after the fact, it works fine.
type MyLibAAS.DataStore.MyLibAASDbContext with
member this.GetEvents streamId =
query {
for event in this.Events do
where (event.StreamId = streamId)
select event
} |> Seq.map (fun event ->
{
Id = event.Id
StreamId = streamId
Created = event.Timestamp
Item = (JsonConvert.DeserializeObject<QuestionnaireEvent> event.Payload)
}
)
Why does this make a difference? The Envelope type is not even a EF type.
How F# records work
F# records get compiled into .NET classes with read-only properties and a constructor that takes values of all fields as parameters (plus a few interfaces).
For example, your record would be expressed in C# roughly as follows:
public class Envelope<T> : IComparable<Envelope<T>>, IEquatable<Envelope<T>>, ...
{
public Guid Id { get; private set; }
public Guid StreamId { get; private set; }
public DateTimeOffset Created { get; private set; }
public T Item { get; private set; }
public Envelope( Guid id, Guid streamId, DateTimeOffset created, T item ) {
this.Id = id;
this.StreamId = streamId;
this.Created = created;
this.Item = item;
}
// Plus implementations of IComparable, IEquatable, etc.
}
When you want to create an F# record, the F# compiler emits a call to this constructor, supplying values for all fields.
For example, the select part of your query would look in C# like this:
select new Envelope<QuestionnaireEvent>(
event.Id, streamId, event.Timestamp,
JsonConvert.DeserializeObject<QuestionnaireEvent>(event.Payload) )
Entity Framework limitations
It so happens that Entity Framework does not allow calling non-default constructors in queries. There is a good reason: if it did allow it, you could, in principle, construct a query like this:
from e in ...
let env = new Envelope<E>( e.Id, ... )
where env.Id > 0
select env
Entity Framework wouldn't know how to compile this query, because it doesn't know that the value of e.Id passed to the constructor becomes the value of the property env.Id. This is always true for F# records, but not for other .NET classes.
Entity Framework could, in principle, recognize that Envelope is an F# record and apply the knowledge of the connection between constructor arguments and record properties. But it doesn't. Unfortunately, the designers of Entity Framework did not think of F# as a valid use case.
(fun fact: C# anonymous types work the same way, and EF does make an exception for them)
How to fix this
In order to make this work, you need to declare Envelope as a type with default constructor. The only way to do this is to make it a class, not a record:
type Envelope<'T>() =
member val Id : Guid = Guid.Empty with get, set
member val StreamId : Guid = Guid.Empty with get, set
member val Created : DateTimeOffset = DateTimeOffset.MinValue with get, set
member val Item : 'T = Unchecked.defaultof<'T> with get, set
And then create it using property initialization syntax:
select Envelope<_>( Id = event.Id, StreamId = streamId, ... )
Why does moving the select to a Seq.map work
The Seq.map call is not part of the query expression. It does not end up as part of the IQueryable, so it does not end up compiled to SQL by Entity Framework. Instead, EF compiles just what's inside query and returns you the resulting sequence, after fetching it from SQL Server. And only after that you apply Seq.map to that sequence.
The code inside Seq.map is executed on CLR, not compiled to SQL, so it can call anything it wants, including non-default constructors.
This "fix" comes with a cost though: instead of just the fields you need, the whole Event entity gets fetched from DB and materialized. If this entity is heavy, this may have a performance impact.
Another thing to watch out for
Even if you fix the problem by making Envelope a type with default constructor (as suggested above), you'll still hit the next problem: the method JsonConvert.DeserializeObject can't be compiled to SQL, so Entity Framework will complain about it, too. The way you should do it is fetch all fields to the CLR side, then apply whatever non-SQL-compilable transformations you need.
Using LINQ to Entities, everything that happens in the query computational expression is actually executed within the database engine, which may reside on a remote server. Everything outside of it is executed in the running application on the client.
So, in your first snippet, Entity Framework refuses to execute Envelope<'T>'s constructor because, in order to do so, it would need to translate that into SQL commands for the server. This is plainly not something it can guarantee, because the constructor could potentially contain any sort of side effects and .NET framework code - it could request user input, read files from the client's hard disk, whatever.
What EF can do, in your second snippet, is sending its own POCO event objects back to the client, which is then tasked with Seq.mapping them to your fancy Envelopes, which it can do because it's running on your client machine with access to the full .NET framework.
Addendum: So why are parameterless constructors ok? What if I were to call MsgBox() in a parameterless constructor? I think that parameterless constructors work by having the client construct the objects (without knowing the query results), sending them to the server in serialised form, and having the server just fill the object's properties with the query results.
I haven't actually tested that. But F# record types have no parameterless constructors anyway, so the point is moot in your case.

How to support OData query syntax but return non-Edm models

Exposing my EF models to an API always seemed wrong. I'd like my API to return a custom entity model to the caller but use EF on the back.
So I may have PersonRestEntity and a controller for CRUD ops against that and a Person EF code-first entity behind in and map values.
When I do this, I can no longer use the following to allow ~/people?$top=10 etc. in the URL
[EnableQuery]
public IQueryable<Person> Get(ODataQueryOptions<Person> query) { ... }
Because that exposes Person which is private DB implementation.
How can I have my cake and eat it?
I found a way. The trick is not to just return the IQueryable from the controller, because you need to materialise the query first. This doesn't mean materialising the whole set into RAM, the query is still run at the database, but by explicitly applying the query and materialising the results you can return mapped entities thereafter.
Define this action, specifying the DbSet entity type:
public async Task<HttpResponseMessage> Get(ODataQueryOptions<Person> oDataQuery)
And then apply the query manually to the DbSet<Person> like so:
var queryable = oDataQuery.ApplyTo(queryableDbSet);
Then use the following to run the query and turn the results into the collection of entities you publicly expose:
var list = await queryable.ToListAsync(cancellationToken);
return list
.OfType<Person>()
.Select(p => MyEntityMapper.MapToRestEntity(p));
Then you can return the list in an HttpResponseMessage as normal.
That's it, though obviously where the property names between the entities don't match or are absent on either class, there's going to be some issues, so its probably best to ensure the properties you want to include in query options are named the same in both entities.
Else, I guess you could choose to not support filters and just allow $top and $skip and impose a default order yourself. This can be achieved like so, making sure to order the queryable first, then skip, then top. Something like:
IQueryable queryable = people
.GetQueryable(operationContext)
.OrderBy(r => r.Name);
if (oDataQuery.Skip != null)
queryable = oDataQuery.Skip.ApplyTo(queryable, new System.Web.OData.Query.ODataQuerySettings());
if (oDataQuery.Top != null)
queryable = oDataQuery.Top.ApplyTo(queryable, new System.Web.OData.Query.ODataQuerySettings());
var list = await queryable.ToListAsync(operationContext.CreateToken());
return list
.OfType<Person>()
.Select(i => this.BuildPersonEntity(i));
More information:
If you simply use the non-generic ODataQueryOptions you get
Cannot create an EDM model as the action 'Get' on controller 'People'
has a return type 'System.Net.Http.HttpResponseMessage' that does not
implement IEnumerable
And other errors occur under different circumstances.

Entity Framework using Generic Predicates

I use DTO's to map between my Business and Entity Framework layer via the Repository Pattern.
A Standard call would look like
public IClassDTO Fetch(Guid id)
{
var query = from s in _db.Base.OfType<Class>()
where s.ID == id
select s;
return query.First();
}
Now I wish to pass in filtering criteria from the business layer so I tried
public IEnumerable<IClassDTO> FetchAll(ISpecification<IClassDTO> whereclause)
{
var query = _db.Base.OfType<Class>()
.AsExpandable()
.Where(whereclause.EvalPredicate);
return query.ToList().Cast<IClassDTO>();
}
The Call from the business layer would be something like
Specification<IClassDTO> school =
new Specification<IClassDTO>(s => s.School.ID == _schoolGuid);
IEnumerable<IClassDTO> testclasses = _db.FetchAll(school);
The problem I am having is that the .Where clause on the EF query cannot be inferred from the usage. If I use concrete types in the Expression then it works find but I do not want to expose my business layer to EF directly.
Try making FetchAll into a generic on a class instead, like this:-
public IEnumerable<T> FetchAll<T> (Expression<Func<T,bool>> wherePredicate)
where T:IClassDTO //not actually needed
{
var query = _db.Base.OfType<T>()
.AsExpandable()
.Where(wherePredicate);
return query;
}
pass in school.Evalpredicate instead. FetchAll doesn't appear to need to know about the whole specification, it just needs the predicate, right? If you need to cast it to IClassDTO, do that after you have the results in a List.