We are porting an existing application from EF6 to EF Core 3.1. In the process of doing so, we came across a situation in which a query that was previously working now fails at runtime because it contains an Expression that can't be translated.
public class Task : SqlUserUpdatableSequenceBase
{
...
public override Expression<Func<TEntity, bool>> GetSameSequencePredicate<TEntity>()
=> entity => (entity as Task).ProjectId == ProjectId;
...
}
private List<ItemSequence> UpdateSequenceInternal<TEntity>(TEntity entity, ItemSequence sequence)
where TEntity : SqlUserUpdatableSequenceBase
{
...
var query = Context.Set<TEntity>().AsQueryable();
query = query.Where(entity.GetSameSequencePredicate<TEntity>());
...
}
So it seems that the cast (entity as Task) in the Expression is what cannot be translated by EF Core.
I know this behavior of failing when an Expression can't be translated is listed as one of the breaking changes with mayor impact. However, the documentation talks about it in relation to previous versions of EF Core, not EF 6.
Before 3.0, when EF Core couldn't convert an expression that was part of a query to either SQL or a parameter, it automatically evaluated the expression on the client. By default, client evaluation of potentially expensive expressions only triggered a warning.
I have figured out some workarounds to solve this issue, but I am still wondering if EF 6 was able to translate this expression to SQL or was it also silently executing it client side?
Related
Suppose I want to enhance a model like
public class Person
{
public string Name { get; set; }
public string Address { get; set; }
}
so that I use a complex object for Address:
public class Person
{
public string Name { get; set; }
public Address Address { get; set; }
}
EF Core is quite nice in allowing this with the HasConversion feature:
modelBuilder.Entity<Person>
.Property(p => p.Address)
.HasConversion(addr => addr.ToString(), str => Address.FromString(str));
I even tested and this works in queries with == operator: The following will successfully convert to SQL
var whiteHouse = Address.Parse("1600 Pennsylvania Avenue NW");
var matches = from person in people
where person.Address == whiteHouse
select person;
However, suppose I want to string.Contains on the string version of Address, something like
var search = "1600";
var matches = from person in people
where person.Address.ToString().Contains(search)
select person;
This will fail to convert. Is there any feature of EF Core to map the ToString() method or otherwise map a complex object that converts to a string / VARCHAR so that I can write a query like this?
The problem with EF Core value converters and LINQ queries is that the LINQ query is against the CLR entity property, hence the CLR type rather than the provider type. This is partially mentioned under currently Limitations of the value conversion system section of the EF Core documentation:
Use of value conversions may impact the ability of EF Core to translate expressions to SQL. A warning will be logged for such cases. Removal of these limitations is being considered for a future release.
So having query expression against the CLR type combined with the inability to translate custom methods is causing your issue. Technically it's possible to add custom method/property translation, but it's quite complicated because requires a lot of non user friendly infrastructure plumbing code, which makes practically unusable in real life application development.
In this particular case though, you know that the provider type is string, and the database table values are generated by ToString method. So you just need to let the query use the provider type. And you can do that by using cast operator.
Normally C# compiler won't allow you to cast known object type to another known object type if there is no conversion between them. But you can trick it by using the "double cast" technique by first casting to object and then to the desired type. Fortunately EF Core translator supports such casts and properly (sort of) translates them to SQL. By sort of I mean it emits unnecessary (redundant) CAST inside the query, but at least it translates and executes server side.
With that being said, the solution for your example is
where ((string)(object)person.Address).Contains(search)
As a default behavior, EF Core use Server-side evaluation, EF Core try to translate your expression to standard DB provider T-SQL code (based on selected DB provider)
you expression can't translate to T-SQL code and the DB provider can't handle it (Because the logic you write in the overridden version of ToString() is in your C# code and is unknown to the database provider)
You should force EF Core to use client-side evaluation by fetching all data to memory and then query on the loaded entities, something like this:
var search = "1600";
var matches = from person in people.ToList()
where person.Address.ToString().Contains(search)
select person;
Note that fetching all data to memory in huge databases have performance impacts and use client-side evaluation carefully.
I'm using EF's Code First approach with MySQL.
I'm wondering if EF in that approach has built in protection against the SQL Injection, or do I have to create SQL string query in the MySqlCommand and add some parameters to be safe from that attacks ?
I think I don't have to but I'd like to be sure about that.
Edit (code snippets example):
MyContext cont = new MyContext();
cont.Comment.AddObject(new Comment { Content = "my string" });
cont.SaveChanges();
or
string query = "INSERT INTO Comment(Content)VALUES(#myVal)";
MySqlCommand comm = new MySqlCommand(query);
comm.CommandType = CommandType.Text;
comm.Parameters.AddWithValue("#myVal", "my string");
...and later execute that query
The 1st approach is much faster to code for me
Your first code is SQL Injection proof.
Anything that you pass to EntityFramework are passed as Command in the inner IDbCommand.
But beware, if you are executing direct query with EntityFramework.
A sentence from MSDN.
Although query composition is possible in LINQ to Entities, it is
performed through the object model API. Unlike Entity SQL queries,
LINQ to Entities queries are not composed by using string manipulation
or concatenation, and they are not susceptible to traditional SQL
injection attacks. "
EF's LINQ to Entities (as used in your first example) takes care of preventing SQL injection attacks.
I know some differences of LINQ to Entities and LINQ to Objects which the first implements IQueryable and the second implements IEnumerable and my question scope is within EF 5.
My question is what's the technical difference(s) of those 3 methods? I see that in many situations all of them work. I also see using combinations of them like .ToList().AsQueryable().
What do those methods mean, exactly?
Is there any performance issue or something that would lead to the use of one over the other?
Why would one use, for example, .ToList().AsQueryable() instead of .AsQueryable()?
There is a lot to say about this. Let me focus on AsEnumerable and AsQueryable and mention ToList() along the way.
What do these methods do?
AsEnumerable and AsQueryable cast or convert to IEnumerable or IQueryable, respectively. I say cast or convert with a reason:
When the source object already implements the target interface, the source object itself is returned but cast to the target interface. In other words: the type is not changed, but the compile-time type is.
When the source object does not implement the target interface, the source object is converted into an object that implements the target interface. So both the type and the compile-time type are changed.
Let me show this with some examples. I've got this little method that reports the compile-time type and the actual type of an object (courtesy Jon Skeet):
void ReportTypeProperties<T>(T obj)
{
Console.WriteLine("Compile-time type: {0}", typeof(T).Name);
Console.WriteLine("Actual type: {0}", obj.GetType().Name);
}
Let's try an arbitrary linq-to-sql Table<T>, which implements IQueryable:
ReportTypeProperties(context.Observations);
ReportTypeProperties(context.Observations.AsEnumerable());
ReportTypeProperties(context.Observations.AsQueryable());
The result:
Compile-time type: Table`1
Actual type: Table`1
Compile-time type: IEnumerable`1
Actual type: Table`1
Compile-time type: IQueryable`1
Actual type: Table`1
You see that the table class itself is always returned, but its representation changes.
Now an object that implements IEnumerable, not IQueryable:
var ints = new[] { 1, 2 };
ReportTypeProperties(ints);
ReportTypeProperties(ints.AsEnumerable());
ReportTypeProperties(ints.AsQueryable());
The results:
Compile-time type: Int32[]
Actual type: Int32[]
Compile-time type: IEnumerable`1
Actual type: Int32[]
Compile-time type: IQueryable`1
Actual type: EnumerableQuery`1
There it is. AsQueryable() has converted the array into an EnumerableQuery, which "represents an IEnumerable<T> collection as an IQueryable<T> data source." (MSDN).
What's the use?
AsEnumerable is frequently used to switch from any IQueryable implementation to LINQ to objects (L2O), mostly because the former does not support functions that L2O has. For more details see What is the effect of AsEnumerable() on a LINQ Entity?.
For example, in an Entity Framework query we can only use a restricted number of methods. So if, for example, we need to use one of our own methods in a query we would typically write something like
var query = context.Observations.Select(o => o.Id)
.AsEnumerable().Select(x => MySuperSmartMethod(x))
ToList – which converts an IEnumerable<T> to a List<T> – is often used for this purpose as well. The advantage of using AsEnumerable vs. ToList is that AsEnumerable does not execute the query. AsEnumerable preserves deferred execution and does not build an often useless intermediate list.
On the other hand, when forced execution of a LINQ query is desired, ToList can be a way to do that.
AsQueryable can be used to make an enumerable collection accept expressions in LINQ statements. See here for more details: Do i really need use AsQueryable() on collection?.
Note on substance abuse!
AsEnumerable works like a drug. It's a quick fix, but at a cost and it doesn't address the underlying problem.
In many Stack Overflow answers, I see people applying AsEnumerable to fix just about any problem with unsupported methods in LINQ expressions. But the price isn't always clear. For instance, if you do this:
context.MyLongWideTable // A table with many records and columns
.Where(x => x.Type == "type")
.Select(x => new { x.Name, x.CreateDate })
...everything is neatly translated into a SQL statement that filters (Where) and projects (Select). That is, both the length and the width, respectively, of the SQL result set are reduced.
Now suppose users only want to see the date part of CreateDate. In Entity Framework you'll quickly discover that...
.Select(x => new { x.Name, x.CreateDate.Date })
...is not supported (at the time of writing). Ah, fortunately there's the AsEnumerable fix:
context.MyLongWideTable.AsEnumerable()
.Where(x => x.Type == "type")
.Select(x => new { x.Name, x.CreateDate.Date })
Sure, it runs, probably. But it pulls the entire table into memory and then applies the filter and the projections. Well, most people are smart enough to do the Where first:
context.MyLongWideTable
.Where(x => x.Type == "type").AsEnumerable()
.Select(x => new { x.Name, x.CreateDate.Date })
But still all columns are fetched first and the projection is done in memory.
The real fix is:
context.MyLongWideTable
.Where(x => x.Type == "type")
.Select(x => new { x.Name, DbFunctions.TruncateTime(x.CreateDate) })
(But that requires just a little bit more knowledge...)
What do these methods NOT do?
Restore IQueryable capabilities
Now an important caveat. When you do
context.Observations.AsEnumerable()
.AsQueryable()
you will end up with the source object represented as IQueryable. (Because both methods only cast and don't convert).
But when you do
context.Observations.AsEnumerable().Select(x => x)
.AsQueryable()
what will the result be?
The Select produces a WhereSelectEnumerableIterator. This is an internal .Net class that implements IEnumerable, not IQueryable. So a conversion to another type has taken place and the subsequent AsQueryable can never return the original source anymore.
The implication of this is that using AsQueryable is not a way to magically inject a query provider with its specific features into an enumerable. Suppose you do
var query = context.Observations.Select(o => o.Id)
.AsEnumerable().Select(x => x.ToString())
.AsQueryable()
.Where(...)
The where condition will never be translated into SQL. AsEnumerable() followed by LINQ statements definitively cuts the connection with entity framework query provider.
I deliberately show this example because I've seen questions here where people for instance try to 'inject' Include capabilities into a collection by calling AsQueryable. It compiles and runs, but it does nothing because the underlying object does not have an Include implementation anymore.
Execute
Both AsQueryable and AsEnumerable don't execute (or enumerate) the source object. They only change their type or representation. Both involved interfaces, IQueryable and IEnumerable, are nothing but "an enumeration waiting to happen". They are not executed before they're forced to do so, for example, as mentioned above, by calling ToList().
That means that executing an IEnumerable obtained by calling AsEnumerable on an IQueryable object, will execute the underlying IQueryable. A subsequent execution of the IEnumerable will again execute the IQueryable. Which may be very expensive.
Specific Implementations
So far, this was only about the Queryable.AsQueryable and Enumerable.AsEnumerable extension methods. But of course anybody can write instance methods or extension methods with the same names (and functions).
In fact, a common example of a specific AsEnumerable extension method is DataTableExtensions.AsEnumerable. DataTable does not implement IQueryable or IEnumerable, so the regular extension methods don't apply.
ToList()
Execute the query immediately
AsEnumerable()
lazy (execute the query later)
Parameter: Func<TSource, bool>
Load EVERY record into application memory, and then handle/filter them. (e.g. Where/Take/Skip, it will select * from table1, into the memory, then select the first X elements) (In this case, what it did: Linq-to-SQL + Linq-to-Object)
AsQueryable()
lazy (execute the query later)
Parameter: Expression<Func<TSource, bool>>
Convert Expression into T-SQL (with the specific provider), query remotely and load result to your application memory.
That’s why DbSet (in Entity Framework) also inherits IQueryable to get the efficient query.
Do not load every record, e.g. if Take(5), it will generate select top 5 * SQL in the background. This means this type is more friendly to SQL Database, and that is why this type usually has higher performance and is recommended when dealing with a database.
So AsQueryable() usually works much faster than AsEnumerable() as it generate T-SQL at first, which includes all your where conditions in your Linq.
ToList() will being everything in memory and then you will be working on it.
so, ToList().where ( apply some filter ) is executed locally.
AsQueryable() will execute everything remotely i.e. a filter on it is sent to the database for applying.
Queryable doesn't do anything til you execute it. ToList, however executes immediately.
Also, look at this answer Why use AsQueryable() instead of List()?.
EDIT :
Also, in your case once you do ToList() then every subsequent operation is local including AsQueryable(). You can't switch to remote once you start executing locally.
Hope this makes it a little bit more clearer.
Encountered a bad performance on below code.
void DoSomething<T>(IEnumerable<T> objects){
var single = objects.First(); //load everything into memory before .First()
...
}
Fixed with
void DoSomething<T>(IEnumerable<T> objects){
T single;
if (objects is IQueryable<T>)
single = objects.AsQueryable().First(); // SELECT TOP (1) ... is used
else
single = objects.First();
}
For an IQueryable, stay in IQueryable when possible, try not be used like IEnumerable.
Update. It can be further simplified in one expression, thanks Gert Arnold.
T single = objects is IQueryable<T> q?
q.First():
objects.First();
I made a SqlDependency service in my application. It works perfectly when I type the queries by hand but I cannot include wildcards (I don't really know why).
For example:
//Using this SqlCommand will work
new SqlCommand("SELECT [employees].[name] FROM [dbo].[employees]", sqlNotificationConn)
//But this one won't
new SqlCommand("SELECT [employees].* FROM [dbo].[employees]", sqlNotificationConn)
//And this one won't either
new SqlCommand("SELECT * FROM [dbo].[employees]", sqlNotificationConn)
So basically, I want to get my DbContext to generate a full SELECT command with every fields it deals with.
In Linq 2 SQL, I used this service using dbContext.GetCommand(.....);
In EF 4.0 (or was it 4.1?), I used dbContext.employee.ToTraceString();
But in EF 4.4, I can't find anything to generate that SELECT query string....
With DbContext (DbQuery) it is as simple as:
query.ToString()
With ObjectContext (ObjectQuery):
((ObjectQuery)query).ToTraceString()
By the way, query can be any expression based on a DbSet (or ObjectSet, respectively). So something like dbContext.employee.Where(e => e.Name == "Gates").ToString() will also show the generated SQL query.
A LINQ statement that forces execution, like ToList(), Single(), FirstOrDefault(), etc, creates a new object and ToString() will return the object's type name.
ToTraceString() is still in EF, same place it always was. However, it's on ObjectQuery, not DbQuery. Having said that, I've never seen the SQL Server provider for EF use *; it always uses discrete fields in SQL.
I have the code below (this is actually part of a much more complicated query, but I have isolated the issue to this particular line to help with debugging) which per everything I have read should create an IN clause in SQL, assuming I am using EF4. As far as I can tell, I am using EF4 (We are using .NET Framework 4 for our projects and when I look at the System.Data and System.Data.Entity they both say version 4.0.0.0 for all the projects)
int[] assessmentIDs; // this is just here to show what this is,
// but it is a params parameter passed to this methed
var assessments = from cert in container.ProctorAssessmentCertifications
where assessmentIDs.Contains(cert.AssessmentID)
select cert.ID;
However, when I run this, I get the runtime error:
LINQ to Entities does not recognize the method 'Boolean Contains[Int32](Int32[], Int32)' method, and this method cannot be translated into a store expression.
When I use LinqPad, it does correctly output an IN clause like one would expect in EF4.
My questions are:
A. What am I doing wrong and how do I make this work?
B. How do I force EF4 to be called if in fact it's not? I can find no reference in any web.config file that point it to the older version.
Contains does not get translated into valid SQL because assessmentIDs is not IQueryable, it is an in-memory object. So you'll have to pull the data out first, and then do the check.
var assessments = (from cert in container.ProctorAssessmentCertifications
select cert.ID).ToList() //no longer IQueryable.
var result = assessments.Intersect(assessmentIDs);