I am new to EF Core and am trying to figure out why I keep having to increase my SQl timeout setting to make a specific query work. In the process I developed a theory. What if EF has a que of pending requests and when there are too many it just stops working instead of removing older requests from the que. That might explain why increasing timeout time is the only way to keep my browser from showing "err connection closed" when I load pages using a specific query.
I tried posting the query on the Microsoft Q and A website with my problem a couple days ago (https://learn.microsoft.com/en-us/answers/questions/601858/aspnet-core-app-only-works-for-few-minutes-after-r.html), but as usual Microsoft has not responded with an explanation that specifically explains why their product does not work and how to fix it. The best responses I've gotten are vague suggestions that don't really explain what to do.
For instance, one guy suggest not using ToArray() but doesn't say how the same functionality can be obtained without it. Another guy suggests researching connection resiliency but doesn't say how doing that might solve my problem.
So far I have managed to keep the app running longer without this problem occurring by using LazyCache to cache results of the problem query, but after a couple days the app is having the same problem again. Why is it that Microsoft is quick to tell people not to do something, but never tells people what to do instead?
UPDATE: Since Microsoft requires people to sign into their account to access the above link and nobody here want to sign in here is part of what is posted there. It is an error message from running my app locally
An unhandled exception occurred while processing the request.
Win32Exception: The wait operation timed out.
Unknown location
SqlException: Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
Microsoft.Data.SqlClient.SqlCommand+<>c.<ExecuteDbDataReaderAsync>b__169_0(Task<SqlDataReader> result)
Stack Query Cookies Headers Routing
Win32Exception: The wait operation timed out.
Show raw exception details
SqlException: Execution Timeout Expired. The timeout period elapsed prior to completion of the operation or the server is not responding.
Microsoft.Data.SqlClient.SqlCommand+<>c.<ExecuteDbDataReaderAsync>b_169_0(Task<SqlDataReader> result)
System.Threading.Tasks.ContinuationResultTaskFromResultTask<TAntecedentResult, TResult>.InnerInvoke()
System.Threading.Tasks.Task+<>c.<.cctor>b277_0(object obj)
System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, object state)
System.Threading.Tasks.Task.ExecuteWithThreadLocal(ref Task currentTaskSlot, Thread threadPoolThread)
Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
Microsoft.EntityFrameworkCore.Storage.RelationalCommand.ExecuteReaderAsync(RelationalCommandParameterObject parameterObject, CancellationToken cancellationToken)
Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable<T>+AsyncEnumerator.InitializeReaderAsync(DbContext , bool result, CancellationToken cancellationToken)
Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.ExecuteImplementationAsync<TState, TResult>(Func<DbContext, TState, CancellationToken, Task<TResult>> operation, Func<DbContext, TState, CancellationToken, Task<ExecutionResult<TResult>>> verifySucceeded, TState state, CancellationToken cancellationToken)
Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.ExecuteImplementationAsync<TState, TResult>(Func<DbContext, TState, CancellationToken, Task<TResult>> operation, Func<DbContext, TState, CancellationToken, Task<ExecutionResult<TResult>>> verifySucceeded, TState state, CancellationToken cancellationToken)
Microsoft.EntityFrameworkCore.Query.Internal.SingleQueryingEnumerable<T>+AsyncEnumerator.MoveNextAsync()
Microsoft.EntityFrameworkCore.EntityFrameworkQueryableExtensions.ToListAsync<TSource>(IQueryable<TSource> source, CancellationToken cancellationToken)
Microsoft.EntityFrameworkCore.EntityFrameworkQueryableExtensions.ToListAsync<TSource>(IQueryable<TSource> source, CancellationToken cancellationToken)
Microsoft.EntityFrameworkCore.EntityFrameworkQueryableExtensions.ToArrayAsync<TSource>(IQueryable<TSource> source, CancellationToken cancellationToken)
PostAlmostAnything.SiteServices.PostsService.GetPosts() in PostsService.cs
}
public async Task<Posts[]> GetPosts()
{
IAppCache cache = new CachingService(CachingService.DefaultCacheProvider) { DefaultCachePolicy = new CacheDefaults { DefaultCacheDurationSeconds = GlobalStatic.SITEPOSTSCACHEDURATION() } };
string cacheKey = $"GetPosts";
var cacheobject = await (from post in _context.Posts
where post.Active == (bool?)true && post.Adminban == (bool?)false && post.Site == (int?)GlobalStatic.SITENUMBER()
select post into pts
select new Posts
{
Postid = pts.Postid,
Title = pts.Title,
PostAlmostAnything.SiteServices.PostsService.GetPaginatedResult(int currentPage, int pageSize) in PostsService.cs
Sc2url = linkgenerator.subcategory2link($"{pts.CategoryNavigation.Categoryurl}", $"{pts.SubcategoryNavigation.Subcategoryurl}", $"{pts.Subcategory2Navigation.Subcategory2url}")
}
}).ToArrayAsync();
}
public async Task<List<Posts>> GetPaginatedResult(int currentPage, int pageSize)
{
return (await GetPosts()).OrderByDescending((Posts d) => d.Postid).Skip((currentPage - 1) * pageSize).Take(pageSize)
.ToList();
}
public async Task<List<Posts>> GetPaginatedResultByCategoryCity(string catslug, string waslug, string regionslug, string cityslug, int currentPage, int pageSize)
{
return (from d in await GetPosts()
PostAlmostAnything.Pages.IndexModel.OnGetAsync() in Index.cshtml.cs
{
PostService = postService;
}
public async Task<PageResult> OnGetAsync()
{
Posts = await PostService.GetPaginatedResult(CurrentPage, PageSize);
return Page();
}
}
}
Microsoft.AspNetCore.Mvc.RazorPages.Infrastructure.ExecutorFactory+GenericTaskHandlerMethod.Convert<T>(object taskAsObject)
Microsoft.AspNetCore.Mvc.RazorPages.Infrastructure.ExecutorFactory+GenericTaskHandlerMethod.Execute(object receiver, object[] arguments)
Microsoft.AspNetCore.Mvc.RazorPages.Infrastructure.PageActionInvoker.InvokeHandlerMethodAsync()
Microsoft.AspNetCore.Mvc.RazorPages.Infrastructure.PageActionInvoker.InvokeNextPageFilterAsync()
Microsoft.AspNetCore.Mvc.RazorPages.Infrastructure.PageActionInvoker.Rethrow(PageHandlerExecutedContext context)
Microsoft.AspNetCore.Mvc.RazorPages.Infrastructure.PageActionInvoker.Next(ref State next, ref Scope scope, ref object state, ref bool isCompleted)
Microsoft.AspNetCore.Mvc.RazorPages.Infrastructure.PageActionInvoker.InvokeInnerFilterAsync()
Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g_Awaited|24_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, object state, bool isCompleted)
Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)
Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(ref State next, ref Scope scope, ref object state, ref bool isCompleted)
Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>gAwaited|19_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, object state, bool isCompleted)
Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>gLogged|17_1(ResourceInvoker invoker)
Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g_AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
Microsoft.AspNetCore.Diagnostics.StatusCodePagesMiddleware.Invoke(HttpContext context)
Microsoft.AspNetCore.Diagnostics.EntityFrameworkCore.MigrationsEndPointMiddleware.Invoke(HttpContext context)
Microsoft.AspNetCore.Diagnostics.EntityFrameworkCore.DatabaseErrorPageMiddleware.Invoke(HttpContext httpContext)
Microsoft.AspNetCore.Diagnostics.EntityFrameworkCore.DatabaseErrorPageMiddleware.Invoke(HttpContext httpContext)
Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware.Invoke(HttpContext context)
UPDATE: Here is where I am now as far as trying to paginate the query. I cannot find a way to use an IQueryable and access it with an async task without getting errors for lacking GetAwaiter
public IQueryable<Posts> GetPosts(bool active)
{
var query = _context.Posts.Where(post => post.Active == active && post.Adminban == (bool?)false && post.Site == GlobalStatic.SITENUMBER());
return query;
}
public IQueryable<Posts> GetPaginatedResult(bool active, int currentPage, int pageSize)
{
//IAppCache cache = new CachingService(CachingService.DefaultCacheProvider) { DefaultCachePolicy = new CacheDefaults { DefaultCacheDurationSeconds = GlobalStatic.SITEPOSTSCACHEDURATION() } };
//string cacheKey = $"GetPosts";
var cacheobject = GetPosts(active)
.Select(pts => new Posts
{
Postid = pts.Postid,
Title = pts.Title,
Description = pts.Description,
Dateposted = pts.Dateposted,
Datemodified = pts.Datemodified,
Video = pts.Video,
Videostream = pts.Videostream,
Location = pts.Location,
Tags = pts.Tags,
Cap = pts.Cap,
Titletag = pts.Titletag,
Metatag = pts.Metatag,
Link = pts.Link,
Linkurl = pts.Linkurl,
Category = pts.Category,
Subcategory = pts.Subcategory,
Subcategory2 = pts.Subcategory2,
Worldarea = pts.Worldarea,
Region = pts.Region,
City = pts.City,
Sendemail = pts.Sendemail,
Userid = pts.Userid,
Active = pts.Active,
Adminban = pts.Adminban,
Posturl = linkgenerator.postlink($"{pts.Postid}", $"{pts.Title}"),
Comments = (from cm in pts.Comments
where cm.Active == (bool?)true && cm.Adminblock == (bool?)false
select cm into cmts
select new Comments
{
Commentid = cmts.Commentid,
Comment = cmts.Comment,
Date = cmts.Date,
Active = cmts.Active,
Adminblock = cmts.Adminblock,
Userid = cmts.Userid,
Post = cmts.Post,
Ratings = cmts.Ratings.Select((Ratings rts) => new Ratings
{
Commentnumber = rts.Commentnumber,
Rating = rts.Rating
}).ToArray(),
Images = cmts.Images.Select((Images imgs) => new Images
{
Image = imgs.Image,
Imagename = imgs.Imagename
}).ToArray(),
Likes = cmts.Likes.Select((Likes lks) => new Likes
{
Likeid = lks.Likeid,
Commentid = lks.Commentid
}).ToArray()
}).ToArray(),
Commentcount = pts.Comments.Where((Comments cm) => cm.Active == (bool?)true && cm.Adminblock == (bool?)false).Count(),
Ratings = (from r in pts.Ratings
where r.CommentnumberNavigation.Active == (bool?)true && r.CommentnumberNavigation.Adminblock == (bool?)false
select r into rtgs
select new Ratings
{
Ratingid = rtgs.Ratingid,
Rating = rtgs.Rating,
Daterated = rtgs.Daterated,
CommentnumberNavigation = new Comments
{
Commentid = rtgs.CommentnumberNavigation.Commentid
}
}).ToArray(),
Ratingcount = pts.Ratings.Where((Ratings rc) => rc.CommentnumberNavigation.Active == (bool?)true && rc.CommentnumberNavigation.Adminblock == (bool?)false).Count(),
Ratingavg = ratingavg((from rac in pts.Ratings
where rac.CommentnumberNavigation.Active == (bool?)true && rac.CommentnumberNavigation.Adminblock == (bool?)false
select rac into ra
select ra.Rating).Average()),
Images = pts.Images.Select((Images imgs) => new Images
{
Imageid = imgs.Imageid,
Imagename = imgs.Imagename,
Image = imgs.Image,
Imagetype = imgs.Imagetype,
Postid = imgs.Postid,
Comment = imgs.Comment,
Userid = imgs.Userid
}).ToArray(),
Primaryimage = computations.primaryimage((from pic in pts.Images
where pic.Imagetype == (int?)1
select pic into pi
select new Images
{
Imageid = pi.Imageid,
Image = pi.Image,
Imagename = pi.Imagename
}).DefaultIfEmpty().First()),
Likes = pts.Likes.Select((Likes lks) => new Likes
{
Likeid = lks.Likeid,
Post = lks.Post,
Commentid = lks.Commentid
}).ToArray(),
Likecount = pts.Likes.Where((Likes lc) => lc.Commentid == null).Count(),
WorldareaNavigation = new Worldarea
{
Worldarea1 = pts.WorldareaNavigation.Worldarea1,
Worldareaacronym = pts.WorldareaNavigation.Worldareaacronym,
Wurl = linkgenerator.worldarealink($"{pts.WorldareaNavigation.Worldareaacronym}")
},
RegionNavigation = new Regions
{
Regionname = pts.RegionNavigation.Regionname,
Regionacronym = pts.RegionNavigation.Regionacronym,
Rurl = linkgenerator.regionlink($"{pts.WorldareaNavigation.Worldareaacronym}", $"{pts.RegionNavigation.Regionacronym}")
},
CityNavigation = new Cities
{
City = pts.CityNavigation.City,
Cityacronym = pts.CityNavigation.Cityacronym,
Cityurl = linkgenerator.citylink($"{pts.WorldareaNavigation.Worldareaacronym}", $"{pts.RegionNavigation.Regionacronym}", $"{pts.CityNavigation.Cityacronym}")
},
CategoryNavigation = new Categories
{
Categoryname = pts.CategoryNavigation.Categoryname,
Categorylongname = pts.CategoryNavigation.Categorylongname,
Categorytitle = pts.CategoryNavigation.Categorytitle,
Categorydescription = pts.CategoryNavigation.Categorydescription,
Categorykeywords = pts.CategoryNavigation.Categorykeywords,
Categorymeta = pts.CategoryNavigation.Categorymeta,
Categoryurl = pts.CategoryNavigation.Categoryurl,
Curl = linkgenerator.categorylink($"{pts.CategoryNavigation.Categoryurl}")
},
SubcategoryNavigation = new Subcategories
{
Subcategoryname = pts.SubcategoryNavigation.Subcategoryname,
Subcategorylongname = pts.SubcategoryNavigation.Subcategorylongname,
Subcategorytitle = pts.SubcategoryNavigation.Subcategorytitle,
Subcategorydescription = pts.SubcategoryNavigation.Subcategorydescription,
Subcategorymeta = pts.SubcategoryNavigation.Subcategorymeta,
Subcategorykeywords = pts.SubcategoryNavigation.Subcategorykeywords,
Subcategoryurl = pts.SubcategoryNavigation.Subcategoryurl,
Scurl = linkgenerator.subcategorylink($"{pts.CategoryNavigation.Categoryurl}", $"{pts.SubcategoryNavigation.Subcategoryurl}")
},
Subcategory2Navigation = new Subcategory2
{
Subcategory2name = pts.Subcategory2Navigation.Subcategory2name,
Subcategory2longname = pts.Subcategory2Navigation.Subcategory2longname,
Subcategory2title = pts.Subcategory2Navigation.Subcategory2title,
Subcategory2description = pts.Subcategory2Navigation.Subcategory2description,
Subcategory2meta = pts.Subcategory2Navigation.Subcategory2meta,
Subcategory2keywords = pts.Subcategory2Navigation.Subcategory2keywords,
Subcategory2url = pts.Subcategory2Navigation.Subcategory2url,
Sc2url = linkgenerator.subcategory2link($"{pts.CategoryNavigation.Categoryurl}", $"{pts.SubcategoryNavigation.Subcategoryurl}", $"{pts.Subcategory2Navigation.Subcategory2url}")
}
}).OrderByDescending((Posts d) => d.Postid).Skip((currentPage - 1) * pageSize).Take(pageSize);
//Func<List<Posts>> actionThatWeWantToCache = () => cacheobject;
//var CachedAdapter = cache.GetOrAdd(cacheKey, actionThatWeWantToCache);
return cacheobject;
}```
Ok, here's your problem with code like this:
public async Task<Posts[]> GetPosts()
{
IAppCache cache = new CachingService(CachingService.DefaultCacheProvider) { DefaultCachePolicy = new CacheDefaults { DefaultCacheDurationSeconds = GlobalStatic.SITEPOSTSCACHEDURATION() } };
string cacheKey = $"GetPosts";
var cacheobject = await (from post in _context.Posts
where post.Active == (bool?)true && post.Adminban == (bool?)false && post.Site == (int?)GlobalStatic.SITENUMBER()
select post into pts
select new Posts
{
Postid = pts.Postid,
Title = pts.Title,
Sc2url = linkgenerator.subcategory2link($"{pts.CategoryNavigation.Categoryurl}", $"{pts.SubcategoryNavigation.Subcategoryurl}", $"{pts.Subcategory2Navigation.Subcategory2url}")
}
}).ToArrayAsync();
}
public async Task<List<Posts>> GetPaginatedResult(int currentPage, int pageSize)
{
return (await GetPosts()).OrderByDescending((Posts d) => d.Postid).Skip((currentPage - 1) * pageSize).Take(pageSize)
.ToList();
}
GetPosts is doing a ToArray which will look to materialize your ENTIRE table into entities BEFORE doing pagination.... This is also calling C# code like this linkgenerator(...) method which will trigger client-side evaluation which is a huge performance no-no since that will also materialize the related entities for all rows. You are then selecting the Post Entities into another class called Posts which is either a Post entity instance or a separate DTO/ViewModel in order to capture this link.
This is a common sin when I see developers pursuing DNRY (Do Not Repeat Yourself) to keep common logic in one place, and then employ patterns that treat everything consistently for consistency's sake.
GetPosts() is a dangerous method if it returns an array of posts as it would always return everything where the code you are calling wants to optimize for paginated results. This is very common in Generic implementations which force a one-size-fits-all on operations. Some tables will be small enough that a "GetAll" type operation is reasonable safe, however that common method against another table can halt a system for no good reason.
To satisfy DNRY, centralize the common query into a private method that can be called as needed. A public GetAll() method for Posts is probably never needed for this entity so I would remove it:
private IQueryable<Post> getPosts()
{
var query = _context.Posts
.Where(post => post.Active
&& !post.Adminban
&& post.Site == GlobalStatic.SITENUMBER());
return query;
}
This method can be called by every method that wants to query across Posts and their related details. This centralizes your common rules around active state, and tenancy. (ownership)
Now to your GetPaginatedResult method. Here you're going to want to ensure you are selecting your desired data. I would recommend declaring a ViewModel for the post data given you want to have some computed values.
[Serializable]
public class PostViewModel
{
public int PostId { get; set; }
public string Title { get; set; }
public string CategoryUrl { get; set; }
public string SubcategoryUrl { get; set; }
public string Subcategory2Url { get; set; }
public string Sc2Url
{
get { return linkgenerator.subcategory2link(CategoryUrl, SubcategoryUrl, Subcategory2url); }
}
}
Within Linq expressions we want to select all of the raw data from the entity and associated other entities. Where we want to compose/calculate values, we can expose getters in the view model that do this calculation as needed using the raw data.
And for the GetPaginatedResult method:
public async Task<List<Posts>> GetPaginatedResult(int currentPage, int pageSize)
{
var posts = await getPosts()
.Select(x => new PostViewModel
{
PostId = x.PostId,
Title = x.Title,
CategoryUrl = x.CategoryNavigation.Categoryurl,
SubcategoryUrl = x.SubcategoryNavigation.Subcategoryurl,
SubcategoryUrl = x.Subcategory2Navigation.Subcategory2url
}).Skip((currentPage -1) * pageSize)
.Take(pageSize)
.ToListAsync();
}
Note that the Select only selects the raw values from the Post and associated entities. It doesn't attempt to calculate the full link. The view model will take care of that when the property is accessed, such as when it is referenced to compose a view or serialized. You can perform simple computed fields in the Linq expression, but only basic stuff like appending strings together with "+" and some simple functions that EF can translate. Even String.Format (including $"{value}") is off the cards inside a Linq expression fed to EF. It cannot translate that to SQL so it would fall on client-side evaluation and you want to avoid that at all costs. By default with EF Core 2, client-side evaluation was on by default, with EF Core 3.1 onward it is off by default but some teams turn it on. I highly recommend leaving it off as it's a bigger landmine than lazy loading.
That should eliminate your performance issues. This generates a compact SQL statement taking into account the desired pagination and serializes just those results into a view model to return to the view. The core business rules around ownership can be cetralized, and by exposing IQueryable it is left to the consuming code to project or further modify the results (paginate, etc.) to build an efficient query. Don't expose any method that you don't intend to get called. (Like a GetAll())
Edit: It sounds like you are trying to replace:
public async Task<Posts[]> GetPosts()
with
public async IQueryable<Post> GetPosts()
This is *not the objective. The objective is to centralize the core logic into a protected method which GetPosts as well as GetPaginatedResult can both call rather than having GetPaginatedResult calling GetPosts which is loading all data. You also need to address that you cannot call linkgenerator from within the Select expression without triggering client-side evaluation so the suggestion there is to either define view models for the results, or if you insist on returning the entity models as a view model, adding a non-mapped property for the generated URL.
public async Task<Posts[]> GetPosts()
{
return await _getPosts()
.Select(pts => new Posts
{
Postid = pts.Postid,
Title = pts.Title,
CategoryUrl = pts.CategoryNavigation.CategoryUrl,
SubcategoryUrl = pts.SubcategoryNavigation.SubcategoryUrl,
Subcategory2Url = pts.SubcategoryNavigation.Subcategory2Url
}).ToArrayAsync();
}
public async Task<List<Posts>> GetPaginatedResult(int currentPage, int pageSize)
{
return await _getPosts()
.Select(pts => new Posts
{
Postid = pts.Postid,
Title = pts.Title,
CategoryUrl = pts.CategoryNavigation.CategoryUrl,
SubcategoryUrl = pts.SubcategoryNavigation.SubcategoryUrl,
Subcategory2Url = pts.SubcategoryNavigation.Subcategory2Url
}).Skip((currentPage-1) * pageSize)
.Take(pageSize)
.ToListAsync();
}
Where _getPosts() is a new protected IQueryable method that centralizes the core rules around your isActive and Site. I put an Underscore before the method name to differentiate it from your public GetPosts() method which is probably called in a number of different places.
You initially want to avoid changing the signature of your public methods because this would break anywhere and everywhere else that called this method, where it is better to correct this issue a bit at a time where it is most needed.
Looking at your updated example you don't want to change your public methods to IQueryable, leave those as your original return types and just ensure that these methods that call the new IQueryable method do their respective await with ToArrayAsync or ToListAsync respectively instead of calling the public GetPosts() method. The change you will still need to make is to move ALL properties where you are calling "linkgenerator" or any similar C# objects/functions into unmapped properties and instead use Select to retrieve the raw values these calls will need.
I am one of the many struggling to "upgrade" from ASP.NET to ASP.NET Core.
In the ASP.NET project, I made database calls from my DAL like so:
var result = context.Database.SqlQuery<Object_VM>("EXEC [sp_Object_GetByKey] #Key",
new SqlParameter("#Key", Key))
.FirstOrDefault();
return result;
My viewmodel has additional fields that my object does not, such as aggregates of related tables. It seems unnecessary and counter intuitive to include such fields in a database / table structure. My stored procedure calculates all those things and returns the fields as should be displayed, but not stored.
I see that ASP.NET Core has removed this functionality. I am trying to continue to use stored procedures and load view models (and thus not have the entity in the database). I see options like the following, but as a result I get "2", the number of rows being returned (or another mysterious result?).
using(context)
{
string cmd = "EXEC [sp_Object_getAll]";
var result = context.Database.ExecuteSQLCommand(cmd);
}
But that won't work because context.Database.ExecuteSQLCommand is only for altering the database, not "selecting".
I've also seen the following as a solution, but the code will not compile for me, as "set" is really set<TEntity>, and there isn't a database entity for this viewmodel.
var result = context.Set().FromSql("EXEC [sp_Object_getAll]");
Any assistance much appreciated.
Solution:
(per Tseng's advice)
On the GitHub Entity Framework Issues page, there is a discussion about this problem. One user recommends creating your own class to handle this sort of requests, and another adds an additional method that makes it run smoother. I changed the methods slights to accept slightly different params.
Here is my adaptation (very little difference), for others that are also looking for a solution:
Method in DAL
public JsonResult GetObjectByID(int ID)
{
SqlParameter[] parms = new SqlParameter[] { new SqlParameter("#ID", ID) };
var result = RDFacadeExtensions.GetModelFromQuery<Object_List_VM>(context, "EXEC [sp_Object_GetList] #ID", parms);
return new JsonResult(result.ToList(), setting);
}
Additional Class
public static class RDFacadeExtensions
{
public static RelationalDataReader ExecuteSqlQuery(
this DatabaseFacade databaseFacade,
string sql,
SqlParameter[] parameters)
{
var concurrencyDetector = databaseFacade.GetService<IConcurrencyDetector>();
using (concurrencyDetector.EnterCriticalSection())
{
var rawSqlCommand = databaseFacade
.GetService<IRawSqlCommandBuilder>()
.Build(sql, parameters);
return rawSqlCommand
.RelationalCommand
.ExecuteReader(
databaseFacade.GetService<IRelationalConnection>(),
parameterValues: rawSqlCommand.ParameterValues);
}
}
public static IEnumerable<T> GetModelFromQuery<T>(
DbContext context,
string sql,
SqlParameter[] parameters)
where T : new()
{
DatabaseFacade databaseFacade = new DatabaseFacade(context);
using (DbDataReader dr = databaseFacade.ExecuteSqlQuery(sql, parameters).DbDataReader)
{
List<T> lst = new List<T>();
PropertyInfo[] props = typeof(T).GetProperties();
while (dr.Read())
{
T t = new T();
IEnumerable<string> actualNames = dr.GetColumnSchema().Select(o => o.ColumnName);
for (int i = 0; i < props.Length; ++i)
{
PropertyInfo pi = props[i];
if (!pi.CanWrite) continue;
System.ComponentModel.DataAnnotations.Schema.ColumnAttribute ca = pi.GetCustomAttribute(typeof(System.ComponentModel.DataAnnotations.Schema.ColumnAttribute)) as System.ComponentModel.DataAnnotations.Schema.ColumnAttribute;
string name = ca?.Name ?? pi.Name;
if (pi == null) continue;
if (!actualNames.Contains(name)) { continue; }
object value = dr[name];
Type pt = pi.DeclaringType;
bool nullable = pt.GetTypeInfo().IsGenericType && pt.GetGenericTypeDefinition() == typeof(Nullable<>);
if (value == DBNull.Value) { value = null; }
if (value == null && pt.GetTypeInfo().IsValueType && !nullable)
{ value = Activator.CreateInstance(pt); }
pi.SetValue(t, value);
}//for i
lst.Add(t);
}//while
return lst;
}//using dr
}
Insert is working well. The update is not working for the collections. And do not give any error. What am I doing wrong?
In the code there is a brief explanation of the rules.
public void EditaModelo(Modelo model)
{
try
{
// ProjectResponsible - is an unmapped property. It was used to separate ModeloFuncao between type 2 and type 1 (ProjectResponsible and ProductDevelopment)
// ModeloFuncao is a collection within a model where for each function there is a responsibility within the model.
// In other words, for each record in the function table shows a field on the screen. The same goes for ProductDevelopment.
// When you save the model, there will be incusões, deletions and changes of those responsible.
// The IsUpdate property tells whether the record already exists and will be changed.
var ModelosToAdd = model.ProjectResponsible.Where(x => !String.IsNullOrWhiteSpace(x.Usuario) && !x.IsUpdate).ToList();
List<ModeloFuncao> ModelosToRemove = model.ProjectResponsible.Where(x => String.IsNullOrWhiteSpace(x.Usuario) && x.IsUpdate).ToList();
List<ModeloFuncao> ModelosToUpdate = model.ProjectResponsible.Where(x => !String.IsNullOrWhiteSpace(x.Usuario) && x.IsUpdate).ToList();
ModelosToAdd.AddRange(model.ProductDevelopment.Where(x => !String.IsNullOrWhiteSpace(x.Usuario) && !x.IsUpdate).ToList());
ModelosToRemove.AddRange(model.ProductDevelopment.Where(x => String.IsNullOrWhiteSpace(x.Usuario) && x.IsUpdate).ToList());
ModelosToUpdate.AddRange(model.ProductDevelopment.Where(x => !String.IsNullOrWhiteSpace(x.Usuario) && x.IsUpdate).ToList());
if(ModelosToAdd.Count > 0) context.ModelosFuncoes.AddRange(ModelosToAdd); //Insert is Ok
if (ModelosToRemove.Count > 0) context.ModelosFuncoes.RemoveRange(ModelosToRemove); //Not tested
if (ModelosToUpdate.Count > 0) ModelosToUpdate.ForEach(x => context.ModelosFuncoes.Attach(x)); //Not working
// The ModeloPlanta is a collection in the model table and follow the rules as explained below in ModeloPlantaArea.
List<ModeloPlanta> plantasToUpdate = model.ModelosPlantas.ToList();
plantasToUpdate.ForEach(x => context.ModelosPlantas.Attach(x));//Not working
// The ModeloPlantaArea is a collection in the model table. Each model has a number of plants and each plant has a number of areas.
// Each plant has a responsibility and each area has a responsibility.
// The screen should display a field for each plant x Responsible and for each Area x Responsible
// When you save the model, there will be incusões, deletions and changes of those responsible.
List<ModeloPlantaArea> AreasToAdd = model.PlantasArea.Where(x => !String.IsNullOrWhiteSpace(x.UsuarioResponsavel) && !x.IsUpdate).ToList();
List<ModeloPlantaArea> AreasToUpdate = model.PlantasArea.Where(x => !String.IsNullOrWhiteSpace(x.UsuarioResponsavel) && x.IsUpdate).ToList();
List<ModeloPlantaArea> AreasToRemove = model.PlantasArea.Where(x => String.IsNullOrWhiteSpace(x.UsuarioResponsavel) && x.IsUpdate).ToList();
if (AreasToAdd.Count > 0) context.ModelosPlantasArea.AddRange(AreasToAdd);//Insert is Ok
if (AreasToUpdate.Count > 0) AreasToUpdate.ForEach(x => context.ModelosPlantasArea.Attach(x));//Not working
if (AreasToRemove.Count > 0) context.ModelosPlantasArea.RemoveRange(AreasToRemove);//Not tested
// When saving Model, need to save (add, delete, change) collections. And if a collection fails, the other must not be saved.
// So far this Inclusion OK. The change is not working.
this.Update(model);
}
catch (Exception ex)
{
throw ex;
}
}
Update method:
public void Update(Modelo item, IEnumerable<string> fieldsToUpdate = null)
{
base.context.Modelos.Attach(item);
base.UpdateSave(item, fieldsToUpdate);
}
UpdateSave Method:
protected void UpdateSave<TEntity>(TEntity item, IEnumerable<string> fieldsToUpdate) where TEntity : class
{
var entry = this.context.Entry<TEntity>(item);
if (fieldsToUpdate == null || fieldsToUpdate.Count() == 0)
entry.State = System.Data.Entity.EntityState.Modified;
else
{
this.changeFieldsModified<TEntity>(entry, fieldsToUpdate);
}
this.Save();
}
Save Method:
protected void Save(bool auditing = true)
{
try
{
this.context.SaveChanges(auditing);
}
catch (DbEntityValidationException exception)
{
throw new ValidationException(exception);
}
catch
{
throw;
}
}
SaveChanges Method:
public int SaveChanges(bool auditing)
{
var entriesAdded = new List<DbEntityEntry>();
// Salva o log dos itens alterados e excluídos.
if (auditing && base.ChangeTracker.HasChanges())
foreach (var entry in base.ChangeTracker.Entries())
switch (entry.State)
{
case EntityState.Added:
entriesAdded.Add(entry);
break;
case EntityState.Deleted:
this.saveEntryOperation(entry, EntityState.Deleted);
break;
case EntityState.Modified:
this.saveEntryOperation(entry, EntityState.Modified);
break;
}
// Realiza a persitência de dados
int count = base.SaveChanges();
// Salva o log dos itens adicionados.
if (auditing && entriesAdded.Count > 0)
foreach (var entry in entriesAdded)
this.saveEntryOperation(entry, EntityState.Added);
this.AuditionContext.SaveChanges();
return count;
}
Attach merely adds the entity into change tracking, but if you don't actually change it after that point, its state remains Unchanged. The fact that it was different from what's in the database already before it was attached is meaningless.
To signal that the entity needs to be updated, you need to set it to Modified:
db.Entry(entity).State = EntityState.Modified;
If the entity is not already attached, this will also have the effect of attaching it, so there's no need to call Attach as well.
[EDITED BY Ferreira]
This tip work fine. I made the changes as below.
if (ModelosToUpdate.Count > 0)
{
ModelosToUpdate.ForEach(x =>
{
context.ModelosFuncoes.Attach(x);
var entry = this.context.Entry<ModeloFuncao>(x);
entry.State = System.Data.Entity.EntityState.Modified
});
}
I'm trying to implement a caching scheme for my EF Repository similar to the one blogged here. As the author and commenters have reported the limitation is that the key generation method cannot produce cache keys that vary with a given query's parameters. Here is the cache key generation method:
private static string GetKey<T>(IQueryable<T> query)
{
string key = string.Concat(query.ToString(), "\n\r",
typeof(T).AssemblyQualifiedName);
return key;
}
So the following queries will yield the same cache key:
var isActive = true;
var query = context.Products
.OrderBy(one => one.ProductNumber)
.Where(one => one.IsActive == isActive).AsCacheable();
and
var isActive = false;
var query = context.Products
.OrderBy(one => one.ProductNumber)
.Where(one => one.IsActive == isActive).AsCacheable();
Notice that the only difference is that isActive = true in the first query and isActive = false in the second.
Any suggestions/insight to efficiently generating cache keys which vary by IQueryable parameters would be truly appreciated.
Kudos to Sergey Barskiy for sharing the EF CodeFirst caching scheme.
Update
I took the approach of traversing the IQueryable's expression tree myself with the goal of resolving the values of the parameters used in the query. With maxlego's suggestion, I extended the System.Linq.Expressions.ExpressionVisitor class to visit the expression nodes that we're interested in - in this case, the MemberExpression. The updated GetKey method looks something like this:
public static string GetKey<T>(IQueryable<T> query)
{
var keyBuilder = new StringBuilder(query.ToString());
var queryParamVisitor = new QueryParameterVisitor(keyBuilder);
queryParamVisitor.GetQueryParameters(query.Expression);
keyBuilder.Append("\n\r");
keyBuilder.Append(typeof (T).AssemblyQualifiedName);
return keyBuilder.ToString();
}
And the QueryParameterVisitor class, which was inspired by the answers of Bryan Watts and Marc Gravell to this question, looks like this:
/// <summary>
/// <see cref="ExpressionVisitor"/> subclass which encapsulates logic to
/// traverse an expression tree and resolve all the query parameter values
/// </summary>
internal class QueryParameterVisitor : ExpressionVisitor
{
public QueryParameterVisitor(StringBuilder sb)
{
QueryParamBuilder = sb;
Visited = new Dictionary<int, bool>();
}
protected StringBuilder QueryParamBuilder { get; set; }
protected Dictionary<int, bool> Visited { get; set; }
public StringBuilder GetQueryParameters(Expression expression)
{
Visit(expression);
return QueryParamBuilder;
}
private static object GetMemberValue(MemberExpression memberExpression, Dictionary<int, bool> visited)
{
object value;
if (!TryGetMemberValue(memberExpression, out value, visited))
{
UnaryExpression objectMember = Expression.Convert(memberExpression, typeof (object));
Expression<Func<object>> getterLambda = Expression.Lambda<Func<object>>(objectMember);
Func<object> getter = null;
try
{
getter = getterLambda.Compile();
}
catch (InvalidOperationException)
{
}
if (getter != null) value = getter();
}
return value;
}
private static bool TryGetMemberValue(Expression expression, out object value, Dictionary<int, bool> visited)
{
if (expression == null)
{
// used for static fields, etc
value = null;
return true;
}
// Mark this node as visited (processed)
int expressionHash = expression.GetHashCode();
if (!visited.ContainsKey(expressionHash))
{
visited.Add(expressionHash, true);
}
// Get Member Value, recurse if necessary
switch (expression.NodeType)
{
case ExpressionType.Constant:
value = ((ConstantExpression) expression).Value;
return true;
case ExpressionType.MemberAccess:
var me = (MemberExpression) expression;
object target;
if (TryGetMemberValue(me.Expression, out target, visited))
{
// instance target
switch (me.Member.MemberType)
{
case MemberTypes.Field:
value = ((FieldInfo) me.Member).GetValue(target);
return true;
case MemberTypes.Property:
value = ((PropertyInfo) me.Member).GetValue(target, null);
return true;
}
}
break;
}
// Could not retrieve value
value = null;
return false;
}
protected override Expression VisitMember(MemberExpression node)
{
// Only process nodes that haven't been processed before, this could happen because our traversal
// is depth-first and will "visit" the nodes in the subtree before this method (VisitMember) does
if (!Visited.ContainsKey(node.GetHashCode()))
{
object value = GetMemberValue(node, Visited);
if (value != null)
{
QueryParamBuilder.Append("\n\r");
QueryParamBuilder.Append(value.ToString());
}
}
return base.VisitMember(node);
}
}
I'm still doing some performance profiling on the cache key generation and hoping that it isn't too expensive (I'll update the question with the results once I have them). I'll leave the question open, in case anyone has suggestions on how to optimize this process or has a recommendation for a more efficient method for generating cache keys with vary with the query parameters. Although this method produces the desired output, it is by no means optimal.
i suggest to use ExpressionVisitor
http://msdn.microsoft.com/en-us/library/bb882521(v=vs.90).aspx
Just for the record, "Caching the results of LINQ queries" works well with the EF and it's able to work with parameters correctly, so it can be considered as a good second level cache implementation for EF.
While the solution of the OP works quite well, I found that the performance of the solution is a little bit poor.
The duration of the key generation varied between 300ms and 1200ms for my queries.
However, I've found another solution that has quite better performance (<10ms).
public static string ToTraceString<T>(DbQuery<T> query)
{
var internalQueryField = query.GetType().GetFields(BindingFlags.NonPublic | BindingFlags.Instance).Where(f => f.Name.Equals("_internalQuery")).FirstOrDefault();
var internalQuery = internalQueryField.GetValue(query);
var objectQueryField = internalQuery.GetType().GetFields(BindingFlags.NonPublic | BindingFlags.Instance).Where(f => f.Name.Equals("_objectQuery")).FirstOrDefault();
var objectQuery = objectQueryField.GetValue(internalQuery) as ObjectQuery<T>;
return ToTraceStringWithParameters(objectQuery);
}
private static string ToTraceStringWithParameters<T>(ObjectQuery<T> query)
{
string traceString = query.ToTraceString() + "\n";
foreach (var parameter in query.Parameters)
{
traceString += parameter.Name + " [" + parameter.ParameterType.FullName + "] = " + parameter.Value + "\n";
}
return traceString;
}