I want to find any entity that inherits from DbEntity and has the field DeletedDate set to a value older than 100 days.
I have managed to piece together this code. But I am unsure of how I can go from this to querying the database for relevant entities.
var softDeletableEntities = _applicationContext.Model.GetEntityTypes()
.Where(e =>
typeof(DbEntity).IsAssignableFrom(e.ClrType)
&& e.BaseType == null
);
foreach (var entity in softDeletableEntities)
{
var entry = _applicationContext.Entry(entity.ClrType);
entry.Property<DateTimeOffset?>("DeletedDate");
// how do I find any Entity where (DeletedDate != null) and delete it?
}
I was hoping to be able to do something like this, but Set<> can't be used like that:
// Not ok to use Set<>() as below. Gives: 'entity' is a variable but is used like a type
var dataSet = _applicationContext.Set<entity.ClrType>();
var deleteOlderThan = DateTime.Now.AddDays(-100);
var toDelete = dataSet.Where(x => x.DeletedDate < deleteOlderThan);
dataSet.RemoveRange(toDelete);
_applicationContext.SaveChanges();
Any tips :)?
I would suggest to use EF Core extension linq2db.EntityFrameworkCore, note that I'm one of the creators. Install appropriate version 5.x for EF Core 5, 6.x for EF Core 6, etc.
Then you can execute the following query:
var dataSet = _applicationContext.Set<entity.ClrType>();
var deleteOlderThan = DateTime.Now.AddDays(-100);
var toDelete = dataSet.Where(x => x.DeletedDate < deleteOlderThan);
toDelete.Delete(); // method from Extension
Related
Our current system is using Lazyloading by default (it is something I am going to be disabling but it can't be done right now)
For this basic query I want to return two tables, CustomerNote and Note.
This is my query
using (var newContext = new Entities(true))
{
newContext.Configuration.LazyLoadingEnabled = false;
var result = from customerNotes in newContext.CustomerNotes.Include(d=>d.Note)
join note in newContext.Notes
on customerNotes.NoteId equals note.Id
where customerNotes.CustomerId == customerId
select customerNotes;
return result.ToList();
}
My result however only contains the data in the CustomerNote table
The linked entities Customer and Note are both null, what am I doing wrong here?
I got it working with the following which is much simpler than what I've found elsewhere
Context.Configuration.LazyLoadingEnabled = false;
var result = Context.CustomerNotes.Where<CustomerNote>(d => d.CustomerId == customerId)
.Include(d=>d.Note)
.Include(d=>d.Note.User);
return result.ToList();
This returns my CustomerNote table, related Notes and related Users from the Notes.
That is callled eager loading you want to achieve.
var customerNotes = newContext.CustomerNotes.Include(t=> t.Node).ToList();
This should work, i don't really understand the keyword syntax.
If the code above doesn't work try this:
var customerNotes = newContext.CustomerNotes.Include(t=> t.Node).Select(t=> new {
Node = t.Node,
Item = t
}).ToList();
My goal is to modify "JustMyCode" queries using nDepend API. I am using code like:
var justMyCodeGroup = prj.CodeQueries.CodeQueriesSet.ChildGroups.Single(x => x.Name.Contains("JustMyCode"));
var originalQuery = justMyCodeGroup.ChildQueries
.Single(x => x.QueryString.Contains("Discard generated Types from JustMyCode"));
var changedQuery = originalQuery.Controller.CreateQuery(originalQuery.IsActive,
query,
originalQuery.
DisplayStatInReport,
originalQuery.DisplayListInReport,
originalQuery.DisplaySelectionViewInReport,
originalQuery.IsCriticalRule);
var justMyCodeGroupWithModifiedQuery = justMyCodeGroup.ReplaceQuery(originalQuery, changedQuery);
prj.CodeQueries.CodeQueriesSet.ReplaceGroup(justMyCodeGroup, justMyCodeGroupWithModifiedQuery);
However, when I run the code above I get ArgumentException with message:
newGroup.Controller is different than this groupOfGroups.Controller
Any help ?
Update 1:
I also tried code:
var justMyCodeGroup = prj.CodeQueries.CodeQueriesSet.ChildGroups.Single(x => x.Name.Contains("JustMyCode"));
var originalQuery = justMyCodeGroup.ChildQueries
.Single(x => x.QueryString.Contains("Discard generated Types from JustMyCode"));
var changedQuery = originalQuery.Controller.CreateQuery(originalQuery.IsActive,
query,
originalQuery.
DisplayStatInReport,
originalQuery.DisplayListInReport,
originalQuery.DisplaySelectionViewInReport,
originalQuery.IsCriticalRule);
var justMyCodeGroupWithModifiedQuery = justMyCodeGroup.ReplaceQuery(originalQuery, changedQuery);
var newQueries = new List<IQuery>();
foreach (var q in justMyCodeGroup.ChildQueries)
{
if (q.QueryString.Contains("Discard generated Types from JustMyCode"))
{
continue;
}
newQueries.Add(prj.CodeQueries.CodeQueriesSet.Controller.CreateQuery(q.IsActive, q.QueryString,
q.DisplayStatInReport, q.DisplayListInReport, q.DisplaySelectionViewInReport, q.IsCriticalRule));
}
newQueries.Add(prj.CodeQueries.CodeQueriesSet.Controller.CreateQuery(originalQuery.IsActive, query, originalQuery.DisplayStatInReport, originalQuery.DisplayListInReport, originalQuery.DisplaySelectionViewInReport, originalQuery.IsCriticalRule));
var newGroup = prj.CodeQueries.CodeQueriesSet.Controller.CreateGroup(justMyCodeGroup.Name,
justMyCodeGroup.IsActive, justMyCodeGroup.ShownInReport, newQueries, new List<IGroup>());
prj.CodeQueries.CodeQueriesSet.RemoveGroup(justMyCodeGroup);
prj.CodeQueries.CodeQueriesSet.AddGroup(newGroup);
Right now, RemoveGroup throws exception:
this group of groups doesn't contain groupToRemove.
Update 2:
And I also wonder, why does this code return false ?
var justMyCodeGroup = prj.CodeQueries.CodeQueriesSet.ChildGroups.Single(x => x.Name.Contains("JustMyCode"));
prj.CodeQueries.CodeQueriesSet.ContainsGroup(justMyCodeGroup)
Refer to the PowerTools source file:
$NDependInstallDir$\NDepend.PowerTools.SourceCode\CQL2CQLinq\CQL2CQLinqPowerTool.cs
This PowerTools convert code queries written with old CQL syntax into code queries written with new CQLinq syntax, hence it loads the queries set from a project, update CQL queries, and then save the new queries set in the project.
The queriesController is gathered this way...
var queriesSet = project.CodeQueries.CodeQueriesSet;
var queriesController = queriesSet.Controller;
... and then used this way to modify the queries set:
queriesController.DoUpdateQueryObject(query, newQuery);
What is the best way to deal with batch updates using (Entity Framework) EF5?
I have 2 particular cases I'm interested in:
Updating a field (e.g. UpdateDate) for a list (List) of between 100 and 100.000 Id's, which the primary key. Calling each update separately seem to be to much overhead and takes a long time.
Inserting many, also between the 100 and 100.000, of the same objects (e.g. Users) in a single go.
Any good advice?
There are two open source projects allowing this: EntityFramework.Extended and Entity Framework Extensions. You can also check discussion about bulk updates on EF's codeplex site.
Inserting 100k records through EF is in the first place wrong application architecture. You should choose different lightweight technology for data imports. Even EF's internal operation with such big record set will cost you a lot of processing time. There is currently no solution for batch inserts for EF but there is broad discussion about this feature on EF's code plex site.
I see the following options:
1 . The simplest way - create your SQL request by hands and execute through ObjectContext.ExecuteStoreCommand
context.ExecuteStoreCommand("UPDATE TABLE SET FIELD1 = {0} WHERE FIELD2 = {1}", value1, value2);
2 . Use EntityFramework.Extended
context.Tasks.Update(
t => t.StatusId == 1,
t => new Task {StatusId = 2});
3 . Make your own extension for EF. There is an article Bulk Delete where this goal was achieved by inheriting ObjectContext class. It's worth to take a look. Bulk insert/update can be implemented in the same way.
You may not want to hear it, but your best option is to not use EF for bulk operations. For updating a field across a table of records, use an Update statement in the database (possibly called through a stored proc mapped to an EF Function). You can also use the Context.ExecuteStoreQuery method to issue an Update statement to the database.
For massive inserts, your best bet is to use Bulk Copy or SSIS. EF will require a separate hit to the database for each row being inserted.
Bulk inserts should be done using the SqlBulkCopy class. Please see pre-existing StackOverflow Q&A on integrating the two: SqlBulkCopy and Entity Framework
SqlBulkCopy is a lot more user-friendly than bcp (Bulk Copy command-line utility) or even OPEN ROWSET.
Here's what I've done successfully:
private void BulkUpdate()
{
var oc = ((IObjectContextAdapter)_dbContext).ObjectContext;
var updateQuery = myIQueryable.ToString(); // This MUST be above the call to get the parameters.
var updateParams = GetSqlParametersForIQueryable(updateQuery).ToArray();
var updateSql = $#"UPDATE dbo.myTable
SET col1 = x.alias2
FROM dbo.myTable
JOIN ({updateQuery}) x(alias1, alias2) ON x.alias1 = dbo.myTable.Id";
oc.ExecuteStoreCommand(updateSql, updateParams);
}
private void BulkInsert()
{
var oc = ((IObjectContextAdapter)_dbContext).ObjectContext;
var insertQuery = myIQueryable.ToString(); // This MUST be above the call to get the parameters.
var insertParams = GetSqlParametersForIQueryable(insertQuery).ToArray();
var insertSql = $#"INSERT INTO dbo.myTable (col1, col2)
SELECT x.alias1, x.alias2
FROM ({insertQuery}) x(alias1, alias2)";
oc.ExecuteStoreCommand(insertSql, insertParams.ToArray());
}
private static IEnumerable<SqlParameter> GetSqlParametersForIQueryable<T>(IQueryable<T> queryable)
{
var objectQuery = GetObjectQueryFromIQueryable(queryable);
return objectQuery.Parameters.Select(x => new SqlParameter(x.Name, x.Value));
}
private static ObjectQuery<T> GetObjectQueryFromIQueryable<T>(IQueryable<T> queryable)
{
var dbQuery = (DbQuery<T>)queryable;
var iqProp = dbQuery.GetType().GetProperty("InternalQuery", BindingFlags.Instance | BindingFlags.NonPublic | BindingFlags.Public);
var iq = iqProp.GetValue(dbQuery, null);
var oqProp = iq.GetType().GetProperty("ObjectQuery", BindingFlags.Instance | BindingFlags.NonPublic | BindingFlags.Public);
return (ObjectQuery<T>)oqProp.GetValue(iq, null);
}
public static bool BulkDelete(string tableName, string columnName, List<object> val)
{
bool ret = true;
var max = 2000;
var pages = Math.Ceiling((double)val.Count / max);
for (int i = 0; i < pages; i++)
{
var count = max;
if (i == pages - 1) { count = val.Count % max; }
var args = val.GetRange(i * max, count);
var cond = string.Join("", args.Select((t, index) => $",#p{index}")).Substring(1);
var sql = $"DELETE FROM {tableName} WHERE {columnName} IN ({cond}) ";
ret &= Db.ExecuteSqlCommand(sql, args.ToArray()) > 0;
}
return ret;
}
I agree with the accepted answer that ef is probably the wrong technology for bulk inserts.
However, I think it's worth having a look at EntityFramework.BulkInsert.
I have a local list of values that I need to have entity framework check against the database and return them.
If the list was already in the database, the following would work:
var list = /* some ef query */;
var myList = context.Logs.Where(l => list.Any(li => l.LogNumber == li.LogNumber));
But if the list is local, it would throw an error:
var list = new List<Log>();
var myList = context.Logs.Where(l => list.Any(li => l.LogNumber == li.LogNumber));
Exception: Unable to process the type 'Data.Log[]', because it has no known mapping to the value layer.
So how can I match a local list against a database list using EF?
I got a different error than you with the code sample, but I believe it's the same idea. EF doesn't know how to translate List<Log> into a SQL store expression. It works when you're still in a query because it hasn't been serialized yet.
I realize this is less than ideal, but I was able to make this query work by extracting the scalar values of LogNumber and then using that in the query.
var list = new List<Log>();
list.Add(new Log()
{
LogNumber = 1
});
var numbers = list.Select(l => l.LogNumber);
var myList = m.Logs.Where(l => numbers.Contains(l.LogNumber));
I asked a question about this previously but my database structure has changed, and while it made other things simpler, now this part is more complicated. Here is the previous question.
At the time, my EF Context had a UsersProjects object because there were other properties. Now that I've simplified that table, it is just the keys, so all my EF context knows about is Users and Projects and the M2M relationship between them. There is no more UsersProjects as far as EF knows.
So my goal is to say "show me all the users who are working on projects with me."
in SQL, this would go something like:
SELECT * FROM Users INNER JOIN UsersProjects ON Users.ID=UsersProjects.UserID
WHERE ProjectID IN (SELECT ProjectID FROM UsersProjects WHERE UserID=#UserID)
and I started in EF with something like this:
var myProjects =
(from p in edmx.Projects
where p.Users.Contains(edmx.Users.FirstOrDefault(u => u.Email == UserEmail))
orderby p.Name
select p).ToList();
var associatedUsers =
(from u in edmx.Users
where myProjects.Contains(?????????)
//where myProjects.Any(????????)
select u);
The trick is finding what to put in the ????????. Anyone help here?
var me = context
.Users
.First(user => user.Email = "me#example.com");
// Note that there is no call to ToList() or AsEnumerable().
var myProjects = context
.Projects
.Where(project => project.Users.Contains(me));
var associatedUsers = context
.Users
.Where(user => myProjects.Any(project => user.Project.Contains(project)));
But there are several other possible solutions. For example
var associatedUsers = myProjects
.SelectMany(project => project.Users)
.Distinct();
which I would prefer.
Further note that it is much easier to obtain myProjects using a navigation property instead of using Contains().
var myProjects = me.Projects;
Daniel, I tried what you had and ran into some issues. Can you explain what these errors mean?
I tried:
// Doesn't work.
using (var edmx = new MayflyEntities())
{
var me = edmx.Users.First(user => user.Email == UserEmail);
var myProjects = edmx.Projects.Where(project => project.Users.Contains(me));
var associatedUsers = myProjects.SelectMany(project => project.Users).Distinct();
}
but got the two following exceptions:
Unable to create a constant value of type 'DomainModel.User'. Only primitive types ('such as Int32, String, and Guid') are supported in this context.
and
The ObjectContext instance has been disposed and can no longer be used for operations that require a connection.
So, I moved some things around and this works fine, but now I'm curious as to why? In SQL Profiler, it all executes in one query, so why does it show that the context has been disposed? Also, why can it not use the me object instead of the lambda?
// Works fine
var edmx = new MayflyEntities();
var myProjects = edmx.Projects.Where(project => project.Users.Contains(edmx.Users.First(user => user.Email == UserEmail)));
var associatedUsers = myProjects.SelectMany(project => project.Users).Distinct();