For performance reasons, I cache lots of objects with .NET system.runtime.caching. I'm caching entities coming from a EF datacontext. I'm disposing the datacontext but since the entities coming from it are in the cache, does that mean GC won't be able to collect it?
Example code:
public static Menu GetMenu(string menuName, string languageCode)
{
ObjectCache cache = MemoryCache.Default;
var menu = (Menu)cache[menuName + "cache"];
if (menu == null)
{
using (PageDb db = new PageDb())
{
menu = db.Menus.Where(m => m.Name == menuName && m.LanguageCode == languageCode).FirstOrDefault();
}
CacheItemPolicy policy = new CacheItemPolicy();
policy.AbsoluteExpiration = DateTimeOffset.Now.AddMonths(1);
cache.Set(menuName + "cache", menu, policy);
}
return menu;
}
As you can see I'm storing a menu object in the cache, but that came from the PageDb datacontext. The menu object is supposed to stay in memory for 1 month, but how about the PageDb datacontext?
Related
I'm creating a change log on my records and ChangeTracking only reads records from the main class.
I've already added slow loading and the child classes are being loaded inside my EntityEntry, but I can't capture them through entry.Properties
My class
private void OnBeforeSaveChanges()
{
ChangeTracker.DetectChanges();
var auditEntries = new List<AuditEntry>();
foreach (var entry in ChangeTracker.Entries())
{
if (entry.Entity is AuditLog || entry.State == EntityState.Detached || entry.State == EntityState.Unchanged)
continue;
var auditEntry = new AuditEntry(entry)
{
TableName = entry.Entity.GetType().Name,
UserId = _aplicationUser.GetId
};
auditEntries.Add(auditEntry);
foreach (var property in entry.Properties)
{
string propertyName = property.Metadata.Name;
if (property.Metadata.IsPrimaryKey())
{
auditEntry.KeyValues[propertyName] = property.CurrentValue;
continue;
}
switch (entry.State)
{
case EntityState.Added:
auditEntry.AuditType = AuditType.Create;
auditEntry.NewValues[propertyName] = property.CurrentValue;
break;
case EntityState.Deleted:
auditEntry.AuditType = AuditType.Delete;
auditEntry.OldValues[propertyName] = property.OriginalValue;
break;
case EntityState.Modified:
if (property.IsModified)
{
auditEntry.ChangedColumns.Add(propertyName);
auditEntry.AuditType = AuditType.Update;
auditEntry.OldValues[propertyName] = property.OriginalValue;
auditEntry.NewValues[propertyName] = property.CurrentValue;
}
break;
}
}
}
foreach (var auditEntry in auditEntries)
{
var auditDto = auditEntry.ToAudit();
AuditLogs.Add(new AuditLog(auditDto.Id, auditDto.UserId, auditDto.Type, auditDto.TableName, auditDto.DateTime,
auditDto.OldValues, auditDto.NewValues, auditDto.AffectedColumns, auditDto.PrimaryKey));
}
}
The change tracker reports on a per-entity basis. If you have a Parent entity and a Child collection beneath it and make a change to the parent and one or more children, the parent only shows up in the change tracker because of the changes to the parent's properties. It does not list the properties of all related / child entities. Any modified children would appear in the ChangeTracker as Child entities.
For example:
var parent = context.Parents.Include(x => x.Children).Single(x => x.Id == parentId);
var oldestChild = parent.Children.OrderByDescending(x => x.Age).FirstOrDefault();
parent.SomeValue = newValue;
if (oldestChild != null)
oldestChild.OtherValue = someOtherValue;
context.SaveChanges();
In this case the change tracker would include Parent if, and only if the newValue differed from the existing SomeValue. The change tracker would also include the Child denoted as the oldestChild if such a child actually exists, and the someOtherValue differed from that child's OtherValue. If the Child's value was the only one that actually changed then the Change tracker would not include that child's Parent, only the Child reference. This means you cannot really capture an easy "complete" before and after snapshot of an entire object graph. (Parent and all of it's children before and after, or including the parent when one of it's children happens to change) You can do it, but it will involve piecing the deltas together based on what's in the change tracker.
When it comes to writing audit records I would suggest using a bounded DbContext solely for this purpose that only manages the audit log entities and keeps the entity structure as simple as absolutely possible. This way main app context can intercept and inspect changes without worrying about audit changes, and the audit DbContext avoids this overhead, just performing the raw writes.
I have a console program that moves Data between two different servers (DatabaseA and DatabaseB).
Database B is a Postgres-Server.
It calls a lot of stored procedures and other raw queries.
I use ExecuteSqlRaw a lot.
I also use NpsqlBulk.EfCore.
The program uses the same context instance for DatabaseB during the whole run it takes to finish.
Somehow i get locks on some of my tables on DatabaseB that never get released.
This happens always on my table mytable_fromdatabase_import.
The code run on that is the following:
protected override void AddIdsNew()
{
var toAdd = IdsNotInDatabaseB();
var newObjectsToAdd = GetByIds(toAdd).Select(Converter.ConvertAToB);
DatabaseBContext.Database.ExecuteSqlRaw("truncate mytable_fromdatabase_import; ");
var uploader = new NpgsqlBulkUploader(DatabaseBContext);
uploader.Insert(newObjectsToAdd); // inserts data into mytable_fromdatabase_import
DatabaseBContext.Database.ExecuteSqlRaw("call insert_myTable_from_importTable();");
}
After i run it the whole table is not accessable annymore and when i query the locks on the server i can see there is a process holding it.
How can i make sure this process always closes and releases its locks on tables?
I thought ef-core would do that automaticaly.
-----------Edit-----------
I just wanted to add that this is not a temporary problem during the run of the console. When i run this code and it is finished my table is still locked and nothing can access it. My understanding was that the ef-core context would release everything after it is disposed (if by error or by being finished)
The problem had nothing to do with ef core but with a wrong configured backupscript. The program is running now with no changes to it and it works fine
For concrete task you need right tools. Probably you have locks when retrieve Ids and also when trying to do not load already imported records. These steps are slow!
I would suggest to use linq2db (disclaimer, I'm co-author of this library)
Create two projects with models from different databases:
Source.Model.csproj - install linq2db.SQLServer
Destination.Model.csproj - install linq2db.PostgreSQL
Follow instructions in T4 templates how to generate model from two databases. It is easy and you can ask questions on linq2db`s github site.
I'll post helper class which I've used for transferring tables on my previous project. It additionally uses library CodeJam for mapping, but in your project, for sure, you can use Automapper.
public class DataImporter
{
private readonly DataConnection _source;
private readonly DataConnection _destination;
public DataImporter(DataConnection source, DataConnection destination)
{
_source = source;
_destination = destination;
}
private long ImportDataPrepared<TSource, TDest>(IOrderedQueryable<TSource> source, Expression<Func<TSource, TDest>> projection) where TDest : class
{
var destination = _destination.GetTable<TDest>();
var tableName = destination.TableName;
var sourceCount = source.Count();
if (sourceCount == 0)
return 0;
var currentCount = destination.Count();
if (currentCount > sourceCount)
throw new Exception($"'{tableName}' what happened here?.");
if (currentCount >= sourceCount)
return 0;
IQueryable<TSource> sourceQuery = source;
if (currentCount > 0)
sourceQuery = sourceQuery.Skip(currentCount);
var projected = sourceQuery.Select(projection);
var copied =
_destination.BulkCopy(
new BulkCopyOptions
{
BulkCopyType = BulkCopyType.MultipleRows,
RowsCopiedCallback = (obj) => RowsCopiedCallback(obj, currentCount, sourceCount, tableName)
}, projected);
return copied.RowsCopied;
}
private void RowsCopiedCallback(BulkCopyRowsCopied obj, int currentRows, int totalRows, string tableName)
{
var percent = (currentRows + obj.RowsCopied) / (double)totalRows * 100;
Console.WriteLine($"Copied {percent:N2}% \tto {tableName}");
}
public class ImporterHelper<TSource>
{
private readonly DataImporter _improrter;
private readonly IOrderedQueryable<TSource> _sourceQuery;
public ImporterHelper(DataImporter improrter, IOrderedQueryable<TSource> sourceQuery)
{
_improrter = improrter;
_sourceQuery = sourceQuery;
}
public long To<TDest>() where TDest : class
{
var mapperBuilder = new MapperBuilder<TSource, TDest>();
return _improrter.ImportDataPrepared(_sourceQuery, mapperBuilder.GetMapper().GetMapperExpressionEx());
}
public long To<TDest>(Expression<Func<TSource, TDest>> projection) where TDest : class
{
return _improrter.ImportDataPrepared(_sourceQuery, projection);
}
}
public ImporterHelper<TSource> ImprortData<TSource>(IOrderedQueryable<TSource> source)
{
return new ImporterHelper<TSource>(this, source);
}
}
So begin transferring. Note that I have used OrderBy/ThenBy to specify Id order to do not import already transferred records - important order fields should be Unique Key combination. So this sample is reentrant and can be re-run again when connection is lost.
var sourceBuilder = new LinqToDbConnectionOptionsBuilder();
sourceBuilder.UseSqlServer(SourceConnectionString);
var destinationBuilder = new LinqToDbConnectionOptionsBuilder();
destinationBuilder.UsePostgreSQL(DestinationConnectionString);
using (var source = new DataConnection(sourceBuilder.Build()))
using (var destination = new DataConnection(destinationBuilder.Build()))
{
var dataImporter = new DataImporter(source, destination);
dataImporter.ImprortData(source.GetTable<Source.Model.FirstTable>()
.OrderBy(e => e.Id1)
.ThenBy(e => e.Id2))
.To<Dest.Model.FirstTable>();
dataImporter.ImprortData(source.GetTable<Source.Model.SecondTable>().OrderBy(e => e.Id))
.To<Dest.Model.SecondTable>();
}
For sure boring part with OrderBy can be generated automatically, but this will explode this already not a short answer.
Also play with BulkCopyOptions. Native Npgsql COPY may fail and Multi-Line variant should be used.
I have min 100 000 data into a Job_Details table and I'm using Entity Framework to map the data.
This is the code:
public GetJobsResponse GetImportJobs()
{
GetJobsResponse getJobResponse = new GetJobsResponse();
List<JobBO> lstJobs = new List<JobBO>();
using (NSEXIM_V2Entities dbContext = new NSEXIM_V2Entities())
{
var lstJob = dbContext.Job_Details.ToList();
foreach (var dbJob in lstJob.Where(ie => ie.IMP_EXP == "I" && ie.Job_No != null))
{
JobBO job = MapBEJobforSearchObj(dbJob);
lstJobs.Add(job);
}
}
getJobResponse.Jobs = lstJobs;
return getJobResponse;
}
I found to this line is taking about 2-3 min to execute
var lstJob = dbContext.Job_Details.ToList();
How can i solve this issue?
To outline the performance issues with your example: (see inline comments)
public GetJobsResponse GetImportJobs()
{
GetJobsResponse getJobResponse = new GetJobsResponse();
List<JobBO> lstJobs = new List<JobBO>();
using (NSEXIM_V2Entities dbContext = new NSEXIM_V2Entities())
{
// Loads *ALL* entities into memory. This effectively takes all fields for all rows across from the database to your app server. (Even though you don't want it all)
var lstJob = dbContext.Job_Details.ToList();
// Filters from the data in memory.
foreach (var dbJob in lstJob.Where(ie => ie.IMP_EXP == "I" && ie.Job_No != null))
{
// Maps the entity to a DTO and adds it to the return collection.
JobBO job = MapBEJobforSearchObj(dbJob);
lstJobs.Add(job);
}
}
// Returns the DTOs.
getJobResponse.Jobs = lstJobs;
return getJobResponse;
}
First: pass your WHERE clause to EF to pass to the DB server rather than loading all entities into memory..
public GetJobsResponse GetImportJobs()
{
GetJobsResponse getJobResponse = new GetJobsResponse();
using (NSEXIM_V2Entities dbContext = new NSEXIM_V2Entities())
{
// Will pass the where expression to be DB server to be executed. Note: No .ToList() yet to leave this as IQueryable.
var jobs = dbContext.Job_Details..Where(ie => ie.IMP_EXP == "I" && ie.Job_No != null));
Next, use SELECT to load your DTOs. Typically these won't contain as much data as the main entity, and so long as you're working with IQueryable you can load related data as needed. Again this will be sent to the DB Server so you cannot use functions like "MapBEJobForSearchObj" here because the DB server does not know this function. You can SELECT a simple DTO object, or an anonymous type to pass to a dynamic mapper.
var dtos = jobs.Select(ie => new JobBO
{
JobId = ie.JobId,
// ... populate remaining DTO fields here.
}).ToList();
getJobResponse.Jobs = dtos;
return getJobResponse;
}
Moving the .ToList() to the end will materialize the data into your JobBO DTOs/ViewModels, pulling just enough data from the server to populate the desired rows and with the desired fields.
In cases where you may have a large amount of data, you should also consider supporting server-side pagination where you pass a page # and page size, then utilize a .Skip() + .Take() to load a single page of entries at a time.
I am having an issue understanding why when adding a new entity to a DbSet of ObjectContext, that entity is not found will looking it up again.
using (var db = new SmartrailDB())
{
var cart01 = db.Carts.SingleOrDefault(x => x.Number == 0);
if (cart01 == null)
{
cart01 = new Cart { Number = 0 };
db.Carts.Add(cart01);
}
var cart02 = db.Carts.SingleOrDefault(x => x.Number == 0); // Should find the cart I just added - right?
Assert.IsNotNull(cart02); // Fails because cart02 does not exist in the db.Carts collection
}
Is anyone able to tell me what I am doing wrong here?
Also late on a Friday here so brain half asleep now.
You have to update your context before you try to access the entity. Just do:
db.SaveChanges(); right after db.Cart.Add(cart01);
As you can see from the code below i use AsNoTracking to get my object.
I then even use ObjectSateManager to see what is going on and i can see
nothing being tracked in the l* collections and yet i still get
"An object with the same key already exists in the ObjectStateManager. The ObjectStateManager cannot track multiple objects with the same key".
Any ideas?
==========================================
BasketRepository repo = new BasketRepository();
var ba = repo.GetById(8);
var bpro = new BasketProduct(ba, ba.BasketProducts.First().Product, 3);
repo.AddToBasket(bpro);
repo.Save();
==================================
public Basket GetById(int basketId)
{
// eager-load product info
var basket = dbContext.Baskets.Include("BasketProducts")
.Include("BasketProducts.Product.Brand").AsNoTracking().SingleOrDefault(b => b.BasketId == basketId);;
return basket;
}
======================================
public void AddToBasket(BasketProduct product)
{
var ctx = ((IObjectContextAdapter)dbContext).ObjectContext;
ObjectStateManager objectStateManager = ctx.ObjectStateManager;
var l1 = objectStateManager.GetObjectStateEntries(EntityState.Added);
var l2 = objectStateManager.GetObjectStateEntries(EntityState.Modified);
var l3 = objectStateManager.GetObjectStateEntries(EntityState.Deleted);
//var l4 = objectStateManager.GetObjectStateEntries(EntityState.Detached);
var l5 = objectStateManager.GetObjectStateEntries(EntityState.Unchanged);
var existingProductInBasket = dbContext.BasketProducts.AsNoTracking().SingleOrDefault(b => b.BasketId == product.BasketId && b.ProductId == product.ProductId);
var l6 = objectStateManager.GetObjectStateEntries(EntityState.Added);
var l7 = objectStateManager.GetObjectStateEntries(EntityState.Modified);
var l8 = objectStateManager.GetObjectStateEntries(EntityState.Deleted);
//var l4 = objectStateManager.GetObjectStateEntries(EntityState.Detached);
var l9 = objectStateManager.GetObjectStateEntries(EntityState.Unchanged);
//objectStateManager.
dbContext.Entry<BasketProduct>(product).State = existingProductInBasket == null ? EntityState.Added : EntityState.Modified;
}
When you use asNoTracking() you get an unconnected entity that cannot be updated normally with EntityState.Modified. Then, one trick is to replace the cached entity values with non-cached entity values.
Here is the source code I use for special case where asNoTracking is necessary. It can be
private T ReplaceEntity(T cachedEntity, T nonCachedEntity) {
dbContext.Entry(cachedEntity).CurrentValues.SetValues(nonCachedEntity);
return cachedEntity;
}
A common use can be:
public virtual T FindFirstBy(System.Linq.Expressions.Expression<Func<T, bool>> predicate, bool asNoTracking = false)
{
if (asNoTracking)
{
T cachedEntity = dbContext.Set<T>().FirstOrDefault(predicate);
T nonCachedEntity = dbContext.Set<T>().AsNoTracking().FirstOrDefault(predicate);
return ReplaceEntity(cachedEntity, nonCachedEntity);
}
return dbContext.Set<T>().FirstOrDefault(predicate);
}
For your situation:
var cachedEntity = dbContext.BasketProducts.SingleOrDefault(b => b.BasketId == product.BasketId && b.ProductId == product.ProductId);
var nonCachedEntity = dbContext.BasketProducts.AsNoTracking().SingleOrDefault(b => b.BasketId == product.BasketId && b.ProductId == product.ProductId);
var product= ReplaceEntity(cachedEntity, nonCachedEntity);
dbContext.Entry<BasketProduct>(product).State = EntityState.Modified;
Hope it helps!!
That was actually by bad. It can be a bit difficult conceptually to get used to the idea of
how things get attached and tracked by EF. The secret is to remember that if you have an object
graph (objects with relationships to other objects) whenever an object of the graph gets attached with a state X then all the other objects in the graph seem to be attached as well in that state.
In my scenario i was quite stupidly using
var bpro = new BasketProduct(ba, ba.BasketProducts.First().Product, 3);
to create a test product that actually already existed in the database and it was part of the object
graph containing the basket and its products. When i tried to attach this "new" basket EF rightly complained that an object with the same key already exists in the attached graph!