ASP.NET Core/EF Core 2.0 using too much memory that results SQL Server Timeouts - entity-framework

tl;dr I am not cleaning up the context, as I assume that is all handled with DI/automatically being scoped (this might be my problem)
I have a ASP.NET Core that utilizes EF Core and after a while my site being live, the memory on the dotnet process for the app shoots to 1GB, at that point I see a lot of SQL Timeout errors. If I kill the process then those errors go away after the process re-spawns on a request for a while and it repeats..
I suspect somewhere in my code I have a memory leak or my design is not correct that results in something not cleaning up properly.
Here is my setup.
public void ConfigureServices(IServiceCollection services)
{
services.AddMemoryCache();
var connectionString = Configuration.GetSection("GeneralSettings:ConnectionString").Value;
services.AddDbContext<CryptoCalContext>(options => options.UseSqlServer(connectionString));
...
services.AddScoped<IDbService, DbService>();
}
I pass the context to a service(DbService) handles all my DB calls
public class DbService : IDbService
{
public CryptoCalContext DbContext { get; set; }
public DbService(CryptoCalContext context, ...)
{
DbContext = context;
...
}
public List<CategoryDto> Categories
{
get
{
return MemoryCache.GetOrCreate("Categories", entry =>
{
entry.SlidingExpiration = TimeSpan.FromMinutes(GeneralSettings.CacheExperationMinutes);
return DbContext.Categories.Include(x => x.Articles).Select(x => new CategoryDto(x)).ToList();
});
}
}
}
And in my calls to db, I do not make use of "using blocks" or call dispose.
I assume each call creates a separate db connection when needed so I do not do any of that (maybe this is where I am going wrong)
public class HomeController : BaseController
{
public HomeController(IDbService dbService ,..)
: base(dbService, ...)
{
}
}
public class BaseController : Controller
{
public IDbService DbService { get; set; }
public BaseController(IDbService dbService, ...)
{
DbService = dbService;
}
}
In my controllers, I simply do DbService.Categories, one of many different calls in my DbService, I put this example to illustrate that I do zero cleaning
EDIT:
public class CryptoCalContext : DbContext
{
public CryptoCalContext(DbContextOptions<CryptoCalContext> options)
: base(options)
{
}
public CryptoCalContext()
{
}
...Properties...
protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
{
base.OnConfiguring(optionsBuilder);
}
}

Related

ASP.NET Core 2.2 How to add two constructors to the same class using dependency injection?

I have a class "ConnectorManagement" in which I need to use both SignalR services as well as querying a db table using EF CORE.
I cant work out how to load both dbcontext and hubcontext into the same class using a constructor and dependancy injection. The current result is visual studio fails to load the project when run in debug. Tried researching this but not understanding what needs to be done.
Current code below:
namespace myNamespace.Controller
{
public class ConnectorManagement : IHostedService
{
private static readonly log4net.ILog log = log4net.LogManager.GetLogger(typeof(Logger));
private readonly IHubContext<MessageHub> _hubContext;
public readonly ApplicationDbContext _context;
public ConnectorManagement(IHubContext<MessageHub> hubContext, ApplicationDbContext context)
{
_hubContext = hubContext;
_context = context;
}
public Task StartAsync(CancellationToken cancellationToken)
{
log.Info("Initial Test");
return Task.CompletedTask;
}
public Task StopAsync(CancellationToken cancellationToken)
{
throw new NotImplementedException();
}
dbcontext class:
namespace myNamespace.Data
{
public class ApplicationDbContext : IdentityDbContext
{
public ApplicationDbContext(DbContextOptions<ApplicationDbContext> options)
: base(options)
{
}
public DbSet<myProject.Models.ConnectorInbound> ConnectorInbound { get; set; }
public DbSet<myProject.Models.ConnectorOutbound> ConnectorOutbound { get; set; }
public DbSet<myProject.Models.SystemMapping> SystemMapping { get; set; }
}
}
startup class:
namespace myProjectNamespace
{
public class Startup
{
public Startup(IConfiguration configuration)
{
Configuration = configuration;
}
public IConfiguration Configuration { get; }
// This method gets called by the runtime. Use this method to add services to the container.
public void ConfigureServices(IServiceCollection services)
{
services.Configure<CookiePolicyOptions>(options =>
{
// This lambda determines whether user consent for non-essential cookies is needed for a given request.
options.CheckConsentNeeded = context => true;
options.MinimumSameSitePolicy = SameSiteMode.None;
});
services.AddDbContext<ApplicationDbContext>(options =>
options.UseSqlServer(
Configuration.GetConnectionString("DefaultConnection")));
services.AddDefaultIdentity<IdentityUser>()
.AddDefaultUI(UIFramework.Bootstrap4)
.AddEntityFrameworkStores<ApplicationDbContext>();
// Start up the TcpServerTcpServer engine
services.AddHostedService<ConnectorManagement>();
services.AddMvc().SetCompatibilityVersion(CompatibilityVersion.Version_2_2);
services.AddSignalR();
}
// This method gets called by the runtime. Use this method to configure the HTTP request pipeline.
public void Configure(IApplicationBuilder app, IHostingEnvironment env, ILoggerFactory loggerFactory)
{
if (env.IsDevelopment())
{
app.UseDeveloperExceptionPage();
app.UseDatabaseErrorPage();
}
else
{
app.UseExceptionHandler("/Error");
// The default HSTS value is 30 days. You may want to change this for production scenarios, see https://aka.ms/aspnetcore-hsts.
app.UseHsts();
}
app.UseHttpsRedirection();
app.UseStaticFiles();
app.UseCookiePolicy();
app.UseAuthentication();
app.UseSignalR(routes =>
{
routes.MapHub<MessageHub>("/messageHub");
});
loggerFactory.AddLog4Net();
app.UseMvc();
}
}
}
I wasn't paying attention to the fact that you're injecting this into a hosted service. Hosted services are singletons and both the hub context and database context are scoped services. You need to inject IServiceProvider instead and then create a scope. This will need to be done for every usage; you cannot persist it on an ivar, for example. You can only use it within the using statement.
using (var scope = _serviceProvider.CreateScope())
{
var context = scope.ServiceProvider.GetRequiredService<ApplicationDbContext>();
// Do something
}

How to dispose Entity Framework Core in-memory database

I want to create clean inmemory database in each unit test.
When I run multiple tests, data from previous tests remains in the database. How to dispose existing inmemory database?
I initialize each test with following code:
[TestInitialize]
public void TestInitialize()
{
Services = new ServiceCollection();
Services.AddScoped<DbContextOptions<MyDbContext>>(sp => new DbContextOptionsBuilder<TacsDbContext>()
.UseInMemoryDatabase("MyTestDbContext")
.Options);
Services.AddTransient<InMemoryMyDbContext>();
Services.AddTransient<MyDbContext>(sp => sp.GetService<InMemoryTacsDbContext>());
ServiceProvider = Services.BuildServiceProvider();
}
[TestMethod]
public void Test1()
{
using (var dbContext = ServiceProvider.GetService<MyDbContext>()) ...
}
[TestMethod]
public void Test2()
{
using (var dbContext = ServiceProvider.GetService<MyDbContext>()) ...
}
I use .NET Core 2.0, and Entity Framework Core 2.0
EDIT
I wasn't able to use the standard registration: Services.AddDbContext<InMemoryMyDbContext>(...), because
public class InMemoryMyDbContext : MyDbContext
{
public InMemoryMyDbContext(DbContextOptions<InMemoryMyDbContext> options)
: base(options) { } //compiler error
public InMemoryMyDbContext(DbContextOptions<MyDbContext> options)
: base(options) { } //runtime IoC error
}
public class MyDbContext : DbContext
{
public MyDbContext(DbContextOptions<MyDbContext> options)
: base(options) { }
}
Ok, the solution in my case was to call DbContextOptionsBuilder.UseApplicationServiceProvider()
Services.AddScoped<DbContextOptions<MyDbContext>>(sp => new DbContextOptionsBuilder<MyDbContext>()
.UseApplicationServiceProvider(sp)
.UseInMemoryDatabase("Test")
.Options);
This method is called automatically when you setup ServiceCollection the usuall way, so in following case the database is created from scratch each time
Services.AddDbContext<MyDbContext>(options => options.UseInMemoryDatabase("Test"));
At the end, I was able to modify MyDbContext so that I call the line above:
public class MyDbContext : DbContext
{
protected MyDbContext (DbContextOptions options) : base(options) { }
public MyDbContext (DbContextOptions<MyDbContext> options)
: this((DbContextOptions)options) { }
}
public class InMemoryMyDbContext : MyDbContext
{
public InMemoryMyDbContext (DbContextOptions<InMemoryTacsDbContext> options)
: base(options) { }
}
Services.AddDbContext<InMemoryMyDbContext>(options =>
options.UseInMemoryDatabase("Test"), ServiceLifetime.Transient);
Services.AddTransient<MyDbContext>(sp => sp.GetService<InMemoryMyDbContext>());
Consider using the Nuget package Effort
It is a simple and fast in-memory database ideal for unit-testing
You can start it with an empty database; if desired fill it using a database seeder, or fill it with values from a test CSV file.
See Tutorials Effort - Entity Framework Unit Testing Tool
Your DbContext probably will look similar to:
class MyDbContext : DbContext
{
public MyDbContext() : base() { } // constructor using config file
public BloggingContext(string nameOrConnectionString) :
base(nameOrConnectionString) { }
public DbSet<...> ...{ get; set; }
public DbSet<...> ...{ get; set; }
}
Just add one constructor and you can use your in-memory database as if it was your original database:
public MyDbContext(DbConnection connection) : base(connection, true) { }
You pass the DbConnection to the test database, this connection is passed to the DbContext class. The true parameter says that when the DbContext is disposed, that the DbConnection should also be Disposed.
Usage would be:
[TestMethod]
public void UnitTest_X()
{
var dbConnection = Effort.DbConnectionFactory.CreateTransient();
using (var dbContext = new MyDbContext(dbConnection)
{
// perform your test of MyDbContext as if it was connected to your
// original database
}
}
A simple copy-paste code example for a console program with an Effort database that uses a one-to-many relationship can be found here on StackOverFlow

How to pass options from Startup.cs to DbContextOptions constructor function at ASP.NET Core 2.0

I am using ASP.NET Core 2.0
At Startup.cs I have
services.AddDbContext<MailDBServicesContext>(optionsSqLite =>
{
optionsSqLite.UseSqlite("Data Source=Mail.db");
});
I have created a model and a DbContext where DbContext is:
public class MailDBServicesContext : DbContext
{
public MailDBServicesContext(DbContextOptions<MailDBServicesContext> options)
: base(options)
{
}
public DbSet<MailCountSentErrorMails> DbSetMailCountSentErrorMails { get; set; }
}
from a Class helper I need to pass DbContextOptions and my question is how can I tell to use the options from the Startup.cs ConfigureServices method
using (var db = new MailDBServicesContext())
{
}
It should be enough to simply inject MailDBServicesContext into your controller or a service class, for example.
public class SomeDataService
{
private readonly MailDBServicesContext _dbContext;
public SomeDataService(MailDBServicesContext dbContext)
{
_dbContext = dbContext ?? throw new ArgumentNullException(nameof(dbContext));
}
public async Task AddMailCounts()
{
_dbContext.DbSetMailCountSentErrorMails
.Add(new MailCountSentErrorMails { CountSentMails = 55 });
await _dbContext.SaveChangesAsync();
}
}
Other DB context configuration options are defined in Configuring a DbContext on MSDN.
Update
Make sure to register your service in DI, i.e. ConfigureServices method.
public void ConfigureServices(IServiceCollection services)
{
services.AddTransient<ISomeDataService, SomeDataService>();
services.AddDbContext<MailDBServicesContext>(optionsSqLite =>
{
optionsSqLite.UseSqlite("Data Source=Mail.db");
});
services.AddMvc();
}
Then make a call to AddMailCounts() in your controller.
public class HomeController : Controller
{
private readonly ISomeDataService _dataService;
public HomeController(ISomeDataService dataService)
{
_dataService = dataService ?? throw new ArgumentNullException(nameof(dataService));
}
public IActionResult Index()
{
_dataService.AddMailCounts();
return View();
}
}
Now every time you load homepage, a record is inserted into DbSetMailCountSentErrorMails table.
You can find working solution on my GitHub.

Understanding Asp.Net MVC Identity and Owin

Disclaimer: I have already read and seen LOTS of tutorials/videos and unfortunately am still not clear on how this works. What I know:
At application Startup ConfigureAuth is executed which contains
app.CreatePerOwinContext(GE.Core.DbContexts.ApplicationDbContext.Create);
This in turn goes to ApplicationDbContext class and establishes a connection
public class ApplicationDbContext : IdentityDbContext<ApplicationUser, Role, int, UserLogin, UserRole, UserClaim>
{
public DbSet<User> geUsers { get; set; }
public ApplicationDbContext(string connString)
: base(connString)
{
Database.SetInitializer(new MySqlInitializer());
}
public static ApplicationDbContext Create()
{
return new ApplicationDbContext("name=GEContext");
}
}
Question 1: is there a way to not establish this connection until the login page appears? OR What if i keep a dummy connection in my we.config which should be invalid as soon as i sign-in and replaced with the actual connection?
After the login page is loaded, i take a schoolCode at Login and want to establish a database connection based on that schoolCode. e.g. name = GEContext_[schoolCode]. My Web.config will contain many connectionStrings like this.
Question 2 When I Debug my application, every time a page is loaded, Create method of ApplicationDbContext is hit and a default connection
return new ApplicationDbContext("name=GEContext");
always seems to establish..What's actually going on?
Here is my remaining code:
ApplicationUserManager
public class ApplicationUserManager : UserManager<ApplicationUser, int>
{
public ApplicationUserManager(IUserStore<ApplicationUser, int> store)
: base(store)
{
}
public static ApplicationUserManager Create(IdentityFactoryOptions<ApplicationUserManager> options, IOwinContext context)
{
var manager = new ApplicationUserManager(new UserStore<ApplicationUser, Role, int, UserLogin, UserRole, UserClaim>(context.Get<ApplicationDbContext>()));
....
}
}
I have another GEContext
[DbConfigurationType(typeof(MySql.Data.Entity.MySqlEFConfiguration))]
public class GEContext : DbContext
{
public GEContext(string connString):base(connString)
{
this.Configuration.LazyLoadingEnabled = false;
Database.SetInitializer(new MySqlInitializer());
}
}
which works fine from the controller
private static string schoolCode = (string)System.Web.HttpContext.Current.Session["SchoolCode"];
[Authorize]
public ActionResult About()
{
YearRepository repYear = new YearRepository("name=GEContext_" + schoolCode);
}
How do i do the same with applicationdbcontext? but above all questions how does it work?

UnitOfWork and Entity Framework Contexts

So the problem I am trying to solve is this; We are using Entity Framework to access our Oracle database that has 1200-1500 tables. Now mind you we are not accessing them all, but possibly could have 800+ to access. We are using the UnitOfWork --> Repository --> Service pattern and that works great, but we are trying to figure out if we should have one big DbContext, or multiple little contexts that are specific to the task at hand.
Our UnitOfWork is setup using an EFUnitOfWorkBase like so:
public abstract class EFUnitOfWorkBase : IUnitOfWork
{
private bool isDisposed = false;
public DbContextBase Context { get; set; }
protected EFUnitOfWorkBase(DbContextBase context)
{
Context = context;
}
public int Commit()
{
return Context.SaveChanges();
}
public void Dispose()
{
if (!isDisposed)
Dispose(true);
GC.SuppressFinalize(this);
}
private void Dispose(bool disposing)
{
isDisposed = true;
if (disposing)
{
if (this.Context != null)
this.Context.Dispose();
}
}
public IRepository<TEntity> GetRepository<TEntity>() where TEntity : Common.EntityBase<TEntity>
{
return new Repository<TEntity>(this);
}
}
Any unit of work we create extends that base one and provides the context like so:
public class EmployeeDirectoryUnitOfWork : EFUnitOfWorkBase
{
public EmployeeDirectoryUnitOfWork(string connectionString)
: base(new EmployeeDirectoryContext(connectionString))
{
}
}
The DbContext is passed a connection string through the unit of work.
The Repository looks like this:
public abstract class RepositoryBase<TEntity> : IRepository<TEntity> where TEntity : class
{
protected DbContextBase Context;
protected DbSet<TEntity> EntitySet;
public RepositoryBase(EFUnitOfWorkBase unitOfWork)
{
Enforce.ArgumentNotNull(unitOfWork, "unitOfWork");
Context = unitOfWork.Context;
EntitySet = Context.Set<TEntity>();
}
public TEntity Add(TEntity entity)
{
Enforce.ArgumentNotNull(entity, "entity");
return EntitySet.Add(entity);
}
public TEntity Attach(TEntity entity)
{
Enforce.ArgumentNotNull(entity, "entity");
return EntitySet.Attach(entity);
}
public TEntity Delete(TEntity entity)
{
Enforce.ArgumentNotNull(entity, "entity");
return EntitySet.Remove(entity);
}
public System.Linq.IQueryable<TEntity> Query()
{
return EntitySet.AsQueryable();
}
public TEntity Save(TEntity entity)
{
Enforce.ArgumentNotNull(entity, "entity");
Attach(entity);
Context.MarkModified(entity);
return entity;
}
}
Any suggestions on how to best handle this situation?
In such a case when you have a large application like this, I think you should probably go for a more Domain Driven Design approach and split the contexts into some separate, bounded contexts. This way when later developers are adding features to the program they will be confined to only being able to access certain tables depending on which context they will be using there.
For better information, Julie Lerman recently came out with a course on Pluralsight about Entity Framework in the Enterprise that's really good. She posted a small clip of it (actually about bounded contexts) on this site. It's a very good course, and I highly recommend it, especially for what you appear to be doing.