Proxies retain a reference to pooled DbContext after it is returned to the pool #25486
Comments
|
It's just occurred to me that when switching from AddDbContextPool to AddDbContext, I'm getting database disposed errors, caused by code I've put in an AuthorizationAttribute that accesses the database wrapped in a using statement. I've now fixed that by getting the scope via HttpContext.RequestServices.CreateScope(), so I'll see if it mitigates the issue. |
|
It appears that removing the using statement wrapping the datacontext in the AuthorizationAttribute and instead getting the scope from the httpcontext has stopped the issue. I can only suppose that objects were being left behind by the disposal of the context, when all other cases had it handled either by my web api's controllers via DI or the injection of the scope factory. |
|
It appears that removing the using statement wrapping the datacontext in the AuthorizationAttribute and instead getting the scope from the httpcontext has stopped the issue. I can only suppose that objects were being left behind by the disposal of the context, when all other cases had it handled either by my web api's controllers via DI or the injection of the scope factory. |
|
It seems I still had a small memory issue while using the DbContextPool, which has now reduced by switching to AddDbContext instead. I'm not sure whether this implies there is actually a memory leak in the lazyloading framework.
|
|
@rexxmagnus we're going to have to see some code in order to investigate this, can you please submit a minimal repro project? |
|
Hi, @roji using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Hosting;
using Microsoft.AspNetCore.Mvc;
using Microsoft.EntityFrameworkCore;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using System;
using System.Collections.Generic;
using System.ComponentModel.DataAnnotations;
using System.ComponentModel.DataAnnotations.Schema;
using System.Linq;
using System.Text.Json.Serialization;
using System.Threading;
using System.Threading.Tasks;
namespace WebApp
{
public class Program
{
public static void Main(string[] args)
{
CreateHostBuilder(args).Build().Run();
}
public static IHostBuilder CreateHostBuilder(string[] args) =>
Host.CreateDefaultBuilder(args)
.ConfigureWebHostDefaults(webBuilder =>
{
webBuilder.UseStartup<Startup>();
}).ConfigureServices(services => {
// Launch the daemon background thread after the website has started
services.AddHostedService<Daemon>();
});
}
public class Startup
{
public Startup(IConfiguration configuration, IWebHostEnvironment environment)
{
Configuration = configuration;
HostingEnvironment = environment;
}
public IConfiguration Configuration { get; }
public IWebHostEnvironment HostingEnvironment { get; }
public void ConfigureServices(IServiceCollection services)
{
services.AddDbContextPool<MyDataContext>(options =>
{
options
.UseLazyLoadingProxies()
.UseSqlServer("myconnectionstring")
.UseQueryTrackingBehavior(QueryTrackingBehavior.NoTracking);
});
services.AddAuthentication(Microsoft.AspNetCore.Authentication.Cookies.CookieAuthenticationDefaults.AuthenticationScheme).AddCookie();
services.AddControllers().AddJsonOptions(options =>{
});
services.AddAuthorization();
services.AddCors(options =>
{
options.AddPolicy("all",
builder =>
{
builder.AllowAnyOrigin().AllowAnyMethod().AllowAnyHeader();
});
});
}
public void Configure(IApplicationBuilder app, IWebHostEnvironment env, IServiceProvider serviceProvider, IHostApplicationLifetime applicationLifetime)
{
app.UseDeveloperExceptionPage();
app.UseRouting();
app.UseCors("all");
app.UseAuthentication();
app.UseAuthorization();
app.UseEndpoints(endpoints =>
{
endpoints.MapControllerRoute(name: "default", pattern: "{controller}/{action}");
});
}
}
public class MyDataContext : DbContext
{
public MyDataContext(DbContextOptions<MyDataContext> options) : base(options) { }
public DbSet<User> Users { get; set; }
public DbSet<UserExtended> UserExtended { get; set; }
}
[Table("User")]
public class User
{
[Key, DatabaseGenerated(DatabaseGeneratedOption.None)]
public Guid UserID { get; set; }
// Other user properties
public DateTime? LastLogin { get; set; }
[InverseProperty("User")]
public virtual ICollection<UserExtended> UserOutcodes { get; private set; }
}
[Table("UserExtended")]
public class UserExtended
{
[Key, DatabaseGenerated(DatabaseGeneratedOption.None)]
public Guid UserExtendedID { get; set; }
[ForeignKey("User")]
public Guid UserID { get; set; }
// Other properties related to user
// Foreign key
public virtual User User { get; private set; }
}
internal class Daemon : IHostedService, IDisposable
{
private Timer _timer;
private readonly IServiceScopeFactory _scopeFactory;
public Daemon(IServiceScopeFactory scopeFactory)
{
_scopeFactory = scopeFactory;
}
public Task StartAsync(CancellationToken cancellationToken)
{
if (_timer == null)
{
_timer = new System.Threading.Timer(DaemonAction, null, new TimeSpan(0, 0, 20), new TimeSpan(0, 0, 20));
}
return Task.CompletedTask;
}
private void DaemonAction(object state)
{
// In production, this Daemon tracks various things like user alerts.
// With the incorrect code in the auth attribute (when using AddDbConnectionPool) the database
// context does not get released from memory correctly.
using (var scope = _scopeFactory.CreateScope())
{
MyDataContext dbCtx = scope.ServiceProvider.GetRequiredService<MyDataContext>();
// Update something in the database
dbCtx.Users.AsTracking().First().LastLogin = DateTime.UtcNow;
dbCtx.SaveChanges();
}
}
public Task StopAsync(CancellationToken cancellationToken)
{
_timer?.Change(Timeout.Infinite, Timeout.Infinite);
return System.Threading.Tasks.Task.CompletedTask;
}
public void Dispose()
{
_timer?.Dispose();
}
}
[AttributeUsage(AttributeTargets.Method, Inherited = false, AllowMultiple = false)]
public class AuthAttribute : Microsoft.AspNetCore.Authorization.AuthorizeAttribute, Microsoft.AspNetCore.Mvc.Filters.IAuthorizationFilter
{
#region Internal variables
#endregion
#region exposed parameters
#endregion
public void OnAuthorization(Microsoft.AspNetCore.Mvc.Filters.AuthorizationFilterContext context)
{
// In our production scenario, this method does some work every time a user is authenticated,
// such as extending their session time (there is other logic here).
// With this using block, if running startup with AddDbContextPool, this code works.
// with AddDbContext, it causes a database disposed when the page controller tries to access the db
// (this possibly makes sense if it's accessing and disposing the injected instance.
// With AddDbContextPool, it causes a memory leak by making the daemon's data context hold references to
// the ef core lazyloading library (at least that's what the memory dump implies)
using (MyDataContext dbCtx = context.HttpContext.RequestServices.GetService(typeof(MyDataContext)) as MyDataContext)
{
dbCtx.Users.First().LastLogin = DateTime.UtcNow;
dbCtx.SaveChanges();
}
/* The fix (to work with AddDbContext and prevent the leak)
using (var scope = context.HttpContext.RequestServices.CreateScope())
{
var dbCtx = scope.ServiceProvider.GetRequiredService<MyDataContext>();
dbCtx.Users.First().LastLogin = DateTime.UtcNow;
dbCtx.SaveChanges();
}
*/
}
}
[ApiController]
[Route("Test")]
public class TestController : ControllerBase
{
private readonly MyDataContext _dbCtx;
public TestController(MyDataContext dbCtx)
{
_dbCtx = dbCtx;
}
[Auth] // The authattribute defined above
[HttpPost]
[ActionName("Test")]
public IActionResult Test([FromBody] object data)
{
_dbCtx.Users.First().LastLogin = DateTime.UtcNow;
_dbCtx.SaveChanges();
return Ok(new { });
}
}
} |
|
I've included a screenshot of Azure's memory statistics after switching our production app to correctly use the database (AddDbContext rather than pool, with no usings on the context) but noticed some strange behaviour. The working memory set (the pale blue line) drops to about 30mb when the app becomes less active (the dip is during the night when no users are on) but the private bytes (the darker blue line above it) barely drops and remains around 400-450mb. Is this indicative of a continuing memory leak or something else?
|
|
@rexxmagnus This code is manually disposing the context managed by the request scope. The scope should instead be allowed to dispose the context when the scope is disposed. using (MyDataContext dbCtx = context.HttpContext.RequestServices.GetService(typeof(MyDataContext)) as MyDataContext)
{
dbCtx.Users.First().LastLogin = DateTime.UtcNow;
dbCtx.SaveChanges();
}With regard to holding on to the memory, every EF Core proxy has a reference to the DbContext from which it was queried. This means as long as the proxies are referenced, then the DbContext will also be referenced and not garbage collected. On the other hand, the DbContext should not be holding any references to the proxies once the DbContext has been disposed. If this is happening, then that would be a bug. Is this what you are asserting? If so, we will investigate that. |
|
@ajcvickers Yes, I understand your point about the disposal - once I realised that mistake, I allowed the scope to take control and the memory usage dropped drastically (the chart above shows the working set hovering around 30-50mb overnight now) however the private bytes allocation is still really high and only drops at peculiar times, which made me think the proxies have an issue. |
ajcvickers
changed the title
|
Notes for triage:
We should consider disconnecting proxies from the context when the context is returned to the pool, and possibly also when the context is disposed in non-pooling scenarios. |
|
Hello ! I'm seeing the same behavior. |
ajcvickers
added
consider-for-current-release
and removed
consider-for-next-release
labels
ajcvickers
changed the title
I've been using AddDbContextPool to inject an instance of my database into asp.net core controllers. In one instance, I also have a background task that runs via
The Daemon implements IHostedService and IDisposable and retrieves IServiceScopeFactory in its constructor. A timer uses this to create a database context (with the scope correctly wrapped in a using statement) if it needs to update the database (it's monitoring which users are logged in and updating their session times if they log out). Through the day, the memory usage of the app slowly increases, butforcing periodic garbage collection helps to slow this a little. However, taking a memory dump shows that a lot of memory is being used by EntityFrameworkCore.Internal.LazyLoader at the very top of the flame graph.
Two entries below (on the line below LIB <<System.Private.CoreLib!List>> is a reference to DependencyInjection.ServiceLookup.ServiceProviderEngineScope and then further references to the database context pool (I'm using AddDbContextPool to inject the database)
The timer doesn't capture the database scope other than via ServiceProvider.GetRequiredService() so I can't see where this memory is being held onto, unless there's an issue with the lazyloader (although the code I'm accessing in the database doesn't reference any lazy loaded entities.
EF Core version: 5.0.8
Microsoft.EntityFrameworkCore.SqlServer
Target framework: NET 5.0
Operating system: Windows 10 (publishing to Azure web app)
IDE: Visual Studio 2019 16.9.5
The text was updated successfully, but these errors were encountered: