.NET Entity Framework Core 3.1 - Nlog leaking memory when :format=# when calling update on a big model and a db error occurs - entity-framework

Happens in the save below:
context.Update(myBigObject);
await context.SaveChangesAsync(); -> Hangs here and RAM goes up until all the memory is finished
It only happens when a db error occurs ("String or binary data would be truncated" to be exact, meaning trying to stuff too large string in a field that's too small).
It happens when nlog config is:
${exception:format=#}
One "fix" is to change it to:
${exception:format=toString} -> But then I lose all the inner exception logging
See nlog docs on the difference between :format=# and :format=toString:
https://github.com/NLog/NLog/wiki/Exception-Layout-Renderer
It's happening to more people than me (see bottom comment) and happening in both Serilog and Nlog (so maby it's a EF Core thing):
https://github.com/dotnet/efcore/issues/24663#issuecomment-1349965403
Any idea how to fix without using :format=toString in nlog config?

NLog 4.7 allows you to override the reflection for a specific exception-type (Ex. Microsoft.EntityFrameworkCore.DbUpdateException) like this:
LogManager.Setup().SetupSerialization(s =>
s.RegisterObjectTransformation<Microsoft.EntityFrameworkCore.DbUpdateException>(ex => new {
Type = ex.GetType().ToString(),
Message = ex.Message,
StackTrace = ex.StackTrace,
Source = ex.Source,
InnerException = ex.InnerException,
})
);
See also: https://github.com/NLog/NLog/wiki/How-to-use-structured-logging#customize-object-reflection

Related

Exception Thrown by Manatee.Trello.RestSharp.RestSharpResponse

I just came across an exception while trying something with Manatee.Trello. I was trying to create a Func like this:
var criteria = new List<string>
{
"(put a board ID here)"
};
var query = new Func<IEnumerable<Manatee.Trello.Member>, IEnumerable<Manatee.Trello.Board>>(
members =>
{
foreach (var member in members)
{
var selectedBoards = member.Boards.Where(b => criteria.Contains(b.Id, StringComparer.Ordinal));
boards.AddRange(selectedBoards); // Exception thrown here
}
return boards;
});
But the line marked above throws this exception:
System.TypeLoadException
{"Method 'get_StatusCode' in type 'Manatee.Trello.RestSharp.RestSharpResponse' from assembly 'Manatee.Trello.RestSharp, Version=0.0.0.0, Culture=neutral, PublicKeyToken=783b036be1eaf5a7' does not have an implementation.":"Manatee.Trello.RestSharp.RestSharpResponse"}
I'm not sure if this is something I'm doing wrong with my code, or some kind of setup error I made in setting up my project with Manatee.Trello, perhaps the NuGet packages are jacked up...
Any tips on where to start looking would be much appreciated.
This situation was remedied by changing from the RestSharp provider included in Manatee.Trello to the provider in Manatee.Trello.WebApi. As I discovered in the library's documentation:
Manatee.Trello.RestSharp is backed by (you guessed it) RestSharp.
There have been some issues with the .Net 4.5+ versions, so it's
suggested you don't use this one unless you are using .Net 4.0 or
earlier.
Indeed, I am using .Net 4.5.2.

Data driven unit test breaking entity framework connection

I have an application that uses entity framework. I am writing a unit test in which I would like to use data driven testing from a CSV file.
However, when I run the test, I get an error that the sqlserver provider cannot be loaded:
Initialization method UnitTest.CalculationTest.MyTestInitialize threw
exception. System.InvalidOperationException:
System.InvalidOperationException: The Entity Framework provider type
'System.Data.Entity.SqlServer.SqlProviderServices,
EntityFramework.SqlServer' registered in the application config file
for the ADO.NET provider with invariant name 'System.Data.SqlClient'
could not be loaded. Make sure that the assembly-qualified name is
used and that the assembly is available to the running application.
If I remove the data driven aspects and just test a single value, then the test works.
If I just use the data driven aspects and remove the Entity Framework stuff, then the test works.
So, its only when I try to use data driven test with entity framework active at the same time do I get the error. So, where am I going wrong here?
Here's my test method:
[TestMethod, TestCategory("Calculations")
, DataSource("Microsoft.VisualStudio.TestTools.DataSource.CSV"
, "ConvertedMeanProfileDepth.csv", "ConvertedMeanProfileDepth#csv"
, Microsoft.VisualStudio.TestTools.UnitTesting.DataAccessMethod.Sequential)
, DeploymentItem("ConvertedMeanProfileDepth.csv")]
public void ConvertedMeanProfileDepthTest()
{
ConvertedMeanProfileDepth target = new ConvertedMeanProfileDepth();
Decimal mpd = decimal.Parse(this.TestContext.DataRow["mpd"].ToString());
Decimal expected = decimal.Parse(this.TestContext.DataRow["converted"].ToString());
Decimal actual;
actual = target.Calculate(mpd);
Assert.AreEqual(expected, actual);
}
So I managed to work it out in the end. For future reference, here's the solution:
Rob Lang's post, Entity Framework upgrade to 6 configuration and nuget magic, reminded me of the issue here:
When a type cannot be loaded for a DLL that is referenced in a
project, it usually means that it has not been copied to the output
bin/ directory. When you're not using a type from a referenced
library, it will not be copied.
And this will raise its ugly head the moment you use deployment items in your tests. If you use a deployment item in your test, then all of the required binaries are copied to the deployment directory. Problem is, if you are using dynamically loaded items, then the test suite does not know it has to copy those items.
With Entity Framework, this means that your providers will not be copied to the deployment location and you will receive the error as per my question.
To resolve the issue, simply ensure that your entity framework provider is also marked as a deployment item.
So, note the inclusion of DeploymentItem(#"EntityFramework.SqlServer.dll") in my test attributes. All works perfectly from here:
[TestMethod, TestCategory("Calculations")
, DataSource("Microsoft.VisualStudio.TestTools.DataSource.CSV"
, "ConvertedMeanProfileDepth.csv", "ConvertedMeanProfileDepth#csv"
, Microsoft.VisualStudio.TestTools.UnitTesting.DataAccessMethod.Sequential)
, DeploymentItem("ConvertedMeanProfileDepth.csv")
, DeploymentItem(#"EntityFramework.SqlServer.dll")]
public void ConvertedMeanProfileDepthTest()
{
ConvertedMeanProfileDepth target = new ConvertedMeanProfileDepth();
Decimal mpd = decimal.Parse(this.TestContext.DataRow["mpd"].ToString());
Decimal expected = decimal.Parse(this.TestContext.DataRow["converted"].ToString());
Decimal actual;
actual = target.Calculate(mpd);
Assert.AreEqual(expected, actual);
}

Breeze: cannot execute _executeQueryCore until metadataStore is populated

I was using Breeze v1.1.2 that came with the Hot Towel template which has now been extended to form my project. I made the mistake of updating the NuGet package to the current 1.3.3 (I never learn). Anyway, all was well, and now not so much!
I followed the instructions in the release notes and other docs to change my BreezeWebApiConfig file to:
[assembly: WebActivator.PreApplicationStartMethod(
typeof(BreezeWebApiConfig), "RegisterBreezePreStart")]
namespace MyApp.App_Start {
public static class BreezeWebApiConfig {
public static void RegisterBreezePreStart() {
GlobalConfiguration.Configuration.Routes.MapHttpRoute(
name: "BreezeApi",
routeTemplate: "breeze/{controller}/{action}"
);}}}
And the config.js file (which provides the serviceName to the EntityManager constructor) to:
var remoteServiceName = 'breeze/breeze'; // NEW version
//var remoteServiceName = 'api/breeze'; // OLD version
And my BreezeController if you're interested:
[BreezeController]
public class BreezeController : ApiController
{
readonly EFContextProvider<MyDbContext> _contextProvider =
new EFContextProvider<MyDbContext>();
[HttpGet]
public string Metadata()
{
return _contextProvider.Metadata();
}
[HttpGet]
public IQueryable<SomeItem> SomeItems()
{
// Do stuff here...
}
}
Now I get the "cannot execute _executeQueryCore until metadataStore is populated" error.
What am I missing here?
EDIT:
I perhaps left out the part you needed... Above in the SomeItems() method, the stuff that actually gets done is a call to the GetMeSomeData() method in the MyDBContext class. This method makes the following call to a stored procedure to get the data.
public virtual ObjectResult<SomeItem> GetMeSomeData(string inParam)
{
var p = new object[] { new SqlParameter("#inParam", inParam) };
var retVal = ((IObjectContextAdapter)this).ObjectContext.ExecuteStoreQuery<SomeItem>("exec GetData #SN", p);
return retVal;
}
Now given my limited understanding, the call to Metadata() is not failing, but I don't think it has any idea what the entity model is when coming back, even though somewhere along the line, it should figure that out from the entity model I do have (i.e. SomeItem)? The return string from Metadata() doesn't have any information about the entity. Is there a way to make it aware? Or am I just completely off in left field playing with the daisies?
Hard to say based on this report. Let's see if Breeze is right.
Open the browser debugging tools and look at the network traffic. Do you see an attempt to get metadata from the server before you get that error? If so, did it succeed? Or 404? Or 500? What was the error?
I'm betting it didn't even try. If it didn't, the usual reason is that you tried some Breeze operation before your first query ... and you didn't ask for metadata explicitly either. Did you try to create an entity? That requires metadata.
The point is, you've got to track down the Breeze operation that precipitates the error. Sure everything should just work. The world should be rainbows and unicorns. When it isn't, we heave a sigh, break out the debugger, and start with the information that the error gave us.
And for the rest of you out there ... upgrading to a new Breeze version is a good thing.
Happy coding everyone.
Follow-up to your update
Breeze doesn't know how you get your data on the back-end. If the query result has a recognizable entity in it, Breeze will cache that. It's still up to you in the query callback to ensure that what you deliver to the caller is something meaningful.
You say that you're server-side metadata method doesn't have any idea what SomeItem is? Then it's not much use to the client. If it returns a null string, Breeze may treat that as "no metadata at all" in which case you should be getting the "cannot execute _executeQueryCore until metadataStore is populated" error message. Btw, did you check the network traffic to determine what your server actually returned in response to the metadata request (or if there was such a request)?
There are many ways to create Metadata on the server. The easiest is to use EF ... at least as a modeling tool at design time. What's in that MyDbContext of yours? Why isn't SomeItem in there?
You also can create metadata on the client if you don't want to generate it from the server. You do have to tell the Breeze client that you've made that choice. Much of this is explained in the documentation "Metadata Format".
I get the feeling that you're kind of winging it. You want to stray from the happy path ... and that's cool. But most of us need to learn to walk before we run.

How do I resolve "No views were found in assemblies or could be generated for Table"?

I am working with a database first model in Entity Framework 5 and when I attempt to add a row, I get the following error:
"No views were found in assemblies or could be generated for Table 'ui_renewals'."
The table exists in my EDMX and the template generated a ui_renewals class. I've deleted the table from the EDMX and added it again using the Update Model from Database option and I get the same error. Creating a separate connection for it resolves the issue, but that seems like a less-than-ideal solution (more like a kludge) not to mention it makes it more difficult to maintain in the future.
Any ideas on how to fix this so that I can add or update (I've tried both) a row in ui_renewals?
Here is the code I'm currently using - only difference before was using db as a DBContext instead of ui (yes, receipt is misspelled - gotta love legacy stuff)
[HttpPost]
public bool UpdateTeacher(string login_id, string password, UIRenewal data)
{
if (ModelState.IsValid)
{
// map from UIRenewal VM to ui_renewal
ui_renewals Renewal = Mapper.Map<UIRenewal, ui_renewals>(data);
// check to see if this is a new entry or not
var tmp = ui.ui_renewals.Find(Renewal.reciept);
if (tmp == null)
ui.ui_renewals.Add(Renewal);
else
{
// mark as modified
db.Entry(Renewal).State = EntityState.Modified;
}
// save it
try
{
ui.SaveChanges();
}
catch (DBConcurrencyException)
{
return false;
}
return true;
}
return false;
}
I should mention that I do have a view in the model (v_recent_license).
I know this is a very old question, however as I haven't found any other topics like this, I'll post my answer.
I have had the same Exception thrown. I found that, in a failed attempt to optimize EF performance, following the advices found here, I left behind this piece of code in EF .edmx code-behind:
<EntityContainerMapping StorageEntityContainer="XXXModelStoreContainer" CdmEntityContainer="YYYEntities" GenerateUpdateViews="false">
I removed the GenerateUpdateViews="false" string, and all is working again.
(The Exception message is a little misleading in my opinion).

XML serialization errors when trying to serialize Entity Framework objects

I have entities that I am getting via Entity Framework. I'm using Code-First so they're POCOs. When I try to XML Serialize them using XmlSerializer, I get the following error:
The type
System.Data.Entity.DynamicProxies.Song_C59F4614EED1B7373D79AAB4E7263036C9CF6543274A9D62A9D8494FB01F2127
was not expected. Use the XmlInclude
or SoapInclude attribute to specify
types that are not known statically.
Anybody got any ideas on how to get around this (short of creating a whole new object)?
Just saying POCO doesn't really help (especially in this case since it looks like you're using proxies). Proxies come in handy in a lot of cases but make things like serialization more difficult since the actual object being serialized is not really your object but an instance of a proxy.
This blog post should give you your answer.
http://blogs.msdn.com/b/adonet/archive/2010/01/05/poco-proxies-part-2-serializing-poco-proxies.aspx
Sorry, I know I'm coming at this a bit late (a couple YEARS late), but if you don't need the proxy objects for lazy loading, you can do this:
Configuration.ProxyCreationEnabled = false;
in your Context. Worked like a charm for me. Shiv Kumar actually gives better insight into why, but this at least will get you back to work (again, assuming you don't need the proxies).
Another way that works independent of the database configuration is by doing a deep clone of your object(s).
I use Automapper (https://www.nuget.org/packages/AutoMapper/) for this in my code-first EF project. Here is some sample code that exports a list of an EF objects called 'IonPair':
public bool ExportIonPairs(List<IonPair> ionPairList, string filePath)
{
Mapper.CreateMap<IonPair, IonPair>(); //creates the mapping
var clonedList = Mapper.Map<List<IonPair>>(ionPairList); // deep-clones the list. EF's 'DynamicProxies' are automatically ignored.
var ionPairCollection = new IonPairCollection { IonPairs = clonedList };
var serializer = new XmlSerializer(typeof(IonPairCollection));
try
{
using (var writer = new StreamWriter(filePath))
{
serializer.Serialize(writer, ionPairCollection);
}
}
catch (Exception exception)
{
string message = string.Format(
"Trying to export to the file '{0}' but there was an error. Details: {1}",
filePath, exception.Message);
throw new IOException(message, exception);
}
return true;
}