I've set up mvc-mini-profiler against my Entity Framework-powered MVC 3 site. Everything is duly configured; Starting profiling in Application_Start, ending it in Application_End and so on. The profiling part works just fine.
However, when I try to swap my data model object generation to providing profilable versions, performance slows to a grind. Not every SQL query, but some queries take about 5x the entire page load. (The very first page load after firing up IIS Express takes a bit longer, but this is sustained.)
Negligible time (~2ms tops) is spent querying, executing and "data reading" the SQL, while this line:
var person = dataContext.People.FirstOrDefault(p => p.PersonID == id);
...when wrapped in using(profiler.Step()) is recorded as taking 300-400 ms. I profiled with dotTrace, which confirmed that the time is actually spent in EF as usual (the profilable components do make very brief appearances), only it is taking much longer.
This leads me to believe that the connection or some of its constituent parts are missing sufficient data, making EF perform far worse.
This is what I'm using to make the context object (my edmx model's class is called DataContext):
var conn = ProfiledDbConnection.Get(
/* returns an SqlConnection */CreateConnection());
return CreateObjectContext<DataContext>(conn);
I originally used the mvc-mini-profiler provided ObjectContextUtils.CreateObjectContext method. I dove into it and noticed that it set a wildcard metadata workspace path string. Since I have the database layer isolated to one project and several MVC sites as other projects using the code, those paths have changed and I'd rather be more specific. Also, I thought this was the cause of the performance issue. I duplicated the CreateObjectContext functionality into my own project to provide this, as such:
public static T CreateObjectContext<T>(DbConnection connection) where T : System.Data.Objects.ObjectContext {
var workspace = new System.Data.Metadata.Edm.MetadataWorkspace(
GetMetadataPathsString().Split('|'),
// ^-- returns
// "res://*/Redacted.csdl|res://*/Redacted.ssdl|res://*/Redacted.msl"
new Assembly[] { typeof(T).Assembly });
// The remainder of the method is copied straight from the original,
// and I carried over a duplicate CtorCache too to make this work.
var factory = DbProviderServices.GetProviderFactory(connection);
var itemCollection = workspace.GetItemCollection(System.Data.Metadata.Edm.DataSpace.SSpace);
itemCollection.GetType().GetField("_providerFactory", // <==== big fat ugly hack
BindingFlags.NonPublic | BindingFlags.Instance).SetValue(itemCollection, factory);
var ec = new System.Data.EntityClient.EntityConnection(workspace, connection);
return CtorCache<T, System.Data.EntityClient.EntityConnection>.Ctor(ec);
}
...but it doesn't seem to make much of a difference. The problem still exists whether I use the above hacked version that's more specific with metadata workspace paths or the mvc-mini-profiler provided version. I just thought I'd mention that I've tried this too.
Having exhausted all this, I'm at my wits' end. Once again: when I just provide my data context as usual, no performance is lost. When I provide a "profilable" data context, performance plummets for certain queries (I don't know what influences this either). What could mvc-mini-profiler do that's wrong? Am I still feeding it the wrong data?
I think this is the same problem as this person ran into.
I just resolved this issue today.
see: http://code.google.com/p/mvc-mini-profiler/issues/detail?id=43
It happened cause some of our fancy hacks were not cached well enough. In particular:
var workspace = new System.Data.Metadata.Edm.MetadataWorkspace(
new string[] { "res://*/" },
new Assembly[] { typeof(T).Assembly });
Is a very expensive call, so we need to cache the workspace.
Profiling, by definition, will effect performance of the application being profiled. The profiler needs to insert it's own method calls throughout the application, intercept low level system calls, and record all that data someplace (meaning writes to disk). All of those tasks take up precious CPU cycles, memory, and disk access.
Related
I was wondering what happens when callr_function = NULL?
Is it just issues with things maybe being in the environment/side effects?
Mainly wondering because I was passing quite large spatio-temporal arrays (0.5 to 5 gigs) and callr serialization via saveRDS is quite slow.
The two things I was thinking about was forking callr and dropping in a different save function or just using callr_function = NULL.
Ordinarily, targets runs the pipeline in a fresh new reproducible external R session. callr_function = NULL just says to run the pipeline in the current R session. I only recommend this for debugging because in serious use cases you could accidentally invalidate some targets based on changed data in your global environment. callr_function = NULL will probably not help solve issues with large memory. For that, I recommend selecting a more efficient storage format for your data, e.g. tar_target(..., format = "feather"). You could also try tar_option_set(memory = "transient", garbage_collection = TRUE) for better memory efficiency.
I have been reading everywhere that the 'right' way to use a context is the following:
using (var Db = new MyDatabase())
{
DoStuff(Db);
}
But I have found this to be painfully slow in some cases.
I have a function that is called a lot, from different threads, through web requests and is READ ONLY.
This is my test code:
var S = new Stopwatch()
S.Start();
Parallel.For(
0,
15000,
I =>
{
var P = GetPostWebContent(I);
});
S.Stop();
Console.WriteLine(S.ElapsedMilliseconds);
With the 'recommended' method the execution time is 303.6s for 15000 calls, 20.24ms per call.
Now, by keeping one context per thread, I get very different results, using the following code:
[ThreadStatic]
private static MyDatabase _Db;
if (_Db == null) _Db = new DatabaseWebsite();
DoStuff(Db);
And the execution time is 52.4s for 15000 calls, 3.5ms per call.
Which means that this solution is 5.8x faster!
I am a little bit shocked: I keep reading online that the context creation is not a big deal, etc; but in my case, it has a huge impact.
I am aware that with this implementation the context will keep growing in size, but I could implement a counter and re-create the context regularly.
Is there anything that would prevent this from being used? As I said above, all the operation are READ ONLY, I am not writing anything to the database.
It's just a recommendation. Context is not thread safe, so having one per thread is a good idea if performance is important for you.
I tend to pass contexts around methods that run on the same thread because I've found performance issues with creating contexts. It's worked on production code for me.
I have a class inheriting from DbContext implementing a code-first Entity Framework object.
I would like to have a unit test that exercises the model builder in this dbcontext -- this would be useful for detecting things like 'key not defined on entity' errors.
But I may not have an actual database available during unit testing time. Is there a way to exercise this code without actually trying to make a database connection with the context. I tried something innocuous like:
var ctx = new MyDbContext("Data Source=(local);Initial Catalog=Dummy;<.......>");
var foo = ctx.GetValidationErrors(); //triggers DBModelBuilder, should throw if my DBModel is goofed up
This does technically work. However this takes a very long time to run -- if I pause and inspect the call stack it is triggering a call to System.Data.SqlClient.SqlInternalConnectionTds.AttemptOneLogin
Eventually it times out, swallows the connection error and finishes my test.
Is there any way to do this without the connection attempt?
The solution here is to just kind of go with the problem and use the lowest possible connection timeout. I'm using this connection string:
Server=localhost; Database=tempdb; Integrated Security=true;Connection Timeout=1;ConnectRetryCount=0
It still triggers the problem, but with a one second timeout and no automatic retries (for only this one test in the system) it's totally acceptable.
And if the developer has a local db installed, it will accidentally work even faster.
I am new to Lift and I am thinking whether I should investigate it more closely and start using it as my main platform for the web development. However I have few "fears" which I would be happy to be dispelled first.
Security
Assume that I have the following snippet that generates a form. There are several fields and the user is allowed to edit just some of them.
def form(in : NodeSeq): NodeSeq = {
val data = Data.get(...)
<lift:children>
Element 1: { textIf(data.el1, data.el1(_), isEditable("el1")) }<br />
Element 2: { textIf(data.el2, data.el2(_), isEditable("el2")) }<br />
Element 3: { textIf(data.el3, data.el3(_), isEditable("el3")) }<br />
{ button("Save", () => data.save) }
</lift:children>
}
def textIf(label: String, handler: String => Any, editable: Boolean): NodeSeq =
if (editable) text(label, handler) else Text(label)
Am I right that there is no vulnerability that would allow a user to change a value of some field even though the isEditable method assigned to that field evaluates to false?
Performance
What is the best approach to form processing in Lift? I really like the way of defining anonymous functions as handlers for every field - however how does it scale? I guess that for every handler a function is added to the session with its closure and it stays there until the form is posted back. Doesn't it introduce some potential performance issue when it comes to a service under high loads (let's say 200 requests per second)? And when do these handlers get freed (if the form isn't resubmitted and the user either closes the browser or navigate to another page)?
Thank you!
With regards to security, you are correct. When an input is created, a handler function is generated and stored server-side using a GUID identifier. The function is session specific, and closed over by your code - so it is not accessible by other users and would be hard to replay. In the case of your example, since no input is ever displayed - no function is ever generated, and therefore it would not be possible to change the value if isEditable is false.
As for performance, on a single machine, Lift performs incredibly well. It does however require session-aware load balancing to scale horizontally, since the handler functions do not easily serialize across machines. One thing to remember is that Lift is incredibly flexible, and you can also create stateless form processing if you need to (albeit, it will not be as secure). I have never seen too much of a memory hit with the applications we have created and deployed. I don't have too many hard stats available, but in this thread, David Pollak mentioned that demo.liftweb.net at the time had 214 open sessions consuming about 100MB of ram (500K/session).
Also, here is a link to the Lift book's chapter on Scalability, which also has some more info on security.
The closure and all the stuff is surely cleaned at sessionShutdown. Earlier -- I don't know. Anyway, it's not really a theoretical question -- it highly depends on how users use web forms in practice. So, for a broader answer, I'd ask the question on the main channel of liftweb -- https://groups.google.com/forum/#!forum/liftweb
Also, you can use a "statical" form if you want to. But AFAIK there are no problems with memory and everybody is using the main approach to forms.
If you don't create the handler xml/html -- the user won't be able to change the data, that's for sure. In your code, if I understood it correctly (I'm not sure), you don't create "text(label,handler)" when it's not needed, so everything's secure.
My code is pretty simple:
Context.AddObject("EntitiesSetName", newObjectName);
Context.SaveChanges();
It worked fine, but just one time – the first one. That time, I interrupted my program by Shift+F5 after the SaveChanges() was traced. It was a debug process, so I manually removed a newly created record from a DB and ran a program again in the debug mode. But it does not work anymore – it “hangs” when SaveChanges() is being called.
Another strange thing that I see:
If I write before addObject() and SaveChanges() are called something like:
var tempResult = (from mydbRecord in Context
where Context.myKey == 123
select mydbRecord.myKey).Count();
// 123 is the key value of the record that should be created before the program hangs.
tempResult will have the next value: 1.
So, it seems that the record is created (when the program hung) and now exists, but when I check the DB manually using other tools – it does not!
What do I do wrong? Is it some kind of cache issue or something else?
EDIT:
I've found a source of problem.
It was not EF problem at all, but it's a problem of the tool that I use to control the database manually (Benthic).
My program falls into some kind of deadlock (when I call SaveChanges()) with the tool when the tool is connected into the same DB.
So, the problem is in the synchronization area, imho, so my question can be marked as solved.