I hope my question title is clear enough. What I am trying to do is to concurrently execute update scripts on all Databases in the same Server using SQL Server Management Objects. My current experiment implementation is like this:
var server = new Server(new ServerConnection(new SqlConnection(connString)));
server.ConnectionContext.InfoMessage += new SqlInfoMessageEventHandler(DBOutputReceived);
var databases = server.Databases.Cast<Database>().ToList();
Parallel.ForEach(databases, db => { db.ExecuteNonQuery(script); });
I'm not even sure this is possible using SMO, but this is basically what I want to do. I can see that the ConnectionContext.InfoMessage event will likely have issues, but I'm not sure how to resolve. Should I instead create a separate Server instance for each parallel task?
I would appreciate any advice you can give.
Thanks, Daniel
The solution is to have each thread create it's own Server/ServerConnection object, that way there is no shared memory between the threads.
Related
We have about 200 user define functions in DB2. These UDF are generated by datastudio into a single script file.
When we create a new DB, we need to run the script file several times because some UDF are dependent on other UDF and cannot be create until the precedent functions are created first.
Is there a way to generate a script file so that the order they are deployed take into account this dependency. Or is there some other technique to arrange the order efficiently?
Many thanks in advance.
That problem should only happen if the setting of auto_reval is not correct. See "Creating and maintaining database objects" for details.
Db2 allows to create objects in an "unsorted" order. Only when the object is used (accessed), the objects and its depending objects are checked. The behavior was introduced a long time ago. Only some old, migrated databases keep auto_reval=disabled. Some environments might set it based on some configuration scripts.
if you still run into issues, try setting auto_reval=DEFERRED_FORCE.
The db2look system command can generate DDL by by object creation time with the -ct option, so that can help if you don't want to use the auto_reval method.
Use case: We have a quite large database (about 200 tables) that is used in a large (legacy) system. It's implemented as a database-first approach, with one edmx file defining the entire database. We are using XUnit and Effort for automatic testing. The problem is that these tests are very slow. It takes something like 7-8 minutes to run our current test suite, even though test coverage isn't anywhere near what we want it to be.
I've noticed that if I create a smaller subset of the edmx file, by removing some tables that aren't needed, tests run faster.
I'm looking for a solution where for a particular test, or suite of tests, we can somehow make Effort only create the subset of tables that are needed (I think in many cases, we'll only need one table).
Currently we're setting up our connection like this:
connection = EntityConnectionFactory.CreateTransient("metadata=res://entities.csdl|res://entities.ssdl|res://entities.msl");
Is there some way we can (for instance, by running an XML transformation in runtime), make Effort only create the data structures it need for a subset of tables that we define?
Disclaimer: I'm the owner of the project Entity Framework Effort
Our library has a feature that allows creating a restore point and rollbacking to it.
So by using this trick, you could use the CreateRestorePoint() only once when all tables are created and then for every test, start them with RollbackToRestorePoint. (There is several other ways to make it works but I guess you get the point)
It will without a doubt make your test run A LOT faster since the table will not have to be created every time.
Here is an example:
var conn = Effort.DbConnectionFactory.CreateTransient();
using (var context = new EntityContext(conn))
{
context.EntitySimples.Add(new EntitySimple { ColumnInt = 1 });
context.EntitySimples.Add(new EntitySimple { ColumnInt = 2 });
context.EntitySimples.Add(new EntitySimple { ColumnInt = 3 });
context.SaveChanges();
}
// Create a RestorePoint that will save all current entities in the "Database"
conn.CreateRestorePoint();
// Make any change
using (var context = new EntityContext(conn))
{
context.EntitySimples.RemoveRange(context.EntitySimples);
context.SaveChanges();
}
// Rollback to the restore point to make more tests
conn.RollbackToRestorePoint();
Separate out Unit test and Integration test. For Integration test you can use Database and run on higher environments (to save time) but on local environments you can make use of Faker\Bogus and NBuilder to generate massive data for unit test.
https://dzone.com/articles/using-faker-and-nbuilder-to-generate-massive-data
Other option is you can create resource file corresponding to your unit test cases
https://www.danylkoweb.com/Blog/the-fastest-way-to-mock-a-database-for-unit-testing-B6
I would also like to take you look at InMemoryDB vs SqlLite performance,
http://www.mukeshkumar.net/articles/efcore/unit-testing-with-inmemory-provider-and-sqlite-in-memory-database-in-ef-core
Although above example is for EFCore, in EF6 also we can use SqlLite
https://www.codeproject.com/Tips/1056400/Setting-up-SQLite-and-Entity-Framework-Code-First
So my recommendation for you is to go with sqllite for Integration testing scenarios. For Unit test you can go either with sqllite or with Faker\Bogus and NBuilder.
Hope it helps!
Is it possible to obtain SQL queries with inlined parameters of EF context?
I would like to be able to copy-paste queries into management studio and execute them.
I work with ctx.Database.Log delegate, but it traces queries with parameters' values placed inside comments, eg:
.... WHERE [Extent1].[Name] = #p__linq__0
-- p__linq__0: 'ApproveReminder' (Type = AnsiString, Size = 8000)
Which are not runnable without first editing them.
If I understand your question properly, you can do this with SQL Server Profiler.
Run it, and then set up a profile watching the server that EF is connecting to. While it's running, execute your code so that the EF query you're interested in runs against the server. Then stop the profile.
If you peruse what it's recorded, you should be able to identify your query quite easily. Then click on it and the full query text will appear in the window at the bottom of the screen.
You could use Hibernating Rhinos Entity Framework Profiler (commercial software) or you could do this:
var query = context.Customers.Where(...);
((System.Data.Objects.ObjectQuery) query).ToTraceString();
http://visualstudiomagazine.com/blogs/tool-tracker/2011/11/seeing-the-sql.aspx
I have started using tsqlt, and my question is it possible to have a separate database with just the testing stuff? (tables/sp's/assemblies etc).
This testing database will sit on the same instance as the actual/target database.
If I try to fake a table I get the following error:
FakeTable could not resolve the object name, 'target_db.dbo.Sometable'
Has anyone had any experience with this?
Thanks.
As you discovered, this isn't currently possible as the mocking procedures don't accept three part names. This is something that's been covered at the User feedback forums of SQL Test (RedGate's product that acts as a front end to tSQLt) at : http://sqltest.uservoice.com/forums/140716-sql-test-forum/suggestions/2421628-reduce-the-footprint
Dennis Lloyd, one of the authors of the tSQLt framework wrote towards the end of that thread that support of a separate 'tSQLt' database was something they would keep under consideration.
Also a related issue of mocking remote objects at http://sqltest.uservoice.com/forums/140716-sql-test-forum/suggestions/2423449-being-able-to-mock-fake-remote-objects
I hope that helps,
Dave
You can now do this, so long as the tSQLt framework is in the other database:
EXEC tSQLt.FakeTable '[dbo].[Position]';
EXEC OtherDB.tSQLt.FakeTable '[dbo].[PositionArchive]';
Source
This means that you can at least put your tests where you want them, though you have to install the framework in the actual database under test. Which is not perfect, but it's better.
I am currently using Entity Framework 4.2 with Code First. I currently have a Windows 2008 application server and a database server running on Amazon EC2. The application server has a Windows Service installed that runs once per day. The service executes the following code:
// returns between 2000-4000 records
var users = userRepository.GetSomeUsers();
// do some work
foreach (var user in users)
{
var userProcessed = new UserProcessed { User = user };
userProcessedRepository.Add(userProcessed);
}
// Calls SaveChanges() on DbContext
unitOfWork.Commit();
This code takes a few minutes to run. It also maxes out the CPU on the application server. I have tried the following measures:
Remove the unitOfWork.Commit() to see if it is network related when the application server talks to the database. This did not change the outcome.
Changed my application server from a medium instance to a high CPU instance on Amazon to see if it is resource related. This caused the server not to max out the CPU anymore and the execution time improved slightly. However, the execution time was still a few minutes.
As a test I modified the above code to run three times to see if execution time for the second and third loop using the same DbContext. Every consecutive loop took longer to run that the previous one but that could be related to using the same DbContext.
Am I missing something? Is it really possible that something as simple as this takes minutes to run? Even if I don't commit to the database after each loop? Is there a way to speed this up?
Entity Framework (as it stands) isn't really well suited to this kind of bulk operation. Are you able to use one of the bulk insert methods with EC2? Otherwise, you might find that hand-coding the T-SQL INSERT statements is significantly faster. If performance is important then that probably outweighs the benefits of using EF.
My guess is that your ObjectContext is accumulating a lot of entity instances. SaveChanges seems to have a phase that has time linear in the number of entities loaded. This is likely the reason for the fact that it is taking longer and longer.
A way to resolve this is to use multiple, smaller ObjectContexts to get rid of old entity instances.