In Play (scala) I have a number of evolutions in conf/evolutions/default called 1.sql, 2.sql etc...
Some of these are from playing around, and some are from tutorial code I no longer use.
How do I get rid of these evolutions?
The obvious approach of deleting the evolution files does not seem to work. If you remove the file, the evolution is still applied. Altering the file works, and so the current workaround is emptying the .sql files when they are longer required.
In virtually every migration framework/library/approach it works the same:
if you are using migrations/evolutions seriously (you deploy to production or at least you cooperate with other people, who wouldn't want their environment broken) - you simply don't remove migration. If you want to remove it, write a new migration that reverts the previous one.
if changes are only on your own branch, you haven't deployed it anywhere, you haven't shared your code - remove file, remove lines form file and drop and recreate database - migrations are backed up in execution in the database they are executed against (at least the majority of the tools I used do this), so if you want to get rid of migration it also need to be removed the the table that stores executed migrations. To ensure that things are consistent, the easiest way is to drop and rerun migrations/evolutions.
I cannot stress this enough - if you deployed you code anywhere, do not remove the migration. Hell can break loose. But it you haven't deployed it anywhere, because it's e.g. just a tutorial, just drop the database and do whatever you want.
Related
Need some sophisticated thing - somehow like Liquibase, but not for DB. We have many *.bat files, that need to be executed per each release on PROD. Problem is that only new and updated files should be executed, others shoud be avoided.
Is there some thing to do that? Thanks!
I'm trying to migrate our database engine from MsSql to PostgreSQL. In our automated test, we restore the database back to "clean" state at the start of every test. We do this by comparing the "diff" between the working copy of the database with the clean copy (table by table). Then copying over any records that have changed. Or deleting any records that have been added. So far this strategy seems to be the best way to go about for us because per test, not a lot of data is changed, and the size of the database is not very big.
Now I'm looking for a way to essentially do the same thing but with PostgreSQL. I'm considering doing the exact same thing with PostgreSQL. But before doing so, I was wondering if anyone else has done something similar and what method you used to restore data in your automated tests.
On a side note - I considered using MsSql's snapshot or backup/restore strategy. The main problem with these methods is that I have to re-establish the db connection from the app after every test, which is not possible at the moment.
If you're okay with some extra storage, and if you (like me) are particularly not interested in re-inventing the wheel in terms of checking for diffs via your own code, you should try creating a new DB (per run) via templates feature of createdb command (or CREATE DATABASE statement) in PostgreSQL.
So for e.g.
(from bash) createdb todayDB -T snapshotDB
or
(from psql) CREATE DATABASE todayDB TEMPLATE snaptshotDB;
Pros:
In theory, always exact same DB by design (no custom logic)
Replication is a file-transfer (not DB restore). So far less time taken (i.e. doesn't run SQL again, doesn't recreate indexes / restore tables etc.)
Cons:
Takes 2x the disk space (although template could be on a low performance NFS etc)
For my specific situation. I decided to go back to the original solution. Which is to compare the "working" copy of the database with "clean" copy of the database.
There are 3 types of changes.
For INSERT records - find max(id) from clean table and delete any record on working table that has higher ID
For UPDATE or DELETE records - find all records in clean table EXCEPT records found in working table. Then UPSERT those records into working table.
I am working on a system that currently has a number of environments (test, stage, live, etc) each with their own database. So far these databases have been kept in sync by running the same update scripts on each of them.
We are now moving to using EF6 code first migrations, and would also like to start writing some automated system tests using LocalDB.
I've found https://msdn.microsoft.com/pt-pt/data/dn579398 which describes two options for adding an initial migration.
The first method creates an empty initial migration which will work great for the existing environments but won't help with creating LocalDBs for testing.
The second method creates a migration to bring up the whole database from scratch (minus things EF doesn't care about such as sprocs and views). This would be acceptable for testing, but not good for actually recreating a databse. It also requires you to manually comment out the Up method, run the migration on all existing databases, and then put the Up method back. As it will take a while to get the migration through all the environments I'm not keen on this. It also violates the one of the principles of migrations which is that they shouldn't be edited once they've been released.
Having some kind of conditionality in migrations would solve my problem (e.g. if(tableExists("A_table_in_the_existing_database") return;) but there doesn't seem to be anything like that available.
The best I've come up with is to dump the existing database schema from SQL server to a file (which has the advantage of preserving sprocs, views, etc) and then use option 2 above, except instead of using the generated Up method I'll run the SQL file.
However, this still has the drawbacks of option 2 mentioned above, so I'd be very happy to learn of a better way of handling this situation.
Edit:
Would this work? Run the commented out initial migration on one database, then dumping out the __MigrationHistory table and inserting it into the other databases? That way I wouldn't have to wait for the migration to make it through all the environments before I could safely uncomment it.
EF 6.1.2 has support for running SQL embedded as a resource within the assembly containing the migrations using the SqlResource method.
I think I'd go with scripting out your existing schema and using an initial migration that executes SqlResource as its Up. If all it's doing is a bunch of IF EXISTS then it shouldn't take too long to run. Otherwise scripting out __MigrationHistory will also work if you just want to run it once locally and apply to all your other databases by hand.
I am currently in the processes of setting up Quartz in a load balanced environment using the JDBC job store and I am wondering how everyone manages the quartz job store DB.
For me Quartz (2.2.0) will be deployed as a part of a versioned application with multiple versions potentially existing on the one server at the one time. I am using the notation XXScheduler_v1 to ensure multiple schedulers play nice together. My code is working fine, with the quartz tables being populated with the triggers/jobs/etc as appropriate.
One thing I have noticed though is that there seems to be no database cleanup that occurs when the application is undeployed. What I mean is that the Job/Scheduler data seems to stay in the quartz database even though there is no longer a scheduler active.
This is less than ideal and I can imagine with my model the database would get larger than it needed to be with time. Am I missing how to hook-up some clean-up processes? Or does quartz expect us to do the db cleanup manually?
Cheers!
I got this issue once, and here is what I did to rectify the issue. This will work for sure but in case it does not then we will have backup of table so you don't have anything to loose while trying this.
Take sql dump of following tables using method mentioned at : Taking backup of single table
a) QRTZ_CRON_TRIGGERS
b) QRTZ_SIMPLE_TRIGGERS
c) QRTZ_TRIGGERS
d) QRTZ_JOB_DETAILS
Delete data from above tables in sequence as
delete from QRTZ_CRON_TRIGGERS;
delete from QRTZ_SIMPLE_TRIGGERS;
delete from QRTZ_TRIGGERS;
delete from QRTZ_JOB_DETAILS;
Restart your app which will then freshly insert all deleted tasks and related entries in above tables (Provided your app has its logic right).
This is more like starting your app with all the tasks being scheduled for the first time. So you must keep in mind that tasks will behave as if these are freshly inserted.
NOTE: If this does not work then apply the backup you took for tables and try to debug more closely. As of now, I have not seen this method fail.
It's definitely not doing any DB cleanup when undeploying the application or shutting down the scheduler. You would have to build some cleanup code during application shutdown (i.e. building some sort of StartupServlet or context listener that would do the cleanup on the destroy() event lifecycle)
You're not missing anything.
However, these quartz tables aren't different from any applicative DB objects you use in you data model. You add Employees table and in a later version you don't need it anymore. Who's responsible for deleting the old table? Only you. If you habe a DBA you might roll it on the DBA ;).
This kind of maintenance would typically be done using an uninstall script / wizard, upgrade script / wizard, or during the first startup of the application in its new version.
On a side note, typically different applications use different databases, or different schemas for the least, thus reducing inter-dependencies.
To clean Quartz Scheduler internal data one needs more SQL:
delete from QRTZ_CRON_TRIGGERS;
delete from QRTZ_SIMPLE_TRIGGERS;
delete from QRTZ_TRIGGERS;
delete from QRTZ_JOB_DETAILS;
delete from QRTZ_FIRED_TRIGGERS;
delete from QRTZ_LOCKS;
delete from QRTZ_SCHEDULER_STATE;
I've started looking into Entity Framework migrations on 4.3.1. Have a few questions:
What's preferred during development? Why should I not just drop and recreate my
database always and then reseed. If I use code first migrations, can
I choose to seed my db initially and then add a seed method to each
migration to only add in new data? If i use automatic migrations, is
it possible to do something similar? i.e. seed initially and then
seed as required?
What is the benefit of using migrations during development? I only
actually need migrations when moving to production. So, I need to
create my initial script and then scripts for each migration, so
would it be possible to only use migrations once i want to move to
production and at that point create an initial script and maintain a
migration history from that point onwards?
Well, in our case, we started to use Migrations because in our company, devs don't have the necessary rights to create a DB, which lead to the amusing scenario where I dropped the DB a couple of times and had to ask the db admin to recreate it each time...
In my opinion, it seems easier to incrementally grow your DB, rather than having to recreate it each time. If I were to have to drop and recreate our DB every time a property is added, deleted or changed, I'd never see the end of it.
I've not yet seen a possibility for incremental seeds, unless perhaps if you create manual migration files.
Migrations has the possibility to go to a specific version (either forwards or backwards) and it is possible to generate an SQL script from a migration.
So basically, you don't have to create a migration SQL script by hand anymore, you can get Migrations to do it for you.