I'd like to set up a development environment for our project that reflects the state of a production database, but any data modification on it would not be visible on the production. Does PostgreSQL provide features that would let me do that on the same database, for example making it feel as if the user had its own permamently uncommitted transaction?
How would you want this to work?
Imagine you modified something in the development database, and then the production database gets modified in an incompatible way.
Should the change on the production system fail? Should replication error out?
You could probably accomplish such a setup with Slony, but that is complicated, puts extra load on your production system, makes schema changes complicated and won't make you happy for the reason mentioned above.
The idea to have modifications in the development database take place in an uncommitted transaction is interesting, but how could you develop in a reasonable way if you cannot test transaction handling and other transaction can never see the work of your own transaction?
I think the best solution is to have a test database that is regularly updated from the production system, ideally by automatically restoring your nightly production database backup (that would have the additional advantage of testing your backups).
Related
I have this challenge. I am the DevOps engineer and a software engineer in a team where months back, the developers moved from having a central Oracle DB to having the DB on a CentOS VM on their individual laptops. The move from a central DB was to reduce dependency on the DBAs and also to eliminate issues that stemmed from inconsistent data.
The plan for sharing and ensuring synchronization of the Database with everyone on the team was that each person will share change scripts with everyone. The problem is that we use Skype for communication (we just setup slack but are yet to start using it fully), and although people sometimes post the text of DB change scripts, it could be missed by some. The other problem is that some developers miss posting the changes. Further, new releases are deployed in Production without being deployed on the Test and Demo environments.
This has posed a serious challenge for us, especially myself who of recent, became responsible for ensuring that our Demo deployments were in sync with the Production deployments.
Most of the synchronization issues border on the lack of sync of the Database due to missing change scripts or missing DB objects. Oracle is our DB of preference.
A typical deployment in the Demo environment is a very painful process that involves testing an application and as issues occur due to missing DB table columns, functions, stored procs, we have to look for the missing DB objects, apply them to the DB and then continue until all issues are resolved.
How can I solve this problem to ensure smooth, painless and less time-consuming deployments? Can migrating our applications to Docker help with the DB synchronization issues and the associated lack of discipline of the developers? What process can we put into place to improve in this area?
Thank you very much in advance for your help.
Have a look # http://www.dbmaestro.com
I strongly recommend you to join the live demo session
DBmaetro TeamWork can help you merge the changes from multiple DBs into a single shared DB and to move safely the changes from one environment to the other
Danny
For example when a developer makes changes in any of the database elements in a business critical database it should force them to commit the code before applying the changes to database itself. I came across Redgate sql source control which matches my expectation somehow. Still do we have any more tools or effective database practices that I am missing here?
If you use SQL Source Control or a tool like it (eg, ReadyRoll or VS Database Projects) I'd recommend also using DLM Dashboard.
The reason for this is that no tool can enforce changes to go through a process if people are given (too many) rights and are able to apply changes to production. It's then up to these people to correctly follow the process.
Although DLM Dashboard doesn't enforce changes to your database, it will alert you on changes made to production, warning you when out-of-process changes (aka "drift").
DLM Dashboard is free, which is another reason to use it!
I've been using MongoDB for about a year now, however not nearly up to its potential.
I've been developing new software, out of anyone's eyes except my own, and I've enjoyed the flexibility of the database to its fullest and I've made major structural changes to data on the fly.
Now I'm at a point where I have production server(s) and 3 development servers, I'm having a real problem with changing data structures and syncing them up.
Theoretically the development servers should always have the most current data from production. In a structured database, if I rename something, I can just run a compare tool and do the corresponding change in production after a pull. In MongoDB, this can become incredibly difficult.. there could be hundreds of changes from document to document, much less from database to database.
I've been reviewing my ~/.dbshell file to kinda get the feel of changes I've made, but what about changes made within the program its self? Configuration database changes?
Are there tools or procedures that are around to make this easier?
I've spent hours on Google researching how others do it. I came across Mongeez, but it's more manual and tedious than I need. In the past, I just do a mongodump and mongorestore inside of a git directory to transport data, but these snapshots are too rigid. I read a few blog posts regarding moving new data from production to development, but nothing about updating development documents in production. I could write a comparison script, but I feel like this is reinventing the wheel. There has to be a better way.
TL;DR: What are some ways to version NoSQL data, new entries and changed data, between environments?
I had a similar problem/experience while managing a few production Mongo machines for about a year.
Two quick pieces of advice:
WiredPrairie is right. Version your documents and that will allow you to migrate in a casual/relaxed manner. I wish we had done that up front. One of my biggest regrets.
We used Groovy to connect and do our schema/data changes and I loved it. The language is easy to learn and it works great with JSON. My practice was to back up the collections I'd be operating on, write the scripts in dev, run them and if I messed up, restore the backed up collections. Iterate until I got the scripts perfect and then repeat in production.
This is a question for those of you developing on a team of devs where all of you have separate databases. You're versioning your database using source control and other tools which will automatically bring dev databases up to date to the latest version of the database (schema, data, SP's, functions, etc.).
OK Great! But wait! What if you are developing on version 4.0 of your software, but now you need to switch branches to the 3.2 branch to fix a bug? The schema could be (almost assuredly is) very different by now...
I suppose if you went through the extra effort to write rollback scripts along with your change scripts, this could work. But that seems like a lot of work - is it really worth it?
Much easier would be to create a new 3.2-branch database and work with that while working on the 3.2-branch code. It doesn't seem reasonable to me to require that each developer has exactly one database to work with.
I'm going on a limb and assume that you are versioning the database as a binary? If all your database assets were in the form of constructive code (eg SQL scripts and/or text data dumps), the solution would be simple, as suggested by Mark: store these assets as part of the development branch. To work on version 3.2, switch the branch, re-run the create scripts and presto, 3.2 database. Merging would be just as easy as with regular code (or just as painful, depending on your version control system of choice).
Here are some suggestions to work in this mode:
If creating the database instances from text is too slow, make a cache on a shared disk volume, keyed by the contents of all the schema / data files (or the MD5 sum thereof).
Write a pre-commit hook to ensure that the schema and data dumps in the developer's instance are the same as the ones under version control. This prevents people from making changes to their dev database with an interactive tool, and then forgetting to commit them.
You mention change scripts; treat them as a liability. While they may be required by your deployment scenario (eg for customers who want to upgrade in-place), they duplicate information from the version history of the database, and per Murphy's law duplication means desynchronization sooner or later. Try to auto-generate the change scripts from the versioned database assets using "diff"; or if this cannot be achieved, dedicate some serious unit tests to database upgrades.
I have a database project that goes through iterations (only one so far) and I need to deploy a testing version to a live server. I'm not sure how to go about this.
I can make all the changes in a copy and then remake those changes in the live version. That doesn't make sense.
Is there a way to change a server name to an existing server? What's the best practice for this scenario?
With a Visual Studio Database Project, you should be able to have as many database connections defined as you like. When you go run your scripts, you can pick an menu option called "Run On...." and then pick which server connection to run those scripts on.
Just make sure the database name is the same for both instances, or make sure that you do not specify USE (database) at the top of all your scripts, if the database names are different from target to target.
In the first place, you should have scripts already written for the changes you made. They should be in source control. No changes ever should be made to database structure without a script and versioning.
Since you don't appear to have what you should have to deploy, then you need to tool to check the differences between the databases. Redgate's SQL Compare is the one to buy.
Be careful of simply using the tool without thinking, there may be changes in dev you are not yet ready to promote to prod. Read through the scripts before running them.
Also you may need SQL Data Compare to run against any lookup tables you have to see if new values have been added in dev that need to go to prod. Again these inserts should have been scripted and in source control and then deploying is simple.
Maybe I'm misunderstanding the question, but I don't see how you could just swap the databases. If you make a development version of a database and update the schema, you must surely run some tests and update the data. You can't just make that the development database now because it's full of test data.
What you need to do is run a tool that compares the old schema to the new schema and then apply these changes to the production database. There are tools out there on the market to do this. Failing that, you could dump the old and new schemas, run them through an ordinary file compare to get the differences, and then build an update script out of that.
On my present project we use what I think is a terrible practice: We keep a hand-maintained script of schema updates for each version, and every time someone makes a change they're supposed to update this script. Every now and then someone makes a mistake and we have to scramble to figure out what went wrong. Like we just had a problem deploying to our user acceptance test because someone updated the create statement for a new table to include a foreign key to another new table ... not realizing that the table being referenced was created until further down in the script. It worked fine it test because the tables were created in an order that made it work.
My conclusion is you're much better off to just make changes to the schema on the fly, then when you're done, run an automated compare to generate the ALTER statements.
By the way, on a project I worked on a few years ago, for a desktop application where each customer had their own copy of the database, we put in what I thought was a very nice feature: Every time the program started up, it compared the schema of the database to what it thought it ought to be, and if they didn't match, it automatically updated it. So when they installed a new version, it just automatically updated the database the first time they ran it.