Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 8 years ago.
Improve this question
The goal is to build a concise SQL script to alter/update tables since changes have been made to the schema between any two points in time.
For example, I develop on one machine and on Day "A" I used the dump & restore utilities to install a database on a production machine. Then on Day "B" after making some changes on my development machine and testing them, I need to get those changes to my schema onto my production server.
Short of writing every single command I make to my schema (some of which may be experimental and undone), what is a good way to manage upgrading a schema from point A to point B (or point B to point F for that matter)?
Update:
It seems that diff-like concepts for databases are very much frowned upon with good reason. So this leaves me with new questions.
What is a simple method to distinctly manage your experimental changes from your production-worthy changes? Just keep restoring your dev database to a last known good state when you do something unfavorable?
Can postgresql be configured to log all of your actions in a way that can be pulled out as used as an update script? The reason I ask is that I enjoy working with PgAdminIII, and I would rather use that to work than to write update scripts for building or experimenting.
Short of writing every single command I make to my schema
If you want to do it in a controlled and "professional" way, there is no way around that. You should consider using a schema management tool to help you organize and run those migration scripts:
Liquibase
Flyway
Our experience with Liquibase is very good. We use it for migrations on Oracle, DB2 and PostgreSQL.
For a Postgres specific solution you might want to have a look at Sqitch
Related
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 2 years ago.
Improve this question
I have searched for information about it but still cannot find anything convincing.
I have multiple containerized websites with apache and php, these in turn are exposed through a reverse proxy with virtual hosts for each container, but, I've been thinking about the database, most use mariadb 5.5 but there is one site web required by mariadb 10.
I was wondering if it was a good idea for each container on each website to embed its own instance of mariadb or create a unique container for this, but I have some doubts.
Mariadb uses its own load balancing system, the container will affect its use if it had to raise multiple instances of the same database even though they all use the same data directory? I'm wondering if the engine will have to do the same indexing multiple times or there will be conflict in the use of files.
Having the website in a container has no problems because the files do not undergo changes and the logs and uploaded files are stored in persistent volumes, but in the case of the database it is different because I do not know if it is a good idea that multiple engines make use of the same data directory.
In a productive environment where the database has a high query load, is it recommended to use a container? Or is it better to embed the database inside the website container or do a native installation on the server?
In which cases should I choose one or the other option?
Absolutely do not have 2 databases share the same data directory. Only 1 database server should manage its own volume.
If you need more databases because you want high availability or are worried about load, each needs to manage its own data directory and sync with each other via replication.
I'd say that using containers for database engines is a bit uncommon outside of development setups, but not unheard of especially if you want to be able to scale fast. I don't think it's super easy to automate all of this.
Databases are critical services.
In my opinion, you shouldn't use use Docker for production databases.
But You wouldn’t think twice about using Docker in a local development environment
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 5 years ago.
Improve this question
I'm currently using Firebase as a prototyping tool to showcase a front end design for a documentation tool. In the process we've come to really like the real-time power of Firebase, and are exploring the potential to use it for our production instance of an open source/community version.
The first challenge is version control. Our legacy project used Hibernate/Envers in a Java stack, and we were previously looking at Gitlab as a way to move into a more "familiar" git environment.
This way?
Is there a way to timestamp and version control the data being saved? And thoughts on how to best recall this data without redesigning the wheel (e.g. any open source modules?)?
The real-time aspect of something like Firepad is great for documentation, but we require the means to commit or otherwise distinctly timestamp the state or save of a document.
Or?
Or is it best to use Firebase only for the realtime functionality, and implement Gitlab to commit the instance to a non-realtime database. In other words abstracting the version control entirely out to a more traditional relationship to a db?
Thoughts?
Both options you offer are valid, and feasible. In general, I'd suggest you to use firebase only as your real-time stack (data sync). And connect it to your own backend (gitlib or custom-db).
I've went that path, and find the best solution is to integrate your own backend db with firebase on top. Depend on firebase exclusively for everything, and you'll hit walls sooner or later..
The best solution is to keep full control on your data structure, security and access models, and use firebase where needed to keep clients in sync (online and offline). The integration is simple.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 8 years ago.
Improve this question
I've read many answers to similar questions, but still didn't get to the answer that I was looking for.
We've got a group of about 12 devs and business analysts working on one app. It's an enormous application, I'd guess about 1000+ pages in a mix of ASP and ASP.NET.
What I'm wondering is how the pros manage versioning of a large app like this one? Especially how to manage deployments, database changes and source control. Do they build the source control procedures in such a way that the app can be rolled back to a stable point at any time? How does the database fit in to that? Are all database schema changes and procs etc stored in source control?
I think my ideal solution here is to be able to re-hydrate the entire app from scratch to a specific version, including data. Is that overkill?
Update: my first guess at the size of the app was way off. I did an actual count and came up with a bigger number. 90% of those pages are frozen for development or unused.
While you are certainly asking a question that covers a lot of ground, there are 3 major parts you need to have in place:
Use a version control system. Something like Subversion or GIT is going to automatically let you roll back the code of your application to any point in time, or to any point at which you "tagged" your source code. As long as everyone only commits code that builds and runs successfully, every commit will be a valid point to which you can roll changes back to. Your version control system will manage multiple lines of code for you as well with branches. There are many strategies for handling branches, find one that works best for your process.
Use a build server. A build server like cruise control, Hudson, or TeamCity is going to automatically ensure that every commit is a "good" commit that compiles and can run successfully (assuming you have tests in place)
Include your database schema and static database data with your code in version control. There are tools like LiquiBase that allow you to manage your database structure and are designed to work with multiple developers working concurrently, even across multiple branches. Your code is no good without the corresponding database structure, so you do need to make sure you keep both in sync. Storing your database changes in your source code repository is the easiest way to do that. That being said, you cannot store your full database in your repository. It is too big and it changes too much for your source control's diff/merge support to handle. Depending on your industry, it may also not be legal due to government regulations. If you find you need to roll back your production database due to a bad release, you will need to determine what SQL statements would best undo the changes applied by looking at your stored update scripts.
Perforce has a great white paper addressing some of the issues you raise. From their white paper, that talks about web pages, you can extend the concept to stored procedures and scripts for generating/modifying tables.
As for being able to rehydrate from scratch, that really depends on your disaster recovery plans, as well as, how you setup and configure your development, and integration testing environments; and how much downtime you are willing to accept. I don't deal with these issues day to day, so perhaps people who have a lot more real world experience, specially people on the Operations side of things can impart their wisdom.
120 pages is "enormous"? Wow...
Versioning an app like this is quite straightforward. You have a production rollout process that includes tagging every release in the revision control system before it goes out, and only installing tagged releases. You back up the database at the time of the upgrade, before changing the schema, and store it somewhere with a descriptive name that allows you to link the code and DB backup. If you need to roll back, you just install the old code and restore the backup.
If you need to rollback, but with the current data, well, that's a harder problem, and is more about how your database is structured and how it's structure is evolved, but Rails migrations are an interesting study in how it can be done.
Beyond that, your question is pretty huge, and touches on a lot of areas. It might be best to contract someone who has dealt with all these things before and put you on the right path.
This is really quite a normal, and moderately-sized application. You just need to follow good source control processes, use proper unit testing, continuous integration, etc. Many, many, development organizations have solved this problem.
With all due respect, the fact that you consider 120 pages to be enormous, and the fact that you feel this is a big deal, both indicate to me that you should consider stopping development Right Now, and improving your development process.
I'm concerned that if you don't do that now, you'll learn later why you should have done so.
For something this small, rather then worry about being able to roll way back in time, what you are really after is the ability to roll back the most current release to the last one. That way if you break the build, you can roll back to the build prior. How?
1) Your web server reads from /www/working
2) Push your changes into /www/new
3) Move /www/current to /www/old
4) Move /www/new to /www/current
If /www/current blows up, just move /www/old back in its place.
The only catch is rolling back the database schema isn't part of this. Rolling back schema changes is difficult, if not impossible. All you can really do is restore the old version from backup and loose any activity between the live database and what was on backup. This makes sense if you think about it--if you add three new tables and somehow refactor another table into a better schema, how can you roll back the schema change after that? You can't--you just have to loose all the new data. Moral? Take your database changes seriously--there is a reason DBA's are sometimes considered assholes. They have to be--they can't just roll back mistakes in the database like you can roll back your buggy code.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 6 days ago.
Improve this question
In an ideal world, our development processes would be perfect, resulting in regular releases that were so thoroughly tested that it would never be necessary to "hotfix" a running application.
But, unfortunately, we live in the real world, and sometimes bugs slip past us and don't rear their ugly heads until we're already busy coding away at the next release. And the bug needs to be fixed Now. Not as a part of the next scheduled release. Not tonight when the traffic dies down. Now.
How do you deal with this need? It really can run counter to good design practices, like refactoring your code into nice, discrete class libraries.
Hand-editing markup and stored procedures on a production server can be a recipe for disaster, but it can also avert disaster.
What are some good strategies for application design and deployment techniques to find a balance between maintenance needs and good coding practices?
[Even though we test a lot before we release, ] What we do is this:
Our SVN looks like this:
/repo/trunk/
/repo/tags/1.1
/repo/tags/1.2
/repo/tags/1.3
Now whenever we release, we create a tag which we eventually check out in production. Before we do production, we do staging which is [less servers but] pretty much the same as production.
Reasons to create a "tag" include that some of the settings of our app in production code are slightly different (e.g. no errors are emailed, but logged) from "trunk" anyway, so it makes sense to create the tag and commit those changes. And then checkout on the production cluster.
Now whenever we need to hotfix an issue, we fix it in tags/x first and then we svn update from the tag and are good. Sometimes we go through staging, with some issues (e.g. minor/trivial fixes like spelling) we by-pass staging.
The only thing to remember is to apply all patches from tags/x to trunk.
If you have more than one server, Capistrano (link to capify.org doesn't go to the intended anymore) is extremely helpful to run all those operations.
One strategy is to heavily use declarative-style external configuration files for the different components.
Examples of this:
Database access/object-relational mapping via a tool like IBatis/IBatis.NET
Logging via a tool like JLog/NLog
Dependency injection via a tool like Spring/Spring.NET
In this way, you can often keep key components separated into discrete parts, hotfix a running application without recompile, and seamlessly use source control (particularly in comparison to stored procedures, which usually require manual effort to source control).
We divide our code in framework code and business customizations. Business customization classes are loaded using a separate classloader and we have tool to submit changes to a running instance of production. whenever we need a change in any class we change it and submit it to a running instance. the running instance will reject the old classloader and use a new classloader insance to load the classes again. This is similar to Jboss hot deploy of EJBs.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 3 years ago.
Improve this question
At work we have 4 people working together on a few different projects. For each project we each have a local copy we work on and then there is a development, staging, and live deployment, along with any branches we have (we use subversion). Our database is MySQL.
So my question is, what is a good way to manage which revisions to the database have been made to each deployment (and for the developers their local copies). Right now each change goes into a text file that is timestamped in the name and put into a folder under the project. This isn't working very well to be honest.. I need a solution that will help keep track of what has been applied where.
http://odetocode.com/Blogs/scott/archive/2008/01/30/11702.aspx
The above blog brought us to our current database version control system. Simply put, no DB changes are made without an update script and all update scripts are in our source control repository.
We only manage schema changes but you may also be able/willing to consider keeping dumps of your data available in version control as well; creating such files is a pretty trivial exercise using mysqldump.
Our solution differs from the solution presented in the blog in one key manner: it's not automated. We have to hand apply database updates, etc. Though this can be slightly time consuming, it postponed some of the effort a fully automated system would have required. One thing we did automate however, was the db version tracking in the software: this was pretty simple and it ensures that our software is aware of the database it's running against and will ONLY run if it knows the schema it's working with.
The hardest part of our solution was how to merge updates from our branches into our trunk. We spent some time to develop a workflow to address the possibility of two developers trying to merge branches with DB updates at the same time and how to handle it. We eventually settled on locking a file in version control (the file in question for us is actually a table mapping software version to db version which assists in our manual management strategy), much like you would a thread's critical section, and the developer who gets the lock goes about their update of the trunk. When completed, the other developer would be able to lock and it is their responsibility to make any changes necessary to their scripts to ensure that expected version collisions and other bad juju are avoided.
We keep all of our database scripts (data and schema/ddl) in version control. We also keep a central catalog of the changes. When a developer makes a change to a schema/DDL file or adds a script that changes the data in some way, those files are added to the catalog, along with the SVN commit number.
We have put together a small utility in-house that reads the catalog changes and builds a large update script based on the contents of the catalog by grabbing the contents from each revision in the catalog and applying them. The concept is pretty similar to the DBDeploy tool, which I believe originally came from Thoughtworks, so you may be able to utilize it. It will at least give you a good place to start, from which point you can customize a solution more directly suited to your needs.
Best of luck!
If your database maps nicely to a set of data access objects, consider using 'migrations'. The idea is to store your data model as application code with steps for moving forward and backward through each database version.
I believe Rails did it first.
Java has at least one project.
And here's a .NET migration library.
To change versions, you run a simple script that steps through all of the up or down versions to get you to the version you want. The beauty of it is, you check your migrations into the same source repository as your app code - it's all in one place.
Maybe others can suggest other migration libraries.
Cheers.
Edit: See also https://stackoverflow.com/questions/313/net-migrations-engine and .NET database migration tool roundup (from above post).