Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 6 days ago.
Improve this question
In an ideal world, our development processes would be perfect, resulting in regular releases that were so thoroughly tested that it would never be necessary to "hotfix" a running application.
But, unfortunately, we live in the real world, and sometimes bugs slip past us and don't rear their ugly heads until we're already busy coding away at the next release. And the bug needs to be fixed Now. Not as a part of the next scheduled release. Not tonight when the traffic dies down. Now.
How do you deal with this need? It really can run counter to good design practices, like refactoring your code into nice, discrete class libraries.
Hand-editing markup and stored procedures on a production server can be a recipe for disaster, but it can also avert disaster.
What are some good strategies for application design and deployment techniques to find a balance between maintenance needs and good coding practices?
[Even though we test a lot before we release, ] What we do is this:
Our SVN looks like this:
/repo/trunk/
/repo/tags/1.1
/repo/tags/1.2
/repo/tags/1.3
Now whenever we release, we create a tag which we eventually check out in production. Before we do production, we do staging which is [less servers but] pretty much the same as production.
Reasons to create a "tag" include that some of the settings of our app in production code are slightly different (e.g. no errors are emailed, but logged) from "trunk" anyway, so it makes sense to create the tag and commit those changes. And then checkout on the production cluster.
Now whenever we need to hotfix an issue, we fix it in tags/x first and then we svn update from the tag and are good. Sometimes we go through staging, with some issues (e.g. minor/trivial fixes like spelling) we by-pass staging.
The only thing to remember is to apply all patches from tags/x to trunk.
If you have more than one server, Capistrano (link to capify.org doesn't go to the intended anymore) is extremely helpful to run all those operations.
One strategy is to heavily use declarative-style external configuration files for the different components.
Examples of this:
Database access/object-relational mapping via a tool like IBatis/IBatis.NET
Logging via a tool like JLog/NLog
Dependency injection via a tool like Spring/Spring.NET
In this way, you can often keep key components separated into discrete parts, hotfix a running application without recompile, and seamlessly use source control (particularly in comparison to stored procedures, which usually require manual effort to source control).
We divide our code in framework code and business customizations. Business customization classes are loaded using a separate classloader and we have tool to submit changes to a running instance of production. whenever we need a change in any class we change it and submit it to a running instance. the running instance will reject the old classloader and use a new classloader insance to load the classes again. This is similar to Jboss hot deploy of EJBs.
Related
Closed. This question is not about programming or software development. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed last month.
Improve this question
Any big project is subject of code freeze in special moments of the year, mine included. In our context, we work with microservices architecture, where each team is responsable for the entire cycle coding->deploying, where deployment means changing a k8s deployment.yaml file that points to a new docker image with the latest changes.
However, the way we deal with freezes is that we simply don't merge any changes into the deployment.yaml of any of our services, this way, k8s won't deploy anything new. But IMO this approach is not ideal and easily bypassed because there isn't any real blocker, it's just a common agreement that we won't merge such PRs so that we don't change the deployment.yaml.
This way, my question would be if there is a known way, be that in kubernetes configurations or somewhere else, where I can enforce a real freeze and be 100% sure that nothing will be deployed in the meantime?
Even better if I were allowed to keep merging changes in deployment.yaml of my services, but only have the changes actually deployed upon freezing end.
-> In case you also don't know any existing way to do it, please leave your suggestions on how you think this could be done because Im heading to my final project in college and I think this could be an interesting topic...
The answer to this question will vary according to some parameters, but there are generally 2 main access points that can change production which should be controlled in order to achieve a hermetic code freeze -
CI/CD pipelines - This is the most common way to deploy changes to production these days. In my previous company, when we wanted to prevent developers from deploying changes during a code freeze, we would delete production credentials from the CI/CD system during the code freeze, and that way no change can be deployed even if it is merged to master.
As #larsks mentioned in the comments - if you are using GitOps, you might have to pin the current change to a specific commit/tag in git.
Manual changes - If developers in the organization have access to change production manually, you will have to address this as well. You can either block manual access until the end of the code freeze, or make sure the policy is clearly communicated to everyone with production access (since manual changes don't are less probable to happen unintentionally)
Another point to address when implementing code freeze is out of band access to apply hotfixes and other urgent changes. When cutting access to production there should still be an emergency route allowing changes to production to be applied, and it should be simple and quick - because it will often be applied in times of stress and downtime.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 3 years ago.
Improve this question
when starting with a project and using source control i find it hard to separate the things people are working on so they don't either write duplicate code or think it should be named one thing and so on.
this problem diminishes over time because the general foundation is in place and it's easier to separate the tasks so they don't overlap as much
how do you manage working with source control in the beginning phase?
EDIT:
I can see that it don't really have anything to do with source control, but it gets more apparent when you have source control too. so the question becomes more along the lines of "how do you manage to separate the tasks so they don't overlap too much. I think it's really hard and i haven't really seen much about how to do it.
Well, as far as source control goes, somebody needs to take the lead and set up the basic structure of the project, directories, etc. and communicate it to the team. On projects I work on, this is usually an architect or senior developer, someone who knows the best practices for project organization for the team/company.
With respect to avoiding having multiple people working on the same tasks, that's a project management function; someone needs to determine what tasks need to be done, and communicate it to the team. If you are working in an agile/scrum environment, the team may divide and hand out work items amongst themselves, but in either case you need to communicate to avoid doing the same work twice.
EDIT
To address the issue of multiple people working on the same task, I tend to work on smaller teams, 2-6 people; in this environment, I have had a lot of success with a scrum-influenced approach using the Crystal Clear methodology:
Architect(s)/designer(s) come up with high level design
Architect(s)/designer(s) define iterations/deliveries, the first of which is a "project skeleton" which consists of architectural and back-end components and a thin slice of the app
Lead person breaks up features into 1-3 day tasks/units of work (estimated)
Team meets and discusses priority, timing and dependencies of tasks, and divides up the first set of tasks
The team has brief daily meetings to discuss status/priorities and dependencies, and change direction if necessary
With larger projects/teams, you will almost certainly need someone whose main job is dedicated to tracking status, dependencies and conflicts.
I don't think source control has much to do with the problem of coordinating people's efforts (except that it can catch some "conflicts" when people erroneously try to modify the same files in different ways -- but, that's not as good as preventing conflicts, and even just "preventing conflicts" does not per se ensure that everybody is working on what they should ideally be working right now, in terms of priorities). Coordination is properly managed with practices (and perhaps tools, e.g. Pivotal Tracker -- but, using the right practices is even more important than using nice tools!-) that specifically focus on ensuring coordination. For example, the practices that Tracker is designed to support and enhance, such as story-based iterative planning, and other compatible ones, such as stand-ups, offer ways to meet these needs.
You must be having a base version that everyone is using, check that into the repository, and then make incremental changes to the repository, make sure that everyone works on different part of the code, commit every working change, and resolve conflicts as and when they occur. That is how I would do it.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 8 years ago.
Improve this question
I've read many answers to similar questions, but still didn't get to the answer that I was looking for.
We've got a group of about 12 devs and business analysts working on one app. It's an enormous application, I'd guess about 1000+ pages in a mix of ASP and ASP.NET.
What I'm wondering is how the pros manage versioning of a large app like this one? Especially how to manage deployments, database changes and source control. Do they build the source control procedures in such a way that the app can be rolled back to a stable point at any time? How does the database fit in to that? Are all database schema changes and procs etc stored in source control?
I think my ideal solution here is to be able to re-hydrate the entire app from scratch to a specific version, including data. Is that overkill?
Update: my first guess at the size of the app was way off. I did an actual count and came up with a bigger number. 90% of those pages are frozen for development or unused.
While you are certainly asking a question that covers a lot of ground, there are 3 major parts you need to have in place:
Use a version control system. Something like Subversion or GIT is going to automatically let you roll back the code of your application to any point in time, or to any point at which you "tagged" your source code. As long as everyone only commits code that builds and runs successfully, every commit will be a valid point to which you can roll changes back to. Your version control system will manage multiple lines of code for you as well with branches. There are many strategies for handling branches, find one that works best for your process.
Use a build server. A build server like cruise control, Hudson, or TeamCity is going to automatically ensure that every commit is a "good" commit that compiles and can run successfully (assuming you have tests in place)
Include your database schema and static database data with your code in version control. There are tools like LiquiBase that allow you to manage your database structure and are designed to work with multiple developers working concurrently, even across multiple branches. Your code is no good without the corresponding database structure, so you do need to make sure you keep both in sync. Storing your database changes in your source code repository is the easiest way to do that. That being said, you cannot store your full database in your repository. It is too big and it changes too much for your source control's diff/merge support to handle. Depending on your industry, it may also not be legal due to government regulations. If you find you need to roll back your production database due to a bad release, you will need to determine what SQL statements would best undo the changes applied by looking at your stored update scripts.
Perforce has a great white paper addressing some of the issues you raise. From their white paper, that talks about web pages, you can extend the concept to stored procedures and scripts for generating/modifying tables.
As for being able to rehydrate from scratch, that really depends on your disaster recovery plans, as well as, how you setup and configure your development, and integration testing environments; and how much downtime you are willing to accept. I don't deal with these issues day to day, so perhaps people who have a lot more real world experience, specially people on the Operations side of things can impart their wisdom.
120 pages is "enormous"? Wow...
Versioning an app like this is quite straightforward. You have a production rollout process that includes tagging every release in the revision control system before it goes out, and only installing tagged releases. You back up the database at the time of the upgrade, before changing the schema, and store it somewhere with a descriptive name that allows you to link the code and DB backup. If you need to roll back, you just install the old code and restore the backup.
If you need to rollback, but with the current data, well, that's a harder problem, and is more about how your database is structured and how it's structure is evolved, but Rails migrations are an interesting study in how it can be done.
Beyond that, your question is pretty huge, and touches on a lot of areas. It might be best to contract someone who has dealt with all these things before and put you on the right path.
This is really quite a normal, and moderately-sized application. You just need to follow good source control processes, use proper unit testing, continuous integration, etc. Many, many, development organizations have solved this problem.
With all due respect, the fact that you consider 120 pages to be enormous, and the fact that you feel this is a big deal, both indicate to me that you should consider stopping development Right Now, and improving your development process.
I'm concerned that if you don't do that now, you'll learn later why you should have done so.
For something this small, rather then worry about being able to roll way back in time, what you are really after is the ability to roll back the most current release to the last one. That way if you break the build, you can roll back to the build prior. How?
1) Your web server reads from /www/working
2) Push your changes into /www/new
3) Move /www/current to /www/old
4) Move /www/new to /www/current
If /www/current blows up, just move /www/old back in its place.
The only catch is rolling back the database schema isn't part of this. Rolling back schema changes is difficult, if not impossible. All you can really do is restore the old version from backup and loose any activity between the live database and what was on backup. This makes sense if you think about it--if you add three new tables and somehow refactor another table into a better schema, how can you roll back the schema change after that? You can't--you just have to loose all the new data. Moral? Take your database changes seriously--there is a reason DBA's are sometimes considered assholes. They have to be--they can't just roll back mistakes in the database like you can roll back your buggy code.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 6 years ago.
Improve this question
What is continuous integration and what are its benefits?
This is by far the best explanation I have read so far.
At its simplest, it is simply a mechanism that rebuilds your project whenever a check in is made into some revision control system (CVS etc). This can be extended though to include running tests, all the way through to generating a CD image, mounting it within VMs, installing the product and running full tests on it.
It has the simple advantage of highlighting when code changes break the system as early as possible. Not only does it detect breaks in the code, it highlights who caused the break. This psychological effect is very effective in encouraging good testing prior to check in!
It is the practice of ensuring that all aspects of your software development process are lined up to permit the daily creation of a working version of your product. It is best known as part of Extreme Programming.
This involves things as far afield as build automation, automated testing, daily check-ins, using a source code repository, etc. But the ultimate goal is to help the entire project run according to core Agile Principles so that you deliver early and often. This, in turn, helps you leverage feedback from your users, etc.
+1 for the link to Fowler's page.
Personally, I just found it "nice" to know whenever something didn't compile because we had the poor practice of having a single build (yes, we developed on the production build; we were awesome). We hadn't got the integrated testing phase before I left.
After a while, it did, however, lessen the amount of massive coding changes (compared to the "check in and pray my changes don't conflict" that was rampant). Eventually, most developers started making small changes frequently just to get confirmation from the CC.Net tray icon.
Overall, I found it very comforting to know that we could send out a build immediately if we had to. Had we had just a few smoke tests integrated, I think the stress-level would have been substantially lower.
Just to refresh. At this point there is a huge difference between Continuous Integration (CI) and Continuous Delivery (CD). While most of posts above described CD I'll try to show how CI extends now CD definition. Having all the tools needed to build a package and deploy new version of app automatically is a crucial part of CD. Adding to that test automation (based on three level verification: General Health-check, Detailed Statistics and Historical entries) and a proper governance you're creating a really good piece of CI. Only because of such an extended definition building extraordinary cloud tools is possible. Think about muleESB or esbeetle.com. For both of them CI is something natural although only the second one is supporting both ESB and ETL components.
I hope that it was helpful.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 3 years ago.
Improve this question
At work we have 4 people working together on a few different projects. For each project we each have a local copy we work on and then there is a development, staging, and live deployment, along with any branches we have (we use subversion). Our database is MySQL.
So my question is, what is a good way to manage which revisions to the database have been made to each deployment (and for the developers their local copies). Right now each change goes into a text file that is timestamped in the name and put into a folder under the project. This isn't working very well to be honest.. I need a solution that will help keep track of what has been applied where.
http://odetocode.com/Blogs/scott/archive/2008/01/30/11702.aspx
The above blog brought us to our current database version control system. Simply put, no DB changes are made without an update script and all update scripts are in our source control repository.
We only manage schema changes but you may also be able/willing to consider keeping dumps of your data available in version control as well; creating such files is a pretty trivial exercise using mysqldump.
Our solution differs from the solution presented in the blog in one key manner: it's not automated. We have to hand apply database updates, etc. Though this can be slightly time consuming, it postponed some of the effort a fully automated system would have required. One thing we did automate however, was the db version tracking in the software: this was pretty simple and it ensures that our software is aware of the database it's running against and will ONLY run if it knows the schema it's working with.
The hardest part of our solution was how to merge updates from our branches into our trunk. We spent some time to develop a workflow to address the possibility of two developers trying to merge branches with DB updates at the same time and how to handle it. We eventually settled on locking a file in version control (the file in question for us is actually a table mapping software version to db version which assists in our manual management strategy), much like you would a thread's critical section, and the developer who gets the lock goes about their update of the trunk. When completed, the other developer would be able to lock and it is their responsibility to make any changes necessary to their scripts to ensure that expected version collisions and other bad juju are avoided.
We keep all of our database scripts (data and schema/ddl) in version control. We also keep a central catalog of the changes. When a developer makes a change to a schema/DDL file or adds a script that changes the data in some way, those files are added to the catalog, along with the SVN commit number.
We have put together a small utility in-house that reads the catalog changes and builds a large update script based on the contents of the catalog by grabbing the contents from each revision in the catalog and applying them. The concept is pretty similar to the DBDeploy tool, which I believe originally came from Thoughtworks, so you may be able to utilize it. It will at least give you a good place to start, from which point you can customize a solution more directly suited to your needs.
Best of luck!
If your database maps nicely to a set of data access objects, consider using 'migrations'. The idea is to store your data model as application code with steps for moving forward and backward through each database version.
I believe Rails did it first.
Java has at least one project.
And here's a .NET migration library.
To change versions, you run a simple script that steps through all of the up or down versions to get you to the version you want. The beauty of it is, you check your migrations into the same source repository as your app code - it's all in one place.
Maybe others can suggest other migration libraries.
Cheers.
Edit: See also https://stackoverflow.com/questions/313/net-migrations-engine and .NET database migration tool roundup (from above post).