Which is the more feasible strategy for version upgrades when you are multiple versions behind. For example from 6.4 to 2005.
Should we really do a version by version approach as SAP suggests. I understand it's recommended way but still.
Any one can share their experience regarding this?
What difficulties could be faced when directly migrating multiple versions?
Thanks!
There are several approaches you can take. Which one you take depends on the knowledge your team has and with the amount of customizations you already performed.
Step by Step
This is the recommended way by SAP. This is a more secure strategy, where it's very clear what changed between different versions. With every version, you will experience build failures, startup failures and possibly even data issues that need to be migrated. But it's very clear what version caused those issues. With the SAP help and the upgrade notes, you should be able to easily find what was changed, and how to fix it. Disadvantage with this approach is that you need to download, unzip and build for every version, and that takes time. Sometimes you even need to fix the same code twice, when the implentation was changed multiple times
One Shot
With this approach, you go straight to the latest version. You just put your custom code in the latest version and just see what build failures you get.
With this approach, it will be harder to figure out what exact version upgrade caused a specific issue. You should still check all upgrade notes, to make sure that no migrations are needed. Advantage is that you only perform everything once. If you have an experienced team, this is a feasable approach. If you have a new team, be carefull with this approach. You might encounter some difficult errors where you won't be sure what version caused them, so finding info in the SAP help might be harder
Hybrid approach
A third option would be a hybrid approach, where you upgrade several versions at once (For example to versions that contain big changes, like with the addition of backoffice in 6.3). This makes it easier to apply changes for those big changes, while you don't have to go through every version one by one.
Conclusion
I've tried all approaches in the past. The step by step approach takes a lot of time, but makes the changes easier and clearer. With the One Shot approach, you only need to download the latest version, but it might be somewhat harder to find the bugs. If you have an experienced team, you should go for the one shot approach. When you are a lot of versions behind and there were big changes, you could go for the hybrid approach
I had a similar requirement of upgrading from version 6.2 to 2005, I went with the One-Shot approach as explained by Yoni, and the biggest challenge I faced was due to Java version change.
I believe One-Shot approach and Step by Step approach will take a similar amount of time in the major version upgrade, though Step by Step approach is safe but redundant. My personal favorite is One-Shot.
I recently did a platform upgrade from hybris 6.7 to 2005 and did it step by step, mainly because of the java version change and, other than that, there were certain migration steps in each intermediate version that were needed to be done. Also the customer had a lot of custom promotion rules and they needed some special care.
In my case, the process in each step was this:
Upgrade to new version - there is an help.sap.com page for each step, I recommend you follow it and go through each of the section to see what applies to your project, e.g Upgrading Platform from 6.7 to 1808
Compile the project - some deprecated things will be removed in some steps and you have to refactor where needed. This step took me the most amount of time
Start the hybris server - after you finish the refactoring and your project builds successfully with ant clean all there is the possibility that the platform will fail to start due to some (now) incorrect xml config. The "good" part here is that you can see in the console what the problem is and the fix should go faster than the previous step.
Perform the necessary upgrade steps - here is the tricky part, once your platform starts you have to perform the necessary upgrade steps for each extension and add-on that needs it, otherwise you risk working with some broken business logic. You need to do some regression tests and check that everything works as it should.
All in all, an upgrade takes time and depends on how many versions you have to go through, but I think taking it step by step is the most efficient way to accomplish it.
This is not a "why should I use version control" question :-)
I have always used version control from the first line of code of every project I've written so far. However yesterday I came up with a question (maybe a stupid one) to which I find no answer: when should version control really start during the software development process? Should it start from the first line of code, as I've been doing all of my life, or should it start when you really have an operational version of your code? Put in other words: should version control be used before the first version of your software? (I mean version control, not source backup, of course!).
Pre-development you don't need version control; but what you do need is some form of collaboration mechanism to keep track of changes to the specifications and documentation.
Some teams deploy version control at this stage. Personally I don't find the value of it here, a wiki/trello or similar is more valuable and makes more sense; as you are tracking a lot of abstract ideas.
As soon as you start writing code - you should start the version control process; and through out the development phase before you have deployed you continue to use version control; this is where you start getting value out of it. Especially if you are developing with others. If you are a solo developer, version control may seem like extra work for no use; this is debatable, but when you are working in a team it is essential.
Once the project has deployed; revision control is critical and mandatory. You simply cannot afford to not have it - version control offers you lots of benefits for the type of work you undertake after deployment. Bugfixes, automated testing, deployment - these can easily be automated from your version control system. If you didn't use version control during development; now is the best time to deploy it since you have a solid codebase as your reference point.
Version control is so simple these days with mercurial/git and their online hosting services that it is costs nothing to get started; and the benefits far outweigh any drawbacks.
The question is quite abstract. So, an equally abstract answer.
I think you should use version control on a specific project as soon as it starts to add value.
If you can distinguish between two phases - proof of concept/prototype etc., and product code, I think you should separate the code bases for the two. And you can use version control tools for both (source backup first, then real version control), just avoiding cluttering the production repository with early stuff.
If you are using version control just for the code, you could ask that question.
But ideally, version control should help you reproduce a build, which means the configuration files and other settings can be as important as your first line of code.
See for instance ".classpath and .project - check into version control or not?.
That is the kind of data which will facilitate collaboration, as other developers will be up and running (ie able to build your program) very quickly.
I am pretty comfortable with the producing web apps now. I am using a NodeJs stack on the back-end and usually have a fair amount of Javascript on the front end. Where I really lack understanding is the deployment process.
What is a typical deployment process?
From what I have gathered in my reading a deployment/build process can include several tasks:
Running through unit-test suites
Concatenating script and CSS files
Version numbering your app
Tracing module dependencies (node_modules)
Pushing it to a remote repo (GitHub)
Instructing 'staging' servers to pull down the latest repo
Instructing 'production' server to pull down the latest repo
This has all left me a little overwhelmed. I don't know whether I should be going into this level of detail for my own projects, it seems like a lot of work! I am using Sublime Text 2 IDE and it seems to have a Build Script process, is this suitable? How does one coordinate all these separate tasks? I'm imagining ideally they would all run at the flick of a switch.
Sorry so many questions, but I need to know how people learnt similar principles. Some of my requirements may be specific to NodeJS but I'm sure processes are similar no matter what choice of stack you are developing in.
First off, let's split the job in two: front-end and back-end stuff. For both, you really want some kind of bulid system, but their goals and scope are vastly different.
For the front-end, you want your source to be as small as possible; concatenate/minify JavaScript, CSS and images. A colleague of mine has written a "compiler", Assetgraph, to do this for you. It has a somewhat seep learning-curve, but it does wonders for your code (our dev-builds are usually ~20 megs, production ~500 k).
As to the back-end, you want contained, easily managed bundles of some sort. We re-package our stuff into debian-packages. As long as the makefile is wired up correctly, you get a lot of the boring build- and deploy-time stuff for free. Here's my (pre-NPM 1.0) Debianizing node programs. I've seen other ways to do this in NPM and on Github, but I haven't looked into them, so I can't speak on their quality.
For testing/pusing around/deploying, we use a rather convoluted combination of Debian package-archives, git-hooks, Jenkins-servers and what not. While I highly recommend using the platforms' native package-manager for rolling out stuff, it can be a bit too much. All in all, we usually deploy staging either automatically (on each git push), or semi-automatic for unstable codebases. Production deployments are always done explicitly.
For the assets I use asereje https://github.com/masylum/asereje
I recently documented my nodejs deployment process in a blog post:
http://pau.calepin.co/how-to-deploy-a-nodejs-application-with-monit-nginx-and-bouncy.html
A build script sounds like a good idea indeed.
What should that build script do?
make sure all the test pass, else exit immediately
concatenate your javascript and css files into one single js/css file and minify them also
increment the version number (unless you decide to set that up yourself manually)
push to the remote repo (which instructs the staging and production servers through a git hook)
This is at least my opinion.
Other resources:
http://howtonode.org/deploying-node-with-spark
http://howtonode.org/deploying-node-upstart-monit
http://dailyjs.com/2010/03/15/hosting-nodejs-apps/
How to deploy node app depencies
Given a large scale software project with several components written in different languages, configuration files, configuration scripts, environment settings and database migration scripts - what are the common practices for deployment to production?
What are the difficulties to consider? Can the process be simplified with tools like Ant or Maven? How can rollback and database management be handled? Is it advisable to use version control for the production environment?
As I see it, you're mostly asking about best practices and tools for release engineering AKA releng -- it's important to know the "term of art" for a subject, because it makes it much easier to search for more information.
A configuration management system (CMS -- aka revision control system or version control system) is indispensable for today's software development; if you use one or more IDEs, it's also nice to have good integration between them and the CMS, though that's more of an issue for purposes of development than for purposes of deployment / releng.
From a releng viewpoint, the key thing about a CMS is that it must have good support for "branching" (under whatever name), because releases must be made from a "release branch" where all the code under development and all of its dependencies (code and data) are in a stable "snapshot" from which the exact, identical configuration can be reproduced at will.
The need for good branching support may be more obvious if you have to maintain multiple branches (customized for different uses, platforms, whatever), but even if your releases are always, strictly in a single linear sequence, releng best practices still dictate making a release branch. "Good branching support" includes ease of merging (and "conflict resolution" when different changes are made to a file), "cherry-picking" (taking one patch or changeset from one branch, or the head/trunk, and applying it to another branch), and the like.
In practice, you start off the release process by making a release branch; then, you run exhaustive testing on that branch (typically MUCH more than what you run everyday in your continuous build -- including extensive regression testing, integration testing, load testing, performance verification, etc, and possibly even more costly quality assurance processes, depending). If and when the exhaustive testing and QA reveal defects in the release-candidate (including regressions, performance degradation, etc), they must be fixed; in a large team, development on head/trunk may be continuing while the QA is being done, whence the need for ease of cherry-picking / merging / etc (whether your practice is to perform the fix on head or on the release branch, it still needs to be merged to the other side;-).
Last but not least, you're NOT getting full releng value from your CMS unless you're somehow tracking with it "everything" that your releases depend on -- simplest would be to have copies or hard links to all the binaries for the tools you need to build your release, etc, but that may often be impractical; so at the very least track the exact release, version, bugfix &c numbers of those tools that are used (operating system, compilers, system libraries, tools that preprocess image, sound or video files into final form, etc, etc). The key is being able, at need, to exactly reproduce the environment required to rebuild the exact version that's proposed for release (otherwise you'll go mad tracking down subtle bugs that may depend on third party tools' changes as their versions change;-).
After a CMS, the second most important tool for releng is a good issue tracking system -- ideally one that's well integrated with the CMS. That's also important for the development process (and other aspects of product management), but in terms of the release process the importance of the issue tracker is the ability to easily document exactly what bugs have been fixed, what features have been added, removed, or changed, and what modifications in performance (or other user-observable characteristics) are expected in the new forthcoming release. For the purpose, a key "best practice" in development is that every changeset that gets committed to the CMS must be connected to one (or more) issue in the issue tracking system: after all, there's gotta be some purpose for that change (fix a bug, change a feature, optimize something, or some internal refactor that's supposed to be invisible to the software's user); similarly, every tracked issue that's marked as "closed" must be connected to one (or more) changesets (unless the closing is of the "won't fix / working as intended" kind; issues related to bugs &c in third-party components, which have been fixed by the third-party supplier, are easy to treat similarly if you do manage to keep track of all third-party components in the CMS too, see above; if you don't, at least there should be text files under CMS documenting third-party components and their evolution, again see above, and they need to be changed when some tracked issue on a 3rd party component gets closed).
Automating the various releng processes (including building, automated testing, and deployment tasks) is the third top priority -- automated processes are much more productive and repeatable than asking some poor individual to manually go through a list of steps (for sufficiently complex tasks, of course, the workflow of the automation may need to "get a human being in the loop"). As you surmise, tools such as Ant (and SCons, etc, etc) can help here, but inevitably (unless you're lucky enough to get away with very simple and straightforward processes) you'll find yourself enriching them with ad-hoc scripts &c (some powerful and flexible scripting language such as perl, python, ruby, &c, will help). A "workflow engine" can also be precious when your release workflow is sufficiently complex (e.g. involving specific human beings or groups thereof "signing off" on QA compliance, legal compliance, UI guidelines compliance, and so forth).
Some other specific issues you're asking about vary enormously depending on specifics of your environment. If you can afford programmed downtime, your life is relatively easy, even with a large database in play, as you can operate sequentially and deterministically: you shut the existing system down gracefully, ensure the current database is saved and backed up (easing rollback, in the hopefully VERY rare case it's needed), run the one-off scripts for schema migration or other "irreversible" environment changes, fire the system back up again in a mode that's still unaccessible to general users, run another extensive suite of automated tests -- and finally if everything's gone smoothly (including the saving and backup of the DB in its new state, if relevant) the system's opened up to general use again.
If you need to update a "live" system, without downtime, this may range anywhere from a slight inconvenience to a systematic nightmare. In the best case, transactions are reasonably short, and synchronization between the state set by transactions may be delayed a bit without damage... and you have a reasonable abundance of resources (CPUs, storage, &c). In this case, you run two systems in parallel -- the old one and the new one -- and just make sure all new transactions target the new system, while letting old ones complete on the old system. A separate task periodically syncs "new data in the old system" to the new system, as transactions on the old system terminate. Eventually you can determine that no transactions are running on the old system and all changes that happened there are synced up to the new one -- and at that time you can finally shut down the old system. (You need to be prepared to "reverse sync" too of course, in case a rollback of the change is needed).
This is the "simple, sweet" extreme for live system updating; at the other extreme, you can find yourself in such an overconstrained situation that you can prove the task is impossible (you just cannot logically meet all the stated requirements with the given resources). Long sessions opened on the old system which just cannot be terminated -- scarce resources that make it impossible to run two systems in parallel -- core requirements for hard-real-time sync of every transaction -- etc, etc, can all make your life miserable (and as I noticed, at the extreme, can make the stated task absolutely impossible). The two best things you can do about this: (1) ensure you have abundant resources (this will also save your skin when some server unexpected goes belly-up... you'll have another one to fire up to meet the emergency!-); (2) consider this predicament from the start, when initially defining the architecture of the overall system (e.g.: prefer short-lived transactions to long-lived sessions that just can't be "snapshot, closed down, and restarted seamlessly from the snapshot", is one good arcitectural pointer;-).
disclaimer: where I work we use something I wrote that I'm going to mention
I'll tell you how I do it.
For configuration settings, and general deployment of code and content, I use a combination of NAnt, a CI server, and dashy (an automatic deployment tool). You can replace dashy with any other 'thing', that automates the uploads to your server (perhaps capistrano).
For DB scripts, we use RedGate SQL Compare to get the scripts, and then for most changes, I actually make them manually, where appropriate. This is because some of the changes are slightly complex, and I feel more comfortable doing it by hand. You can actually use this tool to do it for you, (or at least generate the scripts).
Depending on your language, there are some tools that can script the DB updates for you (I think someone on this forum wrote one; hopefully he'll reply), but I have no experience with those. It's something I'd like to add though.
Difficulties
I forgot to answer one of your questions.
The biggest problem in updating any significantly complicated/distributed site is DB synchronisation. You need to consider if you will have any downtime, and if you do, what will happen to the DBs. Will you shutdown everything, so that no transactions can be processed? Or will you shift everything to one server, update DB B, and then sync DB A & B, then update DB B? Or something else?
Whatever you choose, you need to choose it, and either say "Okay for each update, there will be X downtime", or whatever. Just have it documented.
The worst thing you can do is have someones transaction fail, mid processing, because you were updating that server; or to somehow leave only part of your system operational.
I don't think you have any option about using versioning or not.
You cannot go without versioning (as mentioned in a comment).
I am talking from experience since I am currently working on a website where we have several items that have to work together:
The software itself, its functionalities at a given point in time
The external systems (6 of them) we are linked with (messages are versioned)
A database, which holds the configuration (and translations)
The static resources (images, css, javascripts) hosted on an Apache server (several in fact)
A java applet, which has to be synchronized with the javascript and software
This works because we use versioning, although I must admit that our database is pretty simple, and because the deployment is automated.
Versioning means that we have actually at any point in time several concurrent versions of the messages, the database, the static resources and the java applet.
It is especially important in the event of a 'fallback'. If you discover a flaw when loading you new software and suddenly you cannot afford to load, you'll have a crisis if you have not versioned, and you'll just have to load the old soft if you have.
Now as I said, the deployment is scripted:
- the static resources are deployed first (+ the java applet)
- the database goes next, several configurations cohabit since they are versioned
- the soft is queued for load in a 'cool' time-window (when our traffic is at its lowest point, so at night)
And of course we take care of backward / forward compatibility issues with the external servers, which should not be underestimated.
I'm not sure If any of you have encountered this scenario.We have small small projects or
fixes that we need to push to LIVE every other day.
There is one set of team that says "do a clean build and deploy on prod server".
The other set says we don't have to do a full deploy we will just do a dll drop or aspx drop.
They have listed out few pros and cons for each method. But want to know what method you generally follow any major set backs for each method.
I would first of all try to minimize the cost of building a new release by trying to automate as much as possible. (Which might be easier said than done but it is usually well invested time and money, especially if you release often.)
The big issue I have with dropping in new binaries is that it is often a manual process and those are performed by humans who tend to mess up the easies tasks over and over again.
Aren’t you really looking for a “patch management system” that could help you distribute the changes in a controlled manner and getting rid of the manual work?
That way you still would have pretty good integrity since the patches should be versioned and hopefully carefully tested. But still they can be developed and deployed which much less overhead then a full release.
For sites under my control, I use SVN or Git for revision control, and update the source and compile on the server directly, if necessary. That ensures the integrity of the release, which I suspect is the argument put forth by the "do a clean build and deploy on prod server" team. For servers not under my control, I do whatever I am told :)