Related
I am working on a website of 3,000+ pages that is updated on a daily basis. It's already built on an open source CMS. However, we cannot simply continue to apply hot fixes on a regular basis. We need to replace the entire system and I anticipate the need to replace the entire system on a 1-2 year basis. We don't have the staff to work on a replacement system while the other is being worked on, as it results in duplicate effort. We also cannot have a "code freeze" while we work on the new site.
So, this amounts to changing the tire while driving. Or fixing the wings while flying. Or all sorts of analogies.
This brings me to a concept called "continuous migration." I read this article here: https://www.acquia.com/blog/dont-wait-migrate-drupal-continuous-migration
The writer's suggestion is to use a CDN like Fastly. The idea is that a CDN allows you to switch between a legacy system and a new system on a URL basis. This idea, in theory, sounds like a great idea that would work. This article claims that you can do this with Varnish but Fastly makes the job easier. I don't work much with Varnish, so I can't really verify its claims.
I also don't know if this is a good idea or if there are better alternatives. I looked at Fastly's pricing scheme, and I simply cannot translate what it means to a specific price point. I don't understand these cryptic cloud-service pricing plans, they don't make sense to me. I don't know what kind of bandwidth the website uses. Another agency manages the website's servers.
Can someone help me understand whether or not using an online CDN would be better over using something like Varnish? Is there free or cheaper solutions? Can someone tell me what this amounts to, approximately, on a monthly or annual basis? Any other, better ways to roll out a new website on a phased basis for a large website?
Thanks!
I think I do not have the exact answers to your question but may be my answer helps a little bit.
I don't think that the CDN gives you an advantage. It is that you have more than one system.
Changes to the code
In professional environments I'm used to have three different CMS installations. The fist is the development system, usually on my PC. That system is used to develop the extensions, fix bugs and so on supported by unit-tests. The code is committed to a revision control system (like SVN, CVS or Git). A continuous integration system checks the commits to the RCS. When feature is implemented (or some bugs are fixed) a named tag will be created. Then this tagged version is installed on a test-system where developers, customers and users can test the implementation. After a successful test exactly this tagged version will be installed on the production system.
A first sight this looks time consuming. But it isn't because most of the steps can be automated. And the biggest advantage is that the customer can test the change on a test system. And it is very unlikely that an error occurs only on your production system. (A precondition is that your systems are build on a similar/equal environment. )
Changes to the content
If your code changes the way your content is processed it is an advantage when your
CMS has strong workflow support. Than you can easily add a step to your workflow
which desides if the content is old and has to be migrated for the current document.
This way you have a continuous migration of the content.
HTH
Varnish is a cache rather than a CDN. It intercepts page requests and delivers a cached version if one exists.
A CDN will serve up contents (images, JS, other resources etc) from an off-server location, typically in the cloud.
The cloud-based solutions pricing is often very cryptic as it's quite complicated technology.
I would be careful with continuous migration. I've done both methods in the past (continuous and full migrations) and I have to say, continuous is a pain. It means double the admin time for everything, and assumes your requirements are the same at all points in time.
Unfortunately, I would say you're better with a proper rebuilt on a 1-2 year basis than a continuous migration, but obviously you know best about that.
I would suggest you maybe also consider a hybrid approach? Build yourself an export tool to keep all of your content in a transferrable state like CSV/XML/JSON so you can just import into a new system when ready. This means you can incorporate new build requests when you need them in a new system (what's the point in a new system if it does exactly the same as the old one) and you get to keep all your content. Plus you don't need to build and maintain two CMS' all the time.
Problem:
A huge code base spanning to several
million SLoC,
maintained(support/active enhancements
etc) by a horde of second/third rate
programmers(most of them who do not
really care). Decades ago, few smart
guys had put in place a wrapper that
uses CVS underneath and this system is
being used by current generation of
developers (90% of them have not used
CVS directly, or heard of/used another
command line source control system).
Effect:
Usage of CVS and multiple teams
working across multiple modules,
inevitably => CVS branch merges to
trunk. This would be an activity
practised in the most religiously and
ritualistically possible way. [=>
brute force; planned weeks ahead,
involving a dozen guys for 2/4 days. Hundreds(sometimes thousands) of sources handled, manually.
Funny part is, the people involved are not the original owners of the
fix, and they simply go by checking
the diffs; really, am not kidding!]
This leads to a lot of inconsistency
in the sanity of the
libraries/modules/functionality and
too much of effort is spent to correct
defects due to regression during these
merges.
And now, the question:
What alternative source control system
can bring in some positive change and
improve the lives of
programmers/managers and everyone else
in the environment?
Since everyone around there seems to have drunk the KoolAid (and sing "This-is-how-things-are-done-everywhere") without even giving a thought about finding an alternative, it is high time someone does that. But considering the sort of people who would use the system, the following aspects are to be kept in mind.
Simple to use && understand, even Joe Coder should be able to use it without fuss. (Anyway this won't be needed, as the wrapper would hide real under-the-hood stuff from folks)
A HUGE codebase (consisting of sources across multiple languages), with multiple(around 30) active branches at any given time.
Easy merges to various branches. (considering that volume of changes is quite huge)
Commercial support for the system would be sweet, if available.
Development happens on UNIX servers (Should run on HP-UX/Solaris at least)
Should scale well (thousands of users/hundreds of thousands of sources)
Good Documentation
Simple/lucid browser based interface to compare/view changes/copies.
There are no binary files in repository, so need not bother about them.
Provision for importing current repository contents into the new system.
So, please suggest. Is there hope && a way out? :) Am pretty sure stuff like git would be rejected outright (they believe "git is only for smart folks")
EDIT: I too have Mercurial and BitKeeper in mind, and have mentioned it to folks up the chain. Hoping for the best!
Thanks! :)
You may have heard the old saying about good, fast and cheap. The same applies here. With a rich feature set come some degree of complexity. I don't believe that you can fulfill that list of requirements without picking a tool that is going to involve some complexity. I wish you luck in trying, but if it were me I would pick a good tool and invest some time into training the users.
Am pretty sure stuff like git would be rejected outright (they believe "git is only for smart folks")
If it's purely a matter of "perception" - that they "perceive" git to be too complex, try suggesting Mercurial or Bazaar - they might not be familiar enough with them to have formed an inaccurate preconception.
Mercurial would be my suggestion. To avoid the "complexity," perception, take a look at
this site.
It is "A friendly introduction to the Mercurial DVCS by Joel Spolsky," and it provides an excellent tutorial for users (and a subversion recovery portion) which takes them through step by step editing, committing, merging, etc.
I'll throw out the suggestion of moving to Subversion. It's not the sexy distributed source control that all the cool kids are using, but, and this is the reason I am suggesting it, SVN should be an easy migration from CVS. It's established, well-used, and conceptually similar to CVS in some respects. (Now, if your developers are so far removed from even that, it might not help.)
Many, many people have done this migration. There are utilities out there to migrate your code from CVS to SVN (but not without some pain).
It should address most of your requirements (though how easy merging can be is debatable).
Bottom line is: your challenge isn't the technology. It's the adoption. If your team doesn't want to change, it won't. And sadly, any solution will be doomed. You have to convince them they need a change first. The best bet is to appeal to their laziness (I mean that in a positive way) and show that "life will be better" if they change.
Any tool would need training. I would say that if you are confident that you will be moving to some other system then it makes complete sense to spend sometime on training the guys so that in the longer run it would be very helpful.
By looking at your requirement set I'd say try looking at svn or sos. Advantage of having sos is that you get 24*7 support on issues. They can handle large data. They also have a cool browser interface.
I am curious as to how other development teams spec out new features. The team I have just moved up to lead has no real specification process. I have just implemented a proper development process with CI, auto deployment and logging all bugs using Trac and I am now moving on to deal with changes.
I have a list of about 20 changes to our product to have done over the next 2 months. Normally I would just spec out each change going into detail of what should be done but I am curious as to how other teams handle this. Any suggestions?
I think we had a successful approach in my last job as we delivered the project on time and with only a couple of issues found in production. However, there were only 3 people working on the product, so I'm not entirely sure how it would scale to larger teams.
We wrote specs upfront for the whole product but without going into too much detail and with an emphasis on the UI. This was a means for us to get a feel for what had to be done and for the scope of the project.
When we started implementing things, we had to work everything out in a lot more detail (and inevitably had to do some things differently from the spec). To that end, we got together and worked out the best approach to implementing each feature (sometimes with prototypes). We didn't update the original spec but we did make notes after the meetings as it's very easy to forget the details afterwards.
So in summary, my approach is to treat specs as an exploratory tool and to work out finer details during implementation. Depending on the project, it may also be a good idea to keep the original spec up to date as the application evolves (which we didn't need to do this time).
Good question but its can be subjective. I guess it depends on the strategy of the product, if its to be deployed to multiple clients in the same way or to a single client on a bespoke project, the impact, dependency these changes have on the system and each other and the priority these changes need to be made.
I would look at the priority and the dependency, that will naturally start grouping things?
I recently joined the IT department of a big insurance company. Although the department's title is "IT", a lot of code gets written here; Java, JSP, JavaScript, COBOL and even some C++ from what I've heard. All the programs that allow insurers, brokers and the rest of the tie-wearing, white-collar workers to issue new contracts and deal with clients runs on the code produced by this department. I've been told that the department is pretty good by the parent company's standards and that we've even received an internal award or two. We're 17 people in the department, split in smaller groups of 2 or 3. As you might've guessed from the COBOL part further up, the average age is over 40 years (as a point of reference I'm 29 yo).
Right now, there is no version control system in place (there exists a general backup scheme though). When needed, files are passed around through shared folders. Usually there's one person in every group responsible for copying the "final" version of the files back to the production server. I find this absurd and even a bit dangerous.
How may I try to convince management that we should implement a VCS scheme in our department? I've never deployed a VCS myself but every other place I've worked at had one. I think I'll hit a "we've been OK until now, why bother" wall from the first step, coupled with the age of most of my co-workers that will feel this step is an unnecessary hurdle.
I know the basic advantages of VCS (traceability, granular backups, accountability etc). I'm looking to back my case with realistic cases and examples of real added value over the implementation costs, not just a "but-but-but, we must have a VCS you fools!" :-)
You don't necessarily need their permission.
Install svn on your machine, start using it, and then start convincing your fellow team members to use it too.
Then watch and see what happens.
Edits
The basic idea of this is that it's easier to show than to tell.
It's a great idea to support your ideas with a working implementation/solution.
Of course, if you succeed, and they want the system used department/company wide, you must be prepared to support the transition, know how the software is to be installed and used.
Going ahead and using something accepted in the industry is faster than having discussions on what system should be used.
There is a good change that this will get you noticed. You may also get your peers respect and support.
As suggested, the same approach can be made on other areas:
issue/bug tracking systems
quality tools
time tracking
continuous integration
a wiki for knowledge base, HOWTO's, guidelines, tutorials, presentations, screencasts
different IDEs and tools
build tools
automated deployment
various scripts that would save your team time
.. any item that will visibly add quality to your work, but doesn't (yet) disrupt existing methodologies and practices.
Joel Spolsky has an excellent article: Getting Things Done When You're Only a Grunt
Quote
Nobody on your team wants to use
source control? Create your own CVS
repository, on your own hard drive if
necessary. Even without cooperation,
you can check your code in
independently from everybody else's.
Then when they have problems that
source control can solve (someone
accidentally types rm * ~ instead of
rm *~), they'll come to you for help.
Eventually, people will realize that
they can have their own checkouts,
too.
Management? I will put bold the expressions and words you should use:
Your should display some examples how a VCS will prevent losing money to the company if some error/bugs or disaster happens. It will be faster to solve all problems, so maintaning the systens won't be so lazy and people become more productive.
You should also mention that implementing a VCS has no costs.
VCS will also give advantages for backup all the existing code. So, all the code will be safe.
My opinion on how to go about doing this, is that you should try to convince your fellow developers first. The way I see it, there are two ways this might go about:
You give the right arguments to the other developers (possibly only the head developers will suffice), they like the idea, and the suggest it to management. Management is easy to convince at that point, so everyone is happy.
You give the right arguments to management, who get all excited (great!) and mandate that version control has to be installed and used by everyone. Here's the thing: If at this point the other developers are not sold to the idea already, then (a) they might be hostile to an idea that management is forcing upon them, and (b) they might not like you for being the cause of it all.
So what are good arguments to convince fellow developers? As someone who uses subversion (which is the one I recommend in this case, by the way) even for his solo projects, here's a few advantages I can think of:
Using version control forces people to think of code modification in terms of a series of small, self-contained changes. This is an extremely beneficial way of working: where otherwise people would be inclined to make lots of changes all over the place, leaving the code in a mess, version control kinda forces them to change the code in bite-size, easy-to-swallow bits, keeping the code compiling at all times, easing the cost of integration with other modules, etc.
Version control makes it very easy to see what has changed in the code each time. This might sound trivial, but when you start modifying code it's easy to lose track after a while. But with VC it's all an "svn diff" (or equivalent) away, always.
Version control makes it very easy to see who has changed the code each time. So that, for example, when something breaks, you know who to blame. (It's not an accident that the subversion command which shows who last changed each line is called "svn blame".)
Version control makes it very easy to see why a piece of code was changed. Commit messages, if used properly, essentially provide continual documentation of the ongoing development process. Documentation that otherwise wouldn't be written.
Version control makes it very easy to track down regressions and see where they appeared. In the easy case, you just track down commit messages and spot the culprit. In the average case, you have to consult the diffs too. In the hard case, you have to do regression testing of previous versions using what amounts to binary search, which is still better than the no-VC case, where you simply have no clue.
This list is not exhaustive, of course, but these are the main benefits that come to mind right now. Obviously, as others have already mentioned, it's easier to show all this to your colleagues than to describe it to them, and setting it up for yourself first (but importing everyone's code, mind) sounds like a great idea.
As Joel points out on one of his articles, start using your own one man version control system and market its benefit on every opportunity you get. Show them the benefits of traceability, granular backups etc from your single instance. People will start realizing its benefits irrespective of their age.
I agree with the answer that are referring to the Joel Guerrilla article.
Install/Use some thing with a low overhead. Hg (Mercurial) is easy in a mixed eniroment and is good because you can bail out and use something else in an easy way.
You must share your things without making a fuzz about it. When someone needs your code, export it and use the "standard" corporate method (shared folder or whatever)
When you get code, always import it into a repos, if you think it is a new commit of a repos you already have, try to get it into that one.
Sooner or later you will have a code for several project and hopefully some commits on some repos. Then you can expose those with the mercurials webserver interface (hg serve -p XXXX).
When the times comes when someone don't know why something suddenly don't works as it should be and is trying to figure out why becase it was working last monday ... and you know that you have that code in a repos step up and ask if you can be of any assistance. Get the falty code, commit it into your repos and expose with hg serve. Look at it in the browser.
My point is that you must prove with real cases to your colleges that this stuff has a value.
If the haven't figured it out by themselves after some many years you have a mountain to climb but it can be done. You must be patience though. It could very well take a year to convert one man (old dog). If you have any younger coworkers try to do this together, the more code you can get hold on the better.
I would point out the hazards of not having one - lost code, developers over writing each other changes, ability to rollback problems, etc.
Also since Subversion and some others are free, point out there is no real cost to purchase, jsut the time to implement.
The biggest issue you will have as the new guy is that you will be seen as rockign the boat, if they had no issues to date they will be hard to convince. Perhaps start using it locally jsut for yourself and maybe they will like what they see and start to adopt it.
I would try small steps, maybe ask the others if they ever used one, point out the benefits, when an issue arises that a system would have prevented or aided in point it out delicately.
From a purely business perspective, and depending on the size and nature of your parent company an IT auditor may consider your lack of a VCS a finding (i.e. something that needs fixing). I believe you could improve your pitch to management by telling them that any CVS is a great way of showing that your department respects its resources and works in a structured way and efficient way, something auditors always like to see.
I don't know how your corporate culture works but I'd be careful about rolling out your own CVS since if it does see use it suddenly becomes your responsibility when things go wrong, even if you were not at fault. To cover your ass (and keep the aforementioned auditors even happier) I'd roll the system out with a full set of written procedures for its use and maintenance.
Finally while I myself am a big fan of initiative at any level of the enterprise don't expect people to remember to say thanks when they figure out how great it is. Some might, but for the most part you're doing this to make your own life easier and for your own karma.
Remember, there are plenty of version control systems that are absolutely free. And the amount of time spent installing and maintaining a version control system should be somewhere near 0 (they shouldn't require any maintenance). There isn't even a space penalty for most systems, as they can compress things internally.
You have listed some advantages, and there are others. But more importantly, I can't think of a single disadvantage.
I would also recommend starting with implementing VCS (Version Control System) for yourself first. I'd recommend using one of distributed VCS (Git, Mercurial, Bazaar) rather than centralized Subversion, because it would be easier to create central repository (or repositories) by cloning than moving your Subversion repository to central place. Distributed SCM can be also used in a smaller group to exchange ideas.
A few advantages of (modern) version control systems:
You can always revert (go back) to last working version of your code (provided that you follow some sane version control conventions, like at least tagging only tested code). With code shared via folders it might turn out that no one version works, backup copies were deleted to save space, and recovering code from backup is tedious / was never tested.
You can switch between working on some new feature (some experimental work), and working on urgent fix in currently deployed version (maintenace work) thanks to branches (and stash / shelve for uncommitted work).
If you follow good practices for version control (small and often commits, changes being about one single thing, writing good commit message describing change and whys of change) you would have much, much easier finding bugs, be it by bisecting history to find which change introduced bug, or by using version control system to look up who was responsible for given area of code (annotate / blame).
Start talking to the other developers about problems thay have had in the past as a way to get to know the system and how it evolved (sneaky, sneaky, sneaky, but hey this information will probably come in handy at some poitn even aside from the version control issue). You are bound to sooner or later find some wonderful examples of things that have already happened which would have been far less painful if you had version control. Use these examples when you present the idea to management.
I agree with the idea that you can probably start using your own version control and eventually will be able to help thm out of a fix, but I'd bet money they have been in some of those fixes already and if they already remember how painful the problme was before, it will help sell the new idea.
Look for another job.
Seriously.
There are way better jobs out there that don't require you to teach the existing staff.
Ones where you could go into work and just, y'know, work.
Also, keep in mind that 30 isn't far off. That's the age at which most people
stop suffering fools gladly.
Just a heads up.
EDIT
It's been suggested that quitting a bad job is for quitters.
Maybe so, but keep in mind that you're supposed to
put your employer to the Joel test before you accept the job, not after.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 3 years ago.
Improve this question
We have written a software package for a particular niche industry. This package has been pretty successful, to the extent that we have signed up several different clients in the industry, who use us as a hosted solution provider, and many others are knocking on our doors. If we achieve the kind of success that we're aiming for, we will have literally hundreds of clients, each with their own web site hosted on our servers.
Trouble is, each client comes in with their own little customizations and tweaks that they need for their own local circumstances and conditions, often (but not always) based on local state or even county legislation or bureaucracy. So while probably 90-95% of the system is the same across all clients, we're going to have to build and support these little customizations.
Moreover, the system is still very much a work in progress. There are enhancements and bug fixes happening continually on the core system that need to be applied across all clients.
We are writing code in .NET (ASP, C#), MS-SQL 2005 is our DB server, and we're using SourceGear Vault as our source control system. I have worked with branching in Vault before, and it's great if you only need to keep 2 or 3 branches synchronized - but we're looking at maintaining hundreds of branches, which is just unthinkable.
My question is: How do you recommend we manage all this?
I expect answers will be addressing things like object architecture, web server architecture, source control management, developer teams etc. I have a few ideas of my own, but I have no real experience in managing something like this, and I'd really appreciate hearing from people who have done this sort of thing before.
Thanks!
I would recommend against maintaining separate code branches per customer. This is a nightmare to maintain working code against your Core.
I do recommend you do implement the Strategy Pattern and cover your "customer customizations" with automated tests (e.g. Unit & Functional) whenever you are changing your Core.
UPDATE:
I recommend that before you get too many customers, you need to establish a system of creating and updating each of their websites. How involved you get is going to be balanced by your current revenue stream of course, but you should have an end in mind.
For example, when you just signed up Customer X (hopefully all via the web), their website will be created in XX minutes and send the customer an email stating it's ready.
You definitely want to setup a Continuous Integration (CI) environment. TeamCity is a great tool, and free.
With this in place, you'll be able to check your updates in a staging environment and can then apply those patches across your production instances.
Bottom Line: Once you get over a handful of customers, you need to start thinking about automating your operations and your deployment as yet another application to itself.
UPDATE: This post highlights the negative effects of branching per customer.
Our software has very similar requirements and I've picked up a few things over the years.
First of all, such customizations will cost you both in the short and long-term. If you have control over it, place some checks and balances such that sales & marketing do not over-zealously sell customizations.
I agree with the other posters that say NOT to use source control to manage this. It should be built into the project architecture wherever possible. When I first began working for my current employer, source control was being used for this and it quickly became a nightmare.
We use a separate database for each client, mainly because for many of our clients, the law or the client themselves require it due to privacy concerns, etc...
I would say that the business logic differences have probably been the least difficult part of the experience for us (your mileage may vary depending on the nature of the customizations required). For us, most variations in business logic can be broken down into a set of configuration values which we store in an xml file that is modified upon deployment (if machine specific) or stored in a client-specific folder and kept in source control (explained below). The business logic obtains these values at runtime and adjusts its execution appropriately. You can use this in concert with various strategy and factory patterns as well -- config fields can contain names of strategies etc... . Also, unit testing can be used to verify that you haven't broken things for other clients when you make changes. Currently, adding most new clients to the system involves simply mixing/matching the appropriate config values (as far as business logic is concerned).
More of a problem for us is managing the content of the site itself including the pages/style sheets/text strings/images, all of which our clients often want customized. The current approach that I've taken for this is to create a folder tree for each client that mirrors the main site - this tree is rooted at a folder named "custom" that is located in the main site folder and deployed with the site. Content placed in the client-specific set of folders either overrides or merges with the default content (depending on file type). At runtime the correct file is chosen based on the current context (user, language, etc...). The site can be made to serve multiple clients this way. Efficiency may also be a concern - you can use caching, etc... to make it faster (I use a custom VirtualPathProvider). The largest problem we run into is the burden of visually testing all of these pages when we need to make changes. Basically, to be 100% sure you haven't broken something in a client's custom setup when you have changed a shared stylesheet, image, etc... you would have to visually inspect every single page after any significant design change. I've developed some "feel" over time as to what changes can be comfortably made without breaking things, but it's still not a foolproof system by any means.
In my case I also have no control other than offering my opinion over which visual/code customizations are sold so MANY more of them than I would like have been sold and implemented.
This is not something that you want to solve with source control management, but within the architecture of your application.
I would come up with some sort of plugin like architecture. Which plugins to use for which website would then become a configuration issue and not a source control issue.
This allows you to use branches, etc. for the stuff that they are intended for: parallel development of code between (or maybe even over) releases. Each plugin becomes a seperate project (or subproject) within your source code system. This also allows you to combine all plugins and your main application into one visual studio solution to help with dependency analisys etc.
Loosely coupling the various components in your application is the best way to go.
As mention before, source control does not sound like a good solution for your problem. To me it sounds that is better yo have a single code base using a multi-tenant architecture. This way you get a lot of benefits in terms of managing your application, load on the service, scalability, etc.
Our product using this approach and what we have is some (a lot) of core functionality that is the same for all clients, custom modules that are used by one or more clients and at the core a the "customization" is a simple workflow engine that uses different workflows for different clients, so each clients gets the core functionality, its own workflow(s) and some extended set of modules that are either client specific or generalized for more that one client.
Here's something to get you started on multi-tenancy architecture:
Multi-Tenant Data Architecture
SaaS database tenancy patterns
Without more info, such as types of client specific customization, one can only guess how deep or superficial the changes are. Some simple/standard approaches to consider:
If you can keep a central config specifying the uniqueness from client to client
If you can centralize the business rules to one class or group of classes
If you can store the business rules in the database and pull out based on client
If the business rules can all be DB/SQL based (each client having their own DB
Overall hard coding differences based on client name/id is very problematic, keeping different code bases per client is costly (think of the complete testing/retesting time required for the 90% that doesn't change)...I think more info is required to properly answer (give some specifics)
Layer the application. One of those layers contains customizations and should be able to be pulled out at any time without affect on the rest of the system. Application- and DB-level "triggers" (quoted because they may or many not employ actual DB triggers) that call customer-specific code or are parametrized with customer keys) are very helpful.
Core should never be customized, but you must layer it in somewhere, even if it is simplistic web filtering.
What we have is a a core datbase that has the functionality that all clients get. Then each client has a separate database that contains the customizations for that client. This is expensive in terms of maintenance. The other problem is that when two clients ask for a simliar functionality, it is often done differnetly by the two separate teams. There is currently little done to share custiomizations between clients and make common ones become part of the core application. Each client has their own application portal, so we don't have the worry about a change to one client affecting some other client.
Right now we are looking at changing to a process using a rules engine, but there is some concern that the perfomance won't be there for the number of records we need to be able to process. However, in your circumstances, this might be a viable alternative.
I've used some applications that offered the following customizations:
Web pages were configurable - we could drag fields out of view, position them where we wanted with our own name for the field label.
Add our own views or stored procedures and use them in: data grids (along with an update proc) and reports. Each client would need their own database.
Custom mapping of Excel files to import data into system.
Add our own calculated fields.
Ability to run custom scripts on forms during various events.
Identify our own custom fields.
If you clients are larger companies, you're almost going to need your own SDK, API's, etc.