My co-workers and I are relatively need to the stream idea with Clearcase UCM. Currently management has created streams for each functional software package, each of which has defined interfaces and lives within a layered architecture. Developers create child streams depending on the package they are working in, and attempt to develop their code independently, however they normally have dependencies on other packages during initial development. This has caused our integration group to create system builds that developers then use to create an adequate environment to develop their software and manually pull in dependencies (i.e. zip files, patches, etc.).
My contention is that this is wrong and not how UCM was intended to be used, but I needed someone more familiar with UCM to confirm my beliefs.
I believe that streams should be created from a functional point of view (while each package does some function, multiple architectural packages contribute to achieving some customer function, call it "ABC"). Then, the component for each architectural component which is perfomring initial development for function "ABC" is added to the stream. All developers for function "ABC" now work in the stream (or in some set of child streams) to complete that function. Once complete you have a baseline for each UCM component, and no "binding" between components exists from UCM's point of view (someone claimed this could somehow happen within UCM due to Activity Records).
NOTE: I agree that maybe you don't work this way FOREVER, but during initial development where interfaces commonly change, you haven't implemented all the interfaces for all functions, and so having multiple components working together in a stream makes the most sense. Later you can transition to a "architectural package-centric" way of working where each package is independent of changes in another.
Thoughts? Sorry for the long post, I felt the detail was necessary.
created streams for each functional software package
All developers for function "ABC" now work in the stream (or in some set of child streams) to complete that function
Yes, that's pretty much the two UCM normal usages of stream
(the only very bad usage is the one involving one stream per developer, just for isolation purpose, and that would be madness, as specified before)
Those two modes are system approach and component approach, detailed in this answer.
Basically, you want to avoid too much merges or rebase during the initial phase of development and keep one coherent system (with all components writable) at the beginning.
Then, when API is stabilized, you can go one stream per writable component.
Note: that does not prevent you to establish "system integration" streams, when you have a set of well-defined baselines referencing a stable state for all your components (read-only), and where you can deploy and test your system.
Those streams are maintained on one or several separate "integration" UCM projects.
I do agree with VonC. I'd prefer the functional approach.
There is a ClearCase plug-in that can help you to establish environments for your users (stream, views, project strategy) whatever approach you take. Just google about "clearEnv"
Related
Background: I'm using jira for bug tracking, and git for source control. I've got a complete end-to-end system comprising of an iOS front end, and a Java/Tomcat back end that provides web services and a GUI. Right now I've got a single git repository holding all the software and a single jira project tracking issues for the whole system.
Now that the software is live, I'm finding that changes are being made to either the iOS application or the server, but generally not both. The version numbers of the two components have diverged somewhat.
It's probably too late for this project, but in future:
Should I pursue the path of having all related components in a single source repository and tracked using a single bug-tracking project; or
Should each component be in a separate repository and be managed by a separate bug-tracking project?
I can see pro's and con's for both approaches, and I can also see that the answer could easily be "it depends".
Which way would you lean, and why?
I'd go with distinct source repositories for a few reasons
The developers working on the two are likely to have distinct
skill sets. Also, you may have management reasons for wanting to
segregate who sees what.
They should not be tightly tied at a protocol level - different versions need to interact.
The first point becomes even more important when you do another front end
The second reason is my main one.
However, I'd go with a common bug database. Defects/features may need changes on both ends. Also, it is extremely likely you will have bugs that are believed to be in one component but actually end up fixed in the other. If you try to migrate across databases, information will get lost. I've seen that too many times.
we are creating hospital information system software. The project will be different hospital to hospital and contain different use cases. But lots of parts will be the same. So we will use branching mechanism of the source control. If we find a bug in one hospital, how can we know the other branches have the same bug or not.
IMAGE http://img85.imageshack.us/img85/5074/version.png
The numbers in the picture which we attached show the each hospital software.
Do you have a solution about this problem ?
Which source control(SVN,Git,Hg) we will be suitable about this problem ?
Thank you.!
Ok, this is not really a VCS question, this is first and foremost and architectural problem - how do you structure and build your application(s) in such a way that you can deliver the specific use cases to each hospital as required whilst being able, as you suggest, to fix bugs in commom code.
I think one can state with some certainty that if you follow the model suggested in your image you aren't going to be able to do so consistently and effectively. What you will end up with is a number of different, discrete, applications that have to be maintained separately even though they have at some point come from a common set of code.
Its hard to make more than good generalisations but the way I would think about this would be something along the following lines:
Firstly you need a core application (or set of applications or set of application libraries) these will form the basis of any delivered system and therefore a single maintained set of code (although this core may itself include external libraries).
You then have a number of options for your customised applications (the per hospital instance) you can define the available functionality a number of means:
At one extreme, by configuration - having one application containing all the code and effectively switching things on and off on a per instance basis.
At the other extreme by having an application per hospital that substantially comprises the core code with customisation.
However the likelyhood is that whilst the sum of the use cases for each hospital is different individual use cases will be common across a number of instances so you need to aim for a modular system i.e. something that starts with the common core and that can be extended and configured by composition as much as by any other means.
This means that you are probably want to make extensive use of Inversion of Control and Dependency Injection to give you flexibility within a common framework. You want to look at extensibility frameworks (I do .NET so I'd be looking at the Managed Extensibility Framework - MEF) that allow you to go some way towards "assembling" an application at runtime rather than at compile time.
You also need to pay attention to how you're going to deploy - especially how you're going to update - your applications and at this point you're right you're going to need to have both your version control and you build environment right.
Once you know how you're going build your application then you can look at your version control system - #VonC is spot on when he says that the key feature is to be able to include code from shared projects into multiple deliverable projects.
If it were me, now, I'd probably have a core (which will probably itself be multiple projects/solutions) and then one project/solution per hospital but I would be aiming to have as little code as possible in the per hospital projects - ideally just enough framework to define the instance specific configuration and UI customisation
As to which to use... if you're a Microsoft shop then take a good long hard look at TFS, the productivity gains from having a well integrated environment can be considerable.
Otherwise (and in any case), DVCS (Mercurial, Git, Bazaar, etc) seem to me to be gaining an edge on the more traditional systems as they mature. I think SVN is an excellent tool (I use it and it works), and I think that you need a central repository for this kind of development - not least because you need somewhere that triggers your Continuous Integration Server - however you can achieve the same thing with DVCS and the ability to do frequent local, incremental, commits without "breaking the build" and the flexibility that DVCS gives you means that if you have a choice now then that is almost certainly the way to go (but you do need to ensure that you establish good practices in ensuring that code is pushed to your core repositories early)
I think there is still a lot to address purely from the VCS question - but you can't get to that in useful detail 'til you know how you're going to structure your delivered solution.
All of those VCS (Version Control System) you mention are compatible with the notion of "shared component", which allows you to define a common shared and deployed code base, plus some specialization in each branch:
CVCS (Centralized)
Subversion externals
DVCS (Distributed)
Git submodules (see true nature of submodules)
Hg SubRepos
Considering the distributed aspect of the release management process, a DVCS would be more appropriate.
If the bug is located in the common code base, you can quickly see in the other branches if:
what exact version of the common component they are referring to.
they refer the same or older version of that common component than the one in which the bug has been found (in which case chances are they also do have the bug)
Given a large scale software project with several components written in different languages, configuration files, configuration scripts, environment settings and database migration scripts - what are the common practices for deployment to production?
What are the difficulties to consider? Can the process be simplified with tools like Ant or Maven? How can rollback and database management be handled? Is it advisable to use version control for the production environment?
As I see it, you're mostly asking about best practices and tools for release engineering AKA releng -- it's important to know the "term of art" for a subject, because it makes it much easier to search for more information.
A configuration management system (CMS -- aka revision control system or version control system) is indispensable for today's software development; if you use one or more IDEs, it's also nice to have good integration between them and the CMS, though that's more of an issue for purposes of development than for purposes of deployment / releng.
From a releng viewpoint, the key thing about a CMS is that it must have good support for "branching" (under whatever name), because releases must be made from a "release branch" where all the code under development and all of its dependencies (code and data) are in a stable "snapshot" from which the exact, identical configuration can be reproduced at will.
The need for good branching support may be more obvious if you have to maintain multiple branches (customized for different uses, platforms, whatever), but even if your releases are always, strictly in a single linear sequence, releng best practices still dictate making a release branch. "Good branching support" includes ease of merging (and "conflict resolution" when different changes are made to a file), "cherry-picking" (taking one patch or changeset from one branch, or the head/trunk, and applying it to another branch), and the like.
In practice, you start off the release process by making a release branch; then, you run exhaustive testing on that branch (typically MUCH more than what you run everyday in your continuous build -- including extensive regression testing, integration testing, load testing, performance verification, etc, and possibly even more costly quality assurance processes, depending). If and when the exhaustive testing and QA reveal defects in the release-candidate (including regressions, performance degradation, etc), they must be fixed; in a large team, development on head/trunk may be continuing while the QA is being done, whence the need for ease of cherry-picking / merging / etc (whether your practice is to perform the fix on head or on the release branch, it still needs to be merged to the other side;-).
Last but not least, you're NOT getting full releng value from your CMS unless you're somehow tracking with it "everything" that your releases depend on -- simplest would be to have copies or hard links to all the binaries for the tools you need to build your release, etc, but that may often be impractical; so at the very least track the exact release, version, bugfix &c numbers of those tools that are used (operating system, compilers, system libraries, tools that preprocess image, sound or video files into final form, etc, etc). The key is being able, at need, to exactly reproduce the environment required to rebuild the exact version that's proposed for release (otherwise you'll go mad tracking down subtle bugs that may depend on third party tools' changes as their versions change;-).
After a CMS, the second most important tool for releng is a good issue tracking system -- ideally one that's well integrated with the CMS. That's also important for the development process (and other aspects of product management), but in terms of the release process the importance of the issue tracker is the ability to easily document exactly what bugs have been fixed, what features have been added, removed, or changed, and what modifications in performance (or other user-observable characteristics) are expected in the new forthcoming release. For the purpose, a key "best practice" in development is that every changeset that gets committed to the CMS must be connected to one (or more) issue in the issue tracking system: after all, there's gotta be some purpose for that change (fix a bug, change a feature, optimize something, or some internal refactor that's supposed to be invisible to the software's user); similarly, every tracked issue that's marked as "closed" must be connected to one (or more) changesets (unless the closing is of the "won't fix / working as intended" kind; issues related to bugs &c in third-party components, which have been fixed by the third-party supplier, are easy to treat similarly if you do manage to keep track of all third-party components in the CMS too, see above; if you don't, at least there should be text files under CMS documenting third-party components and their evolution, again see above, and they need to be changed when some tracked issue on a 3rd party component gets closed).
Automating the various releng processes (including building, automated testing, and deployment tasks) is the third top priority -- automated processes are much more productive and repeatable than asking some poor individual to manually go through a list of steps (for sufficiently complex tasks, of course, the workflow of the automation may need to "get a human being in the loop"). As you surmise, tools such as Ant (and SCons, etc, etc) can help here, but inevitably (unless you're lucky enough to get away with very simple and straightforward processes) you'll find yourself enriching them with ad-hoc scripts &c (some powerful and flexible scripting language such as perl, python, ruby, &c, will help). A "workflow engine" can also be precious when your release workflow is sufficiently complex (e.g. involving specific human beings or groups thereof "signing off" on QA compliance, legal compliance, UI guidelines compliance, and so forth).
Some other specific issues you're asking about vary enormously depending on specifics of your environment. If you can afford programmed downtime, your life is relatively easy, even with a large database in play, as you can operate sequentially and deterministically: you shut the existing system down gracefully, ensure the current database is saved and backed up (easing rollback, in the hopefully VERY rare case it's needed), run the one-off scripts for schema migration or other "irreversible" environment changes, fire the system back up again in a mode that's still unaccessible to general users, run another extensive suite of automated tests -- and finally if everything's gone smoothly (including the saving and backup of the DB in its new state, if relevant) the system's opened up to general use again.
If you need to update a "live" system, without downtime, this may range anywhere from a slight inconvenience to a systematic nightmare. In the best case, transactions are reasonably short, and synchronization between the state set by transactions may be delayed a bit without damage... and you have a reasonable abundance of resources (CPUs, storage, &c). In this case, you run two systems in parallel -- the old one and the new one -- and just make sure all new transactions target the new system, while letting old ones complete on the old system. A separate task periodically syncs "new data in the old system" to the new system, as transactions on the old system terminate. Eventually you can determine that no transactions are running on the old system and all changes that happened there are synced up to the new one -- and at that time you can finally shut down the old system. (You need to be prepared to "reverse sync" too of course, in case a rollback of the change is needed).
This is the "simple, sweet" extreme for live system updating; at the other extreme, you can find yourself in such an overconstrained situation that you can prove the task is impossible (you just cannot logically meet all the stated requirements with the given resources). Long sessions opened on the old system which just cannot be terminated -- scarce resources that make it impossible to run two systems in parallel -- core requirements for hard-real-time sync of every transaction -- etc, etc, can all make your life miserable (and as I noticed, at the extreme, can make the stated task absolutely impossible). The two best things you can do about this: (1) ensure you have abundant resources (this will also save your skin when some server unexpected goes belly-up... you'll have another one to fire up to meet the emergency!-); (2) consider this predicament from the start, when initially defining the architecture of the overall system (e.g.: prefer short-lived transactions to long-lived sessions that just can't be "snapshot, closed down, and restarted seamlessly from the snapshot", is one good arcitectural pointer;-).
disclaimer: where I work we use something I wrote that I'm going to mention
I'll tell you how I do it.
For configuration settings, and general deployment of code and content, I use a combination of NAnt, a CI server, and dashy (an automatic deployment tool). You can replace dashy with any other 'thing', that automates the uploads to your server (perhaps capistrano).
For DB scripts, we use RedGate SQL Compare to get the scripts, and then for most changes, I actually make them manually, where appropriate. This is because some of the changes are slightly complex, and I feel more comfortable doing it by hand. You can actually use this tool to do it for you, (or at least generate the scripts).
Depending on your language, there are some tools that can script the DB updates for you (I think someone on this forum wrote one; hopefully he'll reply), but I have no experience with those. It's something I'd like to add though.
Difficulties
I forgot to answer one of your questions.
The biggest problem in updating any significantly complicated/distributed site is DB synchronisation. You need to consider if you will have any downtime, and if you do, what will happen to the DBs. Will you shutdown everything, so that no transactions can be processed? Or will you shift everything to one server, update DB B, and then sync DB A & B, then update DB B? Or something else?
Whatever you choose, you need to choose it, and either say "Okay for each update, there will be X downtime", or whatever. Just have it documented.
The worst thing you can do is have someones transaction fail, mid processing, because you were updating that server; or to somehow leave only part of your system operational.
I don't think you have any option about using versioning or not.
You cannot go without versioning (as mentioned in a comment).
I am talking from experience since I am currently working on a website where we have several items that have to work together:
The software itself, its functionalities at a given point in time
The external systems (6 of them) we are linked with (messages are versioned)
A database, which holds the configuration (and translations)
The static resources (images, css, javascripts) hosted on an Apache server (several in fact)
A java applet, which has to be synchronized with the javascript and software
This works because we use versioning, although I must admit that our database is pretty simple, and because the deployment is automated.
Versioning means that we have actually at any point in time several concurrent versions of the messages, the database, the static resources and the java applet.
It is especially important in the event of a 'fallback'. If you discover a flaw when loading you new software and suddenly you cannot afford to load, you'll have a crisis if you have not versioned, and you'll just have to load the old soft if you have.
Now as I said, the deployment is scripted:
- the static resources are deployed first (+ the java applet)
- the database goes next, several configurations cohabit since they are versioned
- the soft is queued for load in a 'cool' time-window (when our traffic is at its lowest point, so at night)
And of course we take care of backward / forward compatibility issues with the external servers, which should not be underestimated.
As an alternative to this question what is the best way to manage custom versions of software for a particular client?
Most of the differences between client releases is changes to the user interface to customize the software to look like it is owned by the client. More often then not, this is a simple logo change. Occasionaly the color scheme will change as well. But there are occassions where features will be enabled or disabled based on the client. What is the best way to keep all of these releases up to date and make them easily available to the users of a particular client?
At this point we have five different clients and each has their own build of the software and their own installer (complete with their logo in the installer). This is becoming a pain to manage, and it will only get worse as more and more clients start using our software.
So assuming the linked question isn't the way to go, what is the best way to manage these releases?
"This is becoming a pain to manage,
and it will only get worse as more and
more clients start using our
software."
The only way to really solve this part is to make sure your software has an architecture that supports the customizations as a layer on top of your core product. Ideally, they would simply be runtime configuration options (colours, logos and enable/disable are perfect for this). If you get (or want) a very large number of clients, push the configurability right into the core product and enable the clients to do the customizing themselves.
When updating, you build (and test) the core product once, and build the custom versions by simply linking to (referencing) the already-built core product as a library. You could have each client in a separate build, or you could have a single build process that generates updates for all the currently-maintained client builds (the latter is probably better for a larger number of clients). If you have a "vanilla" version, you could build that as part of the core or along with the client versions, it would depend on your particular case.
Depending on your technology, it might be possible to have the customization layer be built independently from the core product. In this case, client rebuilds will rarely be necessary (perhaps only for certain significant changes) -- you can simply link to the updated core product at runtime. For deployment to the client, you would only need to deploy the updated core, if you have a deployment method that supports this.
It's hard to say more detail without knowing your platform, but you've graciously kept this question agnostic.
Separate the common and the custom parts in your source tree. This can eliminate the great majority of merges, depending on your testing and release policies. There is always a way to abstract out and customize a build resource, even if your build process has to invoke script to rewrite some file.
Judicious use of branching in a good Source Code Management system. These aren't called "Configuration Management" systems for nothing. They are the tools optimized for precisely this task, so you're probably not going to get anything better without building on top of one.
Subversion is good at setting up multiple branches, but last I checked it could only do the 3-way merge on a single file at a time. Perforce is awesome in this department because it tracks your branching history on a file-by-file basis and uses this to automate merging of whole sets of changes between whole branches. It really is the cat's pajamas here.
Git and darcs may have similar power, but I haven't used them. They seem to be based on the idea that each working checkout tree will have a copy all changes ever since the beginning of time. That sounds impractical to me since I need to keep some large and changing SDKs under SCM control.
Arguably, the only difference needs to be a branch of whatever release you have, with the customer-unique changes in it. So for example:
/project
/trunk
/branches
/release-v1
/customer-branches
/release-v1-google
/release-v1-microsoft
....
Where your client releases are branches of your customer releases. Since these branches won't ever get rolled up into trunk or another release branch, don't pollute the regular development /branches tree.
I guess it depends on the level of customization you need for each client, but could you make your base app "customize" itself based on a config file?, so you could have the trunk being the full app and then have special config files for each customer that control the look & feel, as well as features that are enabled, and your release process would take the correct config files to deploy in your installers.
Seems that if you are always delivering custom(ish) versions of your app for each client, it would be worth the time to extend the app in this way.
I set up a header file called branding.h which contains a bunch of #ifdefs like the following example to change whatever bits need changing. In Visual Studio, it is easy to set up multiple builds with the appropriate client symbol defined.
#if defined BRAND_CLIENT1
# define COMPANY_NAME "Client 1"
# define PRODUCT_NAME "The Client 1 Widget App"
# define LOGO_FILE "res/logoClient1.ico"
#elif defined BRAND_CLIENT2
# define COMPANY_NAME "Client 2"
# define PRODUCT_NAME "The Client 2 Super Widget App"
# define ENABLE_EXTRA_MENU
# define LOGO_FILE "res/logoClient2.ico"
#endif
This is all assuming C++ of course.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 3 years ago.
Improve this question
We have written a software package for a particular niche industry. This package has been pretty successful, to the extent that we have signed up several different clients in the industry, who use us as a hosted solution provider, and many others are knocking on our doors. If we achieve the kind of success that we're aiming for, we will have literally hundreds of clients, each with their own web site hosted on our servers.
Trouble is, each client comes in with their own little customizations and tweaks that they need for their own local circumstances and conditions, often (but not always) based on local state or even county legislation or bureaucracy. So while probably 90-95% of the system is the same across all clients, we're going to have to build and support these little customizations.
Moreover, the system is still very much a work in progress. There are enhancements and bug fixes happening continually on the core system that need to be applied across all clients.
We are writing code in .NET (ASP, C#), MS-SQL 2005 is our DB server, and we're using SourceGear Vault as our source control system. I have worked with branching in Vault before, and it's great if you only need to keep 2 or 3 branches synchronized - but we're looking at maintaining hundreds of branches, which is just unthinkable.
My question is: How do you recommend we manage all this?
I expect answers will be addressing things like object architecture, web server architecture, source control management, developer teams etc. I have a few ideas of my own, but I have no real experience in managing something like this, and I'd really appreciate hearing from people who have done this sort of thing before.
Thanks!
I would recommend against maintaining separate code branches per customer. This is a nightmare to maintain working code against your Core.
I do recommend you do implement the Strategy Pattern and cover your "customer customizations" with automated tests (e.g. Unit & Functional) whenever you are changing your Core.
UPDATE:
I recommend that before you get too many customers, you need to establish a system of creating and updating each of their websites. How involved you get is going to be balanced by your current revenue stream of course, but you should have an end in mind.
For example, when you just signed up Customer X (hopefully all via the web), their website will be created in XX minutes and send the customer an email stating it's ready.
You definitely want to setup a Continuous Integration (CI) environment. TeamCity is a great tool, and free.
With this in place, you'll be able to check your updates in a staging environment and can then apply those patches across your production instances.
Bottom Line: Once you get over a handful of customers, you need to start thinking about automating your operations and your deployment as yet another application to itself.
UPDATE: This post highlights the negative effects of branching per customer.
Our software has very similar requirements and I've picked up a few things over the years.
First of all, such customizations will cost you both in the short and long-term. If you have control over it, place some checks and balances such that sales & marketing do not over-zealously sell customizations.
I agree with the other posters that say NOT to use source control to manage this. It should be built into the project architecture wherever possible. When I first began working for my current employer, source control was being used for this and it quickly became a nightmare.
We use a separate database for each client, mainly because for many of our clients, the law or the client themselves require it due to privacy concerns, etc...
I would say that the business logic differences have probably been the least difficult part of the experience for us (your mileage may vary depending on the nature of the customizations required). For us, most variations in business logic can be broken down into a set of configuration values which we store in an xml file that is modified upon deployment (if machine specific) or stored in a client-specific folder and kept in source control (explained below). The business logic obtains these values at runtime and adjusts its execution appropriately. You can use this in concert with various strategy and factory patterns as well -- config fields can contain names of strategies etc... . Also, unit testing can be used to verify that you haven't broken things for other clients when you make changes. Currently, adding most new clients to the system involves simply mixing/matching the appropriate config values (as far as business logic is concerned).
More of a problem for us is managing the content of the site itself including the pages/style sheets/text strings/images, all of which our clients often want customized. The current approach that I've taken for this is to create a folder tree for each client that mirrors the main site - this tree is rooted at a folder named "custom" that is located in the main site folder and deployed with the site. Content placed in the client-specific set of folders either overrides or merges with the default content (depending on file type). At runtime the correct file is chosen based on the current context (user, language, etc...). The site can be made to serve multiple clients this way. Efficiency may also be a concern - you can use caching, etc... to make it faster (I use a custom VirtualPathProvider). The largest problem we run into is the burden of visually testing all of these pages when we need to make changes. Basically, to be 100% sure you haven't broken something in a client's custom setup when you have changed a shared stylesheet, image, etc... you would have to visually inspect every single page after any significant design change. I've developed some "feel" over time as to what changes can be comfortably made without breaking things, but it's still not a foolproof system by any means.
In my case I also have no control other than offering my opinion over which visual/code customizations are sold so MANY more of them than I would like have been sold and implemented.
This is not something that you want to solve with source control management, but within the architecture of your application.
I would come up with some sort of plugin like architecture. Which plugins to use for which website would then become a configuration issue and not a source control issue.
This allows you to use branches, etc. for the stuff that they are intended for: parallel development of code between (or maybe even over) releases. Each plugin becomes a seperate project (or subproject) within your source code system. This also allows you to combine all plugins and your main application into one visual studio solution to help with dependency analisys etc.
Loosely coupling the various components in your application is the best way to go.
As mention before, source control does not sound like a good solution for your problem. To me it sounds that is better yo have a single code base using a multi-tenant architecture. This way you get a lot of benefits in terms of managing your application, load on the service, scalability, etc.
Our product using this approach and what we have is some (a lot) of core functionality that is the same for all clients, custom modules that are used by one or more clients and at the core a the "customization" is a simple workflow engine that uses different workflows for different clients, so each clients gets the core functionality, its own workflow(s) and some extended set of modules that are either client specific or generalized for more that one client.
Here's something to get you started on multi-tenancy architecture:
Multi-Tenant Data Architecture
SaaS database tenancy patterns
Without more info, such as types of client specific customization, one can only guess how deep or superficial the changes are. Some simple/standard approaches to consider:
If you can keep a central config specifying the uniqueness from client to client
If you can centralize the business rules to one class or group of classes
If you can store the business rules in the database and pull out based on client
If the business rules can all be DB/SQL based (each client having their own DB
Overall hard coding differences based on client name/id is very problematic, keeping different code bases per client is costly (think of the complete testing/retesting time required for the 90% that doesn't change)...I think more info is required to properly answer (give some specifics)
Layer the application. One of those layers contains customizations and should be able to be pulled out at any time without affect on the rest of the system. Application- and DB-level "triggers" (quoted because they may or many not employ actual DB triggers) that call customer-specific code or are parametrized with customer keys) are very helpful.
Core should never be customized, but you must layer it in somewhere, even if it is simplistic web filtering.
What we have is a a core datbase that has the functionality that all clients get. Then each client has a separate database that contains the customizations for that client. This is expensive in terms of maintenance. The other problem is that when two clients ask for a simliar functionality, it is often done differnetly by the two separate teams. There is currently little done to share custiomizations between clients and make common ones become part of the core application. Each client has their own application portal, so we don't have the worry about a change to one client affecting some other client.
Right now we are looking at changing to a process using a rules engine, but there is some concern that the perfomance won't be there for the number of records we need to be able to process. However, in your circumstances, this might be a viable alternative.
I've used some applications that offered the following customizations:
Web pages were configurable - we could drag fields out of view, position them where we wanted with our own name for the field label.
Add our own views or stored procedures and use them in: data grids (along with an update proc) and reports. Each client would need their own database.
Custom mapping of Excel files to import data into system.
Add our own calculated fields.
Ability to run custom scripts on forms during various events.
Identify our own custom fields.
If you clients are larger companies, you're almost going to need your own SDK, API's, etc.