I have decided to finally nail down my team's deployment processes, soup-to-nuts. The last remaining pain point for us is managing database and runtime data migration/management. Here are two examples, though many exist:
If releasing a new "Upload" feature, automatically create upload directory and configure permisions. In later releases, verify existence/permissions - forever, automatically.
If a value in the database (let's say an Account Status of "Signup") is no longer valid, automatically migrate data in database to proper values, given some set of business rules.
I am interested in implementing a framework that allows developers to manage and deploy these changes with the same ease that we manage and deploy our code.
So the first question is: 1. What tools/frameworks are out there that provide this capacity?
In general, this seems to be an issue in any given language and platform. In my specific case, I am deploying a .NET MVC2 application which uses Fluent NHibernate for database abstraction. I already have in my deployment process a tool which triggers NHibernate's SchemaUpdate - which is awesome.
What I have built up to address this issue in my own way, is a tool that will scan target assemblies for classes which inherit from a certain abstract class (Deployment). That abstract class exposes hooks which you can override and implement your own arbitrary deployment code - in the context of your application's codebase. the Deployment class also provides for a versioning mechanism and the tool manages the current "deployment version" of a given running app. Then, a custom NAnt task glues this together with the NAnt deployment script, triggering the hooks at the appropriate times.
This seems to work well, and does meet my goals - but here's my beef, and leads to my second question: 2. Surely what I just wrote has to already exist. If so, can you point me to it? and 3. Has anyone started down this path and have insight into problems with this approach?
Lastly, if something like this exists, but not on the .NET platform, please still let me know - as I would be more interested in porting a known solution than starting from zero on my own solution.
Thanks everyone, I really appreciate your feedback!
Each major release, have a script to create the environment with the exact requirements you need.
For minor releases, have a script that is split into the various releases and incrementally alters the environment. There are some big benefits to this
You can look at the changes to the environment over time by reading the script and matching it with release notes and change logs.
You can create a brand new environment by running the latest major and then latest minor scripts.
You can create a brand new environment of a previous version (perhaps for testing purposes) by specifying it to stop at a certain minor release.
Related
I'm facing an OSGi problem, and I'm not sufficiently well versed in OSGi details to figure out a way forward.
My problem is this:
I have a service, which lives behind a well-defined interface, and periodically emits a file in a particular location. This is controlled by the config admin (via a config file in Karaf)
Some components provide this service to others via a Karaf feature file, bundling my service in a particular version (1.X.0)
Other components provide this service in a newer version (1.Y.0, where Y > X), either via another feature file, or just by adding it to their kar file.
As these are just minor version changes, the consuming services don't really care which service they talk to (the API is the same).
My problem is that both of these bundles are Active in karaf, and there is a race condition as to who gets to overwrite who's output file.
I tried making the #Component into a Singleton (using scope = ServiceScope.SINGLETON), and while this might solve the case of every service consumer using the same implementation, the issue of file overwriting persists, as both services are Active.
Basically, I'm looking for a way to tell OSGi to "don't bother with the older versions, the new version (which is the same major as the others) are fine for all of the consumers (who use the default of [1.X,2[)
As the config file is akin to an "API" for enabling my service I would like to avoid having multiple config files for the different versions.
If possible, I would like to keep the version location logic outside of my service. I guess in theory, the service could listen for other versions of bundles providing the same service interface, and take appropriate action - but this seems very cumbersome to me. Surely there is a better way, which has less impact on the business logic code (i.e. my service)?
The simple answer is of course, why bother with the old bundle? Just uninstall it?
Anyway, the usual answer is then: I can't for some reason. Some solutions in preferred (my) order:
Remove older bundle
Make your components require a configuration and configure the appropriate component, the other one won't run. This is basically the pattern that gave us the Configurator specification. This is actually a really good solution that I use everywhere. It allows the application to be configured in fine detail.
Just solve the configuration file conflict in the bundles.
Use startlevels to never start the older bundle. A bit of a hack.
Register a service property with the service and let the references filter on that property. Rabbit hole.
Use Service Hooks to filter out the old service. This introduces ordering since the service hook must be registered before anyone using it. So I tend to shy away from it. Here is an implementation
This is imho a typical use case that, in retrospect, made the system much more complicated than expected. One hack like this does not seem to be too bad but these hacks tend to multiply like rabbits. OSGi is about clean modules communicating with well defined services. Your description seems you're there but then not correctly solving the problem will lead you down the path to the big ball of mud again :-(
For Apache Karaf there is a special way to implement the first solution from Peter (Remove older bundle).
Set dependency=true in the feature file for the bundle that provides the service.
This way Apache Karaf will automatically install the best bundle for the requirements of your other bundles. In this case it should only install the providing bundle with the highest minor version number.
Imagine there is an application consisting from bunch of microservices. All of these microservices can be developed/deployed completely independently from each other. Each microservice can be "described" with several attributes - e.g. current API version, release version, commit hash etc. Along with that, there are several environments used in development process - e.g. Testing environment (often called Sandbox), Staging environment, Pre-Release environment and obviously Production environment.
Is there a convenient tool/way/approach to track, basically, what attribute is currently deployed to which environment? For instance, get a quick access to information like "what is the current version of Restful API at Pre-Release environment"? Or more complex one - "what was this version two month ago"? And of course see the "global picture" as well?
Theres no ready to use solution on the market yet according to my knowledge.
Some teams are using git ops https://www.twistlock.com/2018/08/06/gitops-101-gitops-use/ to get ahead of the chaos challenge a lot of different micro services usually ship with.
Another technology in a somewhat different, yet related direction are micro service meshes, istio https://istio.io/ being one of them.
There are also test approaches like contract testing or heavy integration tests, that are more expensive, but also provide more confidence.
we are a team of 5 people working with microstrategy. We share every role, but we have no worklfow.
Everybody may build or change attributes and change the schema. This leads often to reports not working. Furthermore, there is no "good" documentation. We tried to establish a documentation with sharepoint, but there we also had no workflow.
Originally, we had an old project where for every report all the attributes where constructed newly. So we did not reuse any existing schema object.
Hence, we started a new project. We realized that due to lack of understanding and lack of workflow we make and made a lot of mistakes. We feel that we understand things better slowly (parent child), but the workflow is still horrible.
We have a development project and a lice project, but with the way we are working now, we have a lot of problems. Particularly, the missing version control system is killing us. We perform changes and forget what we did. Hence, we have to use backups, destroying useful work on a given day
So what are best practices to:
* deploy new attributes, facts and reports
* ensure that old reports work after constructing new attributes and facts
* improve documentation
* attributes defined on fact tables and parent-child relationships
Any help is appreciated
MicroStrategy development in a team environment, deploying from development to live, can be very challenging. As you rightly point out, the lack of version control, and unknown interdependencies between objects can cause untold problems. There's no one right answer to this question, but I would suggest the following:
Use all the tools provided by MicroStrategy. When you're deploying from one project to another, don't just drag and drop in Object Manager, create a package. When you deploy that package, make sure you choose to create an undo package, so you can rollback changes if you encounter any problems.
On that note, try to catch these problems in advance. Running Integrity Manager before and after a deployment, even if it's just to generate SQL for the reports, will point out if you've broken anything. On that note:
Create a third environment/project. Call this test/release control, whatever you prefer. Here you can test packages created in Object Manager, to ensure that they have the desired effect, and don't break anything. In effect, this is a dry run for your deployment to live. This environment should be regularly refreshed from live (via project duplication), to make sure it doesn't get in an unexpected state (as the result of a broken Object Manager package import for example).
Over and above that, I can only offer organisational advice. It's not uncommon for one person to take responsibility for schema objects (i.e. facts, attributes, transformations) so that developers don't undo each other's changes. If you have a large project, these objects could be split into functional areas, and individuals assigned.
Documentation is always tricky, but I like to put as much as possible into the object descriptions. This has the advantage of being visible in the Web interface (via tooltips), and included in the automated project documentation, should you choose to generate that. There is obviously the change log functionality for each object, but in my experience, those logs are soon not completed by developers, as saving happens too frequently. Still, if you can get people to populate that, you'd have a head start on understanding the change in your project.
To summarise:
Use Object Manager packages to deploy changes
Test changes with Integrity Manager, to catch any issues as early as possible
Use a release control project/environment, so you're not catching issues in your production environment
Assign responsibility for schema objects to a specific person or persons where possible.
I recently joined a project where Workflow Foundation 4.0 is being used to model business processes.
We have a designer tool so that consultants for clients can customise the workflow definitions. We also persist the workflow instance along with the definition. The workflows can be long running (e.g. months or potentially years).
My question is how do we manage backward compatibility for each release given we don't necessarily know what customisations have been made and what legacy workflows are still in flight? We are loading from XAML but even seemingly minor changes to workflow definitions prevent them from loading. Migration scripts was my initial thought but it seems non-trivial given the complexity of WF workflows.
First off, XOML is 3.0; WF4 uses straight up XAML.
There are two options for doing this. It depends on whether you need to upgrade a long-running workflow in process, or if you want to update the workflow and use it for all new instances, while keeping current instances running on the previous version. Lets call these two options the upgrade and the multiversion strategies.
Re multiversion:
I'm doing this currently. Essentially, you must isolate every different version of the same workflow within an AppDomain. Deserializing from xaml or creating a new instance of a type is the same thing--they both result in an assembly being loaded into the current AppDomain. If v1 of the workflow is defined in assembly A.1, and v2 of the workflow is defined in assembly A.2, you can experience binding issues if you aren't careful. Isolating each version within its own AppDomain helps reduce the chance of this happening.
Re upgrade:
This currently isn't being supported, but there are plans on including this in a (near) future release. Ron Jacobs gave a presentation at PDC10 last October detailing WF4 futures. Three things (I can remember) were mentioned in the presentation--Metadata errors breaking the build, the state machine, and providing an upgrade path for workflows during execution. I can tell you that the state machine was released in the recent Platform Update, and I've been told the metadata-error-breaks-the-build feature is coming soon as well. I'd assume that the upgrade path feature will be coming soon as well.
I've done some research on this.
WF4 Workflow Versioning Spike
WF4 Versioning Spike: Planning for Change
Distributed System Versioning Myth #1
As an alternative to this question what is the best way to manage custom versions of software for a particular client?
Most of the differences between client releases is changes to the user interface to customize the software to look like it is owned by the client. More often then not, this is a simple logo change. Occasionaly the color scheme will change as well. But there are occassions where features will be enabled or disabled based on the client. What is the best way to keep all of these releases up to date and make them easily available to the users of a particular client?
At this point we have five different clients and each has their own build of the software and their own installer (complete with their logo in the installer). This is becoming a pain to manage, and it will only get worse as more and more clients start using our software.
So assuming the linked question isn't the way to go, what is the best way to manage these releases?
"This is becoming a pain to manage,
and it will only get worse as more and
more clients start using our
software."
The only way to really solve this part is to make sure your software has an architecture that supports the customizations as a layer on top of your core product. Ideally, they would simply be runtime configuration options (colours, logos and enable/disable are perfect for this). If you get (or want) a very large number of clients, push the configurability right into the core product and enable the clients to do the customizing themselves.
When updating, you build (and test) the core product once, and build the custom versions by simply linking to (referencing) the already-built core product as a library. You could have each client in a separate build, or you could have a single build process that generates updates for all the currently-maintained client builds (the latter is probably better for a larger number of clients). If you have a "vanilla" version, you could build that as part of the core or along with the client versions, it would depend on your particular case.
Depending on your technology, it might be possible to have the customization layer be built independently from the core product. In this case, client rebuilds will rarely be necessary (perhaps only for certain significant changes) -- you can simply link to the updated core product at runtime. For deployment to the client, you would only need to deploy the updated core, if you have a deployment method that supports this.
It's hard to say more detail without knowing your platform, but you've graciously kept this question agnostic.
Separate the common and the custom parts in your source tree. This can eliminate the great majority of merges, depending on your testing and release policies. There is always a way to abstract out and customize a build resource, even if your build process has to invoke script to rewrite some file.
Judicious use of branching in a good Source Code Management system. These aren't called "Configuration Management" systems for nothing. They are the tools optimized for precisely this task, so you're probably not going to get anything better without building on top of one.
Subversion is good at setting up multiple branches, but last I checked it could only do the 3-way merge on a single file at a time. Perforce is awesome in this department because it tracks your branching history on a file-by-file basis and uses this to automate merging of whole sets of changes between whole branches. It really is the cat's pajamas here.
Git and darcs may have similar power, but I haven't used them. They seem to be based on the idea that each working checkout tree will have a copy all changes ever since the beginning of time. That sounds impractical to me since I need to keep some large and changing SDKs under SCM control.
Arguably, the only difference needs to be a branch of whatever release you have, with the customer-unique changes in it. So for example:
/project
/trunk
/branches
/release-v1
/customer-branches
/release-v1-google
/release-v1-microsoft
....
Where your client releases are branches of your customer releases. Since these branches won't ever get rolled up into trunk or another release branch, don't pollute the regular development /branches tree.
I guess it depends on the level of customization you need for each client, but could you make your base app "customize" itself based on a config file?, so you could have the trunk being the full app and then have special config files for each customer that control the look & feel, as well as features that are enabled, and your release process would take the correct config files to deploy in your installers.
Seems that if you are always delivering custom(ish) versions of your app for each client, it would be worth the time to extend the app in this way.
I set up a header file called branding.h which contains a bunch of #ifdefs like the following example to change whatever bits need changing. In Visual Studio, it is easy to set up multiple builds with the appropriate client symbol defined.
#if defined BRAND_CLIENT1
# define COMPANY_NAME "Client 1"
# define PRODUCT_NAME "The Client 1 Widget App"
# define LOGO_FILE "res/logoClient1.ico"
#elif defined BRAND_CLIENT2
# define COMPANY_NAME "Client 2"
# define PRODUCT_NAME "The Client 2 Super Widget App"
# define ENABLE_EXTRA_MENU
# define LOGO_FILE "res/logoClient2.ico"
#endif
This is all assuming C++ of course.