I've only got a small amount of experience with A/B testing; but from what I've seen it seems like the standard approach to do an A/B test is to introduce some conditional logic in an application's code. This can be tricky to implement properly (depending on the complexity of the test) and requires extra work both for setup and cleanup.
It got me wondering: are there any frameworks or approaches to A/B testing that simplify matters using, e.g., Git branches? I'm envisioning something at the load balancer level, which directs half of traffic to a server where "master" or "default" has been deployed, and the other half to a server with "experiment" deployed. This way the code itself could always be completely agnostic of any A/B tests going on; and presumably the act of choosing either A or B for full deployment would be a simple flip of a switch.
I'm sure this would not be trivial to set up properly. But still I wonder if it's possible, and if in fact it's already been done.
it's relatively easy to build and definitely doable. you need to implement a deployment system where you deploy all branches starting with "ab_* for an example into different folders on your servers. then at some point in your code you can decide which folder should be included in the actual user session based on your actual test. it's not really a 'framework', it's a simple architecture design pattern you have to add to your own system, i was doing the same in production before.
Related
I'm testing out Azure Service Fabric and started adding a lot of actors and services to the same project - is this okay to do or will I lose any of service fabric features as fail overs, scaleability etc?
My preference here is clearly 1 actor/1 service = 1 project. The big win with a platform like this is that it allows you to write proper microservice-oriented applications at close to no cost, at least compared to the implementation overhead you have when doing similar implementations on other, somewhat similar platforms.
I think it defies the point of an architecture like this to build services or actors that span multiple concerns. It makes sense (to me at least) to use these imaginary constraints to force you to keep the area of responsibility of these services as small as possible - and rather depend on/call other services in order to provide functionality outside of the responsibility of the project you are currently implementing.
In regards to scaling, it seems you'll still be able to scale your services/actors independently even though they are a part of the same project - at least that's implied by looking at the application manifest format. What you will not be able to do, though, are independent updates of services/actors within your project. As an example; if your project has two different actors, and you make a change to one of them, you will still need to deploy an update to both of them since they are part of the same code package and will share a version number.
Introduction
Drools Guvnor has it's own versioning system, that in production use allows the users of an application to modify the rules and decision tables in order to adapt to change in their business. Yet, the same assets continue to live on the development version control system, where new features to the app are developed.
This post is for looking insight/ideas/experience on rule development and deployment when working with Drools rules and Guvnor.
Below are some key concepts I've been puzzling about.
Deployment to Guvnor
First of all, what is the best way to deploy the drl files and decision tables to production environment? Just simply put them on a zip package and then unzip to Web-Dav folder? What I have navigated around Drools, I haven't found a way to import more than one file at a time. The fact model can be added as a jar archive, though. Guvnor seems to have a REST API of some sort, but using that would require custom deployment scripts.
Change management
Secondly, once the application is in production, the users will likely want to change the values in decision tables in order to set the discount percentages to higher for premium clients etc. This is all fine and dandy, until comes the time to start development of version 2.0 of the app.
Now what we have at this point is
drl files and decision tables in version controlling system
drl files and decision tables in production environment with user modifications, versioned by Guvnor
Now we are in the point of getting the rules and decision tables back from the Guvnor. And again is the Web-Dav folder the best for this, what other options there are?
Merge tools today can even handle Excel file diffs, but sounds like a merge hell to me on a big scale projects.
Keeping the fact model backwards compatible
Yet another topic is fact model integrity. For the assumed version 2.0, developers always want to make refactoring and tear the whole fact model upside down. Still, it must remain backwards compatible with the previous versions as there may be user modified rules that depend on that. Any tips on this? Just keep the fact model simple and clean? Plan ahead / suggest what the users could want to change?
Summary
I'm certain I'm not the first, and surely not the last, to consider options on deployment and change management with Drools and Guvnor. So, what I'd like to hear is comment, discussion, tips etc. on some best (and also the worst in order to avoid them) practices to handle these situations.
Thanks.
The best way to do things depends very much on your specific application and the environment you work in. However the following are pointers from my own experience. Note that I'll add just a few points for now. I'll probably come back to this answer when things come to me.
After the initial go-live, keep releases incremental and small
For your first release you have the opportunity to try things out. Take advantage of this opportunity, and do as much refactoring as possible, because...
Your application has gone live and your business users are maintaining rules in decision tables. You have made great gains in what folks in the industry like to call "business agility". Unfortunately, this tends to be at the expense of application development agility. All of your guided editor rules and decision table rules are tied to your fact model, so any changes existing properties of that fact model will break your decision tables. Unlike in most IDEs these days, you can't just right-click on a fact's getX() method, rename it, and expect all code which relies on that property to be updated.
Decision tables and guided rules are hard to refactor. If a fact has been renamed, then in many (all?) versions of Guvnor, that rule/table will no longer open. You need to get at the underlying XML file via WebDav and do some text searching and replacing. That may be very difficult, considering that to do this, you need to download the file from production to a test environment, make changes, test them, deploy them to a test environment. When you're happy with your changes you need to push them back up to the 'production' Guvnor. Unfortunately, while you were doing that, the users have updated a number of decision tables and you need to either start again, or re-apply the past couple of days' changes. In an ideal world, your business users will agree to make no rule changes for a period of time. But the only way to make that feasible is to keep the changes small so that you can make them in a couple of hours or a day depending on how generous they feel.
To mitigate this:
Keep facts used within Guvnor separate from your application domain classes. Your application developers can then refactor the internal application model to their hearts content, but such changes will not affect the business model. Maintain mappings between the two and ensure there are tests covering those mappings.
Avoid changes such as renaming facts or their properties. Make sure that facts you create and their properties have names which suit the domain and agree these with the business. On the other hand, adding a new property is relatively painless. It is well worth prompting the users to give you an eye on their future plans.
Keep facts as simple as possible. Don't go more complex than name-value pairs unless you really need to. For one thing, your business users will find it much easier to maintain rules. Your priority with anything managed within Guvnor should always be about making it easy for business users to maintain.
Keep external dependencies out of your facts. A developer may think it's a good idea to annotate a fact as a JPA #Entity, for easy persistence. Unfortunately, that adds dependencies which need to be added to Guvnor's classpath, requiring a restart.
Tips & tricks
My personal technique for making cross-environment changes is to connect Eclipse to two Guvnor WebDav directories, and checkout the rules into local directories, where each local directory maps to an environment. I then use the Eclipse diff tooling.
When building a Guvnor-managed knowledge base, I create a separate Maven project containing only the facts, and with no dependencies on anything else. It makes it a lot easier to keep them clean this way. Also, when I really do need to add a dependency (i.e. I use JodaTime where possible), then the build can have a step to generate a shaded JAR containing all the dependencies. That way you only ever deploy one JAR to Guvnor, which is guaranteed to contain the correct versions of your dependencies.
I'm sure there will be more that I think of. I'll try to remember to come back to this...
I am curious as to how other development teams spec out new features. The team I have just moved up to lead has no real specification process. I have just implemented a proper development process with CI, auto deployment and logging all bugs using Trac and I am now moving on to deal with changes.
I have a list of about 20 changes to our product to have done over the next 2 months. Normally I would just spec out each change going into detail of what should be done but I am curious as to how other teams handle this. Any suggestions?
I think we had a successful approach in my last job as we delivered the project on time and with only a couple of issues found in production. However, there were only 3 people working on the product, so I'm not entirely sure how it would scale to larger teams.
We wrote specs upfront for the whole product but without going into too much detail and with an emphasis on the UI. This was a means for us to get a feel for what had to be done and for the scope of the project.
When we started implementing things, we had to work everything out in a lot more detail (and inevitably had to do some things differently from the spec). To that end, we got together and worked out the best approach to implementing each feature (sometimes with prototypes). We didn't update the original spec but we did make notes after the meetings as it's very easy to forget the details afterwards.
So in summary, my approach is to treat specs as an exploratory tool and to work out finer details during implementation. Depending on the project, it may also be a good idea to keep the original spec up to date as the application evolves (which we didn't need to do this time).
Good question but its can be subjective. I guess it depends on the strategy of the product, if its to be deployed to multiple clients in the same way or to a single client on a bespoke project, the impact, dependency these changes have on the system and each other and the priority these changes need to be made.
I would look at the priority and the dependency, that will naturally start grouping things?
I'm sure that this is a common issue that other have solved before so I'm calling on the collective wisdom of other developers/project managers out there for some help.
I've got a number of projects:
Application
WebApp
ServerApp
Dev Utils
ORM
All of the apps/utils depend on the ORM: When the ORM changes, it needs to be compiled and all of the apps need to be re-compiled against it and then deployed. Right now my VCS structure is kind of a jumble:
AppName
Trunk
Application
WebApp
ServerApp
Dev Utils (around 4 folders right now, but growing)
ORM
Relase
ProjectName (be it Application or WebApp) version.number
Branches
ExperimentName_DevName
Ideally, I'd like to have a root folder per application (Application/WebApp/ORM etc.), each with its own Trunk/Branches/Releases etc. to logically and physically separate them. My reasoning is that because lots of work gets done on the Application, and it gets released far more often, each release branch has identical copies of the same utils etc. Also checking out the Trunk to work on it always means that all of the other projects come along for the ride.
However, separating would mean ripping certain projects out of solutions and making modifying any projects simultaneously a pain - jumping between 2-3 IDEs (especially when making changes to the ORM).
I'm thinking about this because very soon I'm putting together a CI machine (be ready for questions about that in a week or so) and I'm trying to figure out a way to have the releases automatically created/deployed. Normally, just the Application gets deployed, via script copy to the server where all of the workstations pull from on startup, but like I said before, if the ORM changes/releases, all of the other apps should re-build and deploy.
(Today I broke our website and 3 utilities cause I changed the ORM and deployed it with an updated version of the Application, but forgot to rebuild/deploy the other apps with the new ORM - oops.)
Put yourself in your developer's shoes. This shouldn't be hard because it sounds like you're one of them :-) Look at your version control layout not from an architectural point of view, but from a usability point of view. Ask yourself: on a daily basis what are the most common things that we as developers do with our source code? Think specifically with regards to your interactions with your VCS system and the project layouts. You want to make the common things brain-dead easy. It's ok for the less common use cases to be harder to get to, as long as you have a record of how to do them, so that when (not if!) people forget, they'll know where to look to remind themselves how.
I've seen a lot of VCS layouts that try to be architecturally "perfect", but end up causing no end of hassles and headaches from a day-to-day usage point of view. I'm not saying the two can't coincide and mesh well together, I'm just saying think about it from the users point of view, and let that be your guide.
From a CI point of view, I'd still take this approach. Even if your setup ends up becoming more complex to define in your CI of choice, you only have to do the setup there once. If the layout is easy to use during development, most of your CI setup should also be relatively easy. You then just focus on the last bits that will take more time.
Splitting up your code base into different "projects" can be very difficult. We have done this at each separately "deployable" boundary. Including a "platform" layer that is common/used by the others, as a separate project. But this isn't really perfect either.
The one thing that I cannot stress enough is that one needs to have some form of continuous regression/testing that runs after checkins and before you actually deploy anything. In addition a "release" process that might even involve some manual testing can be helpful and has definitely prevented a few egg on the face situations. (Better to release it a couple days late then broken.)
(Sorry to not directly address your problem)
Is writing deployment friendly code considered a good virtue on the part of a programmer?
If yes, then what are the general considerations to be kept in mind when coding so that deployment of the same code later does not become a nightmare?
The biggest improvement to deployment is to minimize manual intervention and manual steps. If you have to type in configuration values or manually navigate through configuration screens there will be errors in your deployment.
If your code needs to "call home", make sure that the user understands why, and can turn the functionality off if necessary. This might only be a big deal if you are writing off-the-shelf software to be deployed on corporate networks.
It's also nice to not have your program be dependent on too many environmental things to run properly. To combat this, I like to define a directory structure with my own bin, etc and other folders so that everything can be self-contained.
The whole deployment process should be automated to minimize the human errors. The software should not be affected by the envrionment. Any new deployment should be easily rolled back in case any problem occurs. While coding you should not hard code configuration values which may be different for each environment. Configuration should be done in such a way that it can be easily automated depending upon enviroment.
Client or server?
In general, deployment friendly means that you complete and validate deployment as you complete a small story / unit of work. It comes from continual QA more than style. If you wait till the last minute to build and validate deployment, the cleanest code will never be friendly.
Everything else deployment, desktop or server, follows from early validation. You can add all the whacky dependencies you want, if you solve the delivery of those dependencies early. Some very convenient desktop deployment mechanisms result in a sand-boxed / partially trusted applications. Better to discover early that you can't do something (e.g. write your log to c:\log.txt) than to find out late that your customers can't install.
I'm not entirely sure what you mean by "deployment friendly code." What are you deploying? What do you mean by "deploying"?
If you mean that your code should be transferable between computers, I guess the best things to do would be to minimize unnecessary (with a given definition of "unnecessary") dependencies to external libraries, and document well the libraries that you do depend on.