I am trying to simplify our firestore security rules and their deployment in different environments (prod / dev / test). I came across the recently added https://firebase.google.com/docs/reference/admin/node/admin.securityRules.html which looked like it would be very helpful to me, as I can programmatically create and deploy rules. However the concept of Ruleset and RulesFile is very confusing.
In particular, what does it mean to have multiple Rulesets deployed? I never knew this, but when I run admin.securityRules().listRulesetMetadata(), I see a list of over 100 such rulesets. Is it to support rollbacks? If so, how could one do that? Is the latest deployed Ruleset the one which is always enforced and it overrides everything from past? What is RulesFile then? Is there a case where there isn't a one to one mapping between RulesFile and Ruleset?
Some clarifications on these would be very helpful in determining if this is the correct solution for me.
Each time you change your security rules, you're indeed creating a new Ruleset. The latest deployed Ruleset is active.
So when you do a rollback to an older set of rules, Firebase reads that Ruleset and redeploys it.
Currently there is only one RulesFile per Ruleset at least as far as the main API surfaces go (CLI, Console). This may (and very likely will) change in the future, to allow more common powerful conventions (think include files, and standard libraries) to also be applied to security rules.
Related
I'm facing an OSGi problem, and I'm not sufficiently well versed in OSGi details to figure out a way forward.
My problem is this:
I have a service, which lives behind a well-defined interface, and periodically emits a file in a particular location. This is controlled by the config admin (via a config file in Karaf)
Some components provide this service to others via a Karaf feature file, bundling my service in a particular version (1.X.0)
Other components provide this service in a newer version (1.Y.0, where Y > X), either via another feature file, or just by adding it to their kar file.
As these are just minor version changes, the consuming services don't really care which service they talk to (the API is the same).
My problem is that both of these bundles are Active in karaf, and there is a race condition as to who gets to overwrite who's output file.
I tried making the #Component into a Singleton (using scope = ServiceScope.SINGLETON), and while this might solve the case of every service consumer using the same implementation, the issue of file overwriting persists, as both services are Active.
Basically, I'm looking for a way to tell OSGi to "don't bother with the older versions, the new version (which is the same major as the others) are fine for all of the consumers (who use the default of [1.X,2[)
As the config file is akin to an "API" for enabling my service I would like to avoid having multiple config files for the different versions.
If possible, I would like to keep the version location logic outside of my service. I guess in theory, the service could listen for other versions of bundles providing the same service interface, and take appropriate action - but this seems very cumbersome to me. Surely there is a better way, which has less impact on the business logic code (i.e. my service)?
The simple answer is of course, why bother with the old bundle? Just uninstall it?
Anyway, the usual answer is then: I can't for some reason. Some solutions in preferred (my) order:
Remove older bundle
Make your components require a configuration and configure the appropriate component, the other one won't run. This is basically the pattern that gave us the Configurator specification. This is actually a really good solution that I use everywhere. It allows the application to be configured in fine detail.
Just solve the configuration file conflict in the bundles.
Use startlevels to never start the older bundle. A bit of a hack.
Register a service property with the service and let the references filter on that property. Rabbit hole.
Use Service Hooks to filter out the old service. This introduces ordering since the service hook must be registered before anyone using it. So I tend to shy away from it. Here is an implementation
This is imho a typical use case that, in retrospect, made the system much more complicated than expected. One hack like this does not seem to be too bad but these hacks tend to multiply like rabbits. OSGi is about clean modules communicating with well defined services. Your description seems you're there but then not correctly solving the problem will lead you down the path to the big ball of mud again :-(
For Apache Karaf there is a special way to implement the first solution from Peter (Remove older bundle).
Set dependency=true in the feature file for the bundle that provides the service.
This way Apache Karaf will automatically install the best bundle for the requirements of your other bundles. In this case it should only install the providing bundle with the highest minor version number.
I'm testing out Azure Service Fabric and started adding a lot of actors and services to the same project - is this okay to do or will I lose any of service fabric features as fail overs, scaleability etc?
My preference here is clearly 1 actor/1 service = 1 project. The big win with a platform like this is that it allows you to write proper microservice-oriented applications at close to no cost, at least compared to the implementation overhead you have when doing similar implementations on other, somewhat similar platforms.
I think it defies the point of an architecture like this to build services or actors that span multiple concerns. It makes sense (to me at least) to use these imaginary constraints to force you to keep the area of responsibility of these services as small as possible - and rather depend on/call other services in order to provide functionality outside of the responsibility of the project you are currently implementing.
In regards to scaling, it seems you'll still be able to scale your services/actors independently even though they are a part of the same project - at least that's implied by looking at the application manifest format. What you will not be able to do, though, are independent updates of services/actors within your project. As an example; if your project has two different actors, and you make a change to one of them, you will still need to deploy an update to both of them since they are part of the same code package and will share a version number.
I've been reading up on versioning strategies for ReST APIs, and something none of them appear to address is how you manage the underlying codebase.
Let's say we're making a bunch of breaking changes to an API - for example, changing our Customer resource so that it returns separate forename and surname fields instead of a single name field. (For this example, I'll use the URL versioning solution since it's easy to understand the concepts involved, but the question is equally applicable to content negotiation or custom HTTP headers)
We now have an endpoint at http://api.mycompany.com/v1/customers/{id}, and another incompatible endpoint at http://api.mycompany.com/v2/customers/{id}. We are still releasing bugfixes and security updates to the v1 API, but new feature development is now all focusing on v2. How do we write, test and deploy changes to our API server? I can see at least two solutions:
Use a source control branch/tag for the v1 codebase. v1 and v2 are developed, and deployed independently, with revision control merges used as necessary to apply the same bugfix to both versions - similar to how you'd manage codebases for native apps when developing a major new version whilst still supporting the previous version.
Make the codebase itself aware of the API versions, so you end up with a single codebase that includes both the v1 customer representation and the v2 customer representation. Treat versioning as part of your solution architecture instead of a deployment issue - probably using some combination of namespaces and routing to make sure requests are handled by the correct version.
The obvious advantage of the branch model is that it's trivial to delete old API versions - just stop deploying the appropriate branch/tag - but if you're running several versions, you could end up with a really convoluted branch structure and deployment pipeline. The "unified codebase" model avoids this problem, but (I think?) would make it much harder to remove deprecated resources and endpoints from the codebase when they're no longer required. I know this is probably subjective since there's unlikely to be a simple correct answer, but I'm curious to understand how organisations who maintain complex APIs across multiple versions are solving this problem.
I've used both of the strategies you mention. Of those two, I favor the second approach, being simpler, in use cases that support it. That is, if the versioning needs are simple, then go with a simpler software design:
A low number of changes, low complexity changes, or low frequency change schedule
Changes that are largely orthogonal to the rest of the codebase: the public API can exist peacefully with the rest of the stack without requiring "excessive" (for whatever definition of of that term you choose to adopt) branching in code
I did not find it overly difficult to remove deprecated versions using this model:
Good test coverage meant that ripping out a retired API and the associated backing code ensured no (well, minimal) regressions
Good naming strategy (API-versioned package names, or somewhat uglier, API versions in method names) made it easy to locate the relevant code
Cross-cutting concerns are harder; modifications to core backend systems to support multiple APIs have to be very carefully weighed. At some point, the cost of versioning backend (See comment on "excessive" above) outweighs the benefit of a single codebase.
The first approach is certainly simpler from the standpoint of reducing conflict between co-existing versions, but the overhead of maintaining separate systems tended to outweigh the benefit of reducing version conflict. That said, it was dead simple to stand up a new public API stack and start iterating on a separate API branch. Of course, generational loss set in almost immediately, and the branches turned into a mess of merges, merge conflict resolutions, and other such fun.
A third approach is at the architectural layer: adopt a variant of the Facade pattern, and abstract your APIs into public facing, versioned layers that talks to the appropriate Facade instance, which in turn talks to the backend via its own set of APIs. Your Facade (I used an Adapter in my previous project) becomes its own package, self-contained and testable, and allows you to migrate frontend APIs independently of the backend, and of each other.
This will work if your API versions tend to expose the same kinds of resources, but with different structural representations, as in your fullname/forename/surname example. It gets slightly harder if they start relying on different backend computations, as in, "My backend service has returned incorrectly calculated compound interest that has been exposed in public API v1. Our customers have already patched this incorrect behavior. Therefore, I cannot update that computation in the backend and have it apply until v2. Therefore we now need to fork our interest calculation code." Luckily, those tend to be infrequent: practically speaking, consumers of RESTful APIs favor accurate resource representations over bug-for-bug backwards compatibility, even amongst non-breaking changes on a theoretically idempotent GETted resource.
I'll be interested to hear your eventual decision.
For me the second approach is better. I have use it for the SOAP web services and plan to use it for REST also.
As you write, the codebase should be version aware, but a compatibility layer can be used as separate layer. In your example, the codebase can produce resource representation (JSON or XML) with first and last name, but the compatibility layer will change it to have only name instead.
The codebase should implement only the latest version, lets say v3. The compatibility layer should convert the requests and response between the newest version v3 and the supported versions e.g v1 and v2.
The compatibility layer can have a separate adapters for each supported version which can be connected as chain.
For example:
Client v1 request: v1 adapt to v2 ---> v2 adapt to v3 ----> codebase
Client v2 request: v1 adapt to v2 (skip) ---> v2 adapt to v3 ----> codebase
For the response the adapters function simply in the opposite direction. If you are using Java EE, you can you the servlet filter chain as adapter chain for example.
Removing one version is easy, delete the corresponding adapter and the test code.
Branching seems much better for me, and i used this approach in my case.
Yes as you already mentioned - backporting bug fixes will require some effort, but at the same time supporting multiple versions under one source base (with routing and all other stuff) will require you if not less, but at least same effort, making system more complicated and monstrous with different branches of logic inside (at some point of versioning you definetely will come to huge case() pointing to version modules having code duplicated, or having even worse if(version == 2) then...) .
Also dont forget that for regression purposes you still have to keep tests branched.
Regarding versioning policy: i would keep as max -2 versions from current, deprecating support for old ones - that would give some motivation for users to move.
Usually, introduction of a major version of API leading you in a situation of having to maintain multiple versions is an event which does not (or should not) occur very frequently. However, it cannot be avoided completely. I think it is overall a safe assumption that a major version, once introduced, would stay latest version for relatively long period of time. Based on this, I would prefer to achieve simplicity in the code at the expense of duplication as it gives me better confidence of not breaking previous version when I introduce changes in latest one.
Introduction
Drools Guvnor has it's own versioning system, that in production use allows the users of an application to modify the rules and decision tables in order to adapt to change in their business. Yet, the same assets continue to live on the development version control system, where new features to the app are developed.
This post is for looking insight/ideas/experience on rule development and deployment when working with Drools rules and Guvnor.
Below are some key concepts I've been puzzling about.
Deployment to Guvnor
First of all, what is the best way to deploy the drl files and decision tables to production environment? Just simply put them on a zip package and then unzip to Web-Dav folder? What I have navigated around Drools, I haven't found a way to import more than one file at a time. The fact model can be added as a jar archive, though. Guvnor seems to have a REST API of some sort, but using that would require custom deployment scripts.
Change management
Secondly, once the application is in production, the users will likely want to change the values in decision tables in order to set the discount percentages to higher for premium clients etc. This is all fine and dandy, until comes the time to start development of version 2.0 of the app.
Now what we have at this point is
drl files and decision tables in version controlling system
drl files and decision tables in production environment with user modifications, versioned by Guvnor
Now we are in the point of getting the rules and decision tables back from the Guvnor. And again is the Web-Dav folder the best for this, what other options there are?
Merge tools today can even handle Excel file diffs, but sounds like a merge hell to me on a big scale projects.
Keeping the fact model backwards compatible
Yet another topic is fact model integrity. For the assumed version 2.0, developers always want to make refactoring and tear the whole fact model upside down. Still, it must remain backwards compatible with the previous versions as there may be user modified rules that depend on that. Any tips on this? Just keep the fact model simple and clean? Plan ahead / suggest what the users could want to change?
Summary
I'm certain I'm not the first, and surely not the last, to consider options on deployment and change management with Drools and Guvnor. So, what I'd like to hear is comment, discussion, tips etc. on some best (and also the worst in order to avoid them) practices to handle these situations.
Thanks.
The best way to do things depends very much on your specific application and the environment you work in. However the following are pointers from my own experience. Note that I'll add just a few points for now. I'll probably come back to this answer when things come to me.
After the initial go-live, keep releases incremental and small
For your first release you have the opportunity to try things out. Take advantage of this opportunity, and do as much refactoring as possible, because...
Your application has gone live and your business users are maintaining rules in decision tables. You have made great gains in what folks in the industry like to call "business agility". Unfortunately, this tends to be at the expense of application development agility. All of your guided editor rules and decision table rules are tied to your fact model, so any changes existing properties of that fact model will break your decision tables. Unlike in most IDEs these days, you can't just right-click on a fact's getX() method, rename it, and expect all code which relies on that property to be updated.
Decision tables and guided rules are hard to refactor. If a fact has been renamed, then in many (all?) versions of Guvnor, that rule/table will no longer open. You need to get at the underlying XML file via WebDav and do some text searching and replacing. That may be very difficult, considering that to do this, you need to download the file from production to a test environment, make changes, test them, deploy them to a test environment. When you're happy with your changes you need to push them back up to the 'production' Guvnor. Unfortunately, while you were doing that, the users have updated a number of decision tables and you need to either start again, or re-apply the past couple of days' changes. In an ideal world, your business users will agree to make no rule changes for a period of time. But the only way to make that feasible is to keep the changes small so that you can make them in a couple of hours or a day depending on how generous they feel.
To mitigate this:
Keep facts used within Guvnor separate from your application domain classes. Your application developers can then refactor the internal application model to their hearts content, but such changes will not affect the business model. Maintain mappings between the two and ensure there are tests covering those mappings.
Avoid changes such as renaming facts or their properties. Make sure that facts you create and their properties have names which suit the domain and agree these with the business. On the other hand, adding a new property is relatively painless. It is well worth prompting the users to give you an eye on their future plans.
Keep facts as simple as possible. Don't go more complex than name-value pairs unless you really need to. For one thing, your business users will find it much easier to maintain rules. Your priority with anything managed within Guvnor should always be about making it easy for business users to maintain.
Keep external dependencies out of your facts. A developer may think it's a good idea to annotate a fact as a JPA #Entity, for easy persistence. Unfortunately, that adds dependencies which need to be added to Guvnor's classpath, requiring a restart.
Tips & tricks
My personal technique for making cross-environment changes is to connect Eclipse to two Guvnor WebDav directories, and checkout the rules into local directories, where each local directory maps to an environment. I then use the Eclipse diff tooling.
When building a Guvnor-managed knowledge base, I create a separate Maven project containing only the facts, and with no dependencies on anything else. It makes it a lot easier to keep them clean this way. Also, when I really do need to add a dependency (i.e. I use JodaTime where possible), then the build can have a step to generate a shaded JAR containing all the dependencies. That way you only ever deploy one JAR to Guvnor, which is guaranteed to contain the correct versions of your dependencies.
I'm sure there will be more that I think of. I'll try to remember to come back to this...
I have decided to finally nail down my team's deployment processes, soup-to-nuts. The last remaining pain point for us is managing database and runtime data migration/management. Here are two examples, though many exist:
If releasing a new "Upload" feature, automatically create upload directory and configure permisions. In later releases, verify existence/permissions - forever, automatically.
If a value in the database (let's say an Account Status of "Signup") is no longer valid, automatically migrate data in database to proper values, given some set of business rules.
I am interested in implementing a framework that allows developers to manage and deploy these changes with the same ease that we manage and deploy our code.
So the first question is: 1. What tools/frameworks are out there that provide this capacity?
In general, this seems to be an issue in any given language and platform. In my specific case, I am deploying a .NET MVC2 application which uses Fluent NHibernate for database abstraction. I already have in my deployment process a tool which triggers NHibernate's SchemaUpdate - which is awesome.
What I have built up to address this issue in my own way, is a tool that will scan target assemblies for classes which inherit from a certain abstract class (Deployment). That abstract class exposes hooks which you can override and implement your own arbitrary deployment code - in the context of your application's codebase. the Deployment class also provides for a versioning mechanism and the tool manages the current "deployment version" of a given running app. Then, a custom NAnt task glues this together with the NAnt deployment script, triggering the hooks at the appropriate times.
This seems to work well, and does meet my goals - but here's my beef, and leads to my second question: 2. Surely what I just wrote has to already exist. If so, can you point me to it? and 3. Has anyone started down this path and have insight into problems with this approach?
Lastly, if something like this exists, but not on the .NET platform, please still let me know - as I would be more interested in porting a known solution than starting from zero on my own solution.
Thanks everyone, I really appreciate your feedback!
Each major release, have a script to create the environment with the exact requirements you need.
For minor releases, have a script that is split into the various releases and incrementally alters the environment. There are some big benefits to this
You can look at the changes to the environment over time by reading the script and matching it with release notes and change logs.
You can create a brand new environment by running the latest major and then latest minor scripts.
You can create a brand new environment of a previous version (perhaps for testing purposes) by specifying it to stop at a certain minor release.