I have 15 pods that run different PHP applications in a Kubernetes cluster, each application must include a PHP library updated regularly, it's always the same lib/version for all pods.
What is the best approach to share a library with all pods?
I tried to find out a solution but my doubts persist.
What I thought:
Include the lib in the application container during the build, this solution create a consistent application but I have to re-deploy all 15 applications. Make sense?
Include the lib as shared volume among all pods, in this case I can update only the lib and all application will be updated. It seems better even if I don't like this option because each application depends on the shared volume.
What are your thoughts regarding this issue?
Always make your images be self-contained. Don't build an image that can't run unless something outside the container space is present, or try to use a volume to share code. (That is, pick your first option over the second.)
Say you have your shared library, and you have your 15 applications A through O that depend on it. Right now they're running last week's build image: service-a:20210929 and so on. Service A needs a new feature with support in the shared library, so you update that and redeploy it and announce the feature to your users. But then you discover that the implementation in the shared library causes a crash in service B on some specific customer requests. What now?
If you've built each service as a standalone image, this is easy. helm rollback service-b, or otherwise change service B back to the 20210929 tag while service A is still using the updated 20211006 tag. You're running mixed versions, but that's probably okay, and you can fix the library and redeploy everything tomorrow, and your system as a whole is still online.
But if you're trying to share the library, you're stuck. The new version breaks a service, so you need to roll it back; but rolling back the library would require you to also know to roll back the service that depends on the newer version, and so on. You'd have to undeploy your new feature even though the thing that's broken is only indirectly related to it.
If the library is really used everywhere, and if disk space is actually your limiting factor, you could restructure your image setup to have three layers:
The language interpreter itself;
FROM language-interpreter, plus the shared library;
FROM shared-library:20211006, plus this service's application code.
If they're identical, the lower layers can be shared between images, and the image pull mechanics know to not pull layers that are already present on the system. However, this can lead to a more complex build setup that might be a little trickier to replicate in developer environments.
Have you considered making a new microservice that has that shared library and the other 15 pods send requests to that one microservice to get what they need?
If this architecture works you would only have to update one deployment when that shared library is updated.
in our company, we have lots of teams using an common dict library. since we generate this library from tools, and make sure generated code is ok to go, what we do in vm env is to push this library to all servers, all no one need to worry about lib version issue.
we are moving to k8s, and require module/svc owner to keep up with library version changing by deploy new image. this changes work mode. last week, we found a bug caused by one module forget to update dict library :)
so, we are thinking moving some of configuration-like library to deployed by configuration center, and it can publish configurations dynamically.
Related
I'm facing an OSGi problem, and I'm not sufficiently well versed in OSGi details to figure out a way forward.
My problem is this:
I have a service, which lives behind a well-defined interface, and periodically emits a file in a particular location. This is controlled by the config admin (via a config file in Karaf)
Some components provide this service to others via a Karaf feature file, bundling my service in a particular version (1.X.0)
Other components provide this service in a newer version (1.Y.0, where Y > X), either via another feature file, or just by adding it to their kar file.
As these are just minor version changes, the consuming services don't really care which service they talk to (the API is the same).
My problem is that both of these bundles are Active in karaf, and there is a race condition as to who gets to overwrite who's output file.
I tried making the #Component into a Singleton (using scope = ServiceScope.SINGLETON), and while this might solve the case of every service consumer using the same implementation, the issue of file overwriting persists, as both services are Active.
Basically, I'm looking for a way to tell OSGi to "don't bother with the older versions, the new version (which is the same major as the others) are fine for all of the consumers (who use the default of [1.X,2[)
As the config file is akin to an "API" for enabling my service I would like to avoid having multiple config files for the different versions.
If possible, I would like to keep the version location logic outside of my service. I guess in theory, the service could listen for other versions of bundles providing the same service interface, and take appropriate action - but this seems very cumbersome to me. Surely there is a better way, which has less impact on the business logic code (i.e. my service)?
The simple answer is of course, why bother with the old bundle? Just uninstall it?
Anyway, the usual answer is then: I can't for some reason. Some solutions in preferred (my) order:
Remove older bundle
Make your components require a configuration and configure the appropriate component, the other one won't run. This is basically the pattern that gave us the Configurator specification. This is actually a really good solution that I use everywhere. It allows the application to be configured in fine detail.
Just solve the configuration file conflict in the bundles.
Use startlevels to never start the older bundle. A bit of a hack.
Register a service property with the service and let the references filter on that property. Rabbit hole.
Use Service Hooks to filter out the old service. This introduces ordering since the service hook must be registered before anyone using it. So I tend to shy away from it. Here is an implementation
This is imho a typical use case that, in retrospect, made the system much more complicated than expected. One hack like this does not seem to be too bad but these hacks tend to multiply like rabbits. OSGi is about clean modules communicating with well defined services. Your description seems you're there but then not correctly solving the problem will lead you down the path to the big ball of mud again :-(
For Apache Karaf there is a special way to implement the first solution from Peter (Remove older bundle).
Set dependency=true in the feature file for the bundle that provides the service.
This way Apache Karaf will automatically install the best bundle for the requirements of your other bundles. In this case it should only install the providing bundle with the highest minor version number.
First, I'm not sure this question is specific enough for Stack Overflow. Happy to remove or revise if someone has any suggestions.
We use Kubernetes to orchestrate our server side code, and have recently begun using Kustomize to modularize the code.
Most of our backend services fit nicely into that data model. For our main transactional system we have a base configuration that we overlay with tweaks for our development, staging, and different production flavors. This works really well and has helped us clean things up a ton.
We also use TensorFlow Serving to deploy machine learning models, each of which is trained and at this point deployed for each of our many clients. The only way that these configurations differ is in the name and metadata annotations (e.g., we might have one called classifier-acme and another one called classifier-bigcorp), and the bundle of weights that are pulled from our blob storage (e.g., one would pull from storage://models/acme/classifier and another would pull from storage://models/bigcorp/classifier). We also assign different namespaces to segregate between development, production, etc.
From what I understand of the Kustomize system, we would need to have a different base and set of overlays for every one of our customers if we wanted to encode the entire state of our current cluster in Kustomize files. This seems like a huge number of directories as we have many customers. If we have 100 customers and five different elopement environments, that's 500 directories with a kustomize.yml file.
Is there a tool or technique to encode this repeating with Kustomize? Or is there another tool that will work to help us generate Kubernetes configurations in a more systematic and compact way?
You can have more complex overlay structures than just a straight matrix approach. So like for one app have apps/foo-base and then apps/foo-dev and apps/foo-prod which both have ../foo-base in their bases and then those in turn are pulled in by the overlays/us-prod and overlays/eu-prod and whatnot.
But if every combo of customer and environment really does need its own setting then you might indeed end up with a lot of overlays.
I'm testing out Azure Service Fabric and started adding a lot of actors and services to the same project - is this okay to do or will I lose any of service fabric features as fail overs, scaleability etc?
My preference here is clearly 1 actor/1 service = 1 project. The big win with a platform like this is that it allows you to write proper microservice-oriented applications at close to no cost, at least compared to the implementation overhead you have when doing similar implementations on other, somewhat similar platforms.
I think it defies the point of an architecture like this to build services or actors that span multiple concerns. It makes sense (to me at least) to use these imaginary constraints to force you to keep the area of responsibility of these services as small as possible - and rather depend on/call other services in order to provide functionality outside of the responsibility of the project you are currently implementing.
In regards to scaling, it seems you'll still be able to scale your services/actors independently even though they are a part of the same project - at least that's implied by looking at the application manifest format. What you will not be able to do, though, are independent updates of services/actors within your project. As an example; if your project has two different actors, and you make a change to one of them, you will still need to deploy an update to both of them since they are part of the same code package and will share a version number.
I am pretty comfortable with the producing web apps now. I am using a NodeJs stack on the back-end and usually have a fair amount of Javascript on the front end. Where I really lack understanding is the deployment process.
What is a typical deployment process?
From what I have gathered in my reading a deployment/build process can include several tasks:
Running through unit-test suites
Concatenating script and CSS files
Version numbering your app
Tracing module dependencies (node_modules)
Pushing it to a remote repo (GitHub)
Instructing 'staging' servers to pull down the latest repo
Instructing 'production' server to pull down the latest repo
This has all left me a little overwhelmed. I don't know whether I should be going into this level of detail for my own projects, it seems like a lot of work! I am using Sublime Text 2 IDE and it seems to have a Build Script process, is this suitable? How does one coordinate all these separate tasks? I'm imagining ideally they would all run at the flick of a switch.
Sorry so many questions, but I need to know how people learnt similar principles. Some of my requirements may be specific to NodeJS but I'm sure processes are similar no matter what choice of stack you are developing in.
First off, let's split the job in two: front-end and back-end stuff. For both, you really want some kind of bulid system, but their goals and scope are vastly different.
For the front-end, you want your source to be as small as possible; concatenate/minify JavaScript, CSS and images. A colleague of mine has written a "compiler", Assetgraph, to do this for you. It has a somewhat seep learning-curve, but it does wonders for your code (our dev-builds are usually ~20 megs, production ~500 k).
As to the back-end, you want contained, easily managed bundles of some sort. We re-package our stuff into debian-packages. As long as the makefile is wired up correctly, you get a lot of the boring build- and deploy-time stuff for free. Here's my (pre-NPM 1.0) Debianizing node programs. I've seen other ways to do this in NPM and on Github, but I haven't looked into them, so I can't speak on their quality.
For testing/pusing around/deploying, we use a rather convoluted combination of Debian package-archives, git-hooks, Jenkins-servers and what not. While I highly recommend using the platforms' native package-manager for rolling out stuff, it can be a bit too much. All in all, we usually deploy staging either automatically (on each git push), or semi-automatic for unstable codebases. Production deployments are always done explicitly.
For the assets I use asereje https://github.com/masylum/asereje
I recently documented my nodejs deployment process in a blog post:
http://pau.calepin.co/how-to-deploy-a-nodejs-application-with-monit-nginx-and-bouncy.html
A build script sounds like a good idea indeed.
What should that build script do?
make sure all the test pass, else exit immediately
concatenate your javascript and css files into one single js/css file and minify them also
increment the version number (unless you decide to set that up yourself manually)
push to the remote repo (which instructs the staging and production servers through a git hook)
This is at least my opinion.
Other resources:
http://howtonode.org/deploying-node-with-spark
http://howtonode.org/deploying-node-upstart-monit
http://dailyjs.com/2010/03/15/hosting-nodejs-apps/
How to deploy node app depencies
I have decided to finally nail down my team's deployment processes, soup-to-nuts. The last remaining pain point for us is managing database and runtime data migration/management. Here are two examples, though many exist:
If releasing a new "Upload" feature, automatically create upload directory and configure permisions. In later releases, verify existence/permissions - forever, automatically.
If a value in the database (let's say an Account Status of "Signup") is no longer valid, automatically migrate data in database to proper values, given some set of business rules.
I am interested in implementing a framework that allows developers to manage and deploy these changes with the same ease that we manage and deploy our code.
So the first question is: 1. What tools/frameworks are out there that provide this capacity?
In general, this seems to be an issue in any given language and platform. In my specific case, I am deploying a .NET MVC2 application which uses Fluent NHibernate for database abstraction. I already have in my deployment process a tool which triggers NHibernate's SchemaUpdate - which is awesome.
What I have built up to address this issue in my own way, is a tool that will scan target assemblies for classes which inherit from a certain abstract class (Deployment). That abstract class exposes hooks which you can override and implement your own arbitrary deployment code - in the context of your application's codebase. the Deployment class also provides for a versioning mechanism and the tool manages the current "deployment version" of a given running app. Then, a custom NAnt task glues this together with the NAnt deployment script, triggering the hooks at the appropriate times.
This seems to work well, and does meet my goals - but here's my beef, and leads to my second question: 2. Surely what I just wrote has to already exist. If so, can you point me to it? and 3. Has anyone started down this path and have insight into problems with this approach?
Lastly, if something like this exists, but not on the .NET platform, please still let me know - as I would be more interested in porting a known solution than starting from zero on my own solution.
Thanks everyone, I really appreciate your feedback!
Each major release, have a script to create the environment with the exact requirements you need.
For minor releases, have a script that is split into the various releases and incrementally alters the environment. There are some big benefits to this
You can look at the changes to the environment over time by reading the script and matching it with release notes and change logs.
You can create a brand new environment by running the latest major and then latest minor scripts.
You can create a brand new environment of a previous version (perhaps for testing purposes) by specifying it to stop at a certain minor release.