Imagine I am developing an application microservices based. They will be deployed to kubernetes with Helm Package Manager. Some microservices ends having pretty similar YAML files configuration. Some others might be different in terms of YAML configuration. What is the best practice for this? I have a few options:
Use a generic chart and pass different configuration using values.env.yaml for each microservice and then deploy this with a different release name.
Create a chart for every single microservice no matter of they are similar in terms of configuration?
This is an opinion question, so I'll answer with an opinion.
Upside: You would have to change just a few values in your values.yaml depending on the microservice and it would be easier to maintain your values.yml. Your Helm charts repo may not grow as fast.
Downside: It will be harder to create you _helpers.tpl file for example. That file will grow rapidly and it could get confusing for people creating microservices understand it.
Upside: Separation of your microservice as you scale to hundreds. Developers can work only on their microservice deployment.
Downside: File spread, too many files everywhere, and your Helm charts repo can grow rapidly. Also, a risk of large code duplication.
The more general practice is number 2 for the official Helm charts but then again every chart is for a very different application.
Like #Rico mentioned, this is an opinion question. Here is my opinion:
I think it is a good idea to start with one Chart that fits all. But when you have to add very specific stuff for only a few services with special requirements, you should create another Chart. This idea is pretty similar to Monolith first, when it comes to Microservices.
In my company we have one Chart for ~30 Services. They have very similar needs, therefore the template files aren't too complex and the _helpers file has only around 50 lines. We are very happy with this solution, because you only need a few lines of values.yaml to prepare your service for operation.
Related
First, I'm not sure this question is specific enough for Stack Overflow. Happy to remove or revise if someone has any suggestions.
We use Kubernetes to orchestrate our server side code, and have recently begun using Kustomize to modularize the code.
Most of our backend services fit nicely into that data model. For our main transactional system we have a base configuration that we overlay with tweaks for our development, staging, and different production flavors. This works really well and has helped us clean things up a ton.
We also use TensorFlow Serving to deploy machine learning models, each of which is trained and at this point deployed for each of our many clients. The only way that these configurations differ is in the name and metadata annotations (e.g., we might have one called classifier-acme and another one called classifier-bigcorp), and the bundle of weights that are pulled from our blob storage (e.g., one would pull from storage://models/acme/classifier and another would pull from storage://models/bigcorp/classifier). We also assign different namespaces to segregate between development, production, etc.
From what I understand of the Kustomize system, we would need to have a different base and set of overlays for every one of our customers if we wanted to encode the entire state of our current cluster in Kustomize files. This seems like a huge number of directories as we have many customers. If we have 100 customers and five different elopement environments, that's 500 directories with a kustomize.yml file.
Is there a tool or technique to encode this repeating with Kustomize? Or is there another tool that will work to help us generate Kubernetes configurations in a more systematic and compact way?
You can have more complex overlay structures than just a straight matrix approach. So like for one app have apps/foo-base and then apps/foo-dev and apps/foo-prod which both have ../foo-base in their bases and then those in turn are pulled in by the overlays/us-prod and overlays/eu-prod and whatnot.
But if every combo of customer and environment really does need its own setting then you might indeed end up with a lot of overlays.
Similar to posts like this Separate dev and prod Firebase environment
I'm running into similar structuring issues. Unlike other posts like that that i've found, in my case, it's GCP as a whole rather than just Firebase. In addition, i'm looking at separation (or not) of blue and green deployments ALONG with the various environments.
The projects will be handling IoT data; mobile, field sensor/modules, web (in the future). Currently everything is unfortunately in one project.
So, i'm thinking of having 3 different projects for the staging, production, and test environments with each project having both blue and green deployments per, perhaps besides test but that's a different conversation.
Does GCP as a whole have documentation or recommendations about this? Or do you guys have any recommendations?
It's hard to answer because it depends a lot of your organization, your needs and your way of working.
Here you can find a google document about resources hierarchy.
However, I already see some GCP customers using only one project for dev/uat/prod because they share the same K8S cluster and separate the environment thanks to the namespace. By the way, the cluster maintenance cost is done only once for all the different steps of the project.
About the Blue/Green it depends on which component you want to apply this. If it's on the website, App Engine, or a Global Loadbalancer can do this. If it's about IoT Core or PubSub, I fear that you have to manage this by yourself or to create 2 different projects for this.
I have an application with 5 microservices (iam, courses...). I want to know which is the best approach to migrate them to kubernetes. I was thinking to create namespaces by enviroment as google recomendes:
1. prod
2. dev
3. staging
then I thought that may be better create namespace by environment and microservices.
1. iam-prod
2. iam-dev
3. iam-staging
1. courses-prod
2. courses-dev
3. courses-staging
...
but this approach can be a little bit difficult to handle. Because I need to communicate between each other.
Which approach do you think that is better?
Just like the other answer, you should create namespace isolation for prod, dev and staging. This will ensure a couple of nuances are taken care of...
Ideally, your pods in either of the environments should not be talking across environments
You can manage your network policies in a much cleaner and manageable way with this organization of k8s kinds
You can run multiple microservices on the same namespace. So, I would go with prod, dev and staging namespaces where you can have one or multiple instances of each micro-service.
yet, If you want to use separate namespaces for separate microservices environments, they still can communicate using service. The DNS URL will be, SERVICE_NAME.NAMESPACE.SVC.
ref: https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/
If you go with second approach you will create unnecessary complexity without achieving any benefit. Also think of situation if your micro-services grows ,are you going to create new cluster for each one.This is not at all recommended.
Concept of Namespace should not be linked to applications but it is related to users.Refer k8 doc as below
"Namespaces are intended for use in environments with many users spread
across multiple teams, or projects. For clusters with a few to tens of users,
you should not need to create or think about namespaces at all. Start using namespaces when you need the features they provide."
Also even if first approach is recommended, please have separate cluster for prod as this should be more secure and highly available with proper disaster recovery plan ready and tested.
Go with one name space for each environment. You can also define resource quota per names paces. That way each application environment can be independently managed
None of the above are ideal solutions. I’ll go over why.
Security
Namespaces are the easiest boundary to use for managing RBAC permissions. In general, you will want to use the pre-provisioned admin and editor cluster roles to constrain access for users to use namespaces. This means people and services that share namespaces also share visibility of secrets. So the namespace becomes the blast radius for compromising secrets.
In order to reduce the blast radius of secrets exposure you can either micromanage resource level role binding (which is unreasonable overhead without additional automation and tooling) or segregate services across namespaces so that only tightly couple services share a namespace.
Isolation
Kubernetes resource isolation is relatively poor between namespaces. There’s no way to force a namespace to deploy into a different node pool than another namespace without custom admission controllers. So resource isolation is effectively opt-in, which is both insecure and unenforceable.
Because of this, is actually more secure and better resource isolated to have different environments (dev, staging, prod) in seperate K8s clusters all together. But this is obviously more expensive and more management overhead. So it’s only cost effective when you have many services and enough resource usage to justify the added overhead.
The consequence of poor resource isolation is that your dev and staging workloads can effectively DOS your prod workloads simply by using shared resources. CPU/memory/disk are the obvious culprits. These can be enforced by custom admission controllers. But the more insidious problem is sharing ingress proxies, load balancer, and networking, which is harder to isolate between namespaces.
Another consequence of poor isolation is that dev services with poor security can be compromised, allowing horizontal access to prod services. Realistically, no one deploys dev apps as production ready and secure. So without hard isolation, your security is at risk too.
Quotas
Quotas are managed at the namespace level. So if you want to isolate quota by environment AND team, you can’t use namespaces for both. And if you want to have quota by project, you’d need a project per namespace. The only way to handle all three is with multiple clusters, multiple namespaces, and multiple node pools with custom deployment/admission enforcement of that creates a makeshift hierarchy or matrix.
Namespace Hierarchy
Namespaces are flat. If you use them for env you can’t use them for org or team level access control. If you use them for team level access control your engineers can use them for component/project/system level abstraction boundaries. You can only choose one or the chaos will be unmanageable.
Conclusion
Unfortunately, the namespace abstraction is being used for 3 or 4 use cases in the Kubernetes community, and it’s the not really ideal for any of them. So either you pick an non-ideal use case to optimize for or you manage multiple clusters and write a bunch of custom automation to handle all the use cases.
I'd like to know what are your best practices (or just your practices) regarding the management of your helm charts versions.
I wonder what is the best way to deal with application versioning, continuous integration/delivery and chart packaging.
Today I have many microservices that live their life. Each one has its own lifecycle and it own versioning in its own git repository.
Beside, we choosed to have one git repository for all our charts.
Now, we have too choices :
Each time a microservice changes, a new docker image is built and a new version of the chart is created too (with just the tag(s) of the docker image(s) that change in the value.yaml file)
Or, even if a microservice changes, we don't create a new version of the chart. The default value of the docker tag in the chart is set to "default" and when we want to upgrade the chart we have to use --set image.tag=vx.x.x flag.
The benefit of the first approach for the "ops" point of view, is that at any time we know what version of each chart (and each docker image) are running on the cluster. The drawback is that at a certain time we will have many many versions of each charts with just a docker tag version that changed.
On the other side, the benefit of the second approach is that the only thing that makes the chart version to change is a modification of the chart code itself, not an application change. It reduces drastically the "uselessly" high version numbers of each chart. The drawback is that we have to override the docker tags at the installation/upgrade time and we lost the observability of what versions are running on the cluster (usefull in case of Disaster Recovery Plan).
So, what are your practices? Perhaps an hybrid approach?
Thank you for your help
I think this is a choice that comes down to the needs of your project. An interesting comparison is the current versioning strategy of the public charts in the Kubernetes charts repo and the current default versioning strategy of Jenkins-X.
The public charts only get bumped when a change is made to the chart. This could be to bump the version of the default image tag that it points to but each time it is an explicit action requiring a pr and review and a decision on whether it is a major, minor or fix version bump.
In a Jenkins-X cluster the default behaviour is that when you make a change to the code of one of your microservices then it's chart version is automatically bumped whether or not the chart itself changes. The chart in the source repo refers to a snapshot but it is auto deployed under an explicit version and that version gets referenced in the environments it is deployed to via a pipeline. The chart refers to a draft/dev tag of the image in the source and that's also automatically replaced with an explicit version during the flow.
The key difference I think is that Jenkins-X is driven towards a highly automated CI/CD flow with particular environments in the flows. Its approach makes sense for handling frequent deployment of changes. The public charts are aimed at reusability and giving a stable experience across a hugely wide range of environments and situations through public contributions. So the strategy there is more aimed at visibility and ease of understanding for changes that you'd expect to be less frequent by comparison.
Is it best practice to place monitoring tools like Prometheus and Grafana inside a Kubernetes cluster or outside a Kubernetes cluster?
I can see the case for either. It seems very easy to place it inside the cluster. But that seems less robust.
It seems people do this typically, likely they are running everything in their environment or app under K8S. In a bigger picture view if you have use cases outside of one specific app it likely makes sense to run this on another architecture. The reason why is that Prometheus doesn't support clustering. You can write to two instances, but there is not really an HA plan for this product. To me, that's a major problem for a monitoring technology.
Most organizations who use this tool heavily end up not meeting use cases which APM (transaction tracing and diagnostics) can do. Additionally, you'll need to deploy an ELK/Splunk type stack, so it gets very complex. They also find it difficult to manage and often will look towards a Datadog, SingalFx, Sysdig, or another system which can do more and is fully managed. Naturally, most of what I have said here has cost, so if you do not have a budget then you'll have to spend your time (time=money) doing all the work.