I’m looking for ways to maintain high availability in the case that one of the policy pods is unavailable and found the following information on the official website:
https://istio.io/docs/reference/config/policy-and-telemetry/istio.mixer.v1.config.client/#NetworkFailPolicy
But I did not find any additional information on how to apply these rules in my deployment. Can someone help me with this and tell me how to change these values?
What you´re looking for can be found here, in the istio documentation Destination Rules
Specifically check this link
This istio blog about Using Network Policy with Istio redirects us to Calico documentation.
Network Policy is universal, highly efficient, and isolated from the pods, making it ideal for applying policy in support of network security goals. Furthermore, having policy that operates at different layers of the network stack is a really good thing as it gives each layer specific context without commingling of state and allows separation of responsibility.
This post is based on the three part blog series by Spike Curtis, one of the Istio team members at Tigera. The full series can be found here: https://www.projectcalico.org/using-network-policy-in-concert-with-istio/
Additional links which could be useful:
Calico Network Policy
Kubernetes Network Policy
Related
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 1 year ago.
Improve this question
I'm wondering about the best practices for architecting my Kubernetes clusters.
For 1 environment (e.g. production), what organisation should I have in my clusters?
Examples: 1 cluster per technology stack, 1 cluster per exposure area (internet, private...), 1 cluster with everything ... ?
Thanks for your help
I'm not a Kubernetes expert, so I'll give you some generic guidance to help until someone who knows more weighs in.
By technology stack - no. That wouldn't provide any value I can think of.
By 'exposure' - yes. If one cluster is compromised the damage will hopefully be limited to that cluster only.
By solution - yes.
Solution vs Technology Stack
"Solution" is where you have a number of systems that exist to addresses a specific business problem or domain. This could be functional e.g. finance vs CRM vs HR.
Technology stacks in the literal sense is not likely to be relevant. True, it's not uncommon for different solutions & systems to be comprised of different technology (is that what you were meaning?) - but that's usually a by-product, not the primary driver.
Let's say you have two major solutions (e.g. the finance and CRM). It's likely that you will have situations that impacts one but shouldn't impact the other.
Planned functional changes: e.g. rolling out a major release. Object Orientated programmers and architects have had this nailed for years through designing systems that are cohesive but loosely-coupled (see: Difference Between Cohesion and Coupling), and through stuff like the Stable Dependencies Principle. Having both solutions dependent on the same cluster makes them coupled in that respect, which.
Planned infrastructure changes: e.g. patching, maintenance, resource reallocation, etc.
Unplanned changes: e.g. un-planned outage, security breaches.
Conclusion
Look at what will be running on the cluster(s), and what solutions they are part of, and consider separation along those lines.
The final answer might be a combination of both, some sort of balance between security concerns and solution (i.e. change) boundaries.
The best way would be is to have 1 kubernetes cluster and have the worker nodes in private subnets. You can choose to have the control plane in a public subnet with restricted access like your VPN cidr etc.
If you have multiple teams or application stacks, I'd suggest having different namespaces for each stack as this creates the logical separation of resources.
Also, check the resource limits and quotas that you can apply on kubernetes to prevent over consumption of the resources.
And, as you mentioned multiple application stacks, I am assuming you would have multiple services being exposed for each application or something similar. I would highly recommend using a ingress controller (nginx or anything) to work as single point of entry for each application. You can have more than 1 application listening to 1 load balancer.
Also, have prometheus or ELK monitoring in place as they are great with monitoring k8s components and metrics.
And, I would highly recommend using a tool kubecost and kubebench for enhancing your k8s cluster.
Kubecost is for cost analytics and reporting for k8s components and kubebench would audit your cluster against CIS standards and give you a report on what improvements are required and where.
Please note that the above recommendations are based on best practises and cost efficiency.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 2 years ago.
Improve this question
I am totally new to this two technologies (I know docker and kubernetes btw).
Haven't find much an the web about this comparison topic.
I have read that Openshift is used by more companies,but a nightmare to install,pricier and on upgrade data loss can occur.
But nothing else.
What should be the deciding factor for which one to use for kubernete cluster orchestration?
I currently work for Rancher. I've also been building Internet infrastructure since 1996 and owned an MSP for 14 years that built and managed Internet datacenters for large US media companies. I've been working with containers since 2014, and since then I've tried pretty much everything that exists for managing containers and Kubernetes.
"The deciding factor" varies by individual and organization. Many companies use OpenShift. Many companies use Rancher. Many companies use something else, and everyone will defend their solution because it fits their needs, or because of the psychological principle of consistency, which states that because we chose to walk a certain path, that path must be correct. More specifically, the parameters around the solution we chose must be what we need because that was the choice we made.
Red Hat's approach to Kubernetes management comes from OpenShift being a PaaS before it was ever a Kubernetes solution. By virtue of being a PaaS, it is opinionated, which means it's going to be prescriptive about what you can do and how you can do it. For many people, this is a great solution -- they avoid the "analysis paralysis" that comes from having too many choices available to them.
Rancher's approach to Kubernetes management comes from a desire to integrate cloud native tooling into a modular platform that still lets you choose what to do. Much like Kubernetes itself, it doesn't tell you how to do it, but rather gives fast access to the tooling to do whatever you want to do.
Red Hat's approach is to create large K8s clusters and manage them independently.
Rancher's approach is to unify thousands of clusters into a single management control plane.
Because Rancher is designed for multi-cluster management, it applies global configuration where it benefits the operator (such as authentication and identity management) but keeps tight controls on individual clusters and namespaces within them.
Within the security boundaries Rancher gives developers access to clusters and namespaces, easy app deployment, monitoring and metrics, service mesh, and access to Kubernetes features without having to go and learn all about Kubernetes first.
But wait! Doesn't OpenShift give developers those things too?
Yes, but often with Red Hat-branded solutions that are modified versions of open source software. Rancher always deploys unadulterated versions of upstream software and adds management value to it from the outside.
The skills you learn using software with Rancher will transfer to using that same software anywhere else. That's not always the case with skills you learn while using OpenShift.
There are a lot of things in Kubernetes that are onerous to configure, independent of the value of using the thing itself. It's easy to spend more time fussing around with Kubernetes than you do using it, and Rancher wants to narrow that gap without compromising your freedom of choice.
What is it that you want to do, not only now, but in the future? You say that you already know Kubernetes, but something has you seeking a management solution for your K8s clusters. What are your criteria for success?
No one can tell you what you need to be successful. Not me, not Red Hat, not Rancher.
I chose to use Rancher and to work there because I believe that they are empowering developers and operators to hit the ground running with Kubernetes. Everything that Rancher produces is free and open source, and although they're a business, the vast majority of Rancher deployments make no money for Rancher.
This forces Rancher to create a product that has true value, not a product that they can convince other people to buy.
The proof is in the deployments - Red Hat has roughly 1,000 OpenShift customers, which means roughly 1,000 OpenShift deployments. Rancher has fewer paying customers than Red Hat, but Rancher has over 30,000 deployments that we know about.
You can be up and running with Rancher in under ten minutes, and you can import the clusters you already have and start working with them a few minutes later. Why not just take it for a spin and see if you like it?
I also invite you to join the Rancher Users slack. There you will not only find a community of Rancher users, but you will be able to find other people who compared Rancher and OpenShift and chose Rancher. They will be happy to help you with information that will lead you to feel confident about whatever choice you make.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 1 year ago.
Improve this question
Not sure if this is the right place, please point me to a different forum if not.
In a multi-cluster kubernetes setup, is cross-cluster communication a valid design? In particular, a pod in one cluster relying on a pod in another cluster.
Or are there limitations or anti-patterns associated with this that we should avoid?
If not, what tools do you use to manage this deployment and monitor load on each cluster?
Multicluster deployments give you a greater degree of isolation and availability but increase complexity. If your systems have high availability requirements, you likely need clusters across multiple zones and regions. You can canary configuration changes or new binary releases in a single cluster, where the configuration changes only affect a small amount of user traffic. Additionally, if a cluster has a problem, you can temporarily route traffic to nearby clusters until you address the issue.
Multiple meshes afford the following capabilities beyond that of a single mesh:
Organizational boundaries: lines of business
Service name or namespace reuse: multiple distinct uses of the default namespace
Stronger isolation: isolating test workloads from production workloads
I have found a very good youtube videos from KubeCon, check it out because it really explains how multi-cluster works, specially the first one with Matt Turner.
https://www.youtube.com/watch?v=FiMSr-fOFKU
https://www.youtube.com/watch?v=-zsThiLvYos
Check out Admiral which provides automatic configuration and service discovery for multicluster Istio service mesh
Istio has a very robust set of multi-cluster capabilities. Managing this configuration across multiple clusters at scale is challenging. Admiral takes an opinionated view on this configuration and provides automatic provisioning and syncing across clusters. This removes the complexity from developers and mesh operators pushing this complexity into automation.
In a multi-cluster kubernetes setup, is cross-cluster communication a valid design? In particular, a pod in one cluster relying on a pod in another cluster.
Based on provided links and my knowledge everything should work fine, pod can rely on a pod in another cluster.
More useful links:
https://istio.io/docs/ops/deployment/deployment-models/#multiple-clusters
https://banzaicloud.com/blog/istio-multicluster-federation-2/
https://github.com/istio-ecosystem/coddiwomple
https://github.com/istio-ecosystem/multi-mesh-examples
EDIT
how do the different frameworks of Kubefed and Admiral fit with each other? Can we use both or only use one?
I would not use kubefed since it's in alpha as far as i know, unless you really need it. I dont know how both of them would work together, I can only assume that they should both work.
what considerations should we have in deciding between different mesh architecture to facilitate cross-cluster communication?
Above, there is a link to youtube video, istio Multi-Cluster Service Mesh Patterns Explained, I would say it's up to you to decide which one you want to use based on your needs, the simplest one is the first described in the video, single control plane, single network. More about it there.
I have an application with 5 microservices (iam, courses...). I want to know which is the best approach to migrate them to kubernetes. I was thinking to create namespaces by enviroment as google recomendes:
1. prod
2. dev
3. staging
then I thought that may be better create namespace by environment and microservices.
1. iam-prod
2. iam-dev
3. iam-staging
1. courses-prod
2. courses-dev
3. courses-staging
...
but this approach can be a little bit difficult to handle. Because I need to communicate between each other.
Which approach do you think that is better?
Just like the other answer, you should create namespace isolation for prod, dev and staging. This will ensure a couple of nuances are taken care of...
Ideally, your pods in either of the environments should not be talking across environments
You can manage your network policies in a much cleaner and manageable way with this organization of k8s kinds
You can run multiple microservices on the same namespace. So, I would go with prod, dev and staging namespaces where you can have one or multiple instances of each micro-service.
yet, If you want to use separate namespaces for separate microservices environments, they still can communicate using service. The DNS URL will be, SERVICE_NAME.NAMESPACE.SVC.
ref: https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/
If you go with second approach you will create unnecessary complexity without achieving any benefit. Also think of situation if your micro-services grows ,are you going to create new cluster for each one.This is not at all recommended.
Concept of Namespace should not be linked to applications but it is related to users.Refer k8 doc as below
"Namespaces are intended for use in environments with many users spread
across multiple teams, or projects. For clusters with a few to tens of users,
you should not need to create or think about namespaces at all. Start using namespaces when you need the features they provide."
Also even if first approach is recommended, please have separate cluster for prod as this should be more secure and highly available with proper disaster recovery plan ready and tested.
Go with one name space for each environment. You can also define resource quota per names paces. That way each application environment can be independently managed
None of the above are ideal solutions. I’ll go over why.
Security
Namespaces are the easiest boundary to use for managing RBAC permissions. In general, you will want to use the pre-provisioned admin and editor cluster roles to constrain access for users to use namespaces. This means people and services that share namespaces also share visibility of secrets. So the namespace becomes the blast radius for compromising secrets.
In order to reduce the blast radius of secrets exposure you can either micromanage resource level role binding (which is unreasonable overhead without additional automation and tooling) or segregate services across namespaces so that only tightly couple services share a namespace.
Isolation
Kubernetes resource isolation is relatively poor between namespaces. There’s no way to force a namespace to deploy into a different node pool than another namespace without custom admission controllers. So resource isolation is effectively opt-in, which is both insecure and unenforceable.
Because of this, is actually more secure and better resource isolated to have different environments (dev, staging, prod) in seperate K8s clusters all together. But this is obviously more expensive and more management overhead. So it’s only cost effective when you have many services and enough resource usage to justify the added overhead.
The consequence of poor resource isolation is that your dev and staging workloads can effectively DOS your prod workloads simply by using shared resources. CPU/memory/disk are the obvious culprits. These can be enforced by custom admission controllers. But the more insidious problem is sharing ingress proxies, load balancer, and networking, which is harder to isolate between namespaces.
Another consequence of poor isolation is that dev services with poor security can be compromised, allowing horizontal access to prod services. Realistically, no one deploys dev apps as production ready and secure. So without hard isolation, your security is at risk too.
Quotas
Quotas are managed at the namespace level. So if you want to isolate quota by environment AND team, you can’t use namespaces for both. And if you want to have quota by project, you’d need a project per namespace. The only way to handle all three is with multiple clusters, multiple namespaces, and multiple node pools with custom deployment/admission enforcement of that creates a makeshift hierarchy or matrix.
Namespace Hierarchy
Namespaces are flat. If you use them for env you can’t use them for org or team level access control. If you use them for team level access control your engineers can use them for component/project/system level abstraction boundaries. You can only choose one or the chaos will be unmanageable.
Conclusion
Unfortunately, the namespace abstraction is being used for 3 or 4 use cases in the Kubernetes community, and it’s the not really ideal for any of them. So either you pick an non-ideal use case to optimize for or you manage multiple clusters and write a bunch of custom automation to handle all the use cases.
How exactly it helps if recommended labels from kubernetes 1.12 are added in helm charts?
Since this question (as revealed in the comments) is about the application-related recommended labels that are prefixed with app.kubernetes.io, the appropriate place to look is the kubernetes documentation on this. These labels serve to identify various kubernetes objects (Pods, Services, ConfigMaps etc.) as part of a single application. Having a "common set of labels allows tools to work interoperably, describing objects in a common manner that all tools can understand". The idea is that you should be able to go into tools like the kubernetes dashboard or a monitoring tool and see a list of applications and then drill into the individual objects under the applications. However, 1.12 was only released a month ago so it will take time for the common labels to be adopted and for tools to offer support for querying based on them. Having the labels present in helm charts is a step towards adoption.