Currently we could create configmaps, deployment, pods, jobs, namespaces, ingress.
https://github.com/grafana/xk6-kubernetes
But I would like to deploy custom resources. Is it possible?
But I would like to deploy custom resources. Is it possible?
Creation of CR is possible. If definitions are maintained in yaml files for Custom resources, then CR can be applied or created using kubectl tool.
Usually Custom resources are associated with CRD's or API aggregators. Please refer kubernetes documentation for more details.
Adding custom resources
Kubernetes provides two ways to add custom resources to your cluster:
CRDs are simple and can be created without any programming.
API Aggregation requires programming, but allows more control over API behaviors like how data is stored and conversion between API versions.
This is not presently possible. Even when the recently introduced generic interface there are some issues regarding how Kubernetes API handles CRDs. Please refer to this open issue for tracking the progress of this requirement.
Related
I need to automate the provisioning of a complex application in Kubernetes. It's a complex, multi-step process that involves provisioning of some cluster-wide resources and some app-specific resources. The cluster-wide resources are:
Istio
A few Operators (Cert Manager, Prometheus Operator, Postgres Operator, among others)
Then I want to create an application (let's call it Foo) which leverages Istio and the aforementioned operators. It will create statefulsets, services, Certificates, a Postgres database, Istio gateways, Prometheus PodMonitors, etc.
There will be multiple Foo's created, each configured differently (since the Kubernetes cluster will be used to provide Foo applications as a multi-tenant service).
What's the idiomatic way to do this? I think I should write a Foo controller which assumes that Istio and the other operators (prometheus, cert-manager, postgres, etc) already exist.
Is it possible to write a meta ClusterOfFoos operator that installs Istio, installs the required operators, and then installs the Foo controller?
If so, how does one go about provisioning operators (normally installed through Helm) from within a controller?
So far I have looked into using helm to do this, but there are too many dependencies and Helm just tends to create all resources at once, which makes some things fail (eg. when a deployment refers to a Secret that hasn't yet been created by cert-manager).
The Operator Lifecycle Manager is really well suited for the task.
When you create operator Foo, you can package it in the OLM way by creating a bundle which contains the ClusterServiceVersion needed to inform OLM of dependencies that need to be resolved before install and during upgrades. These can just be a list of APIs you need - and OLM will find and install the set of latest versions of the operators that own each API.
All your dependencies are operators available in the Operatorhub.io Catalog so they are available for install and dependency resolution as soon as you install OLM.
You can also configure certain dependencies by including these objects in the bundle itself. According to the docs, the following objects are supported as of the time of this post:
Secret
ClusterRole
ClusterRoleBinding
ConfigMap
ServiceAccount
Service
Role
RoleBinding
PrometheusRule
ServiceMonitor
PodDisruptionBudget
PriorityClasse
VerticalPodAutoscaler
ConsoleYAMLSample
ConsoleQuickStart
ConsoleCLIDownload
ConsoleLink
The Operator SDK can help you with bootstrapping the bundle.
By using GitOps workflow you can automate complex applications in Kubernetes.
You need to define cluster-wide resources and application specific resources in a YAML file.
By using GitOps tools you can continuously deploy kubernetes resources and they will automatically deploy the changes in the cluster.
Use Helm chart to install Istio and make sure dependencies in the Helm chart are created in order.
You can create a custom controller by FOO where it can read configuration of YAML files.
Use kubernetes CRDs to define configuration of each FOO; they will allow you to create custom resources which are specific for each application.
By using Helm; it will read the configuration from the CRD and generate correct YAML values.
The above described approach will allow you to create multiple FOO applications with different configurations and ensure that the required resources are installed in correct order.
You can check this article from codefresh regarding GitOps Workflow and official kubernetes page.
You can also check Working with Multiple Applications and Environments and how Argo CD is useful for this scenario.
As mentioned here:
Harness takes the artifacts and Kubernetes manifests you provide
and deploys them to the target Kubernetes cluster. You can simply
deploy Kubernetes objects via manifests and you can provide manifests
using remote sources and Helm charts.
Is harness tool equipped with kubectl client tool to perform kubectl apply on kubernetes manifests?
If you're curious about the implementation details of the specific tool that are not explained in its official documentation, you should study directly its source code to find the answer.
But answering your specific question:
Is harness tool equipped with kubectl client tool to perform kubectl
apply on kubernetes manifest?
Well, it doesn't have to. Writing a tool which in its code uses a console kubectl client isn't very optimal and doesn't make much sense. For performing exactly the same actions that kubectl does, such tools use Client Libraries. As you can see in the official docs, there is large variety of them, some of them are officialy supported, others are community-maintained, but altogether they support various programming languages.
Of course, you can write an external tool which doesn't use client libraries but implements the API calls and request/response types on its own.
I’m currently using kubernetes and I came across of helm.
Let’s say I don’t like the idea of “infecting” my kubernetes cluster with a process that is not related to my applications but I would gladly accept it if it could be beneficial.
So I made some researches but I still can’t find anything I can’t easily do by using my yaml descriptor and kubectl so for now I can’t find an use except,maybe, for the environizing.
For example (taking it from guides I read:
you can easily install application, eg. helm install nginx —> I add an nginx image to my deployment descriptor, done
repositories -> I have docker ones (where I pull my images from)
you can easily helm rollback in case of release failure-> I just change the image version to the previous one in my kubernetes descriptor, easy
What bothers me is that, at level of commands, I do pretty much the same effort (helm update->kubectl apply).
In exchange for that I have a lot of boilerplate because of keeping the directory structure helm wants and I feel like missing the control I have with plain deployment descriptors ...what am I missing?
It is totally understandable your question. For small and simple deploys the benefits is not actually that great. But when the deploy of something is very complex Helm helps a lot.
Think that you have a couple squads that develop microservice for some company. If you can make a Chart that works for most of them, the deploy of each microservices would differ only by the image and the resources required. This way you get an standardized deployment and easier to all developers.
Another use case is deploying applications which requires a lot of moving parts. For example, if you want to deploy a Grafana server on Kubernetes you're probably going to need at least a Deployment and a Configmap, then you would need a service that matches this deployment. And if you want to expose it to the internet you need an ingress too.
One relatively simple application, would require 4 different YAMLs that you would to manually configure and make sure everything is correct instead you could do a simple helm install and reuse the configuration that someone has already made, sometimes even the company who created the Application.
There are a lot of other use cases, but these two are the ones that I would say are the most common.
Here's three suggestions of ways Helm can be useful:
Your continuous deployment system somewhat routinely produces new builds and wants to send them to the Kubernetes cluster. You can use templating to specify the image name and tag in a deployment, and so helm upgrade ... --set tag=201907211931 to request a specific tag.
You might have various service-specific controls like the log level or external database hostnames. The Helm values mechanism gives a uniform way to specify them, without having to know the details of the Kubernetes YAML files.
There is a repository of pre-packaged application charts, so if you want replicated PostgreSQL with in-cluster persistent storage, that's already built for you and you can just depend on it, rather than figuring out the right combination of StatefulSets and PersistentVolumeClaims yourself.
You can combine these in interesting (and potentially complex) ways: use an in-cluster database for developer testing but use a cloud-hosted and backed-up database for production, for example, and compute the database host name based on what combination of settings are provided.
There are, of course, alternative ways to do all of these things. Kustomize in particular can change the image value fairly straightforwardly, and is notable for having been included in the kubectl tool since Kubernetes 1.14 (see also Declarative Management of Kubernetes Objects Using Kustomize in the Kubernetes documentation). The "operator" pattern gives an alternate path to install software in your cluster, but even more so than Helm you're trusting an arbitrary program with API access.
What is the difference and benefit of the Operator Lifecycle Manager (OLM) vs Helm?
OLM - https://github.com/operator-framework/operator-lifecycle-manager
Helm - https://helm.sh/
I understand that Helm is a general purpose package manager for Kubernetes where as OLM is specific to operators. But, Helm can be used to deploy operators. So, how is OLM different/better than Helm for operators?
Helm provides the ability to install applications onto Kubernetes via Helm Charts, which themselves are a collection of templatized K8s manifests. It handles only the basic lifecycle of these applications (install/delete/rollback/upgrade) by rendering these templates and feeding them to the K8s API server. Based on the version of Helm, there are limitations related to dependency management and which resources can be created in which namespaces.
OLM (Operator Lifecycle Manager), as the previous user mentioned, is a declarative based system which is meant to support installation of Operators, which themselves are responsible for providing the logic and instructions to manage the lifecycle of an application (install/create/delete/upgrade). OLM is an opinionated approach to managing the lifecycle and packaging of these Operators. There is also an SDK which help users create Operators from Helm/Ansible/Go to fit into this system. It has various components which talk to each other through the K8s APIServer, heavily leveraging CRDs and custom resources to make this all happen.
Benefits/differences:
Both can be used to install/delete/rollback/upgrade an Operator, but OLM offers a model whereby you can craft various methods of deployment operations for your application deployment (think alpha vs stable) into different subscribable "channels". As you update these the methods in these "channels", those that are subscribed automatically gain the ability to upgrade/install a newer version according to these methods. Dependencies in OLM are also handled in a different way, and you can have a chain of dependent Operators installed, in order, in various namespaces. Helm is a bit more restricted in this regard.
Lastly OLM make the assumption that your container images are publicly reachable and their use in manifests are built into containers (CatalogSource, Operators, etc), whereas Helm charts much more easily modifiable using various Helm based CLI commands (or 3rd party tools) to override template values before creation.
Well, Helm cannot deploy itself. It only provides primitives for Helm Charts, which you can install when your infrastructure is set up accordingly. In order to deploy anything you need some sort of pipeline that puts all the pieces together.
OLM is a declarative approach to solve some kind of release management where you define different versions of "deployables," which are then upgraded. I have yet to understand how this can be used with custom services. As far as I was digging some time ago, you could only use some predefined applications. Also note that OLM does not replace Helm. I would assume whatever "deployable" OLM manages can also be something that is installed via Helm at the end of the day.
I have made some custom changes to k8s.io/kubernetes/kubernetes, specifically in the pkg/controllers to support some functionality of a project i’m working on. What is the canonical way to deploy & test these changes? Is there like an optional in kubectl to use a custom image of kubernetes? Thanks.
There is a ton of documentation. Try: https://github.com/kubernetes/community/blob/master/contributors/devel/testing.md
kubectl just talks to what it hopes is an API service. kubectl and the various server bits are versioned independently, and there are documented expectations for cross-version interoperability.