I am trying to find a DB (object storage) for my application. The application is really a wrapper over ISTIO Network Routing API. Basically simplifying the ISTIO configuratin for my network. Kubernetes (k8s) Custom Resource Definition (CRD) seems to fit my requirements. Also love the watch and REST API capability provided by CRD.
DB requirement
Few 100 MBs data - worst case
Ability to watch objects
REST API support for object
Persistence
Around 2k writes/sec and similar/more reads/sec. Although I do have my application acting as a proxy for CRD where things can be cached.
Why using CRD will be a good or a bad idea? Are there any performance implication using CRD. This 2016 stackflow answer suggest that etcd data is not in RAM. Whereas etcd link suggest that etcd can do 10k writes/sec (so even things are not in RAM and purely in disk, who cares).
I am seeing multiple application using k8s CRDs.
Helm uses CRDs to store releases
Istio uses CRDs to store their networking routing API objects
Considering that (CRD page)
A resource is an endpoint in the Kubernetes API that stores a collection of API objects of a certain kind. For example, the built-in pods resource contains a collection of Pod objects.
A custom resource is an extension of the Kubernetes API that is not necessarily available on every Kubernetes cluster. In other words, it represents a customization of a particular Kubernetes installation.
A CRD is for extending Kubernetes itself, it is not for application data.
The helm-crd/examples/mariadb.yaml is about lightweight metadata which will enable Kubernetes to download the right release and through Helm install it.
It is not to store data for a random application which could exist without Kubernetes (as opposed to Helm releases, which make sense only in a Kubernetes deployment scenario)
Similarly, Istio CRD makes sense only in a Kubernetes context:
Kubernetes currently implements the Istio configuration on Custom Resource Definitions (CRDs). These CRDs correspond to namespace-scope and cluster-scope CRDs and automatically inherit access protection via the Kubernetes RBAC.
That approach (using etcd to store any application data) would not scale.
Related
Problem setting. Suppose I have 2 pods, A and B. I want to be able to dynamically scale pod A based on some arbitrary number from some arbitrary source. Suppose that pod B is such a source: for example, it can have an HTTP server with an endpoint which responds with the number of desired replicas of pod A at the moment of request. Or maybe it is an ES server or a SQL DB (does not matter).
Question. What kubernetes objects do I need to define to achieve this (apart from HPA)? What configuration should HPA have to know that it needs to look up B for current metric? How should API of B look like (or is there any constraints?)?
Research I have made. Unfortunately, the official documentation does not say much about it, apart from declaring that there is such a possibility. There are also two repositories, one with some go boilerplate code that I have trouble building and another one that has no usage instructions whatsoever (though allegedly does fulfil the "external metrics over HTTP" requirement).
By having a look at the .yaml configs in those repositories, I have reached a conclusion that apart from Deployment and Service one needs to define an APIService object that registers the external or custom metric in the kubernetes API and links it with a normal service (where you would have your pod) and a handful of ClusterRole and ClusterRoleBinding objects. But there is no explanation about it. Also I could not even list existing APIServices with kubectl in my local cluster (of 1.15 version) like other objects.
The easiest way will be to feed metrics into Prometheus (which is a commonly solved problem), and then setup a Prometheus-based HPA (also a commonly solved problem).
1. Feed own metrics to Prometheus
Start with Prometheus-Operator to get the cluster itself monitored, and get access to ServiceMonitor objects. ServiceMonitors are pointers to services in the cluster. They let your pod's /metrics endpoint be discovered and scraped by a prometheus server.
Write a pod that reads metrics from your 3rd party API and shows them in own /metrics endpoint. This will be the adapter between your API and Prometheus format. There are clients of course: https://github.com/prometheus/client_python#exporting
Write a Service of type ClusterIP that represents your pod.
Write a ServiceMonitor that points to a service.
Query your custom metrics thru Prometheus dashboard to ensure this stage is done.
2. Setup Prometheus-based HPA
Setup Prometheus-Adapter and follow the HPA walkthrough.
Or follow the guide https://github.com/stefanprodan/k8s-prom-hpa
This looks like a huge pile of work to get the HPA. However, only the adapter pod is a custom part here. Everything else is a standard stack setup in most of the clusters, and you will get many other use cases for it anyways.
I recently started to explore k8s extensions and got introduced to two concepts:
CRD.
Service catalogs.
They look pretty similar to me. The only difference to my understanding is, CRDs are deployed inside same cluster to be consumed; whereas, catalogs are deployed to be exposed outside the cluster for example as database service (client can order cluster of mysql which will be accessible from his cluster).
My query here is:
Is my understanding correct? if yes, can there be any other scenario where I would like to create catalog and not CRD.
Yes, your understanding is correct. Taken from official documentation:
Example use case
An application developer wants to use message queuing as part of their application running in a Kubernetes cluster.
However, they do not want to deal with the overhead of setting such a
service up and administering it themselves. Fortunately, there is a
cloud provider that offers message queuing as a managed service
through its service broker.
A cluster operator can setup Service Catalog and use it to communicate
with the cloud provider’s service broker to provision an instance of
the message queuing service and make it available to the application
within the Kubernetes cluster. The application developer therefore
does not need to be concerned with the implementation details or
management of the message queue. The application can simply use it as
a service.
With CRD you are responsible for provisioning resources, running backend logic and so on.
More info can be found in this KubeCon 2018 presentation.
We are working on provisioning our service using Kubernetes and the service needs to register/unregister some data for scaling purposes. Let's say the service handles long-held transactions so when it starts/scales out, it needs to store the starting and ending transaction ids somewhere. When it scales out further, it will need to find the next transaction id and save it with the ending transaction id that is covered. When it scales in, it needs to delete the transaction ids, etc. ETCD seems to make the cut as it is used (by Kubernetes) to store deployment data and not only that it is close to Kubernetes, it is actually inside and maintained by Kubernetes; thus we'd like to find out if that is open for our use. I'd like to ask the question for both EKS, AKS, and self-installed. Any advice welcome. Thanks.
Do not use the kubernetes etcd directly for an application.
Access to read/write data to the kubernetes etcd store is root access to every node in your cluster. Even if you are well versed in etcd v3's role based security model avoid sharing that specific etcd instance so you don't increase your clusters attack surface.
For EKS and GKE, the etcd cluster is hidden in the provided cluster service so you can't break things. I would assume AKS takes a similar approach unless they expose the instances to you that run the management nodes.
If the data is small and not heavily updated, you might be able to reuse the kubernetes etcd store via the kubernetes API. Create a ConfigMap or a custom resource definition for your data and edit it via the easily securable and namespaced functionality in the kubernetes API.
For most application uses run your own etcd cluster (or whatever service) to keep Kubernetes free to do it's workload scheduling. The coreos etcd operator will let you define and create new etcd clusters easily.
What is the difference between Objects and Resouces in Kubernetes world?
I couldn't find it from https://kubernetes.io/docs/concepts/ . I wonder they make no distinction about them but seems they see objects as a high-level concept of resources.
A representation of a specific group+version+kind is an object. For example, a v1 Pod, or an apps/v1 Deployment. Those definitions can exist in manifest files, or be obtained from the apiserver.
A specific URL used to obtain the object is a resource. For example, a list of v1 Pod objects can be obtained from the /api/v1/pods resource. A specific v1 Pod object can be obtained from the /api/v1/namespaces/<namespace-name>/pods/<pod-name> resource.
API discovery documents (like the one published at /api/v1) can be used to determine the resources that correspond to each object type.
Often, the same object can be retrieved from and submitted to multiple resources. For example, v1 Pod objects can be submitted to the following resource URLs:
/api/v1/namespaces/<namespace-name>/pods/<pod-name>
/api/v1/namespaces/<namespace-name>/pods/<pod-name>/status
Distinct resources allow for different server-side behavior and access control. The first URL only allows updating parts of the pod spec and metadata. The second URL only allows updating the pod status, and access is typically only given to kubelets.
Authorization rules are based on the resources for particular requests.
Kubernetes Objects - are something like order in the restaurant. You define a state of the cluster you want to get finally, just like an order to a waiter. kubectl defines your order and delivers it to a cook, just like a waiter. And the API Server prepares your order just like a cook. You define objects in .yaml or .json files.
So, resources are something like menu items. Imagine the Pod is a meat. Meat could be cooked different ways: fried or boiled, for example, but finally it will be a meat in both cases. The similar with the Kubernetes resources. StatefulSet will create Pods with fixed names from 0 to N, while Deployment won't. DaemonSet will create Pods on the each of you nodes, while Deployment or StatefulSet will create as many Pods, as you point in replicas. But finally it will be the Pods, no matter what you chose. You might want to order fried meat, but medium-rare with mustard. What the restaurant will do with your order if it was not in item list? You are Welcome to Kubernetes CRD or CustomResourceDefinition.
P.S: it's a very abstract description and actually StatefulSet/DaemonSets/Deployments or Ingress are also the objects, but they are often referred to as the "resources"
A Kubernetes object is a persistent entities in the Kubernetes system.
A Kubernetes resource is an endpoint in the Kubernetes API that stores a collection of API objects of a certain kind; for example, the built-in pods resource contains a collection of Pod objects.
Objects
Kubernetes uses these entities to represent the state of your cluster.
Specifically, they can describe:
What containerized applications are running (and on which nodes)
The resources available to those applications
The policies around how those applications behave, such as restart policies, upgrades, and fault-tolerance
A Kubernetes object is a "record of intent"--once you create the object, the Kubernetes system will constantly work to ensure that object exists.
By creating an object, you're effectively telling the Kubernetes system what you want your cluster's workload to look like; this is your cluster's desired state.
Note: as commented by VASャ:
Objects usually don't disappear from the cluster on their own, with one exception:
if you create a Pod object manually, and it is scheduled on a node:
it will be completely vanished from the cluster after some time, in case of the node failure or shut down, and
the Pod won't be recreated on another node as you may expect
Resource
A resource is part of a declarative API, used by Kubernetes client libraries and CLIs, or kubectl.
It can lead to "custom resource", an extension of the Kubernetes API that is not necessarily available in a default Kubernetes installation.
As we tried to explain in our blog post, resources are representations of a system entity sent or retrieved as JSON via HTTP and objects are the respective in-memory Golang structs along with functions and methods defined on them.
Note that, in general and informally we're using the terms resources and objects interchangeably and that's totally fine. Unless you're a Go developer, extending Kubernetes, you probably don't need to bother at all.
https://kubernetes.io/docs/reference/using-api/api-concepts/
A resource type is the name used in the Kubernates API,
objects is the resource types which represent a concrete instance of a concept on the cluster
other resource types are virtual
A single instance of the resource type is called a resource
Most Kubernetes API resource types are objects: they represent a
concrete instance of a concept on the cluster, like a pod or
namespace. A smaller number of API resource types are "virtual" - they
often represent operations rather than objects, such as a permission
check.
I also found API Convention of Kubernetes. https://github.com/kubernetes/community/blob/master/contributors/devel/api-conventions.md#types-kinds
Is there any way to get statistics such as service / endpoint access for services defined in Kubernetes cluster?
I've read about Heapster, but it doesn't seem to provide these statistics. Plus, the whole setup is tremendously complicated and relies on a ton of third-party components. I'd really like something much, much simpler than that.
I've been looking into what may be available in kube-system namespace, and there's a bunch of containers and services, there, Heapster including, but they are effectively inaccessible because they require authentication I cannot provide, and kubectl doesn't seem to have any API to access them (or does it?).
Heapster is the agent that collects data, but then you need a monitoring agent to interpret these data. On GCP, for example, that's fluentd who gets these metrics and sends to Stackdriver.
Prometheus is an excellent monitoring tool. I would recommend this one, if youare not on GCP.
If you would be on GCP, then as mentioned above you have Stackdriver Monitoring, that is configured by default for K8s clusters. All you have to do is to create a Stackdriver accound (this is done by one click from GCP Console), and you are good to go.