What is the difference between Objects and Resouces in Kubernetes world?
I couldn't find it from https://kubernetes.io/docs/concepts/ . I wonder they make no distinction about them but seems they see objects as a high-level concept of resources.
A representation of a specific group+version+kind is an object. For example, a v1 Pod, or an apps/v1 Deployment. Those definitions can exist in manifest files, or be obtained from the apiserver.
A specific URL used to obtain the object is a resource. For example, a list of v1 Pod objects can be obtained from the /api/v1/pods resource. A specific v1 Pod object can be obtained from the /api/v1/namespaces/<namespace-name>/pods/<pod-name> resource.
API discovery documents (like the one published at /api/v1) can be used to determine the resources that correspond to each object type.
Often, the same object can be retrieved from and submitted to multiple resources. For example, v1 Pod objects can be submitted to the following resource URLs:
/api/v1/namespaces/<namespace-name>/pods/<pod-name>
/api/v1/namespaces/<namespace-name>/pods/<pod-name>/status
Distinct resources allow for different server-side behavior and access control. The first URL only allows updating parts of the pod spec and metadata. The second URL only allows updating the pod status, and access is typically only given to kubelets.
Authorization rules are based on the resources for particular requests.
Kubernetes Objects - are something like order in the restaurant. You define a state of the cluster you want to get finally, just like an order to a waiter. kubectl defines your order and delivers it to a cook, just like a waiter. And the API Server prepares your order just like a cook. You define objects in .yaml or .json files.
So, resources are something like menu items. Imagine the Pod is a meat. Meat could be cooked different ways: fried or boiled, for example, but finally it will be a meat in both cases. The similar with the Kubernetes resources. StatefulSet will create Pods with fixed names from 0 to N, while Deployment won't. DaemonSet will create Pods on the each of you nodes, while Deployment or StatefulSet will create as many Pods, as you point in replicas. But finally it will be the Pods, no matter what you chose. You might want to order fried meat, but medium-rare with mustard. What the restaurant will do with your order if it was not in item list? You are Welcome to Kubernetes CRD or CustomResourceDefinition.
P.S: it's a very abstract description and actually StatefulSet/DaemonSets/Deployments or Ingress are also the objects, but they are often referred to as the "resources"
A Kubernetes object is a persistent entities in the Kubernetes system.
A Kubernetes resource is an endpoint in the Kubernetes API that stores a collection of API objects of a certain kind; for example, the built-in pods resource contains a collection of Pod objects.
Objects
Kubernetes uses these entities to represent the state of your cluster.
Specifically, they can describe:
What containerized applications are running (and on which nodes)
The resources available to those applications
The policies around how those applications behave, such as restart policies, upgrades, and fault-tolerance
A Kubernetes object is a "record of intent"--once you create the object, the Kubernetes system will constantly work to ensure that object exists.
By creating an object, you're effectively telling the Kubernetes system what you want your cluster's workload to look like; this is your cluster's desired state.
Note: as commented by VASャ:
Objects usually don't disappear from the cluster on their own, with one exception:
if you create a Pod object manually, and it is scheduled on a node:
it will be completely vanished from the cluster after some time, in case of the node failure or shut down, and
the Pod won't be recreated on another node as you may expect
Resource
A resource is part of a declarative API, used by Kubernetes client libraries and CLIs, or kubectl.
It can lead to "custom resource", an extension of the Kubernetes API that is not necessarily available in a default Kubernetes installation.
As we tried to explain in our blog post, resources are representations of a system entity sent or retrieved as JSON via HTTP and objects are the respective in-memory Golang structs along with functions and methods defined on them.
Note that, in general and informally we're using the terms resources and objects interchangeably and that's totally fine. Unless you're a Go developer, extending Kubernetes, you probably don't need to bother at all.
https://kubernetes.io/docs/reference/using-api/api-concepts/
A resource type is the name used in the Kubernates API,
objects is the resource types which represent a concrete instance of a concept on the cluster
other resource types are virtual
A single instance of the resource type is called a resource
Most Kubernetes API resource types are objects: they represent a
concrete instance of a concept on the cluster, like a pod or
namespace. A smaller number of API resource types are "virtual" - they
often represent operations rather than objects, such as a permission
check.
I also found API Convention of Kubernetes. https://github.com/kubernetes/community/blob/master/contributors/devel/api-conventions.md#types-kinds
Related
I'm trying to figure out which tools from GKE stack I should apply to my use case which is a dynamic deployment of stateful application with dynamic HTTP endpoints.
Stateful in my case means that I don't want any replicas and load-balancing (because the app doesn't scale horizontally at all). I understand though that in k8s/gke nomenclature I'm still going to be using a 'load-balancer' even though it'll act as a reverse proxy and not actually balance any load.
The use case is as follows. I have some web app where I can request for a 'new instance' and in return I get a dynamically generated url (e.g. http://random-uuid-1.acme.io). This domain should point to a newly spawned, single instance of a container (Pod) hosting some web application. Again, if I request another 'new instance', I'll get a http://random-uuid-2.acme.io which will point to another (separate), newly spawned instance of the same app.
So far I figured out following setup. Every time I request a 'new instance' I do the following:
create a new Pod with dynamic name app-${uuid} that exposes HTTP port
create a new Service with NodePort that "exposes" the Pod's HTTP port to the Cluster
create or update (if exists) Ingress by adding a new http rule where I specify that domain X should point at NodePort X
The Ingress mentioned above uses a LoadBalancer as its controller, which is automated process in GKE.
A few issues that I've already encountered which you might be able to help me out with:
While Pod and NodePort are separate resources per each app, Ingress is shared. I am thus not able to just create/delete a resource but I'm also forced to keep track of what has been added to the Ingress to be then able to append/delete from the yaml which is definitely not the way to do that (i.e. editing yamls). Instead I'd probably want to have something like an Ingress to monitor a specific namespace and create rules automatically based on Pod labels. Say I have 3 pods with labels, app-1, app-2 and app-3 and I want Ingress to automatically monitor all Pods in my namespace and create rules based on the labels of these pods (i.e. app-1.acme.io -> reverse proxy to Pod app-1).
Updating Ingress with a new HTTP rule takes around a minute to allow traffic into the Pod, until then I keep getting 404 even though both Ingress and LoadBalancer look as 'ready'. I can't figure out what I should watch/wait for to get a clear message that the Ingress Controller is ready for accepting traffic for newly spawned app.
What would be the good practice of managing such cluster where you can't strictly define Pods/Services manifests because you are creating them dynamically (with different names, endpoints or rules). You surely don't want to create bunch of yaml-s for every application you spawn to maintain. I would imagine something similar to consul templates in case of Consul but for k8s?
I participated in a similar project and our decision was to use Kubernetes Client Library to spawn instances. The instances were managed by a simple web application, which took some customisation parameters, saved them into its database, then created an instance. Because of the database, there was no problem with keeping track of what have been created so far. By querying the database we were able to tell if such deployment was already created or update/delete any associated resources.
Each instance consisted of:
a deployment (single or multi-replica, depending on the instance);
a ClusterIp service (no reason to reserve machine port with NodePort);
an ingress object for shared ingress controller;
and some shared configMaps.
And we also used external DNS and cert manager, one to manage DNS records and another to issue SSL certificates for the ingress. With this setup it took about 10 minutes to deploy a new instance. The pod and ingress controller were ready in seconds but we had to wait for the certificate and it's readiness depended on whether issuer's DNS got our new record. This problem might be avoided by using a wildcard domain but we had to use many different domains so it wasn't an option in our case.
Other than that you might consider writing a Helm chart and make use of helm list command to find existing instances and manage them. Though, this is a rather 'manual' solution. If you want this functionality to be a part of your application - better use a client library for Kubernetes.
Problem setting. Suppose I have 2 pods, A and B. I want to be able to dynamically scale pod A based on some arbitrary number from some arbitrary source. Suppose that pod B is such a source: for example, it can have an HTTP server with an endpoint which responds with the number of desired replicas of pod A at the moment of request. Or maybe it is an ES server or a SQL DB (does not matter).
Question. What kubernetes objects do I need to define to achieve this (apart from HPA)? What configuration should HPA have to know that it needs to look up B for current metric? How should API of B look like (or is there any constraints?)?
Research I have made. Unfortunately, the official documentation does not say much about it, apart from declaring that there is such a possibility. There are also two repositories, one with some go boilerplate code that I have trouble building and another one that has no usage instructions whatsoever (though allegedly does fulfil the "external metrics over HTTP" requirement).
By having a look at the .yaml configs in those repositories, I have reached a conclusion that apart from Deployment and Service one needs to define an APIService object that registers the external or custom metric in the kubernetes API and links it with a normal service (where you would have your pod) and a handful of ClusterRole and ClusterRoleBinding objects. But there is no explanation about it. Also I could not even list existing APIServices with kubectl in my local cluster (of 1.15 version) like other objects.
The easiest way will be to feed metrics into Prometheus (which is a commonly solved problem), and then setup a Prometheus-based HPA (also a commonly solved problem).
1. Feed own metrics to Prometheus
Start with Prometheus-Operator to get the cluster itself monitored, and get access to ServiceMonitor objects. ServiceMonitors are pointers to services in the cluster. They let your pod's /metrics endpoint be discovered and scraped by a prometheus server.
Write a pod that reads metrics from your 3rd party API and shows them in own /metrics endpoint. This will be the adapter between your API and Prometheus format. There are clients of course: https://github.com/prometheus/client_python#exporting
Write a Service of type ClusterIP that represents your pod.
Write a ServiceMonitor that points to a service.
Query your custom metrics thru Prometheus dashboard to ensure this stage is done.
2. Setup Prometheus-based HPA
Setup Prometheus-Adapter and follow the HPA walkthrough.
Or follow the guide https://github.com/stefanprodan/k8s-prom-hpa
This looks like a huge pile of work to get the HPA. However, only the adapter pod is a custom part here. Everything else is a standard stack setup in most of the clusters, and you will get many other use cases for it anyways.
From the Kubernetes' validation source code, at least those resources are immutable after creation:
Persistent Volumes
Storage Classes
Why is that ?
This is a core concept on Kubernetes. A few specs are immutable because their change has impact in the basic structure of the resource it's connected.
For example, changing the Persistent Volumes may impact pods that are using this PV. Let's suppose you have a mysql pod running on a PV and you change it in a way that all the data is gone.
On Kubernetes 1.18 Secrets and ConfigMaps also became immutable as an Alpha feature, meaning that this will be the new default soon. Check the GitHub Issue here.
What is it good for?
The most popular and the most convenient way of consuming Secrets and
ConfigMaps by Pods is consuming it as a file. However, any update to a
Secret or ConfigMap object is quickly (roughly within a minute)
reflected in updates of the file mounted for all Pods consuming them.
That means that a bad update (push) of Secret and/or ConfigMap can
very quickly break the entire application.
Here you can read more about the motivation behind this decision.
In this KEP, we are proposing to introduce an ability to specify that
contents of a particular Secret/ConfigMap should be immutable for its
whole lifetime. For those Secrets/ConfigMap, Kubelets will not be
trying to watch/poll for changes to updated mounts for their Pods.
Given there are a lot of users not really taking advantage of
automatic updates of Secrets/ConfigMaps due to consequences described
above, this will allow them to:
protect themselves better for accidental bad updates that could cause outages of their applications
achieve better performance of their cluster thanks to significant reduction of load on apiserver
I'm new to K8s, so still trying to get my head around things. I've been looking at deployments and can appreciate how useful they will be. However, I don't understand why they don't support services (only replica sets and pods).
Why is this? Does this mean that services would typically be deployed outside of a deployment?
To answer your question, Kubernetes deployments are used for managing stateless services running in the cluster instead of StatefulSets which are built for the stateful application run-time. Actually, with deployments you can describe the update strategy and road map for all underlying objects that have to be created during implementation.Therefore, we can distinguish separate specification fields for some objects determination, like needful replica number of Pods, template for Pod by describing a list of containers that should be in the Pod, etc.
However, as #P Ekambaram already mention in his answer, Services represent abstraction layer of network communication model inside Kubernetes cluster, and they declare a way to access Pods within a cluster via corresponded Endpoints. Services are separated from deployment object manifest specification, because of their mission to dynamically provide specific network behavior for the nested Pods without affecting or restarting them in case of any communication modification via appropriate Service Types.
Yes, services should be deployed as separate objects. Note that deployment is used to upgrade or rollback the image and works above ReplicaSet
Kubernetes Pods are mortal. They are born and when they die, they are not resurrected. ReplicaSets in particular create and destroy Pods dynamically (e.g. when scaling out or in). While each Pod gets its own IP address, even those IP addresses cannot be relied upon to be stable over time. This leads to a problem: if some set of Pods (let’s call them backends) provides functionality to other Pods (let’s call them frontends) inside the Kubernetes cluster, how do those frontends find out and keep track of which backends are in that set?
Services.come to the rescue.
A Kubernetes Service is an abstraction which defines a logical set of Pods and a policy by which to access them. The set of Pods targeted by a Service is (usually) determined by a Label Selector
Something I've just learnt that is somewhat related to my question: multiple K8s objects can be included in the same yaml file, separate by ---. Something like:
apiVersion: v1
kind: Deployment
# other stuff here
---
apiVersion: v1
kind: Service
# other stuff here
i think it intends to decoupled and fine-grained.
I am trying to find a DB (object storage) for my application. The application is really a wrapper over ISTIO Network Routing API. Basically simplifying the ISTIO configuratin for my network. Kubernetes (k8s) Custom Resource Definition (CRD) seems to fit my requirements. Also love the watch and REST API capability provided by CRD.
DB requirement
Few 100 MBs data - worst case
Ability to watch objects
REST API support for object
Persistence
Around 2k writes/sec and similar/more reads/sec. Although I do have my application acting as a proxy for CRD where things can be cached.
Why using CRD will be a good or a bad idea? Are there any performance implication using CRD. This 2016 stackflow answer suggest that etcd data is not in RAM. Whereas etcd link suggest that etcd can do 10k writes/sec (so even things are not in RAM and purely in disk, who cares).
I am seeing multiple application using k8s CRDs.
Helm uses CRDs to store releases
Istio uses CRDs to store their networking routing API objects
Considering that (CRD page)
A resource is an endpoint in the Kubernetes API that stores a collection of API objects of a certain kind. For example, the built-in pods resource contains a collection of Pod objects.
A custom resource is an extension of the Kubernetes API that is not necessarily available on every Kubernetes cluster. In other words, it represents a customization of a particular Kubernetes installation.
A CRD is for extending Kubernetes itself, it is not for application data.
The helm-crd/examples/mariadb.yaml is about lightweight metadata which will enable Kubernetes to download the right release and through Helm install it.
It is not to store data for a random application which could exist without Kubernetes (as opposed to Helm releases, which make sense only in a Kubernetes deployment scenario)
Similarly, Istio CRD makes sense only in a Kubernetes context:
Kubernetes currently implements the Istio configuration on Custom Resource Definitions (CRDs). These CRDs correspond to namespace-scope and cluster-scope CRDs and automatically inherit access protection via the Kubernetes RBAC.
That approach (using etcd to store any application data) would not scale.