I am trying to delete a namespace but it is in terminating state, I tried removing the finalizer and applying replace but not able to succeed. Below are the steps and error
[root#~]# kubectl replace "/api/v1/namespaces/service-catalog/finalize" -f n.json
namespace/service-catalog replaced
[root#~]#
[root#~]#
[root#~]# k get ns service-catalog
NAME STATUS AGE
service-catalog Terminating 6d21h
[root#~]# k delete ns service-catalog
Error from server (Conflict): Operation cannot be fulfilled on namespaces "service-catalog": The system is ensuring all content is removed from this namespace. Upon completion, this namespace will automatically be purged by the system.
In the namespace I had created few crd objects and my good guess is those are the thing which are preventing it from deletion. Right now I am not able memorise all the crd object that I created.
Is there a way where I can query all the object with the finalizer: service-catalog?
I was looking for all the finalizers that were used in our cluster and this worked for me. It checks for all types of objects in all namespaces and returns their finalizers -- you can probably use awk and grep to filter it out for what you're looking for
kubectl get all -o custom-columns=Kind:.kind,Name:.metadata.name,Finalizers:.metadata.finalizers --all-namespaces
Note, this doesn't return the cluster scoped resources
To get the registered CRD list, use:
$ kubectl get crds
elasticsearches.kubedb.com 2020-03-03T04:05:13Z
elasticsearchversions.catalog.kubedb.com 2020-03-03T04:05:16Z
etcds.kubedb.com 2020-03-03T04:05:13Z
etcdversions.catalog.kubedb.com 2020-03-03T04:05:16Z
ingresses.voyager.appscode.com 2020-03-03T05:07:42Z
m3dbclusters.operator.m3db.io 2020-03-02T10:56:55Z
Once you have the CRDs, you can find the objects of that type in given namespace:
$ kubectl get m3dbclusters.operator.m3db.io -n m3db
NAME AGE
m3db-cluster 47h
To list all objects along with the finalizers, you can use custom-columns.
# kubectl get <crd-name> -n <namespace> -o custom-columns=Kind:.kind,Name:.metadata.name,Finalizers:.metadata.finalizers
$ kubectl get m3dbclusters.operator.m3db.io -n m3db -o custom-columns=Kind:.kind,Name:.metadata.name,Finalizers:.metadata.finalizers
Kind Name Finalizers
M3DBCluster m3db-cluster [operator.m3db.io/etcd-deletion]
Related
I am looking for a way to get all object's metadata within a k8s cluster and send it out to an external server.
By metadata, I refer to objects Name, Kind, Labels, Annotations, etc.
The intention is to build an offline inventory of a cluster.
What would be the best approach to build it? Is there any tool that already does something similar?
Thanks
Posting this as a community wiki, feel free to edit and expand.
There are different ways to achieve it.
From this GitHub issue comment it's possible to iterate through all resources to get all available objects.
in yaml:
kubectl api-resources --verbs=list -o name | xargs -n 1 kubectl get --show-kind --ignore-not-found -o yaml
in json:
kubectl api-resources --verbs=list -o name | xargs -n 1 kubectl get --show-kind --ignore-not-found -o json
And then parse the output.
Use kubernetes clients.
There are already developed kubernetes clients (available for different languages) which can be used to get required information and work with it later.
Use kubectl plugin - ketall (didn't test it)
There's a developed plugin for kubectl which returns all cluster resources. Please find github repo - ketall. Again after cluster objects are gotten, you will need to parse/work with them.
Try this commands
kubectl get all --all-namespaces -o yaml
or
kubectl get all --all-namespaces -o json
you can parse and use as you find fit
I am looking to apply existing annotations on a Kubernetes resource to the underlying YAML configuration files. For example, this command will successfully find all pods with a label of "app=helloworld" or "app=testapp" and annotate them with "xyz=test_anno":
kubectl annotate pods -l 'app in (helloworld, testapp)' xyz=test_anno
However, this only applies the annotations to the running pods and doesn't change the YAML files. How do I force those changes to the YAML files so they're permanent, either after the fact or as part of kubectl annotate to start with?
You could use the kubectl patch command with a little tricks
kubectl patch $(k get po -l 'app in (helloworld, testapp)') -p '{"metadata":{"annotations":{"xyz":"test_anno"}}}'
I'm expecting that kubectl get nodes <node> -o yaml to show the spec.providerID (see reference below) once the kubelet has been provided the additional flag --provider-id=provider://nodeID. I've used /etc/default/kubelet file to add more flags to the command line when kubelet is start/restarted. (On a k8s 1.16 cluster) I see the additional flags via a systemctl status kubelet --no-pager call, so the file is respected.
However, I've not seen the value get returned by kubectl get node <node> -o yaml call. I was thinking it had to be that the node was already registered, but I think kubectl re-registers when it starts up. I've seen the log line via journalctl -u kubelet suggest that it has gone through registration.
How can I add a provider ID to a node manually?
https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.17/#nodespec-v1-core
How a kubelet is configured on the node itself is separate (AFAIK) from its definition in the master control plane, which is responsible for updating state in the central etcd store; so it's possible for these to fall out of sync. i.e., you need to communicate to the control place to update its records.
In addition to Subramanian's suggestion, kubectl patch node would also work, and has the added benefit of being easily reproducible/scriptable compared to manually editing the YAML manifest; it also leaves a "paper trail" in your shell history should you need to refer back. Take your pick :) For example,
$ kubectl patch node my-node -p '{"spec":{"providerID":"foo"}}'
node/my-node patched
$ kubectl describe node my-node | grep ProviderID
ProviderID: foo
Hope this helps!
You can edit the node config and append providerID information under spec section.
kubectl edit node <Node Name>
...
spec:
podCIDR:
providerID:
In kubectl, both describe and get -o <format> can be used to get the details of a resource, I'm wondering what's the difference between the two? why does describe even exist if get can do the same thing and more?
kubectl get shows tables by default. (You can view/visualize large no of objects easily)
kubectl describe shows the detailed description. (Better for a single object)
kubectl describe is more flattened, has lesser data and easier to read than the full object data given by kubectl get -o yaml
Help output for reference.
kubectl describe -h
Show details of a specific resource or group of resources
Print a detailed description of the selected resources, including related resources such as events or controllers. You
may select a single object by name, all objects of that type, provide a name prefix, or label selector. For example:
$ kubectl describe TYPE NAME_PREFIX
will first check for an exact match on TYPE and NAME_PREFIX. If no such resource exists, it will output details for
every resource that has a name prefixed with NAME_PREFIX.
Use "kubectl api-resources" for a complete list of supported resources.
kubectl get -h
Display one or many resources
Prints a table of the most important information about the specified resources. You can filter the list using a label
selector and the --selector flag. If the desired resource type is namespaced you will only see results in your current
namespace unless you pass --all-namespaces.
Uninitialized objects are not shown unless --include-uninitialized is passed.
By specifying the output as 'template' and providing a Go template as the value of the --template flag, you can filter
the attributes of the fetched resources.
Use "kubectl api-resources" for a complete list of supported resources.
A simple explanation could be:
kubectl describe .. == Filterred kubectl get .. -o <format> + relevant events
For debugging purposes it can be useful to look at both describe and get -o <format> since each one has relevant information that is not shown in the other.
Note that events can also be shown by running kubectl get events (for default namespace), or kubectl get event --all-namespaces (for all namespaces).
Some digging:
Running kubectl describe with extended logging --v=8 shows that it calls the events API (the third call) with involvedObject.name%3Dsome-pod.
$ kubectl --v=8 describe pod some-pod 2>&1 | grep GET
I1216 17:09:00.453529 6918 round_trippers.go:416] GET https://100.190.50.200/api/v1/namespaces/default/pods/some-pod
I1216 17:09:01.098053 6918 round_trippers.go:416] GET https://100.190.50.200/api/v1/namespaces/default/pods/some-pod
I1216 17:09:01.265924 6918 round_trippers.go:416] GET https://100.190.50.200/api/v1/namespaces/default/events?fieldSelector=involvedObject.name%3Dsome-pod%2CinvolvedObject.namespace%3Ddefault%2CinvolvedObject.uid%3Dbf664be1-1cde-11ea-bce6-42010af00267
kubectl get pod some-pod -o yaml only calls the pods API.
kubectl --v=8 get pod some-pod -o yaml 2>&1 | grep GET
I1216 17:13:21.084386 28256 round_trippers.go:416] GET https://100.190.50.200/api/v1/namespaces/default/pods/some-pod
According to kubernetes documentation:
kubectl -n <NAMESPACE> get <NAME_OF_RESOURCE>
Prints a table of the most important information about the specified
resources. You can filter the list using a label selector and the
--selector flag. If the desired resource type is namespaced you will only see results in your current namespace unless you pass
--all-namespaces.
Source: https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#get
kubectl -n <NAMESPACE> describe <NAME_OF_RESOURCE>
Print a detailed description of the selected resources, including
related resources such as events or controllers. You may select a
single object by name, all objects of that type, provide a name
prefix, or label selector.
Source: https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#describe
kubectl describe is supposed to give you more information .Even if, I agree with you, some resources have quiete the same informations with kubectl get or kubectl describe.
One important difference need to mention: describe accepts the prefix but get need the exactly name.
$ kubectl describe TYPE NAME_PREFIX
$ kubectl get TYPE NAME
You can get the summary of a specific resource using:
kubectl get < resource > < name >
While you can get detailed information with:
kubectl describe < resource > < name >
I was trying to get the Ready status of pods by using -o=jsonpath.
To be more clear of what I want, I would like to get the value 1/1 of the following example using -o=jsonpath.
NAME READY STATUS RESTARTS AGE
some_pod 1/1 Running 1 34d
I have managed to get some information such as the pod name or namespace.
kubectl get pods --all-namespaces -o=jsonpath='{range .items[*]}{"\n"}{.metadata.namespace}{"\t"}{.metadata.name}{"\t"}{end}'
And I get somthing like:
some_namespace1 pod_name1
However, I don't know how to get the Ready status. What I would like to have is an aoutput similar to this:
some_namespace1 pod_name1 1/1
I know I can use bash commands like cut:
kubectl get pods --all-namespaces| tail -1 | cut -d' ' -f8
However, I would like to get it by using kubectl
You can get all the pods status using the following command:
kubectl get pods -o jsonpath={.items[*].status.phase}
Similar commands you can use for the name
kubectl get pods -o jsonpath={.items[*].metadata.name}
EDIT:
You need to compare the .status.replicas and .status.readyReplicas to get how many ready replicas are there.
I think this isn't directly reported in the Kubernetes API.
If you kubectl get pod ... -o yaml (or -o json) you'll get back an object matching a List (not included in the API docs) where each item is a Pod in the Kubernetes API, and -o jsonpath values follow that object structure. In particular a PodStatus has a list of ContainerStatus, each of which may or may not be ready, but the API itself doesn't return the counts as first-class fields.
There are a couple of different JSONPath implementations. I think Kubernetes only supports the syntax in the Kubernetes documentation, which doesn't include any sort of "length" function. (The original JavaScript implementation and a ready Googlable Java implementation both seem to, with slightly different syntax.)
The best I can come up with playing with this is to report all of the individual container "ready" statuses
kubectl get pods \
-o $'jsonpath={range .items[*]}{.metadata.name}\t{.status.containerStatuses[*].ready}\n{end}'
($'...' is bash/zsh syntax) but this still requires some post-processing to get back the original counts.