I'm trying to search previous versions of my secrets inside Microk8s etcd instance but the number of revisions change everytime I refresh my screen and I don't know why.
When I try to access an older version I get the error below:
etcdctl --endpoints=127.0.0.1:2380 get --rev=9133 -w fields /registry/secrets/default/mysql-test-password
{"level":"warn","ts":"2020-09-14T13:40:08.594Z","caller":"clientv3/retry_interceptor.go:61","msg":"retrying of unary invoker failed","target":"endpoint://client-8b3e59a8-efd2-4f77-a96f-5ec3c451b9b7/127.0.0.1:2380","attempt":0,"error":"rpc error: code = OutOfRange desc = etcdserver: mvcc: required revision has been compacted"}
Error: etcdserver: mvcc: required revision has been compacted
I also added the configuration on my etcd config file and restarted the service but didn't help:
--auto-compaction-mode=periodic
--auto-compaction-retention=72h
It seems that every time I refresh my screen the number of revisions increased a lot without doing anything.
{"header":{"cluster_id":14841639068965178418,"member_id":10276657743932975437,"revision":15322,"raft_term":7}
1 second later
student#desktop:~$ etcdctl --endpoints=127.0.0.1:2380 get /registry/secrets/default/mysql-root-password -w json
{"header":{"cluster_id":14841639068965178418,"member_id":10276657743932975437,"revision":16412,"raft_term":7}
Have someone faced something like that?
"etcdserver: mvcc: required revision has been compacted." is not an error, it's an expected message when a watch is attempted to be established at a resource version that has already been compacted.
Something is wrong with your etcd cluster, this is almost certainly not an apiserver problem. There's an alarm you might need to disable.
Also remember that you have to update etcd version to 3.0.11 or later -
https://github.com/kubernetes/kubernetes/issues/45506.
Take a look: mvcc-issues, reloader.
Related
kubectl version does not show when the cluster was upgraded with the current version. Is there a way to get the cluster upgrade timestamp or better timestamps of all previous upgrades?
We can see a list of every running and completed operations in the cluster using the following command :
gcloud beta container operations list
Each operation is assigned an operation ID , operation type, start and end times, target cluster, and status.
To get more information about a specific operation, specify the operation ID in the following command:
gcloud beta container operations describe OPERATION_ID -z region
I am receiving an error message when trying to access details from VS Code of my Azure Kubernetes Cluster. This problem prevents me from attaching a debugger to the pod.
I receive the following error message:
Error loading document: Error: cannot open k8smsx://loadkubernetescore/pod-kube-system%20%20%20coredns-748cdb7bf4-q9f9x.yaml?ns%3Dall%26value%3Dpod%2Fkube-system%20%20%20coredns-748cdb7bf4-q9f9x%26_%3D1611398456559. Detail: Unable to read file 'k8smsx://loadkubernetescore/pod-kube-system coredns-748cdb7bf4-q9f9x.yaml?ns=all&value=pod/kube-system coredns-748cdb7bf4-q9f9x&_=1611398456559' (Get command failed: error: there is no need to specify a resource type as a separate argument when passing arguments in resource/name form (e.g. 'kubectl get resource/<resource_name>' instead of 'kubectl get resource resource/<resource_name>'
)
My Setup
I have VS Code installed, with "Kubernetes", "Bridge to Kubernetes" and "Azure Kubernetes Service" installed
I have connected my Cluster through az login and can already access different information (e.g. my nodes, etc.)
When trying to access the workloads / pods on my cluster, I receive the above error message - and in the Kubernetes View in VS Code I get an error for the details of the pod.
Error in Kubernetes-View in VS Code
What I tried
I tried to reinstall the AKS Cluster and completely logging in freshly to it
I tried to reinstall all extensions mentioned above in VS Code
Browsing the internet, I do not find any comparable error message
The strange thing is that it used to work two weeks ago - and I did not change or update anything (as far as I remember)
Any ideas / hints that I can try further?
Thank you
As #mdaniel wrote: the Node view is just for human consumption, and that the tree item you actually want to click on is underneath Namespaces / kube-system / coredns-748cdb7bf4-q9f9x. Give that a try, and consider reporting your bad experience to their issue tracker since it looks like release 1.2.2 just came out 2 days ago and might not have been tested well.
final solution is to attach debugger in the other way - through Workloads / Deployments.
I updated my EKS from 1.16 to 1.17. All of sudden I started getting this error:
pkg/mod/k8s.io/client-go#v0.0.0-20180806134042-1f13a808da65/tools/cache/reflector.go:99: watch of *v1.Pod ended with: too old resource version
Checked on git and people were saying that's not an error but my question is how to stop getting these messages? I was not getting this message when I was having EKS 1.16?
Source.
This is a community wiki answer. Feel free to expand it.
In short, there is nothing to worry about when encountering these messages. They mean that there are newer version(s) of the watched resource after the time the client API last acquired a list within that watch window. In other words: a watch against the Kubernetes API is timing out, and it is being restarted, which is a intended behavior.
You can also see that being mentioned here:
this is perfectly expected, no worries. The messages are several hours
apart.
When nothing happens in your cluster, the watches established by the
Kubernetes client don't get a chance to get refreshed naturally, and
eventually time out. These messages simply indicate that these watches
are being re-created.
and here:
these are nothing to worry about. This is a known occurrence in
Kubernetes and is not an issue [0]. The API server ends watch requests
when they are very old. The operator uses a client-go informer, which
takes care of automatically re-listing the resource and then
restarting the watch from the latest resource version.
So answering your question:
my question is how to stop getting these messages
Simply, you don't because:
This is working as expected and is not going to be fixed.
I initialized k8 using kubeadm and now when I try to reset using kubeadm reset , am getting the following error. I searched for several forums but couldn't find any answers
> "level":"warn","ts":"2020-05-28T11:57:52.940+0200","caller":"clientv3/retry_interceptor.go:61","msg":"retrying
> o
> f unary invoker
> failed","target":"endpoint://client-e6d5f25b-0ed2-400f-b4d7-2ccabb09a838/192.168.178.200:2379","a
> ttempt":0,"error":"rpc error: code = Unknown desc = etcdserver:
> re-configuration failed due to not enough started
> members"}
The master node status is showing as not ready and I have not been able to reset the network plugin (weave)
ubuntu#ubuntu-nuc-masternode:~$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
ubuntu-nuc NotReady,SchedulingDisabled master 20h v1.18.3
I tried forcing reset but hasn't worked. Any help is much appreciated
This seems to be reported issue kubeadm reset takes more than 50 seconds to retry deleting the last etcd member which was moved here.
Fix was committed kubeadm: skip removing last etcd member in reset phase on May 28th.
What type of PR is this?
/kind bug
What this PR does / why we need it:
If this is the last etcd member of the cluster, it cannot be removed due to "not enough started members". Skip it as the cluster will be destroyed in the next phase, otherwise the retries with exponential backoff will take more than 50 seconds to proceed.
Which issue(s) this PR fixes:
Fixes kubernetes/kubeadm#2144
Special notes for your reviewer:
Does this PR introduce a user-facing change?:
kubeadm: during "reset" do not remove the only remaining stacked etcd member from the cluster and just proceed with the cleanup of the local etcd storage.
I found the apiserver has a flag named api-prefix, so I have tried to use it.
I set the flag to api1 and to keep compatible, I modified the flags of other components like below:
kubelet: --api-servers=https://master:6443/api1
kube-proxy: --master=https://master:6443/api1
I also tried to modified the flag of kube-controller-manager and kube-scheduler, but the error below was reported:
host must be a URL
So I keep the flag of these two components as before.
Now I am wondering about it is because the flag api-prefix is not implemented or the way I used them is wrong?
I don't think the api-prefix is completed yet. There are still pending PRs to get it fully operational. This is based on this comment to the already merged #12951 PR.