Fresh cluster and linkerd - viz doesn't startup - kubernetes

I've got an issue, I'm trying to install linkerd on my cluster, all is going well
I went exactly with this official README
https://linkerd.io/2.11/tasks/install-helm/
installed it via helm
MacBook-Pro-6% helm list -n default
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
linkerd2 default 1 2021-12-15 15:47:10.823551 +0100 CET deployed linkerd2-2.11.1 stable-2.11.1
linkerd itself works, and the linkerd check command as well
MacBook-Pro-6% linkerd version
Client version: stable-2.11.1
Server version: stable-2.11.1
but when I try to install viz dashboard as described in the getting-started page I run
linkerd viz install | kubectl apply -f -
and when going with
linkerd check
...
Status check results are √
Linkerd extensions checks
=========================
/ Running viz extension check
and it keeps on checking the viz extensions, and when I ran linkerd dashboard (deprecated I know) shows the same error
Waiting for linkerd-viz extension to become available
anyone got any clue what I'm doing wrong ? Been stuck at this part for 2hrs &_& and noone seem to have any answers
note, when I ran, linkerd check after instalation of viz I get
linkerd-viz
-----------
√ linkerd-viz Namespace exists
√ linkerd-viz ClusterRoles exist
√ linkerd-viz ClusterRoleBindings exist
√ tap API server has valid cert
√ tap API server cert is valid for at least 60 days
‼ tap API service is running
FailedDiscoveryCheck: failing or missing response from https://10.190.101.142:8089/apis/tap.linkerd.io/v1alpha1: Get "https://10.190.101.142:8089/apis/tap.linkerd.io/v1alpha1": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
see https://linkerd.io/2.11/checks/#l5d-tap-api for hints
‼ linkerd-viz pods are injected
could not find proxy container for grafana-8d54d5f6d-cv7q5 pod
see https://linkerd.io/2.11/checks/#l5d-viz-pods-injection for hints
√ viz extension pods are running
× viz extension proxies are healthy
No "linkerd-proxy" containers found in the "linkerd" namespace
see https://linkerd.io/2.11/checks/#l5d-viz-proxy-healthy for hints
debugging

From your problem descripiton:
‼ linkerd-viz pods are injected
could not find proxy container for grafana-8d54d5f6d-cv7q5 pod
see https://linkerd.io/2.11/checks/#l5d-viz-pods-injection for hints
and:
MacBook-Pro-6% helm list -n default
I encountered a similar problem but with flagger pod rather than grafana pod (I didn't attempt to install grafana component like you did).
A side effect of my problem is this:
$ linkerd viz dashboard
Waiting for linkerd-viz extension to become available
Waiting for linkerd-viz extension to become available
Waiting for linkerd-viz extension to become available
... ## repeating for 5 minutes or so before popping up the dashboard in browser.
The cause for my problem turned out to be that I installed the viz extension into the linkerd namespace. It should belong to the linkerd-viz namespace.
Looking at your original problem description, it seems that you installed the control plane into the default namespace (as opposed to the linkerd namespace.) While you can use any namespace you want, the control plane must be in a separate namespace from the viz extension. Details can be seen in the discussion I wrote here:
https://github.com/linkerd/website/issues/1309

Related

Telepresence can't find `Deployments, StatefulSets, or ReplicaSets`

I installed the traffic-manager by using the official helm chart by datawire and flux components. But when i try to list interceptable services with telepresence list i get following message:
No Workloads (Deployments, StatefulSets, or ReplicaSets)
First i used the default namespace ambassador without further configurations. Then i tried to activate the RBCA users and restricted the namespaces. In the cluster are several namespaces with different purposes like flux-system, kube-system. Services, where i want to intercept, are deployed in the same namespace. Therefore i tried to install the traffic-manager directly into this namespace, but the same message occured (i also configured my kubeconfig, so the traffic-manager can be found, as the documentation says).
In the logs of the traffic-manager i get following warning:
agent-configs : Issuing a systema request without ApiKey or InstallID may result in an error
What does that mean? Could that be part of the issue?
I am new to cluster topics in general but couldn't find anything by research, hence i decided to ask in the community.
Some hints would be very helpful, because i don't know what i could try next. In the first place it would be enough when it works trough the whole cluster without restrictions.
telepresence version:
Client: v2.6.6 (api v3)
Root Daemon: v2.6.6 (api v3)
User Daemon: v2.6.6 (api v3)
kubernetes: v1.22.6

Failing to run Mattermost locally on a Kubernetes cluster using Minikube

Summary in one sentence
I want to deploy Mattermost locally on a Kubernetes cluster using Minikube
Steps to reproduce
I used this tutorial and the Github documentation:
https://mattermost.com/blog/how-to-get-started-with-mattermost-on-kubernetes-in-just-a-few-minutes/
https://github.com/mattermost/mattermost-operator/tree/v1.15.0
To start minikube: minikube start --kubernetes-version=v1.21.5
To start ingress; minikube addons enable ingress
I cloned the Github repo with tag v1.15.0 (second link)
In the Github documentation (second link) they state that you need to install Custom Resources by running: kubectl apply -f ./config/crd/bases
Afterwards I installed MinIO and MySQL operators by running: make mysql-minio-operators
Started the Mattermost-operator locally by running: go run .
In the end I deployed Mattermost (I followed step 2, 7 and 9 from the first link)
Observed behavior
Unfortunately I keep getting the following error in the mattermost-operator:
INFO[1419] [opr.controllers.Mattermost] Reconciling Mattermost Request.Name=mm-demo Request.Namespace=mattermost
INFO[1419] [opr.controllers.Mattermost] Updating resource Reconcile=fileStore Request.Name=mm-demo Request.Namespace=mattermost kind="&TypeMeta{Kind:,APIVersion:,}" name=mm-demo-minio namespace=mattermost patch="{\"status\":{\"availableReplicas\":0}}"
INFO[1419] [opr.controllers.Mattermost.health-check] mattermost pod not ready: pod mm-demo-ccbd46b9c-9nq8k is in state 'Pending' Request.Name=mm-demo Request.Namespace=mattermost
INFO[1419] [opr.controllers.Mattermost.health-check] mattermost pod not ready: pod mm-demo-ccbd46b9c-tp567 is in state 'Pending' Request.Name=mm-demo Request.Namespace=mattermost
ERRO[1419] [opr.controllers.Mattermost] Error checking Mattermost health Request.Name=mm-demo Request.Namespace=mattermost error="found 0 updated replicas, but wanted 2"
By using k9s I can see that mm-demo won't start. See below for photo.
Another variation of deployment
Also tried another variation by following all the steps from the first link (without the licences secret step). At this point the mattermost-operator is visible using k9s and won't getting any errors. But unfortunately the mm-demo pod keeps crashing (empty logs, so seeing no errors or something).
Anybody an idea?
As #Ashish faced the same issue, he fixed it by upgrading the resources.
Minikube will be able to run all the pods by running minikube start --kubernetes-version=v1.21.5 --memory 4000 --cpus 4

Cert-manager fails on kubernetes with webhooks

I'm following the Kubernetes install instructions for Helm: https://docs.cert-manager.io/en/latest/getting-started/install/kubernetes.html
With Cert-manager v0.81 on K8 v1.15, Ubuntu 18.04 on-premise.
When I get to testing the installation, I get these errors:
error when creating "test-resources.yaml": Internal error occurred: failed calling webhook "issuers.admission.certmanager.k8s.io": the server is currently unable to handle the request
Error from server (InternalError): error when creating "test-resources.yaml": Internal error occurred: failed calling webhook "certificates.admission.certmanager.k8s.io": the server is currently unable to handle the request
If I apply the test-resources.yaml before installing with Helm, I'm not getting the errors but it is still not working.
These errors are new to me, as Cert-manager used to work for me on my previous install about a month ago, following the same installation instructions.
I've tried with Cert-Manager 0.72(CRD 0.7) as well as I think that was the last version I managed to get installed but its not working either.
What does these errors mean?
Update: It turned out to be an internal CoreDNS issue on my cluster. Somehow not being configured correctly. Possible related to wrong POD_CIDR configuration.
If you experience this problem, check the logs of CoreDNS(Or KubeDNS) and you may see lots of errors related to contacting services. Unfortunately, I no longer have the errors.
But this is how I figured out that my network setup was invalid.
I'm using Calico(Will apply for other networks as well) and its network was not set to the same as the POD_CIDR network that I initialized my Kubernetes with.
Example
1. Set up K8:
kubeadm init --pod-network-cidr=10.244.0.0/16
Configure Calico.yaml:
- name: CALICO_IPV4POOL_CIDR
value: "10.244.0.0/16"
I also tried cert-manager v0.8.0 a very similar setup on Ubuntu 18.04 and k8s v1.14.1 and I began to get the same error when i tore down cert-manager using kubectl delete and reinstalled it, after experiencing some network issues on the cluster.
I stumbled on a solution that worked. On the master node, simply restart the apiserver container:
$ sudo docker ps -a | grep apiserver
af99f816c7ec gcr.io/google_containers/kube-apiserver#sha256:53b987e5a2932bdaff88497081b488e3b56af5b6a14891895b08703129477d85 "/bin/sh -c '/usr/loc" 15 months ago Up 19 hours k8s_kube-apiserver_kube-apiserver-ip-xxxxxc_0
40f3a18050c3 gcr.io/google_containers/pause-amd64:3.0 "/pause" 15 months ago Up 15 months k8s_POD_kube-apiserver-ip-xxxc_0
$ sudo docker restart af99f816c7ec
af99f816c7ec
$
Then try applying the test-resources.yaml again:
$ kubectl apply -f test-resources.yaml
namespace/cert-manager-test unchanged
issuer.certmanager.k8s.io/test-selfsigned created
certificate.certmanager.k8s.io/selfsigned-cert created
If that does not work, this github issue mentions that the master node might need firewall rules to be able to reach the cert-manager-webhook pod. The exact steps to do so will depend on which cloud platform you are on.

Error install istio in GKE = the server could not find the requested resource (post `gatewaies.networking.istio.io`)

I have the following error when installing istio in GKE
kubernetes ver = 1.11.2-gke.18
Istio ver = 1.0.4
Kubectl = latest from repo google
Error from server (NotFound): error when creating
"`install/kubernetes/istio-demo-auth.yaml`":
the server could not find the requested resource
(post `gatewaies.networking.istio.io`)
I have tried to follow the tutorial on GCP:
https://cloud.google.com/kubernetes-engine/docs/tutorials/installing-istio
You are missing the CustomResourceDefinition required by istio and hence getting this error. You need to apply following command from istio folder:
kubectl apply -f install/kubernetes/helm/istio/templates/crds.yaml
This will create all the CRD's like virtualservice, destinationrules etc.
Try following official documentation of Istio to install it on GKE:
https://istio.io/docs/setup/kubernetes/quick-start-gke-dm/
I am also getting this issue when installing a custom Istio helm chart:
[tiller] 2019/11/15 21:50:52 failed install perform step: release test failed: the server could not find the requested resource (post gatewaies.networking.istio.io)
I've confirmed the Istio CRDs are installed properly. Note how the installed Gateway CRD explicitly notes the accepted plural name:
status:
acceptedNames:
categories:
- istio-io
- networking-istio-io
kind: Gateway
listKind: GatewayList
plural: gateways
shortNames:
- gw
singular: gateway
I created an issue on Helm to see if that is the culprit, otherwise, I can open an issue on Istio to see if that is either. I'm very confused where the source of this issue could be coming from.
**Note: ** The type of the Gateway resource is correct:
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
istio works by defining a series of crds(Custom Resource Definition), for istio to work, you first need to run command like this:
kubectl apply -f install/kubernetes/helm/istio/templates/crds.yaml
for my version(istio v1.2.0), the command is
for i in install/kubernetes/helm/istio-init/files/crd*yaml; do kubectl apply -f $i; done
but as I follow the instructions from the documentatino, I still get the annoying messages:
Error from server (NotFound): error when creating "samples/bookinfo/networking/bookinfo-gateway.yaml": the server could not find the requested resource (post gatewaies.networking.istio.io)
as the hint implies, the requested resource "gatewaies.networking.istio.io" cannot be found, and then I list the crds:
kubectl get crd
and I got a list like this:
enter image description here
as I see inspect this, I find something wrong.
the message issued by kubectl is (post gatewaies.networking.istio.io), but the crd enlisted is post gateways.networking.istio.io, then everything is clear, the kubectl CLI issued a wrong plural for word "gateway", the correct form is gateways, instead of gatewaies, so to satisfy the command form, the crd must change.
And I edit this file:
vim install/kubernetes/helm/istio-init/files/crd-10.yaml
by changing the name from "gateways.networking.istio.io" to "gatewaies.networking.istio.io", everything is ok now.

Kubernetes 1.11 could not find heapster for metrics

I'm using Kubernetes 1.11 on Digital Ocean, when I try to use kubectl top node I get this error:
Error from server (NotFound): the server could not find the requested resource (get services http:heapster:)
but as stated in the doc, heapster is deprecated and no longer required from kubernetes 1.10
If you are running a newer version of Kubernetes and still receiving this error, there is probably a problem with your installation.
Please note that to install metrics server on kubernetes, you should first clone it by typing:
git clone https://github.com/kodekloudhub/kubernetes-metrics-server.git
then you should install it, WITHOUT GOING INTO THE CREATED FOLDER AND WITHOUT MENTIONING AN SPECIFIC YAML FILE , only via:
kubectl create -f kubernetes-metrics-server/
In this way all services and components are installed correctly and you can run:
kubectl top nodes
or
kubectl top pods
and get the correct result.
For kubectl top node/pod to work you either need the heapster or the metrics server installed on your cluster.
Like the warning says: heapster is being deprecated so the recommended choice now is the metrics server.
So follow the directions here to install the metrics server