What can be the cause for helm error "TILLER: unexpected EOF" - kubernetes

we execute in our automated pipelines this command:
helm list --namespace foobar
We experience rarely that error:
Error: Get https://172.21.0.1:443/api/v1/namespaces/kube-system/configmaps?labelSelector=OWNER%!D(MISSING)TILLER: unexpected EOF
The next run is working fine. in between we didn't change any thing.
So we wonder, what the reason could be. Issue we found, which are more or less similar does not explain it, at least as far we understand them. As it seems to be a random error.
We found these issues:
helm still ignores kubeconfig #2520 - https://github.com/helm/helm/issues/2520
Error: Get http://localhost:8080/api/v1/namespaces/kube-system/configmaps?labelSelector=OWNER%!D(MISSING)TILLER: dial tcp 127.0.0.1:8080: connect: connection refused #4020 - https://github.com/helm/helm/issues/4020
Connection refused error when issuing helm install command #3460 - https://github.com/helm/helm/issues/3460
Helm list Error: Get http://localhost:8080/api/v1/namespaces/kube-system/configmaps?labelSelector=OWNER%!D(MISSING)TILLER: dial tcp 127.0.0.1:8080: connect: connection refused #3870 - https://github.com/helm/helm/issues/3870
Tiller pods can't connect to k8s apiserver #2464 - https://github.com/helm/helm/issues/2464

Related

Getting Readiness problem failed due to connection refused in tidb pods only in tidb cluster

We have been running tidb cluster in k8s. and its working fine since. But suddenly i am getting following issue only in new statsfull pods tidb-tidb-1 after scaling tidb-tidb statsfulset. Interestingly tidb-tidb-2 is running. All others pd and tikv pods are also running fine.I have checked the pd url which is not reachable from problematic pods but fine for other pods.Can you please help me to solve this issue.
tidb-tidb-1 logs:
[2021/04/11 16:15:44.526 +00:00] [WARN] [base_client.go:180] ["[pd] failed to get cluster id"]
[2021/04/11 16:15:48.527 +00:00] [WARN] [base_client.go:180] ["[pd] failed to get cluster id"] [error="[PD:client:ErrClientGetMember]error:rpc error: code = DeadlineExceeded desc = latest connection error: connection error: desc = \"transport: Error while dialing dial tcp: i/o timeout\" target:test-tidb-pd:2379 status:CONNECTING
Could you please show namespace information?
kubectl get all -n -o wide
Please check node information.
https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#node-isolation-restriction
Please check the network. If two nodes could ping successful?
transport: Error while dialing dial TCP: i/o timeout

fluxcd not applying anything with err="running kubectl: error: unable to recognize \"STDIN\": ..."

I recently installed FluxCD 1.19.0 on an Azure AKS k8s cluster using fluxctl install. We use a private git (self hosted bitbucket) which Flux is able to reach and check out.
Now Flux is not applying anything with the error message:
ts=2020-06-10T09:07:42.7589883Z caller=loop.go:133 component=sync-loop event=refreshed url=ssh://git#bitbucket.some-private-server.com:7999/infra/k8s-gitops.git branch=master HEAD=7bb83d1753a814c510b1583da6867408a5f7e21b
ts=2020-06-10T09:09:00.631764Z caller=sync.go:73 component=daemon info="trying to sync git changes to the cluster" old=7bb83d1753a814c510b1583da6867408a5f7e21b new=7bb83d1753a814c510b1583da6867408a5f7e21b
ts=2020-06-10T09:09:01.6130559Z caller=sync.go:539 method=Sync cmd=apply args= count=3
ts=2020-06-10T09:09:20.2097034Z caller=sync.go:605 method=Sync cmd="kubectl apply -f -" took=18.5965923s err="running kubectl: error: unable to recognize \"STDIN\": an error on the server (\"\") has prevented the request from succeeding" output=
ts=2020-06-10T09:09:38.7432182Z caller=sync.go:605 method=Sync cmd="kubectl apply -f -" took=18.5334244s err="running kubectl: error: unable to recognize \"STDIN\": an error on the server (\"\") has prevented the request from succeeding" output=
ts=2020-06-10T09:09:57.277918Z caller=sync.go:605 method=Sync cmd="kubectl apply -f -" took=18.5346491s err="running kubectl: error: unable to recognize \"STDIN\": an error on the server (\"\") has prevented the request from succeeding" output=
ts=2020-06-10T09:09:57.2779965Z caller=sync.go:167 component=daemon err="<cluster>:namespace/dev: running kubectl: error: unable to recognize \"STDIN\": an error on the server (\"\") has prevented the request from succeeding; <cluster>:namespace/prod: running kubectl: error: unable to recognize \"STDIN\": an error on the server (\"\") has prevented the request from succeeding; dev:service/hello-world: running kubectl: error: unable to recognize \"STDIN\": an error on the server (\"\") has prevented the request from succeeding"
ts=2020-06-10T09:09:57.2879489Z caller=images.go:17 component=sync-loop msg="polling for new images for automated workloads"
ts=2020-06-10T09:09:57.3002208Z caller=images.go:27 component=sync-loop msg="no automated workloads"
From what I understand, Flux passes the resource definitions to kubectl, which then applies them?
The way I interpret the error would mean that kubectl isn't passed anything to. However I opened a shell in the container and made sure Flux was in fact checking something out - which it did.
I tried raising the verbosity to 9, but it didn't return anything that I deemed relevant (detailed outputs of the http requests and responses against the Kubernetes API).
So what is happening here?
The problem was with the version of kubectl used in the 1.19 flux release, so I fixed it by using a prerelease: https://hub.docker.com/r/fluxcd/flux-prerelease/tags

Minishift: Problems starting

I am trying to get minishift runnin on my machine (Windows 10) with Virtualbox 5.1.24.
Minishift version: 1.0.0+4f8cb6d
CDK Version: 3.0.0-2
Starting minishift gives me the following:
C:\>minishift start --vm-driver virtualbox
Starting local OpenShift cluster using 'virtualbox' hypervisor...
E0727 18:34:21.682796 17204 start.go:176] Error starting the VM: Error
creating new host: Error attempting to get plugin server address for RPC:
Failed to dial the plugin server in 10s. Retrying.
E0727 18:34:31.740746 17204 start.go:176] Error starting the VM: Error
creating new host: Error attempting to get plugin server address for RPC:
Failed to dial the plugin server in 10s. Retrying.
E0727 18:34:41.770667 17204 start.go:176] Error starting the VM: Error
creating new host: Error attempting to get plugin server address for RPC:
Failed to dial the plugin server in 10s. Retrying.
Error starting the VM: Error creating new host: Error attempting to get
plugin server address for RPC: Failed to dial the plugin server in 10s
Error creating new host: Error attempting to get plugin server address for
RPC: Failed to dial the plugin server in 10s
Error creating new host: Error attempting to get plugin server address for
RPC: Failed to dial the plugin server in 10s
I read the comments that it needs to run from the C:\drive but it looks like this did not fix the problem. I am happy about any hints how to fix this. If there is any additional information you need, just let me know.
Sounds like you got it working.
I usually encourage folks who are having trouble starting their minishift VMs to try the following:
Find your preferred virtualization provider from the list of available options
Install the appropriate driver plugin for your system
Persist your VM provider configuration: minishift config set vm-driver virualbox

Error when running "bee run -e=Godep"

I encountered this error message after running "bee run -e=Godeps":
[ORM] registerdb Ping 'default', dial tcp 127.0.0.1:5432: connectex: No connection could be made because the targer machine actively refused it. must have one register Database alias named 'default'
could anyone give me some idea on how to fix it? thanks in advance!