I googled and searched for the answer to my dilemma all answers I could find are not applicable, but they say this has been discussed many times.
Below is my actual cluster setup. 4 worker nodes, two masters, and one load balancer.
I installed the dashboard
XXXX#master01:~$ kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
default busybox 1/1 Running 30 30h
kube-system coredns-78cb77577b-lbp87 1/1 Running 0 30h
kube-system coredns-78cb77577b-n7rvg 1/1 Running 0 30h
kube-system weave-net-d9jb6 2/2 Running 7 31h
kube-system weave-net-nsqss 2/2 Running 0 39h
kube-system weave-net-wnbq7 2/2 Running 7 31h
kube-system weave-net-zfsmn 2/2 Running 0 39h
kubernetes-dashboard dashboard-metrics-scraper-7b59f7d4df-dhcpn 1/1 Running 0 28h
kubernetes-dashboard kubernetes-dashboard-665f4c5ff-6qnzp 1/1 Running 7 28h
I installed my service accounts and assigned them cluster-admin roles
XXXX#master01:~$ kubectl get sa -n kubernetes-dashboard
NAME SECRETS AGE
default 1 28h
kube-apiserver 1 25h
kubernetes-dashboard 1 28h
I am using the kube-apiserver user service account because it was easy to just load the certs in the browser I already have them.
Now I try to access the dashboard using the load balancer:
https://loadbalancer.local:6443/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/
at this point one would think I should get the dashboard and every question I have encountered makes that assumption but I am getting the following error:
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {
},
"status": "Failure",
"message": "error trying to reach service: dial tcp 10.36.0.1:8443: i/o timeout",
"code": 500
}
so I decided to pull the logs:
kubectl logs -n kubernetes-dashboard service/kubernetes-dashboard
Error from server: Get "https://worker04:10250/containerLogs/kubernetes-dashboard/kubernetes-dashboard-665f4c5ff-6qnzp/kubernetes-dashboard": x509: certificate signed by unknown
authority
all I get is this one line and I had an idea of finding out what the issue is with the certification from this worker node: worker04:10250
I used OpenSSL to check the certificate and I discovered the following:
worker04 has generated its own certificate alright, but it also generated its own CA as well.
and this is where I am with no idea how to fix this and bring up a dashboard.
I also tried a proxy on master01:
kubectl -v=9 proxy --port=8001 --address=192.168.1.24
and all I got was 403 Forbidden!
I made some progress with this, I figured out that when a node generate and registers itself to a cluster, it is generating its own certificate CSR signed by its own generated CA, to fix this I generated the certificates for all the nodes signed by the cluster CA and simply replaced the auto generated certificates and restarted the nodes..
Related
I installed k8s dashboard as written in "https://github.com/kubernetes/dashboard".
However, I cannot access this URL.
http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/
I receive
「Error trying to reach service: 'proxyconnect tcp: dial tcp: lookup $http_proxy: no such host」.
(I can access this URL.
http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/)
Here is my status. My cluster seem to be no problem. (And I'm under proxy circumstances, but there is no problem docker pull.)
# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master Ready master 43h v1.17.1
k8s-worker Ready <none> 39h v1.17.2
# kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-6955765f44-g9nzl 1/1 Running 1 44h
kube-system coredns-6955765f44-vxldl 1/1 Running 1 44h
kube-system etcd-k8s-master 1/1 Running 4 44h
kube-system kube-apiserver-k8s-master 1/1 Running 3 44h
kube-system kube-controller-manager-k8s-master 1/1 Running 3 44h
kube-system kube-proxy-5bzwf 1/1 Running 3 44h
kube-system kube-proxy-5z84p 1/1 Running 0 40h
kube-system kube-scheduler-k8s-master 1/1 Running 4 44h
kube-system weave-net-l5ptw 2/2 Running 3 40h
kube-system weave-net-qcwk2 2/2 Running 0 40h
kubernetes-dashboard dashboard-metrics-scraper-7b64584c5c-wxsd2 1/1 Running 0 57m
kubernetes-dashboard kubernetes-dashboard-566f567dc7-b88n2 1/1 Running 0 57m
# docker info | grep Proxy
HTTP Proxy: http://10.227.74.6:3128
HTTPS Proxy: http://10.227.74.6:3128
No Proxy: localhost,127.0.0.1,10.227.74.4,10.227.74.37,10.96.0.0/12,10.244.0.0/16
# cat /etc/bashrc
export http_proxy=http://10.227.74.6:3128
export https_proxy=http://10.227.74.6:3128
export no_proxy="localhost,127.0.0.1,10.227.74.4,10.227.74.37,10.96.0.0/12,10.244.0.0/16"
I imagine my proxy setting is not good, but I'm not sure how I should fix it.
I'm sorry I'm not good at English. Please answer this question.
# curl http://localhost:8001/api/v1/namespaces/kubernetes-
dashboard/services/
{
"kind": "ServiceList",
"apiVersion": "v1",
"metadata": {
"selfLink": "/api/v1/namespaces/kubernetes-dashboard/services/",
"resourceVersion": "448220"
},
"items": [
{
"metadata": {
"name": "dashboard-metrics-scraper",
...........
# curl http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/
Error trying to reach service: 'proxyconnect tcp: dial tcp: lookup $http_proxy: no such host'
# curl http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/#/login
Error trying to reach service: 'proxyconnect tcp: dial tcp: lookup $http_proxy: no such host'
It might be an intermittent issues due to your corporate proxy settings. The URL is working for me.
http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/#/login
First thing which I noticed is that you use the same url for http and https:
# docker info | grep Proxy
HTTP Proxy: http://10.227.74.6:3128
HTTPS Proxy: http://10.227.74.6:3128
For HTTPS Proxy url should start with https.
Modify /etc/sysconfig/docker file, example:
# cat /etc/sysconfig/docker
HTTP_PROXY="http://10.227.74.6:3128"
HTTPS_PROXY="http://10.227.74.6:3128"
NO_PROXY="localhost,127.0.0.1,10.227.74.4,10.227.74.37,10.96.0.0/12,10.244.0.0/16"
Then configure /etc/bashrc file:
# cat /etc/bashrc
export http_proxy="http://10.227.74.6:3128"
export https_proxy="https://10.227.74.6:3128"
export no_proxy="localhost,127.0.0.1,10.227.74.4,10.227.74.37,10.96.0.0/12,10.244.0.0/16"
After applying changes restart the Docker daemon.
Please follow those steps during proxy setup: proxy-configuration.
More information about setting proxy you can find here: proxy-server.
I have a Minikube Kubernetes cluster running a cockroachdb which looks like:
kubectl get pods
test-cockroachdb-0 1/1 Running 17 95m
test-cockroachdb-1 1/1 Running 190 2d
test-cockroachdb-2 1/1 Running 160 2d
test-cockroachdb-init-m8rzp 0/1 Completed 0 2d
cockroachdb-client-secure 1/1 Running 0 2d
I want to get a connection string that I can use in my application.
To verify my connection string, I am using the tool DBeaver.
My database name is configured to 'defaultdb' which exists on my cluster, and the user with the relevant password. The port is accurate as well (default cockroachdb minikube port).
However as to the certificate aspect of connecting I am at a loss. How do I generate/gather the certificates I need to successfully connect to my cluster? How do I connect to my cluster using DBeaver?
Edit:
$ kubectl get all
NAME READY STATUS RESTARTS AGE
pod/myname-cockroachdb-0 1/1 Running 27 156m
pod/myname-cockroachdb-1 1/1 Running 197 2d1h
pod/myname-cockroachdb-2 1/1 Running 167 2d1h
pod/myname-cockroachdb-init-m8rzp 0/1 Completed 0 2d1h
pod/myname-client-secure 1/1 Running 0 2d1h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/myname-cockroachdb ClusterIP None <none> 26257/TCP,8080/TCP 2d1h
service/myname-cockroachdb-public ClusterIP 10.xxx.xxx.xx <none> 26257/TCP,8080/TCP 2d1h
service/kubernetes ClusterIP 10.xx.0.1 <none> 443/TCP 2d1h
NAME READY AGE
statefulset.apps/myname-cockroachdb 3/3 2d1h
NAME COMPLETIONS DURATION AGE
job.batch/myname-cockroachdb-init 1/1 92s 2d1h
Like #FL3SH already said.
You can use kubectl port-forward <pod_name> <port>
This is nicely explained in Cockroach documentation Step 4. Access the Admin UI, please us it as example and set different ports.
As for the certificates:
As each pod is created, it issues a Certificate Signing Request, or CSR, to have the node's certificate signed by the Kubernetes CA. You must manually check and approve each node's certificates, at which point the CockroachDB node is started in the pod.
Get the name of the Pending CSR for the first pod:
kubectl get csr
NAME AGE REQUESTOR CONDITION
default.node.cockroachdb-0 1m system:serviceaccount:default:default Pending
node-csr-0Xmb4UTVAWMEnUeGbW4KX1oL4XV_LADpkwjrPtQjlZ4 4m kubelet Approved,Issued
node-csr-NiN8oDsLhxn0uwLTWa0RWpMUgJYnwcFxB984mwjjYsY 4m kubelet Approved,Issued
node-csr-aU78SxyU69pDK57aj6txnevr7X-8M3XgX9mTK0Hso6o 5m kubelet Approved,Issued
If you do not see a Pending CSR, wait a minute and try again.
You can check the CSR pod kubectl describe csr default.node.cockroachdb-0
It might look like this:
Name: default.node.cockroachdb-0
Labels: <none>
Annotations: <none>
CreationTimestamp: Thu, 09 Nov 2017 13:39:37 -0500
Requesting User: system:serviceaccount:default:default
Status: Pending
Subject:
Common Name: node
Serial Number:
Organization: Cockroach
Subject Alternative Names:
DNS Names: localhost
cockroachdb-0.cockroachdb.default.svc.cluster.local
cockroachdb-public
IP Addresses: 127.0.0.1
10.48.1.6
Events: <none>
If it does then you can approve the certificate using:
kubectl certificate approve default.node.cockroachdb-0
Please do follow the Orchestrate CockroachDB in a Single Kubernetes Cluster guide.
Let me know if you need any further help.
You can use kubectl port-forward service/myname-cockroachdb 26257 and in DBeaver just use localhost:26257 as a connection string.
I am working on a tutorial on Create a Kubernetes service to point to the Ambassador deployment.
Tutorial: https://www.bogotobogo.com/DevOps/Docker/Docker-Envoy-Ambassador-API-Gateway-for-Kubernetes.php
On running the command
curl $(minikube service --url ambassador)/httpbin/ip
I'm getting error
curl: (7) Failed to connect to 192.168.99.100 port 30790: Connection refused
curl: (3) <url> malformed
I can actually remove the error of
curl: (3) <url> malformed
by running
minikube service --url ambassador
http://192.168.99.100:30790
and then
curl http://192.168.99.100:30790/httpbin/ip
I've already tried this answer curl: (7) Failed to connect to 192.168.99.100 port 31591: Connection refused also the step mentioned in this answer is already in the blog, and it didn't work.
This is the code from the blog, for ambassador-svc.yaml
---
apiVersion: v1
kind: Service
metadata:
labels:
service: ambassador
name: ambassador
annotations:
getambassador.io/config: |
---
apiVersion: ambassador/v0
kind: Mapping
name: httpbin_mapping
prefix: /httpbin/
service: httpbin.org:80
host_rewrite: httpbin.org
spec:
type: LoadBalancer
ports:
- name: ambassador
port: 80
targetPort: 80
selector:
service: ambassador
Can this be a problem related to VM?
Also, I tried to work on this tutorial first but unfortunately, got the same error.
Let me know if anything else is needed from my side.
Edit:
1.As asked in the comment here is the output of
kubectl get pods --namespace=kube-system
NAME READY STATUS RESTARTS AGE
coredns-fb8b8dccf-qkxwm 1/1 Running 0 5h16m
coredns-fb8b8dccf-rrn4f 1/1 Running 0 5h16m
etcd-minikube 1/1 Running 0 5h15m
kube-addon-manager-minikube 1/1 Running 4 5h15m
kube-apiserver-minikube 1/1 Running 0 5h15m
kube-controller-manager-minikube 1/1 Running 0 3h17m
kube-proxy-wfbxs 1/1 Running 0 5h16m
kube-scheduler-minikube 1/1 Running 0 5h15m
storage-provisioner 1/1 Running 0 5h16m
after running
kubectl apply -f https://docs.projectcalico.org/v3.7/manifests/calico.yaml
NAME READY STATUS RESTARTS AGE
calico-kube-controllers-78f8f67c4d-zqtl2 1/1 Running 0 65s
calico-node-27lcq 1/1 Running 0 65s
coredns-fb8b8dccf-qkxwm 1/1 Running 2 22h
coredns-fb8b8dccf-rrn4f 1/1 Running 2 22h
etcd-minikube 1/1 Running 1 22h
kube-addon-manager-minikube 1/1 Running 5 22h
kube-apiserver-minikube 1/1 Running 1 22h
kube-controller-manager-minikube 1/1 Running 0 8m27s
kube-proxy-wfbxs 1/1 Running 1 22h
kube-scheduler-minikube 1/1 Running 1 22h
storage-provisioner 1/1 Running 2 22h
kubectl get pods --namespace=kube-system should have the network service pod
So you have not set up networking policy to used for DNS.
Try using Network Policy Calico
by using command
kubectl apply -f https://docs.projectcalico.org/v3.7/manifests/calico.yaml
check now kubectl get pods --namespace=kube-system
You should get output like this :-
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-kube-controllers-6ff88bf6d4-tgtzb 1/1 Running 0 2m45s
kube-system calico-node-24h85 1/1 Running 0 2m43s
kube-system coredns-846jhw23g9-9af73 1/1 Running 0 4m5s
kube-system coredns-846jhw23g9-hmswk 1/1 Running 0 4m5s
kube-system etcd-jbaker-1 1/1 Running 0 6m22s
kube-system kube-apiserver-jbaker-1 1/1 Running 0 6m12s
kube-system kube-controller-manager-jbaker-1 1/1 Running 0 6m16s
kube-system kube-proxy-8fzp2 1/1 Running 0 5m16s
kube-system kube-scheduler-jbaker-1 1/1 Running 0 5m41s
Here is a checklist to troubleshoot the problem:
Have you created at least one Listener (CRD) ?
If you don't create at least one Listener the edge-stack pod won't respond on any ports for a request. So create the Listeners object as reported in the official doc: https://www.getambassador.io/docs/edge-stack/latest/tutorials/getting-started/
Note that the default port defined in the Ambassador Deployment and in turn in the edge-stack pod are 8080 and 8443 so avoid to change them unless you know what you are doing. You can check if any existing Listener is defined with this command: kubectl get Listener -n ambassador or just kubectl get Listener if maybe you used the default namespace by mistake.
Have you defined at least one Service of type LoadBalancer which reference the Ambassador service ?
Normally this is not needed because once you deployed Ambassador on your node you should have the Service edge-stack in the Namespace ambassador which is configured as LoadBalancer so it exposes some ports in the range 30000-32767 and redirect the traffic to the 8080 and 8443. If you manually created a Service of type LoadBalancer make sure to use the right ports in the port and targetPort fields. The ports are those used by the edge-stack Deployment which are by default 8080 and 8443)
Check that the edge-stack pod is responding to the request on it's port
The easiest way is to use the web based UI of Kubernates or your cloud provider and exec a bash into the edge-stack pod. With the default Kubernates dashboard you can just reach the pod, click on the vertical three dots -> Exec.
If you are running ambassador locally you can use eval $(minikube docker-env), search the related container with docker ps | grep edge-stack and once you got its id you can exec the bash with the command docker exec -it <id> bash
Finally run curl -Lki http://127.0.0.1:8080 You shoudl get something like this: HTTP/1.1 404 Not Founddate: Tue, XX Xxx XXXX XX:XX:XX XXXserver: envoycontent-length: 0
If curl get a response instead of "connection refused" the service is running successfully on the pod and the problem must be in the Service, Deployment or Listener configuration. You can try to use the Cluster IP of the Service edge-stack instead of 127.0.0.1 but you need to run the curl command from a pod different than edge-stack. You can use your own pod or exec into the other ambassador pods which are: edge-stack-agent and edge-stack-redis.
I'm new in Kubernetes and I have a code error 403 trying the access.
kubectl cluster info
Kubernetes master is running at https://x.x.x.x:6443
KubeDNS is running at https://x.x.x.x:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
"status": "Failure",
"message": "namespaces is forbidden: User \"system:anonymous\" cannot list namespaces at the cluster scope",
"reason": "Forbidden",
"details": {
"kind": "namespaces"
},
"code": 403
kubectl get pods --all-namespaces
kube-system calico-etcd-6629s 1/1 Running 0 10h
kube-system calico-kube-controllers-675684d4bb-5h28d 1/1 Running 0 10h
kube-system calico-node-r75wv 2/2 Running 0 10h
kube-system etcd-sp2013a.... 1/1 Running 0 10h
kube-system kube-apiserver-sp2013a ... 1/1 Running 0 10h
kube-system kube-controller-manager-sp2013a.... 1/1 Running 0 10h
kube-system kube-dns-6f4....df-fcqvt 3/3 Running 0 10h
kube-system kube-proxy-mpf2j 1/1 Running 0 10h
kube-system kube-scheduler-sp2013a...... 1/1 Running 0 10h
everything is running..
That sounds like you're being blocked by the cluster's RBAC policies. The system:anonymous user is being prevented from listing the namespaces in the cluster.
(Along the lines of kubectl get namespaces)
Running kubectl create clusterrolebinding cluster-system-anonymous --clusterrole=cluster-admin --user=system:anonymous would create a clusterrolebinding that adds the system:anonymous user to the cluster-admin role.
Blindly elevating accounts to cluster-admin is not recommended in a production cluster but since you are new, this should get you up and running.
All clusters need some form of authorization before accessing the API-server (accessing kubectl) like certificate authentication. RBAC is a way to limit the actions that users (both human users and service accounts) can take in the cluster.
A great RBAC primer from the CNCF can be found here and the official docs are great too! Good Luck!
I have installed kubernetes using minikube on ubuntu 16.04 machine.
I have also installed kubernetes-dashboard.
When i try accessing the dashboard i get
Waiting, endpoint for service is not registered yet
Waiting, endpoint for service is not ready yet...
Waiting, endpoint for service is not ready yet...
Waiting, endpoint for service is not ready yet...
.....
Could not find finalized endpoint being pointed to by kubernetes-dashboard: Temporary Error: Endpoint for service is not ready yet
Temporary Error: Endpoint for service is not ready yet
Temporary Error: Endpoint for service is not ready yet
Temporary Error: Endpoint for service is not ready yet
Temporary Error: Endpoint for service is not ready yet
Temporary Error: Endpoint for service is not ready yet
`
However, when i try a kubectl get pods --all namespacesi get the below output
kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system kube-addon-manager-minikube 1/1 Running 0 11m
kube-system kube-dns-1301475494-xtb3b 3/3 Running 0 8m
kube-system kubernetes-dashboard-2039414953-dvv3m 1/1 Running 0 9m
kube-system kubernetes-dashboard-2crsk 1/1 Running 0 8m
kubectl get endpoints --all-namespaces
NAMESPACE NAME ENDPOINTS AGE
default kubernetes 10.0.2.15:8443 11m
kube-system kube-controller-manager <none> 6m
kube-system kube-dns 172.17.0.4:53,172.17.0.4:53 8m
kube-system kube-scheduler <none> 6m
kube-system kubernetes-dashboard <none> 9m
How can i fix this issue? I don't seem to understand what is wrong. I am completely new to kubernetes
You need to run minikube dashboard. You shouldn't install dashboard separately; it comes with minikube.
some of the minikube commands
./minikube.exe version
./minikube.exe delete
./minikube.exe start --help
./minikube get-k8s-versions
./minikube.exe status
./minikube.exe ip
./minikube.exe dashboard --url=true