Tried to configure kube-ops-view on a local cluster created using Kubernetes KinD not able to access it.
helm install kube-ops-view stable/kube-ops-view
WARNING: This chart is deprecated
NAME: kube-ops-view
LAST DEPLOYED: Wed Dec 2 15:05:45 2020
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
To access the Kubernetes Operational View UI:
1. First start the kubectl proxy:
kubectl proxy
2. Now open the following URL in your browser:
http://localhost:8001/api/v1/proxy/namespaces/default/services/kube-ops-view/
Please try reloading the page if you see "ServiceUnavailable / no endpoints available for service", pod creation might take a moment.
kubectl proxy
Starting to serve on 127.0.0.1:8001
kubectl get pods
NAME READY STATUS RESTARTS AGE
kube-ops-view-7bc8944b46-nmc8k 1/1 Running 0 5m9s
kubectl get service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kube-ops-view ClusterIP 10.96.242.129 <none> 80/TCP 5m28s
kubectl get deployment
NAME READY UP-TO-DATE AVAILABLE AGE
kube-ops-view 1/1 1 1 5m48
kubectl describe deployment kube-ops-view
Name: kube-ops-view
Namespace: default
CreationTimestamp: Wed, 02 Dec 2020 15:05:45 +0800
Labels: app.kubernetes.io/instance=kube-ops-view
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=kube-ops-view
app.kubernetes.io/part-of=kube-ops-view
app.kubernetes.io/version=20.4.0
helm.sh/chart=kube-ops-view-1.2.4
Annotations: deployment.kubernetes.io/revision: 1
meta.helm.sh/release-name: kube-ops-view
meta.helm.sh/release-namespace: default
Selector: app.kubernetes.io/name=kube-ops-view,app.kubernetes.io/part-of=kube-ops-view
Replicas: 1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType: RollingUpdate
MinReadySeconds: 0
RollingUpdateStrategy: 25% max unavailable, 25% max surge
Pod Template:
Labels: app.kubernetes.io/instance=kube-ops-view
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=kube-ops-view
app.kubernetes.io/part-of=kube-ops-view
app.kubernetes.io/version=20.4.0
helm.sh/chart=kube-ops-view-1.2.4
Service Account: default
Containers:
kube-ops-view:
Image: hjacobs/kube-ops-view:20.4.0
Port: 8080/TCP
Host Port: 0/TCP
Limits:
cpu: 100m
memory: 128Mi
Requests:
cpu: 80m
memory: 64Mi
Liveness: http-get http://:8080/health delay=0s timeout=1s period=10s #success=1 #failure=3
Readiness: http-get http://:8080/health delay=0s timeout=1s period=10s #success=1 #failure=3
Environment: <none>
Mounts: <none>
Volumes: <none>
Conditions:
Type Status Reason
---- ------ ------
Available True MinimumReplicasAvailable
Progressing True NewReplicaSetAvailable
OldReplicaSets: <none>
NewReplicaSet: kube-ops-view-7bc8944b46 (1/1 replicas created)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ScalingReplicaSet 5m58s deployment-controller Scaled up replica set kube-ops-view-7bc8944b46 to 1
Trying to access using the URL but it's not working
http://localhost:8001/api/v1/proxy/namespaces/default/services/kube-ops-view/
Got it fixed
Deleted the kube-ops-view deployment and svc.
helm install --set rbac.create=true --set ingress.enabled=true kube-ops-view stable/kube-ops-view
Related
Currently I have a cluster with single controller and single worker, also a nginx reverse-proxy (hhtp only) outside cluster.
Controller is at 192.168.1.65
worker is at 192.168.1.61
reverse proxy at 192.168.1.93 and public ip
here is my ingress-nginx services
bino#corobalap ~/k0s-sriwijaya/ingress-nginx/testapp kubectl -n ingress-nginx get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-nginx-controller LoadBalancer 10.102.58.7 192.168.1.186 80:31097/TCP,443:31116/TCP 56m
ingress-nginx-controller-admission ClusterIP 10.108.233.49 <none> 443/TCP 56m
bino#corobalap ~/k0s-sriwijaya/ingress-nginx/testapp kubectl -n ingress-nginx describe svc ingress-nginx-controller
Name: ingress-nginx-controller
Namespace: ingress-nginx
Labels: app.kubernetes.io/component=controller
app.kubernetes.io/instance=ingress-nginx
app.kubernetes.io/name=ingress-nginx
app.kubernetes.io/part-of=ingress-nginx
app.kubernetes.io/version=1.3.0
Annotations: <none>
Selector: app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
Type: LoadBalancer
IP Family Policy: SingleStack
IP Families: IPv4
IP: 10.102.58.7
IPs: 10.102.58.7
LoadBalancer Ingress: 192.168.1.186
Port: http 80/TCP
TargetPort: http/TCP
NodePort: http 31097/TCP
Endpoints: 10.244.0.23:80
Port: https 443/TCP
TargetPort: https/TCP
NodePort: https 31116/TCP
Endpoints: 10.244.0.23:443
Session Affinity: None
External Traffic Policy: Cluster
Events: <none>
that 192.168.1.186 is assigned by MetalLB.
bino#corobalap ~/k0s-sriwijaya/ingress-nginx/testapp kubectl get IPAddressPools -A
NAMESPACE NAME AGE
metallb-system pool01 99m
bino#corobalap ~/k0s-sriwijaya/ingress-nginx/testapp kubectl -n metallb-system describe IPAddressPool pool01
Name: pool01
Namespace: metallb-system
Labels: <none>
Annotations: <none>
API Version: metallb.io/v1beta1
Kind: IPAddressPool
Metadata:
Creation Timestamp: 2022-07-26T09:08:10Z
Generation: 1
Managed Fields:
API Version: metallb.io/v1beta1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:kubectl.kubernetes.io/last-applied-configuration:
f:spec:
.:
f:addresses:
f:autoAssign:
f:avoidBuggyIPs:
Manager: kubectl-client-side-apply
Operation: Update
Time: 2022-07-26T09:08:10Z
Resource Version: 41021
UID: 2a0dcfb2-bf8f-4b1a-b459-380e78959586
Spec:
Addresses:
192.168.1.186 - 192.168.1.191
Auto Assign: true
Avoid Buggy I Ps: false
Events: <none>
I deploy hello-app at namespace : 'dev'
bino#corobalap ~/k0s-sriwijaya/ingress-nginx/testapp kubectl -n dev get all
NAME READY STATUS RESTARTS AGE
pod/hello-app-5c554f556c-v2gx9 1/1 Running 1 (20m ago) 63m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/hello-service ClusterIP 10.111.161.2 <none> 8081/TCP 62m
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/hello-app 1/1 1 1 63m
NAME DESIRED CURRENT READY AGE
replicaset.apps/hello-app-5c554f556c 1 1 1 63m
bino#corobalap ~/k0s-sriwijaya/ingress-nginx/testapp kubectl -n dev describe service hello-service
Name: hello-service
Namespace: dev
Labels: app=hello
Annotations: <none>
Selector: app=hello
Type: ClusterIP
IP Family Policy: SingleStack
IP Families: IPv4
IP: 10.111.161.2
IPs: 10.111.161.2
Port: <unset> 8081/TCP
TargetPort: 8080/TCP
Endpoints: 10.244.0.22:8080
Session Affinity: None
Events: <none>
Local tests of that service:
bino#k8s-worker-1:~$ curl http://10.111.161.2:8081
Hello, world!
Version: 2.0.0
Hostname: hello-app-5c554f556c-v2gx9
bino#k8s-worker-1:~$ curl http://10.244.0.22:8080
Hello, world!
Version: 2.0.0
Hostname: hello-app-5c554f556c-v2gx9
and the ingress resource of that service:
bino#corobalap ~/k0s-sriwijaya/ingress-nginx/testapp kubectl -n dev describe ingress hello-app-ingress
Name: hello-app-ingress
Labels: <none>
Namespace: dev
Address: 192.168.1.61
Ingress Class: nginx
Default backend: <default>
Rules:
Host Path Backends
---- ---- --------
bino.k8s.jcamp.cloud
/ hello-service:8081 (10.244.0.22:8080)
Annotations: ingress.kubernetes.io/rewrite-target: /
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Sync 23m (x3 over 24m) nginx-ingress-controller Scheduled for sync
When I open http://bino.k8s.jcamp.cloud I got 502
my nginx reverse proxy conf :
server {
listen 80 default_server;
location / {
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_pass http://192.168.1.186;
}
}
The nginx error log say
2022/07/26 06:24:21 [error] 1593#1593: *6 connect() failed (113: No route to host) while connecting to upstream, client: 203.161.185.210, server: , request: "GET /favicon.ico HTTP/1.1", upstream: "http://192.168.1.186:80/favicon.ico", host: "bino.k8s.jcamp.cloud", referrer: "http://bino.k8s.jcamp.cloud/"
from describe ingress-nginx-controller pod
bino#corobalap ~/k0s-sriwijaya/ingress-nginx/testapp kubectl -n ingress-nginx describe pod ingress-nginx-controller-6dc865cd86-9fmsk
Name: ingress-nginx-controller-6dc865cd86-9fmsk
Namespace: ingress-nginx
Priority: 0
Node: k8s-worker-1/192.168.1.61
Start Time: Tue, 26 Jul 2022 16:11:05 +0700
Labels: app.kubernetes.io/component=controller
app.kubernetes.io/instance=ingress-nginx
app.kubernetes.io/name=ingress-nginx
pod-template-hash=6dc865cd86
Annotations: kubernetes.io/psp: 00-k0s-privileged
Status: Running
IP: 10.244.0.23
IPs:
IP: 10.244.0.23
Controlled By: ReplicaSet/ingress-nginx-controller-6dc865cd86
Containers:
controller:
Container ID: containerd://541446c98b55312376aba4744891baa325dca26410abe5f94707d270d378d881
Image: registry.k8s.io/ingress-nginx/controller:v1.3.0#sha256:d1707ca76d3b044ab8a28277a2466a02100ee9f58a86af1535a3edf9323ea1b5
Image ID: registry.k8s.io/ingress-nginx/controller#sha256:d1707ca76d3b044ab8a28277a2466a02100ee9f58a86af1535a3edf9323ea1b5
Ports: 80/TCP, 443/TCP, 8443/TCP
Host Ports: 0/TCP, 0/TCP, 0/TCP
Args:
/nginx-ingress-controller
--election-id=ingress-controller-leader
--controller-class=k8s.io/ingress-nginx
--ingress-class=nginx
--configmap=$(POD_NAMESPACE)/ingress-nginx-controller
--validating-webhook=:8443
--validating-webhook-certificate=/usr/local/certificates/cert
--validating-webhook-key=/usr/local/certificates/key
State: Running
Started: Tue, 26 Jul 2022 16:56:40 +0700
Last State: Terminated
Reason: Unknown
Exit Code: 255
Started: Tue, 26 Jul 2022 16:11:09 +0700
Finished: Tue, 26 Jul 2022 16:56:26 +0700
Ready: True
Restart Count: 1
Requests:
cpu: 100m
memory: 90Mi
Liveness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5
Readiness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
Environment:
POD_NAME: ingress-nginx-controller-6dc865cd86-9fmsk (v1:metadata.name)
POD_NAMESPACE: ingress-nginx (v1:metadata.namespace)
LD_PRELOAD: /usr/local/lib/libmimalloc.so
Mounts:
/usr/local/certificates/ from webhook-cert (ro)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-nfmrc (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
webhook-cert:
Type: Secret (a volume populated by a Secret)
SecretName: ingress-nginx-admission
Optional: false
kube-api-access-nfmrc:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: kubernetes.io/os=linux
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning NodeNotReady 44m node-controller Node is not ready
Warning FailedMount 43m kubelet MountVolume.SetUp failed for volume "webhook-cert" : object "ingress-nginx"/"ingress-nginx-admission" not registered
Warning FailedMount 43m kubelet MountVolume.SetUp failed for volume "webhook-cert" : failed to sync secret cache: timed out waiting for the condition
Warning FailedMount 43m kubelet MountVolume.SetUp failed for volume "kube-api-access-nfmrc" : failed to sync configmap cache: timed out waiting for the condition
Normal SandboxChanged 43m kubelet Pod sandbox changed, it will be killed and re-created.
Normal Pulled 43m kubelet Container image "registry.k8s.io/ingress-nginx/controller:v1.3.0#sha256:d1707ca76d3b044ab8a28277a2466a02100ee9f58a86af1535a3edf9323ea1b5" already present on machine
Normal Created 43m kubelet Created container controller
Normal Started 43m kubelet Started container controller
Warning Unhealthy 42m (x2 over 42m) kubelet Liveness probe failed: Get "http://10.244.0.23:10254/healthz": dial tcp 10.244.0.23:10254: connect: connection refused
Warning Unhealthy 42m (x3 over 43m) kubelet Readiness probe failed: Get "http://10.244.0.23:10254/healthz": dial tcp 10.244.0.23:10254: connect: connection refused
Normal RELOAD 42m nginx-ingress-controller NGINX reload triggered due to a change in configuration
and here is the nft ruleset
bino#k8s-worker-1:~$ su -
Password:
root#k8s-worker-1:~# systemctl status nftables.service
● nftables.service - nftables
Loaded: loaded (/lib/systemd/system/nftables.service; enabled; vendor preset: enabled)
Active: active (exited) since Tue 2022-07-26 05:56:17 EDT; 46min ago
Docs: man:nft(8)
http://wiki.nftables.org
Process: 186 ExecStart=/usr/sbin/nft -f /etc/nftables.conf (code=exited, status=0/SUCCESS)
Main PID: 186 (code=exited, status=0/SUCCESS)
CPU: 34ms
Warning: journal has been rotated since unit was started, output may be incomplete.
[]
Complete ruleset is at https://pastebin.com/xd58rcQp
Kindly please tell me what to do, to check, or to learn for fixing this problem
Sincerely
-bino-
my bad ...
There is a name mismatch between ip pool devinition yaml and the l2 advertisement yaml.
I am getting ServiceUnavailable error when I try to run kubectl top nodes or kubectl top pods command in EKS. I am running my cluster in EKS , and I am not finding any solution for this online. If any one have faced this issue in EKS please let me know how we can resolve this issue
Error from server (ServiceUnavailable): the server is currently unable to handle the request (get pods.metrics.k8s.io)
out put of kubectl get apiservices v1beta1.metrics.k8s.io -o yaml
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"apiregistration.k8s.io/v1","kind":"APIService","metadata":{"annotations":{},"labels":{"k8s-app":"metrics-server"},"name":"v1beta1.metrics.k8s.io"},"spec":{"group":"metrics.k8s.io","groupPriorityMinimum":100,"insecureSkipTLSVerify":true,"service":{"name":"metrics-server","namespace":"kube-system"},"version":"v1beta1","versionPriority":100}}
creationTimestamp: "2022-02-03T08:22:59Z"
labels:
k8s-app: metrics-server
name: v1beta1.metrics.k8s.io
resourceVersion: "1373088"
uid: 2066d4cb-8105-4aea-9678-8303595dc47b
spec:
group: metrics.k8s.io
groupPriorityMinimum: 100
insecureSkipTLSVerify: true
service:
name: metrics-server
namespace: kube-system
port: 443
version: v1beta1
versionPriority: 100
status:
conditions:
- lastTransitionTime: "2022-02-03T08:22:59Z"
message: 'failing or missing response from https://10.16.55.204:4443/apis/metrics.k8s.io/v1beta1:
Get "https://10.16.55.204:4443/apis/metrics.k8s.io/v1beta1": dial tcp 10.16.55.204:4443:
i/o timeout'
reason: FailedDiscoveryCheck
status: "False"
type: Available
metrics-server 1/1 1 1 3d22h
kubectl describe deployment metrics-server -n kube-system
Name: metrics-server
Namespace: kube-system
CreationTimestamp: Thu, 03 Feb 2022 09:22:59 +0100
Labels: k8s-app=metrics-server
Annotations: deployment.kubernetes.io/revision: 2
Selector: k8s-app=metrics-server
Replicas: 1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType: RollingUpdate
MinReadySeconds: 0
RollingUpdateStrategy: 0 max unavailable, 25% max surge
Pod Template:
Labels: k8s-app=metrics-server
Service Account: metrics-server
Containers:
metrics-server:
Image: k8s.gcr.io/metrics-server/metrics-server:v0.6.0
Port: 4443/TCP
Host Port: 0/TCP
Args:
--cert-dir=/tmp
--secure-port=4443
--kubelet-insecure-tls=true
--kubelet-preferred-address-types=InternalIP
--kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
--kubelet-use-node-status-port
--metric-resolution=15s
Requests:
cpu: 100m
memory: 200Mi
Liveness: http-get https://:https/livez delay=0s timeout=1s period=10s #success=1 #failure=3
Readiness: http-get https://:https/readyz delay=20s timeout=1s period=10s #success=1 #failure=3
Environment: <none>
Mounts:
/tmp from tmp-dir (rw)
Volumes:
tmp-dir:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
Priority Class Name: system-cluster-critical
Conditions:
Type Status Reason
---- ------ ------
Progressing True NewReplicaSetAvailable
Available True MinimumReplicasAvailable
OldReplicaSets: <none>
NewReplicaSet: metrics-server-5dcd6cbcb9 (1/1 replicas created)
Events: <none>
Download the components.yaml, find and replace 4443 to 443 and do a kubectl replace -f components.yaml -n kube-system --force.
I'm unable to get the controller working. Tried many times and still I get Error: ImagePullBackOff.
Is there a alternative that I can try or any idea why its failing?
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.27.0/deploy/static/mandatory.yaml
kubectl describe pod nginx-ingress-controller-7fcb6cffc5-m8m5c -n ingress-nginx
Name: nginx-ingress-controller-7fcb6cffc5-m8m5c
Namespace: ingress-nginx
Priority: 0
Node: ip-10-0-0-244.ap-south-1.compute.internal/10.0.0.244
Start Time: Mon, 07 Dec 2020 08:21:13 -0500
Labels: app.kubernetes.io/name=ingress-nginx
app.kubernetes.io/part-of=ingress-nginx
pod-template-hash=7fcb6cffc5
Annotations: kubernetes.io/limit-ranger: LimitRanger plugin set: cpu, memory request for container nginx-ingress-controller
kubernetes.io/psp: eks.privileged
prometheus.io/port: 10254
prometheus.io/scrape: true
Status: Pending
IP: 10.0.0.231
IPs:
IP: 10.0.0.231
Controlled By: ReplicaSet/nginx-ingress-controller-7fcb6cffc5
Containers:
nginx-ingress-controller:
Container ID:
Image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:master
Image ID:
Ports: 80/TCP, 443/TCP
Host Ports: 0/TCP, 0/TCP
Args:
/nginx-ingress-controller
--configmap=$(POD_NAMESPACE)/nginx-configuration
--tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
--udp-services-configmap=$(POD_NAMESPACE)/udp-services
--publish-service=$(POD_NAMESPACE)/ingress-nginx
--annotations-prefix=nginx.ingress.kubernetes.io
State: Waiting
Reason: ErrImagePull
Ready: False
Restart Count: 0
Requests:
cpu: 100m
memory: 90Mi
Liveness: http-get http://:10254/healthz delay=10s timeout=10s period=10s #success=1 #failure=3
Readiness: http-get http://:10254/healthz delay=0s timeout=10s period=10s #success=1 #failure=3
Environment:
POD_NAME: nginx-ingress-controller-7fcb6cffc5-m8m5c (v1:metadata.name)
POD_NAMESPACE: ingress-nginx (v1:metadata.namespace)
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from nginx-ingress-serviceaccount-token-xtnz9 (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
nginx-ingress-serviceaccount-token-xtnz9:
Type: Secret (a volume populated by a Secret)
SecretName: nginx-ingress-serviceaccount-token-xtnz9
Optional: false
QoS Class: Burstable
Node-Selectors: kubernetes.io/os=linux
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 19s default-scheduler Successfully assigned ingress-nginx/nginx-ingress-controller-7fcb6cffc5-m8m5c to ip-10-0-0-244.ap-south-1.compute.internal
Normal Pulling 18s kubelet Pulling image "quay.io/kubernetes-ingress-controller/nginx-ingress-controller:master"
Warning Failed 3s kubelet Failed to pull image "quay.io/kubernetes-ingress-controller/nginx-ingress-controller:master": rpc error: code = Unknown desc = Error response from daemon: Get https://quay.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Warning Failed 3s kubelet Error: ErrImagePull
Normal BackOff 3s kubelet Back-off pulling image "quay.io/kubernetes-ingress-controller/nginx-ingress-controller:master"
Warning Failed 3s kubelet Error: ImagePullBackOff
I had the same problem, with the ingress-nginx installation.
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.2.0/deploy/static/provider/cloud/deploy.yaml
For some reason it couldn't get the ingress-nginx-controller.
$ kubectl get pods --namespace=ingress-nginx
NAME READY STATUS RE
ingress-nginx-admission-create-6q4wx 0/1 Completed 0
ingress-nginx-admission-patch-fr5ct 0/1 Completed 1
ingress-nginx-controller-686556747b-dg68h 0/1 ImagePullBackOff 0
What I did was, I ran $ kubectl describe pod ingress-nginx-controller-686556747b-dg68h --namespace ingress-nginx
and got the following output:
Name: ingress-nginx-controller-686556747b-dg68h
Namespace: ingress-nginx
Priority: 0
Node: docker-desktop/x.x.x.x
Start Time: Wed, 11 May 2022 20:11:55 +0430
Labels: app.kubernetes.io/component=controller
app.kubernetes.io/instance=ingress-nginx
app.kubernetes.io/name=ingress-nginx
pod-template-hash=686556747b
Annotations: <none>
Status: Pending
IP: x.x.x.x
IPs:
IP: x.x.x.x
Controlled By: ReplicaSet/ingress-nginx-controller-686556747b
Containers:
controller:
Container ID:
Image: k8s.gcr.io/ingress-nginx/controller:v1.2.0#sha256:d819
Image ID:
Ports: 80/TCP, 443/TCP, 8443/TCP
Host Ports: 0/TCP, 0/TCP, 0/TCP
Args:
/nginx-ingress-controller
--publish-service=$(POD_NAMESPACE)/ingress-nginx-controller
--election-id=ingress-controller-leader
--controller-class=k8s.io/ingress-nginx
--ingress-class=nginx
--configmap=$(POD_NAMESPACE)/ingress-nginx-controller
--validating-webhook=:8443
--validating-webhook-certificate=/usr/local/certificates/cert
--validating-webhook-key=/usr/local/certificates/key
State: Waiting
Reason: ImagePullBackOff
Ready: False
Restart Count: 0
Requests:
cpu: 100m
memory: 90Mi
Liveness: http-get http://:10254/healthz delay=10s timeout=1s perio
Readiness: http-get http://:10254/healthz delay=10s timeout=1s perio
Environment:
POD_NAME: ingress-nginx-controller-686556747b-dg68h (v1:metad
POD_NAMESPACE: ingress-nginx (v1:metadata.namespace)
LD_PRELOAD: /usr/local/lib/libmimalloc.so
Mounts:
/usr/local/certificates/ from webhook-cert (ro)
From Containers.controller.Image, I got the image name that kubernetes is trying to download but is unsuccessful to do so and tried to docker pull that image myself like so:
docker pull k8s.gcr.io/ingress-nginx/controller:v1.2.0#sha256:d819
Docker could pull the image successfully and after that everything worked just fine.
It's failing because kubernetes cannot download the specified image. Check the events section
Warning Failed 3s kubelet Failed to pull image "quay.io/kubernetes-ingress-controller/nginx-ingress-controller:master": rpc error: code = Unknown desc = Error response from daemon: Get https://quay.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Maybe you dont have internet connectivity or this image does not exist. You can try running docker pull quay.io/kubernetes-ingress-controller/nginx-ingress-controller:master from your computer
As mentioned by John, creating a nat router and nat config allowed docker images to be pulled when I was facing the same issue. If you create a vpc native GKE cluster which is private it by default has no access to the internet. Unless you deploy a NAT router.
gcloud compute routers create nat-router \
--network my-vpc \
--region us-east4
gcloud compute routers nats create nat-config \
--router-region us-east4 \
--router nat-router \
--nat-all-subnet-ip-ranges \
--auto-allocate-nat-external-ips
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-service
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/use-regex: 'true'
spec:
rules:
- host: your host name
http:
paths:
- backend:
service:
name: your service name
port:
number: 3000
path: /api/?(.*)
pathType: Prefix
I think this YAML file solves your problem. I faced the same issue.
I install kube1.10.3 in two virtualbox(centos 7.4) in my win10 machine. I use git clone to get prometheus yaml files.
git clone https://github.com/kubernetes/kubernetes
Then I enter kubernetes/cluster/addons/prometheus annd follow this order to create pods:
alertmanager-configmap.yaml
alertmanager-pvc.yaml
alertmanager-deployment.yaml
alertmanager-service.yaml
kube-state-metrics-rbac.yaml
kube-state-metrics-deployment.yaml
kube-state-metrics-service.yaml
node-exporter-ds.yml
node-exporter-service.yaml
prometheus-configmap.yaml
prometheus-rbac.yaml
prometheus-statefulset.yaml
prometheus-service.yaml
But Prometheus and alertmanage are in pending state:
kube-system alertmanager-6bd9584b85-j4h5m 0/2 Pending 0 9m
kube-system calico-etcd-pnwtr 1/1 Running 0 16m
kube-system calico-kube-controllers-5d74847676-mjq4j 1/1 Running 0 16m
kube-system calico-node-59xfk 2/2 Running 1 16m
kube-system calico-node-rqsh5 2/2 Running 1 16m
kube-system coredns-7997f8864c-ckhsq 1/1 Running 0 16m
kube-system coredns-7997f8864c-jjtvq 1/1 Running 0 16m
kube-system etcd-master16g 1/1 Running 0 15m
kube-system heapster-589b7db6c9-mpmks 1/1 Running 0 16m
kube-system kube-apiserver-master16g 1/1 Running 0 15m
kube-system kube-controller-manager-master16g 1/1 Running 0 15m
kube-system kube-proxy-hqq49 1/1 Running 0 16m
kube-system kube-proxy-l8hmh 1/1 Running 0 16m
kube-system kube-scheduler-master16g 1/1 Running 0 16m
kube-system kube-state-metrics-8595f97c4-g6x5x 2/2 Running 0 8m
kube-system kubernetes-dashboard-7d5dcdb6d9-944xl 1/1 Running 0 16m
kube-system monitoring-grafana-7b767fb8dd-mg6dd 1/1 Running 0 16m
kube-system monitoring-influxdb-54bd58b4c9-z9tgd 1/1 Running 0 16m
kube-system node-exporter-f6pmw 1/1 Running 0 8m
kube-system node-exporter-zsd9b 1/1 Running 0 8m
kube-system prometheus-0 0/2 Pending 0 7m
I checked prometheus pod by command shown below:
[root#master16g prometheus]# kubectl describe pod prometheus-0 -n kube-system
Name: prometheus-0
Namespace: kube-system
Node: <none>
Labels: controller-revision-hash=prometheus-8fc558cb5
k8s-app=prometheus
statefulset.kubernetes.io/pod-name=prometheus-0
Annotations: scheduler.alpha.kubernetes.io/critical-pod=
Status: Pending
IP:
Controlled By: StatefulSet/prometheus
Init Containers:
init-chown-data:
Image: busybox:latest
Port: <none>
Host Port: <none>
Command:
chown
-R
65534:65534
/data
Environment: <none>
Mounts:
/data from prometheus-data (rw)
/var/run/secrets/kubernetes.io/serviceaccount from prometheus-token-f6v42 (ro)
Containers:
prometheus-server-configmap-reload:
Image: jimmidyson/configmap-reload:v0.1
Port: <none>
Host Port: <none>
Args:
--volume-dir=/etc/config
--webhook-url=http://localhost:9090/-/reload
Limits:
cpu: 10m
memory: 10Mi
Requests:
cpu: 10m
memory: 10Mi
Environment: <none>
Mounts:
/etc/config from config-volume (ro)
/var/run/secrets/kubernetes.io/serviceaccount from prometheus-token-f6v42 (ro)
prometheus-server:
Image: prom/prometheus:v2.2.1
Port: 9090/TCP
Host Port: 0/TCP
Args:
--config.file=/etc/config/prometheus.yml
--storage.tsdb.path=/data
--web.console.libraries=/etc/prometheus/console_libraries
--web.console.templates=/etc/prometheus/consoles
--web.enable-lifecycle
Limits:
cpu: 200m
memory: 1000Mi
Requests:
cpu: 200m
memory: 1000Mi
Liveness: http-get http://:9090/-/healthy delay=30s timeout=30s period=10s #success=1 #failure=3
Readiness: http-get http://:9090/-/ready delay=30s timeout=30s period=10s #success=1 #failure=3
Environment: <none>
Mounts:
/data from prometheus-data (rw)
/etc/config from config-volume (rw)
/var/run/secrets/kubernetes.io/serviceaccount from prometheus-token-f6v42 (ro)
Conditions:
Type Status
PodScheduled False
Volumes:
prometheus-data:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: prometheus-data-prometheus-0
ReadOnly: false
config-volume:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: prometheus-config
Optional: false
prometheus-token-f6v42:
Type: Secret (a volume populated by a Secret)
SecretName: prometheus-token-f6v42
Optional: false
QoS Class: Guaranteed
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 42s (x22 over 5m) default-scheduler pod has unbound PersistentVolumeClaims (repeated 2 times)
In the last line, it shows warning message: pod has unbound PersistentVolumeClaims (repeated 2 times)
The Prometheus logs says:
[root#master16g prometheus]# kubectl logs prometheus-0 -n kube-system
Error from server (BadRequest): a container name must be specified for pod prometheus-0, choose one of: [prometheus-server-configmap-reload prometheus-server] or one of the init containers: [init-chown-data]
The I describe alertmanager pod and its logs:
[root#master16g prometheus]# kubectl describe pod alertmanager-6bd9584b85-j4h5m -n kube-system
Name: alertmanager-6bd9584b85-j4h5m
Namespace: kube-system
Node: <none>
Labels: k8s-app=alertmanager
pod-template-hash=2685140641
version=v0.14.0
Annotations: scheduler.alpha.kubernetes.io/critical-pod=
Status: Pending
IP:
Controlled By: ReplicaSet/alertmanager-6bd9584b85
Containers:
prometheus-alertmanager:
Image: prom/alertmanager:v0.14.0
Port: 9093/TCP
Host Port: 0/TCP
Args:
--config.file=/etc/config/alertmanager.yml
--storage.path=/data
--web.external-url=/
Limits:
cpu: 10m
memory: 50Mi
Requests:
cpu: 10m
memory: 50Mi
Readiness: http-get http://:9093/%23/status delay=30s timeout=30s period=10s #success=1 #failure=3
Environment: <none>
Mounts:
/data from storage-volume (rw)
/etc/config from config-volume (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-snfrt (ro)
prometheus-alertmanager-configmap-reload:
Image: jimmidyson/configmap-reload:v0.1
Port: <none>
Host Port: <none>
Args:
--volume-dir=/etc/config
--webhook-url=http://localhost:9093/-/reload
Limits:
cpu: 10m
memory: 10Mi
Requests:
cpu: 10m
memory: 10Mi
Environment: <none>
Mounts:
/etc/config from config-volume (ro)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-snfrt (ro)
Conditions:
Type Status
PodScheduled False
Volumes:
config-volume:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: alertmanager-config
Optional: false
storage-volume:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: alertmanager
ReadOnly: false
default-token-snfrt:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-snfrt
Optional: false
QoS Class: Guaranteed
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 3m (x26 over 9m) default-scheduler pod has unbound PersistentVolumeClaims (repeated 2 times)
And its log:
[root#master16g prometheus]# kubectl logs alertmanager-6bd9584b85-j4h5m -n kube-system
Error from server (BadRequest): a container name must be specified for pod alertmanager-6bd9584b85-j4h5m, choose one of: [prometheus-alertmanager prometheus-alertmanager-configmap-reload]
It has same warning message as Prometheus:
pod has unbound PersistentVolumeClaims (repeated 2 times)
Then I get pvc by issuing command as follows:
[root#master16g prometheus]# kubectl get pvc --all-namespaces
NAMESPACE NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
kube-system alertmanager Pending standard 20m
kube-system prometheus-data-prometheus-0 Pending standard 19m
My question is how to make bound persistentVolumnClaim? Why log says container name must be specified?
===============================================================
Second edition
Since pvc file defined storage class, so I need to define a storage class yaml. How to do it if I want Nfs or GlusterFs? In this way, I could avoid cloud vendor, like Google or AWS.
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: alertmanager
namespace: kube-system
labels:
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: EnsureExists
spec:
storageClassName: standard
accessModes:
- ReadWriteOnce
resources:
requests:
storage: "2Gi"
This log entry:
Error from server (BadRequest): a container name must be specified for pod alertmanager-6bd9584b85-j4h5m, choose one of: [prometheus-alertmanager prometheus-alertmanager-configmap-reload]
means Pod alertmanager-6bd9584b85-j4h5m consists of two containers:
prometheus-alertmanager
prometheus-alertmanager-configmap-reload
When you use kubectl logs for Pod which consists of more then one containers you must specify a name of the container to view its logs. Command template:
kubectl -n <namespace> logs <pod_name> <container_name>
For example, if you want to view logs of the container prometheus-alertmanager which is a part of Pod alertmanager-6bd9584b85-j4h5m in the namespace kube-system you should use this command:
kubectl -n kube-system logs alertmanager-6bd9584b85-j4h5m prometheus-alertmanager
Pending status of the PVCs could mean you have no corresponding PVs
I have deployed kubernetes on a virt-manager vm following this link
https://kubernetes.io/docs/setup/independent/install-kubeadm/
When i join my another vm to the cluster i find that the kube-dns is in pending state.
root#ubuntu1:~# kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system etcd-ubuntu1 1/1 Running 0 7m
kube-system kube-apiserver-ubuntu1 1/1 Running 0 8m
kube-system kube-controller-manager-ubuntu1 1/1 Running 0 8m
kube-system kube-dns-86f4d74b45-br6ck 0/3 Pending 0 8m
kube-system kube-proxy-sh9lg 1/1 Running 0 8m
kube-system kube-proxy-zwdt5 1/1 Running 0 7m
kube-system kube-scheduler-ubuntu1 1/1 Running 0 8m
root#ubuntu1:~# kubectl --namespace=kube-system describe pod kube-dns-86f4d74b45-br6ck
Name: kube-dns-86f4d74b45-br6ck
Namespace: kube-system
Node: <none>
Labels: k8s-app=kube-dns
pod-template-hash=4290830601
Annotations: <none>
Status: Pending
IP:
Controlled By: ReplicaSet/kube-dns-86f4d74b45
Containers:
kubedns:
Image: k8s.gcr.io/k8s-dns-kube-dns-amd64:1.14.8
Ports: 10053/UDP, 10053/TCP, 10055/TCP
Host Ports: 0/UDP, 0/TCP, 0/TCP
Args:
--domain=cluster.local.
--dns-port=10053
--config-dir=/kube-dns-config
--v=2
Limits:
memory: 170Mi
Requests:
cpu: 100m
memory: 70Mi
Liveness: http-get http://:10054/healthcheck/kubedns delay=60s timeout=5s period=10s #success=1 #failure=5
Readiness: http-get http://:8081/readiness delay=3s timeout=5s period=10s #success=1 #failure=3
Environment:
PROMETHEUS_PORT: 10055
Mounts:
/kube-dns-config from kube-dns-config (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-dns-token-4fjt4 (ro)
dnsmasq:
Image: k8s.gcr.io/k8s-dns-dnsmasq-nanny-amd64:1.14.8
Ports: 53/UDP, 53/TCP
Host Ports: 0/UDP, 0/TCP
Args:
-v=2
-logtostderr
-configDir=/etc/k8s/dns/dnsmasq-nanny
-restartDnsmasq=true
--
-k
--cache-size=1000
--no-negcache
--log-facility=-
--server=/cluster.local/127.0.0.1#10053
--server=/in-addr.arpa/127.0.0.1#10053
--server=/ip6.arpa/127.0.0.1#10053
Requests:
cpu: 150m
memory: 20Mi
Liveness: http-get http://:10054/healthcheck/dnsmasq delay=60s timeout=5s period=10s #success=1 #failure=5
Environment: <none>
Mounts:
/etc/k8s/dns/dnsmasq-nanny from kube-dns-config (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-dns-token-4fjt4 (ro)
sidecar:
Image: k8s.gcr.io/k8s-dns-sidecar-amd64:1.14.8
Port: 10054/TCP
Host Port: 0/TCP
Args:
--v=2
--logtostderr
--probe=kubedns,127.0.0.1:10053,kubernetes.default.svc.cluster.local,5,SRV
--probe=dnsmasq,127.0.0.1:53,kubernetes.default.svc.cluster.local,5,SRV
Requests:
cpu: 10m
memory: 20Mi
Liveness: http-get http://:10054/metrics delay=60s timeout=5s period=10s #success=1 #failure=5
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-dns-token-4fjt4 (ro)
Conditions:
Type Status
PodScheduled False
Volumes:
kube-dns-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: kube-dns
Optional: true
kube-dns-token-4fjt4:
Type: Secret (a volume populated by a Secret)
SecretName: kube-dns-token-4fjt4
Optional: false
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: CriticalAddonsOnly
node-role.kubernetes.io/master:NoSchedule
node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 6m (x7 over 7m) default-scheduler 0/1 nodes are available: 1 node(s) were not ready.
Warning FailedScheduling 3s (x19 over 6m) default-scheduler 0/2 nodes are available: 2 node(s) were not ready.
Can anyone just help me how to deconstruct this and find the actual issue??
Any help would be off great use
Thanks in advance.
In addition to what #justcompile has wrote you will need a minimum of 2 CPU cores in order to run all pods from the kube-system namespace without issues.
You need to verify how much resources you have on that box and compare it with CPU reservations which each of Pods make.
For example in the provided by you output I can see that your DNS service tries to make a reservetion for 10% of CPU core:
Requests:
cpu: 100m
You can check each of deployed pods and their CPU reservations using:
kubectl describe pods --namespace=kube-system
in your cause kubectl get pods --all-namespaces output cannot see any about pods network.
so you may choice a network implementation and have to install a Pod Network before then kube-dns may deployed fully. for detail kube-dns is stuck in the Pending state and install pod network solution
Firstly, if you run kubectl get nodes does this show both/all nodes in a Ready state?
If they are, I faced this problem and found that when inspecting kubectl get events it showed that the pods were failing as they required a minimum of 2 CPUs to run.
As I was initially running this on an old Macbook Pro via VirtualBox I had to give up and use AWS (other Cloud Platforms are of course available) in order to get multiple CPUs per node.