Kubernetes Service does not have active Endpoint - kubernetes

I created a Deployment, Service and an Ingress. Unfortunately, the ingress-nginx-controller pods are complaining that my Service does not have an Active Endpoint:
controller.go:920] Service "<namespace>/web-server" does not have any active Endpoint.
My Service definition:
apiVersion: v1
kind: Service
metadata:
annotations:
prometheus.io/should_be_scraped: "false"
creationTimestamp: "2021-06-22T07:07:18Z"
labels:
chart: <namespace>-core-1.9.2
release: <namespace>
name: web-server
namespace: <namespace>
resourceVersion: "9050796"
selfLink: /api/v1/namespaces/<namespace>/services/web-server
uid: 82b3c3b4-a181-4ba2-887a-a4498346bc81
spec:
clusterIP: 10.233.56.52
ports:
- name: http
port: 80
protocol: TCP
targetPort: 8080
selector:
app: web-server
sessionAffinity: None
type: ClusterIP
status:
loadBalancer: {}
My Deployment definition:
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: "1"
creationTimestamp: "2021-06-22T07:07:19Z"
generation: 1
labels:
app: web-server
chart: <namespace>-core-1.9.2
release: <namespace>
name: web-server
namespace: <namespace>
resourceVersion: "9051062"
selfLink: /apis/apps/v1/namespaces/<namespace>/deployments/web-server
uid: fb085727-9e8a-4931-8067-fd4ed410b8ca
spec:
progressDeadlineSeconds: 600
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
app: web-server
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
creationTimestamp: null
labels:
app: web-server
spec:
containers:
- env:
<removed environment variables>
image: <url>/<namespace>/web-server:1.10.1
imagePullPolicy: IfNotPresent
name: web-server
ports:
- containerPort: 8080
name: http
protocol: TCP
- containerPort: 8082
name: metrics
protocol: TCP
readinessProbe:
failureThreshold: 3
httpGet:
path: /actuator/health
port: 8080
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 5
successThreshold: 1
timeoutSeconds: 1
resources:
limits:
memory: 1Gi
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /config
name: <namespace>-config
dnsPolicy: ClusterFirst
hostAliases:
- hostnames:
- <url>
ip: 10.0.1.178
imagePullSecrets:
- name: registry-pull-secret
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 30
volumes:
- configMap:
defaultMode: 420
name: <namespace>-config
name: <namespace>-config
status:
conditions:
- lastTransitionTime: "2021-06-22T07:07:19Z"
lastUpdateTime: "2021-06-22T07:07:19Z"
message: Deployment does not have minimum availability.
reason: MinimumReplicasUnavailable
status: "False"
type: Available
- lastTransitionTime: "2021-06-22T07:17:20Z"
lastUpdateTime: "2021-06-22T07:17:20Z"
message: ReplicaSet "web-server-6df6d6565b" has timed out progressing.
reason: ProgressDeadlineExceeded
status: "False"
type: Progressing
observedGeneration: 1
replicas: 1
unavailableReplicas: 1
updatedReplicas: 1
In the same namespace, I have more Service and Deployment resources, all of them work, except this one (+ another, see below).
# kubectl get endpoints -n <namespace>
NAME ENDPOINTS AGE
activemq 10.233.64.3:61613,10.233.64.3:8161,10.233.64.3:61616 + 1 more... 26d
content-backend 10.233.96.17:8080 26d
datastore3 10.233.96.16:8080 26d
web-server 74m
web-server-metrics 26d
As you can see, the selector/label are the same (web-server) in the Service as well as in the Deployment definition.

C-Nan has solved the problem, and has posted a solution as a comment:
I found the issue. The Pod was started, but not in Ready state due to a failing readinessProbe. I wasn't aware that an endpoint wouldn't be created until the Pod is in Ready state. Removing the readinessProbe created the Endpoint.

From the status of your Deployment, it seems that no pod is running for the Deployment.
status:
conditions:
- lastTransitionTime: "2021-06-22T07:07:19Z"
lastUpdateTime: "2021-06-22T07:07:19Z"
message: Deployment does not have minimum availability.
reason: MinimumReplicasUnavailable
status: "False"
type: Available
- lastTransitionTime: "2021-06-22T07:17:20Z"
lastUpdateTime: "2021-06-22T07:17:20Z"
message: ReplicaSet "web-server-6df6d6565b" has timed out progressing.
reason: ProgressDeadlineExceeded
status: "False"
type: Progressing
observedGeneration: 1
replicas: 1
unavailableReplicas: 1
updatedReplicas: 1
The unavailableReplicas: 1 filed is indicating that the desired pod is not available. As a result the Service has no active endpoint.
You can describe the deployment to see why the pod is unavailable.

Related

Why is GKE HPA not scaling down?

I have a Kubernetes deployment with a Go App in Kubernetes 1.17 on GKE. It has cpu and memory requests and limits. It has 1 replica specified in the deployment.
Furthermore I have this HPA (I have a autoscaling/v2beta2 defined in my Helm chart, but GKE converts it to a v2beta1 apparently):
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
annotations:
meta.helm.sh/release-name: servicename
meta.helm.sh/release-namespace: namespace
creationTimestamp: "2021-02-15T11:30:18Z"
labels:
app.kubernetes.io/managed-by: Helm
name: servicename-service
namespace: namespace
resourceVersion: "123"
selfLink: link
uid: uid
spec:
maxReplicas: 10
metrics:
- resource:
name: memory
targetAverageUtilization: 80
type: Resource
- resource:
name: cpu
targetAverageUtilization: 80
type: Resource
minReplicas: 1
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: servicename-service
status:
conditions:
- lastTransitionTime: "2021-02-15T11:30:33Z"
message: recommended size matches current size
reason: ReadyForNewScale
status: "True"
type: AbleToScale
- lastTransitionTime: "2021-02-15T13:17:20Z"
message: the HPA was able to successfully calculate a replica count from cpu resource
utilization (percentage of request)
reason: ValidMetricFound
status: "True"
type: ScalingActive
- lastTransitionTime: "2021-02-15T13:17:36Z"
message: the desired count is within the acceptable range
reason: DesiredWithinRange
status: "False"
type: ScalingLimited
currentMetrics:
- resource:
currentAverageUtilization: 14
currentAverageValue: "9396224"
name: memory
type: Resource
- resource:
currentAverageUtilization: 33
currentAverageValue: 84m
name: cpu
type: Resource
currentReplicas: 3
desiredReplicas: 3
lastScaleTime: "2021-02-15T13:40:11Z"
Deployment:
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: "456"
meta.helm.sh/release-name: servicename-service
meta.helm.sh/release-namespace: services
creationTimestamp: "2021-02-11T10:00:45Z"
generation: 129
labels:
app: servicename
app.kubernetes.io/managed-by: Helm
chart: servicename
heritage: Helm
release: servicename-service
name: servicename-service-servicename
namespace: namespace
resourceVersion: "123"
selfLink: /apis/apps/v1/namespaces/namespace/deployments/servicename-service-servicename
uid: b1fcc8c6-f3e6-4bbf-92a1-d7ae1e2bb188
spec:
progressDeadlineSeconds: 600
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
app: servicename
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
creationTimestamp: null
labels:
app: servicename
release: servicename-service
spec:
containers:
envFrom:
- configMapRef:
name: servicename-service-servicename
image: image
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 3
httpGet:
path: /health/liveness
port: 8888
scheme: HTTP
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
name: servicename
ports:
- containerPort: 8888
protocol: TCP
readinessProbe:
failureThreshold: 3
httpGet:
path: /health/readiness
port: 8888
scheme: HTTP
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
resources:
limits:
cpu: 500m
memory: 256Mi
requests:
cpu: 150m
memory: 64Mi
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 30
status:
availableReplicas: 3
conditions:
- lastTransitionTime: "2021-02-11T10:00:45Z"
lastUpdateTime: "2021-02-16T14:10:29Z"
message: ReplicaSet "servicename-service-servicename-5b6445fcb" has
successfully progressed.
reason: NewReplicaSetAvailable
status: "True"
type: Progressing
- lastTransitionTime: "2021-02-20T16:19:51Z"
lastUpdateTime: "2021-02-20T16:19:51Z"
message: Deployment has minimum availability.
reason: MinimumReplicasAvailable
status: "True"
type: Available
observedGeneration: 129
readyReplicas: 3
replicas: 3
updatedReplicas: 3
Output of kubectl get hpa --all-namespaces
NAMESPACE NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
namespace servicename-service Deployment/servicename-service 9%/80%, 1%/80% 1 10 2 6d
namespace xyz-service Deployment/xyz-service 18%/80%, 1%/80% 1 10 1 6d
I haven't changed any Kubernetes Controller default settings like --horizontal-pod-autoscaler-downscale-stabilization.
Question:
Why is it not scaling down to 1 replica when the currentAverageUtilization of the cpu is 33 and the target one 80? I waited for more than 1 hour.
Any ideas?

Calico: networkPlugin cni failed to set up pod, i/o timeout

I have got an issue with deploy some pods on my k8s node. The error is following:
Failed create pod sandbox: rpc error: code = Unknown desc = failed to
set up sandbox container
"7da8bce09dd6820a65754073b1b4e52e640291dcb82f1da87ae99570c6964d1b"
network for pod "webservices-8675d4667d-7mdf9": networkPlugin cni
failed to set up pod "webservices-8675d4667d-7mdf9_default" network:
Get https://[10.233.0.1]:443/api/v1/namespaces/default: dial tcp
10.233.0.1:443: i/o timeout
However, some pods are deployed, for example kubernetes-dashboard:
Update:
NAME STATUS ROLES AGE VERSION LABELS
k8s-master.mariyo.eu Ready master 3d15h v1.16.6 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-master.mariyo.eu,kubernetes.io/os=linux,node-role.kubernetes.io/master=
k8s-node-1.mariyo.eu Ready <none> 3d15h v1.16.6 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=k8s-node-1.mariyo.eu,kubernetes.io/os=linux
Deployment for coredns:
kind: Deployment
apiVersion: apps/v1
metadata:
name: coredns
namespace: kube-system
selfLink: /apis/apps/v1/namespaces/kube-system/deployments/coredns
uid: bd5451ec-2a33-443d-8519-ffcec935ac0c
resourceVersion: '397508'
generation: 2
creationTimestamp: '2020-01-24T16:14:37Z'
labels:
addonmanager.kubernetes.io/mode: Reconcile
k8s-app: kube-dns
kubernetes.io/cluster-service: 'true'
kubernetes.io/name: coredns
annotations:
deployment.kubernetes.io/revision: '1'
kubectl.kubernetes.io/last-applied-configuration: >
{"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"labels":{"addonmanager.kubernetes.io/mode":"Reconcile","k8s-app":"kube-dns","kubernetes.io/cluster-service":"true","kubernetes.io/name":"coredns"},"name":"coredns","namespace":"kube-system"},"spec":{"selector":{"matchLabels":{"k8s-app":"kube-dns"}},"strategy":{"rollingUpdate":{"maxSurge":"10%","maxUnavailable":0},"type":"RollingUpdate"},"template":{"metadata":{"annotations":{"seccomp.security.alpha.kubernetes.io/pod":"docker/default"},"labels":{"k8s-app":"kube-dns"}},"spec":{"affinity":{"nodeAffinity":{"preferredDuringSchedulingIgnoredDuringExecution":[{"preference":{"matchExpressions":[{"key":"node-role.kubernetes.io/master","operator":"In","values":[""]}]},"weight":100}]},"podAntiAffinity":{"requiredDuringSchedulingIgnoredDuringExecution":[{"labelSelector":{"matchLabels":{"k8s-app":"kube-dns"}},"topologyKey":"kubernetes.io/hostname"}]}},"containers":[{"args":["-conf","/etc/coredns/Corefile"],"image":"docker.io/coredns/coredns:1.6.0","imagePullPolicy":"IfNotPresent","livenessProbe":{"failureThreshold":10,"httpGet":{"path":"/health","port":8080,"scheme":"HTTP"},"successThreshold":1,"timeoutSeconds":5},"name":"coredns","ports":[{"containerPort":53,"name":"dns","protocol":"UDP"},{"containerPort":53,"name":"dns-tcp","protocol":"TCP"},{"containerPort":9153,"name":"metrics","protocol":"TCP"}],"readinessProbe":{"failureThreshold":10,"httpGet":{"path":"/ready","port":8181,"scheme":"HTTP"},"successThreshold":1,"timeoutSeconds":5},"resources":{"limits":{"memory":"170Mi"},"requests":{"cpu":"100m","memory":"70Mi"}},"securityContext":{"allowPrivilegeEscalation":false,"capabilities":{"add":["NET_BIND_SERVICE"],"drop":["all"]},"readOnlyRootFilesystem":true},"volumeMounts":[{"mountPath":"/etc/coredns","name":"config-volume"}]}],"dnsPolicy":"Default","nodeSelector":{"beta.kubernetes.io/os":"linux"},"priorityClassName":"system-cluster-critical","serviceAccountName":"coredns","tolerations":[{"effect":"NoSchedule","key":"node-role.kubernetes.io/master"},{"key":"CriticalAddonsOnly","operator":"Exists"}],"volumes":[{"configMap":{"items":[{"key":"Corefile","path":"Corefile"}],"name":"coredns"},"name":"config-volume"}]}}}}
spec:
replicas: 2
selector:
matchLabels:
k8s-app: kube-dns
template:
metadata:
creationTimestamp: null
labels:
k8s-app: kube-dns
annotations:
seccomp.security.alpha.kubernetes.io/pod: docker/default
spec:
volumes:
- name: config-volume
configMap:
name: coredns
items:
- key: Corefile
path: Corefile
defaultMode: 420
containers:
- name: coredns
image: 'docker.io/coredns/coredns:1.6.0'
args:
- '-conf'
- /etc/coredns/Corefile
ports:
- name: dns
containerPort: 53
protocol: UDP
- name: dns-tcp
containerPort: 53
protocol: TCP
- name: metrics
containerPort: 9153
protocol: TCP
resources:
limits:
memory: 170Mi
requests:
cpu: 100m
memory: 70Mi
volumeMounts:
- name: config-volume
mountPath: /etc/coredns
livenessProbe:
httpGet:
path: /health
port: 8080
scheme: HTTP
timeoutSeconds: 5
periodSeconds: 10
successThreshold: 1
failureThreshold: 10
readinessProbe:
httpGet:
path: /ready
port: 8181
scheme: HTTP
timeoutSeconds: 5
periodSeconds: 10
successThreshold: 1
failureThreshold: 10
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
imagePullPolicy: IfNotPresent
securityContext:
capabilities:
add:
- NET_BIND_SERVICE
drop:
- all
readOnlyRootFilesystem: true
allowPrivilegeEscalation: false
restartPolicy: Always
terminationGracePeriodSeconds: 30
dnsPolicy: Default
nodeSelector:
beta.kubernetes.io/os: linux
serviceAccountName: coredns
serviceAccount: coredns
securityContext: {}
affinity:
nodeAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
preference:
matchExpressions:
- key: node-role.kubernetes.io/master
operator: In
values:
- ''
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchLabels:
k8s-app: kube-dns
topologyKey: kubernetes.io/hostname
schedulerName: default-scheduler
tolerations:
- key: node-role.kubernetes.io/master
effect: NoSchedule
- key: CriticalAddonsOnly
operator: Exists
priorityClassName: system-cluster-critical
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 0
maxSurge: 10%
revisionHistoryLimit: 10
progressDeadlineSeconds: 600
status:
observedGeneration: 2
replicas: 2
updatedReplicas: 2
readyReplicas: 1
availableReplicas: 1
unavailableReplicas: 1
conditions:
- type: Progressing
status: 'True'
lastUpdateTime: '2020-01-24T16:14:42Z'
lastTransitionTime: '2020-01-24T16:14:37Z'
reason: NewReplicaSetAvailable
message: ReplicaSet "coredns-58687784f9" has successfully progressed.
- type: Available
status: 'False'
lastUpdateTime: '2020-01-27T17:42:57Z'
lastTransitionTime: '2020-01-27T17:42:57Z'
reason: MinimumReplicasUnavailable
message: Deployment does not have minimum availability.
Deployment for webservices:
kind: Deployment
apiVersion: apps/v1
metadata:
name: webservices
namespace: default
selfLink: /apis/apps/v1/namespaces/default/deployments/webservices
uid: da75d3d8-92f4-4d06-86d6-e2fb325806a5
resourceVersion: '398529'
generation: 1
creationTimestamp: '2020-01-27T08:05:16Z'
labels:
run: webservices
annotations:
deployment.kubernetes.io/revision: '1'
spec:
replicas: 5
selector:
matchLabels:
run: webservices
template:
metadata:
creationTimestamp: null
labels:
run: webservices
spec:
containers:
- name: webservices
image: nginx
ports:
- containerPort: 80
protocol: TCP
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
imagePullPolicy: Always
restartPolicy: Always
terminationGracePeriodSeconds: 30
dnsPolicy: ClusterFirst
securityContext: {}
schedulerName: default-scheduler
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 25%
maxSurge: 25%
revisionHistoryLimit: 10
progressDeadlineSeconds: 600
status:
observedGeneration: 1
replicas: 5
updatedReplicas: 5
unavailableReplicas: 5
conditions:
- type: Available
status: 'False'
lastUpdateTime: '2020-01-27T08:05:16Z'
lastTransitionTime: '2020-01-27T08:05:16Z'
reason: MinimumReplicasUnavailable
message: Deployment does not have minimum availability.
- type: Progressing
status: 'False'
lastUpdateTime: '2020-01-27T17:52:58Z'
lastTransitionTime: '2020-01-27T17:52:58Z'
reason: ProgressDeadlineExceeded
message: ReplicaSet "webservices-8675d4667d" has timed out progressing.
Finally, I decided to reinstall nodes from Debian 10 to Ubuntu 18.04 and everything works as expected.
Thank you for your time
Problem is that kube-proxy isn't functioning correctly as I believe the 10.233.0.1 is the kubernetes api service address which it is responsible for configuring/setting up. You should check kube-proxy logs and see that it is healthy and create the iptables rules for the kubernetes services.
Take a look here: calico-timeout-pod.
I had to set the following on the worker node as well, before joining it, for it to work:
sudo sysctl net.bridge.bridge-nf-call-iptables=1
I was having a similar issue. I am using microk8s in my instance. it seems the node needs to advertise itself to the cluster. I hope it points you in the right direction (repost from github):
microk8s stop
# or for workers: sudo snap stop microk8s
sudo vim.tiny /var/snap/microk8s/current/args/kubelet
# Add this to bottom: --node-ip=<this-specific-node-lan-ip>
sudo vim.tiny /var/snap/microk8s/current/args/kube-apiserver
# Add this to bottom: --advertise-address=<this-specific-node-lan-ip>
microk8s start
# or for workers: sudo snap start microk8s

Kubernetes pod cluster ip not responding?

I have two backend services deployed on the Google cloud Kubernetes Engine.
a) Backend Service
b) Admin portal which needed to connect with the Backend Service
Everything is available in one cluster.
As in Workload / Pods,
I have three deployments running whereas fitme:9000 is a backend and nginx-1:9000 is an admin portal service
whereas in Services I have
Visualization
Explanation
1. D1 (fitme), D2 (mongo-mongodb), D3 (nginx-1) are three deployments
2. E1D1 (fitme-service), E2D1 (fitme-jr29g), E1D2 (mongo-mongodb), E2D2 (mongo-mongodb-rcwwc) and E1D3 (nginx-1-service) are Services
3. `E1D1, E1D2 and E1D3` are exposed over `Load Balancer` whereas `E2D1 , E2D2` are exposed over `Cluster IP`.
The reason behind it:
D1 needs to access D2 (internally) -> This is perfectly working fine. I am using E2D2 exposed service (cluster IP) to access the D2 deployment inside from D1
Now, D3 needs access to D1 deployment. So, I exposed D1 as an E2D1 service and trying to access it internally by generated Cluster IP of E2D1 but it's giving me request time out.
YAML for fitme-jr29g service
apiVersion: v1
kind: Service
metadata:
creationTimestamp: "2019-12-02T11:18:55Z"
generateName: fitme-
labels:
app: fitme
name: fitme-jr29g
namespace: default
resourceVersion: "486673"
selfLink: /api/v1/namespaces/default/services/fitme-8t7rl
uid: 875045eb-14f5-11ea-823c-42010a8e0047
spec:
clusterIP: 10.35.240.95
ports:
- port: 9000
protocol: TCP
targetPort: 9000
selector:
app: fitme
sessionAffinity: None
type: ClusterIP
status:
loadBalancer: {}
YAML for nginx-1-service service
apiVersion: v1
kind: Service
metadata:
creationTimestamp: "2019-12-02T11:30:10Z"
labels:
app: admin
name: nginx-1-service
namespace: default
resourceVersion: "489972"
selfLink: /api/v1/namespaces/default/services/admin-service
uid: 195b462e-14f7-11ea-823c-42010a8e0047
spec:
clusterIP: 10.35.250.90
externalTrafficPolicy: Cluster
ports:
- nodePort: 30628
port: 8080
protocol: TCP
targetPort: 9000
selector:
app: admin
sessionAffinity: None
type: LoadBalancer
status:
loadBalancer:
ingress:
- ip: 35.227.26.101
YAML for nginx-1 deployment
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: "1"
creationTimestamp: "2019-12-02T11:24:09Z"
generation: 2
labels:
app: admin
name: admin
namespace: default
resourceVersion: "489624"
selfLink: /apis/apps/v1/namespaces/default/deployments/admin
uid: 426792e6-14f6-11ea-823c-42010a8e0047
spec:
progressDeadlineSeconds: 600
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
app: admin
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
creationTimestamp: null
labels:
app: admin
spec:
containers:
- image: gcr.io/docker-226818/admin#sha256:602fe6b7e43d53251eebe2f29968bebbd756336c809cb1cd43787027537a5c8b
imagePullPolicy: IfNotPresent
name: admin-sha256
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 30
status:
availableReplicas: 1
conditions:
- lastTransitionTime: "2019-12-02T11:24:18Z"
lastUpdateTime: "2019-12-02T11:24:18Z"
message: Deployment has minimum availability.
reason: MinimumReplicasAvailable
status: "True"
type: Available
- lastTransitionTime: "2019-12-02T11:24:09Z"
lastUpdateTime: "2019-12-02T11:24:18Z"
message: ReplicaSet "admin-8d55dfbb6" has successfully progressed.
reason: NewReplicaSetAvailable
status: "True"
type: Progressing
observedGeneration: 2
readyReplicas: 1
replicas: 1
updatedReplicas: 1
YAML for fitme-service
apiVersion: v1
kind: Service
metadata:
creationTimestamp: "2019-12-02T13:38:21Z"
generateName: fitme-
labels:
app: fitme
name: fitme-service
namespace: default
resourceVersion: "525173"
selfLink: /api/v1/namespaces/default/services/drogo-mzcgr
uid: 01e8fc39-1509-11ea-823c-42010a8e0047
spec:
clusterIP: 10.35.240.74
externalTrafficPolicy: Cluster
ports:
- nodePort: 31016
port: 80
protocol: TCP
targetPort: 9000
selector:
app: fitme
sessionAffinity: None
type: LoadBalancer
status:
loadBalancer:
ingress:
- ip: 35.236.110.230
YAML for fitme deployment
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: "1"
creationTimestamp: "2019-12-02T13:34:54Z"
generation: 2
labels:
app: fitme
name: fitme
namespace: default
resourceVersion: "525571"
selfLink: /apis/apps/v1/namespaces/default/deployments/drogo
uid: 865a5a8a-1508-11ea-823c-42010a8e0047
spec:
progressDeadlineSeconds: 600
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
app: drogo
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
creationTimestamp: null
labels:
app: fitme
spec:
containers:
- image: gcr.io/fitme-226818/drogo#sha256:ab49a4b12e7a14f9428a5720bbfd1808eb9667855cb874e973c386a4e9b59d40
imagePullPolicy: IfNotPresent
name: fitme-sha256
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 30
status:
availableReplicas: 1
conditions:
- lastTransitionTime: "2019-12-02T13:34:57Z"
lastUpdateTime: "2019-12-02T13:34:57Z"
message: Deployment has minimum availability.
reason: MinimumReplicasAvailable
status: "True"
type: Available
- lastTransitionTime: "2019-12-02T13:34:54Z"
lastUpdateTime: "2019-12-02T13:34:57Z"
message: ReplicaSet "drogo-5c7f449668" has successfully progressed.
reason: NewReplicaSetAvailable
status: "True"
type: Progressing
observedGeneration: 2
readyReplicas: 1
replicas: 1
updatedReplicas: 1
I am accessing fitme-jr29g by putting 10.35.240.95:9000 ip address in
nginx-1 deployment container.
The deployment object can, and often, should have network properties to expose the applications within the pods.
Pods are networking cappable objects, with virtual ethernet interfaces, needed to receive incoming traffic.
On the other hand, services are completely network oriented objects, meant mostly to relay network traffic into the pods.
You can think of that as pods (grouped in deployments) as backend and services as load balancers. At the end, both need network capabilities.
In your scenario, I'm not sure how are you exposing your deployment via load balancer since its pods doesn't seem to have any open ports.
Since the services exposing your pods are targeting port 9000, you can add it to the pod template in your deployment:
spec:
containers:
- image: gcr.io/fitme-xxxxxxx
name: fitme-sha256
ports:
- containerPort: 9000
Be sure that it matches the port where your container is actually receiving the incoming requests.

Connect back-end with front-end over rest api call running on kubernetes

I have a two tier application.
The frontend calls the webapi layer through simple http rest call
http://mywebapi:5000/
My working docker compose code is below and the application works
version: '3'
services:
webfrontend:
image: webfrontend
build: ./nodeexpress-alibaba-ci-tutorial
ports:
- "3000:3000"
networks:
- my-shared-network
mywebapi:
image: mywebapi
build: ./dotnetcorewebapi-alibaba-ci-tutorial
ports:
- "5000:5000"
networks:
- my-shared-network
networks:
my-shared-network: {}
Now I'm trying to get this to work on kubernetes.
I have created two deployments and two services-loadbalancer for webfrontend and clusterip for mywebapi
But, after deploying, I find that the data from mywebapi is not reaching the frontend. I can view the frontend on the browser through the load balancer public ip.
Mywebapi deployment yaml
apiVersion: apps/v1beta2
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: '3'
creationTimestamp: '2019-09-28T13:31:32Z'
generation: 3
labels:
app: mywebapi
tier: backend
name: mywebapi
namespace: default
resourceVersion: '1047268388'
selfLink: /apis/apps/v1beta2/namespaces/default/deployments/mywebapi
uid: 493ab5e0-e1f4-11e9-9a64-d63fe9981162
spec:
progressDeadlineSeconds: 600
replicas: 2
revisionHistoryLimit: 10
selector:
matchLabels:
app: mywebapi
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
annotations:
aliyun.kubernetes.io/deploy-timestamp: '2019-09-28T14:36:01Z'
labels:
app: mywebapi
spec:
containers:
- image: >-
registry-intl-vpc.ap-southeast-1.aliyuncs.com/devopsci-t/mywebapi:1.0
imagePullPolicy: Always
name: mywebapi
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 30
status:
availableReplicas: 2
conditions:
- lastTransitionTime: '2019-09-28T14:51:18Z'
lastUpdateTime: '2019-09-28T14:51:18Z'
message: Deployment has minimum availability.
reason: MinimumReplicasAvailable
status: 'True'
type: Available
- lastTransitionTime: '2019-09-28T14:49:55Z'
lastUpdateTime: '2019-09-28T14:51:19Z'
message: ReplicaSet "mywebapi-84cf98fb4f" has successfully progressed.
reason: NewReplicaSetAvailable
status: 'True'
type: Progressing
observedGeneration: 3
readyReplicas: 2
replicas: 2
updatedReplicas: 2
Mywebapi service yaml
apiVersion: v1
kind: Service
metadata:
creationTimestamp: '2019-09-28T13:31:33Z'
name: mywebapi-svc
namespace: default
resourceVersion: '1047557879'
selfLink: /api/v1/namespaces/default/services/mywebapi-svc
uid: 49e21207-e1f4-11e9-9a64-d63fe9981162
spec:
clusterIP: None
ports:
- name: mywebapiport
port: 5000
protocol: TCP
targetPort: 5000
selector:
app: mywebapi
sessionAffinity: None
type: ClusterIP
status:
loadBalancer: {}
I even tried updating the http rest call url to http://mywebapi-svc:5000/ but still does not work.
In the webfrontend pod logs I find the below error
Got error: getaddrinfo ENOTFOUND mywebapi-svc mywebapi-svc:5000
Sincerely appreciate all help
Thanks
............................................................
Update..
Changed mywebapi-svc to disable headless. Current YAML is below. Problem still the same..
apiVersion: v1
kind: Service
metadata:
creationTimestamp: '2019-09-29T15:21:44Z'
name: mywebapi-svc
namespace: default
resourceVersion: '667545270'
selfLink: /api/v1/namespaces/default/services/mywebapi-svc
uid: d84503ee-e2cc-11e9-93ec-a65f0b53b1fa
spec:
clusterIP: 172.19.0.74
ports:
- name: mywebapiport
port: 5000
protocol: TCP
targetPort: 5000
selector:
app: mywebapi-default
sessionAffinity: None
type: ClusterIP
status:
loadBalancer: {}
It's because you're using headless service
I guess you don't need a headless service for this job. Because DNS server won't return a single IP address if there are more than one pod match by the label selector. Remove the spec.ClusterIP field from your service.
apiVersion: v1
kind: Service
metadata:
name: mywebapi-svc
namespace: default
spec:
# clusterIP: None <-- remove
ports:
- name: mywebapiport
port: 5000
protocol: TCP
targetPort: 5000
selector:
app: mywebapi
type: ClusterIP
Now you can call at http://service-name.namespace-name.svc:port endpoint from your front-end. It should work.
In case your webapi is stateful, then you should use statefulset instead of deployment.

The l7-default-backend deployment gets reverted when I edit it

I upgraded my GKE clusters to Kubernetes 1.5.6 a couple days ago. I used to be able to scale the l7-default-backend deployment to 3 replicas and increase the CPU resource limits, but now it seems my changes get reverted to their default of 1 replica and 10m CPU limit.
Interestingly enough, when I added the following:
nodeSelector:
cloud.google.com/gke-nodepool: default-pool
it was persisted no problem.
Here's the current deployment manifest:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: "4"
kubectl.kubernetes.io/last-applied-configuration: '{"kind":"Deployment","apiVersion":"extensions/v1beta1","metadata":{"name":"l7-default-backend","namespace":"kube-system","creationTimestamp":null,"labels":{"k8s-app":"glbc","kubernetes.io/cluster-service":"true","kubernetes.io/name":"GLBC"}},"spec":{"replicas":1,"selector":{"matchLabels":{"k8s-app":"glbc"}},"template":{"metadata":{"creationTimestamp":null,"labels":{"k8s-app":"glbc","name":"glbc"}},"spec":{"containers":[{"name":"default-http-backend","image":"gcr.io/google_containers/defaultbackend:1.0","ports":[{"containerPort":8080}],"resources":{"limits":{"cpu":"10m","memory":"20Mi"},"requests":{"cpu":"10m","memory":"20Mi"}},"livenessProbe":{"httpGet":{"path":"/healthz","port":8080,"scheme":"HTTP"},"initialDelaySeconds":30,"timeoutSeconds":5}}]}},"strategy":{}},"status":{}}'
creationTimestamp: 2017-03-23T23:30:12Z
generation: 9
labels:
k8s-app: glbc
kubernetes.io/cluster-service: "true"
kubernetes.io/name: GLBC
name: l7-default-backend
namespace: kube-system
resourceVersion: "40149922"
selfLink: /apis/extensions/v1beta1/namespaces/kube-system/deployments/l7-default-backend
uid: a9772d26-1020-11e7-b9a8-42010af001d0
spec:
replicas: 1
selector:
matchLabels:
k8s-app: glbc
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 1
type: RollingUpdate
template:
metadata:
creationTimestamp: null
labels:
k8s-app: glbc
name: glbc
spec:
containers:
- image: gcr.io/google_containers/defaultbackend:1.0
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 8080
scheme: HTTP
initialDelaySeconds: 30
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 5
name: default-http-backend
ports:
- containerPort: 8080
protocol: TCP
resources:
limits:
cpu: 10m
memory: 20Mi
requests:
cpu: 10m
memory: 20Mi
terminationMessagePath: /dev/termination-log
dnsPolicy: ClusterFirst
nodeSelector:
cloud.google.com/gke-nodepool: default-pool
restartPolicy: Always
securityContext: {}
terminationGracePeriodSeconds: 30
status:
availableReplicas: 1
conditions:
- lastTransitionTime: 2017-04-13T19:19:35Z
lastUpdateTime: 2017-04-13T19:19:35Z
message: Deployment has minimum availability.
reason: MinimumReplicasAvailable
status: "True"
type: Available
observedGeneration: 9
replicas: 1
updatedReplicas: 1
How can I scale the l7-default-backend successfully?