Readiness probe failing when second pod gets scheduled to the same node - kubernetes

I have a k8s service which maps to pod deployment with 2 replicas and is exposed as clusterIp service. I am seeing an issue when the 2nd pod gets scheduled to the same node the readiness probe (http call to an api in container port) is failing with "unable to connect error" . Is this due to some port conflict?
Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
name: demo-deployment
namespace: demo
labels:
app: demo
spec:
replicas: 2
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
selector:
matchLabels:
app: demo
template:
metadata:
annotations:
checksum/config: {{ include (print $.Template.BasePath "/app-configmap.yaml") . | sha256sum }}
labels:
app: demo
spec:
containers:
- name: demo
image: demo-app-image:1.0.1
ports:
- containerPort: 8081
livenessProbe:
httpGet:
path: /healthcheck
port: 8081
initialDelaySeconds: 30
periodSeconds: 60
failureThreshold: 3
successThreshold: 1
timeoutSeconds: 15
readinessProbe:
httpGet:
path: /healthcheck
port: 8081
initialDelaySeconds: 30
periodSeconds: 10
failureThreshold: 3
successThreshold: 1
timeoutSeconds: 15
volumeMounts:
- name: config-volume
mountPath: /config/app
volumes:
- name: config-volume
configMap:
name: demo-configmap
items:
- key: config
path: config.json
nodeSelector:
usage: demo-server
Service
apiVersion: v1
kind: Service
metadata:
name: demo-service
namespace: demo
labels:
app: demo-service
spec:
selector:
app: demo
ports:
- name: admin-port
protocol: TCP
port: 26001
targetPort: 8081

Related

Kubernetes Statefulsets traffic issue

I am trying to deploy 3 replicas using statefulsets.
Now, I have three pods:
statefulset-test-0
statefulset-test-1
statefulset-test-2
Then, I need to use the following to rolling update:
kubectl rollout restart statefulsets/statefulset-test
it will stop statefulset-test-2 pod and create a new statefulset-test-2 pod, then it will stop statefulset-test-1.
At this point, statefulset-test-2 is running the new image, statefulset-test-1 has been stopped so it can not accept requests, and statefulset-test-0 is running the old image.
I was wondering how the k8s handles the request to those pods. Is the k8s send the request to test-0 and test-2 randomly or do they send the request to the new pod?
here is my yaml:
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: statefulset-test
labels:
app: statefulset-test
spec:
minReadySeconds: 20
serviceName: "statefulset-test"
updateStrategy:
type: RollingUpdate
replicas: 3
selector:
matchLabels:
app: statefulset-test
template:
metadata:
labels:
app: statefulset-test
spec:
containers:
- name: statefulset-test
image: ..
imagePullPolicy: Always
ports:
- containerPort: 123
- containerPort: 456
livenessProbe:
httpGet:
path: /api/Health
port: 123
initialDelaySeconds: 180
periodSeconds: 80
timeoutSeconds: 20
failureThreshold: 2
readinessProbe:
httpGet:
path: /api/Health
port: 123
initialDelaySeconds: 20
periodSeconds: 5
successThreshold: 1
---
apiVersion: v1
kind: Service
metadata:
name: statefulset-service
spec:
type: NodePort
selector:
app: statefulset-test
ports:
- name: statefulset-main
protocol: TCP
port: 123
targetPort: 123
nodePort: 678
- name : statefulset-grpc
protocol: TCP
port: 456
targetPort: 456
nodePort: 679

How is it that my k8s service is accessible without explicitly exposing a port?

I am running a http server in k8s, I am able to access the server via it's svc name from another service by making a call to https://e2e-test-runner.hogwarts.svc.cluster.local However, I haven't explicitly configured service port. How does this work?
Does k8s map container port as service port when no service port is present?
This is my service definition:
apiVersion: v1
kind: Service
metadata:
name: e2e-test-runner
namespace: hogwarts
spec:
selector:
app: e2e-test-runner
This is my deployment definition:
apiVersion: apps/v1
kind: Deployment
metadata:
name: e2e-test-runner
namespace: hogwarts
spec:
strategy:
rollingUpdate:
maxSurge: 3
maxUnavailable: 0
revisionHistoryLimit: 3
selector:
matchLabels:
app: e2e-test-runner
template:
metadata:
labels:
app: e2e-test-runner
app.kubernetes.io/name: e2e-test-runner
spec:
containers:
- name: app
image: $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/e2e-test-runner:$IMAGE_VERSION
ports:
- containerPort: 3000
resources:
requests:
memory: "250Mi"
cpu: "100m"
limits:
memory: "1Gi"
cpu: "1000m"
livenessProbe:
httpGet:
path: /health_check
port: 3000
scheme: "HTTPS"
initialDelaySeconds: 20
periodSeconds: 10
timeoutSeconds: 2
readinessProbe:
httpGet:
path: /health_check
port: 3000
scheme: "HTTPS"
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 5
# Gives pod 15 seconds to complete any outstanding requests before being force killed
terminationGracePeriodSeconds: 15

Readiness probe failing when second pod gets scheduled to the same container in k8s

I have a k8s service which maps to pod deployment with 2 replicas and is exposed as clusterIp service. I am seeing an issue when the 2nd pod gets scheduled to the same node the readiness probe (http call to an api in container port) is failing with "unable to connect error" . Is this due to some port conflict?
apiVersion: apps/v1
kind: Deployment
metadata:
name: demo-deployment
namespace: demo
labels:
app: demo
spec:
replicas: 2
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
selector:
matchLabels:
app: demo
template:
metadata:
annotations:
checksum/config: {{ include (print $.Template.BasePath "/app-configmap.yaml") . | sha256sum }}
labels:
app: demo
spec:
containers:
- name: demo
image: demo-app-image:1.0.1
ports:
- containerPort: 8081
livenessProbe:
httpGet:
path: /healthcheck
port: 8081
initialDelaySeconds: 30
periodSeconds: 60
failureThreshold: 3
successThreshold: 1
timeoutSeconds: 15
readinessProbe:
httpGet:
path: /healthcheck
port: 8081
initialDelaySeconds: 30
periodSeconds: 10
failureThreshold: 3
successThreshold: 1
timeoutSeconds: 15
volumeMounts:
- name: config-volume
mountPath: /config/app
volumes:
- name: config-volume
configMap:
name: demo-configmap
items:
- key: config
path: config.json
nodeSelector:
usage: demo-server

Deployment in version "v1" cannot be handled as a Deployment:

helm install failing with the below error
command
helm install --name helloworld helm
Below is the error once I ran above command
Error: release usagemetrics failed: Deployment in version "v1" cannot be handled as a Deployment: v1.Deployment.Spec: v1.DeploymentSpec.Template: v1.PodTemplateSpec.Spec: v1.PodSpec.Containers: []v1.Container: v1.Container.LivenessProbe: readObjectStart: expect { or n, but found 9, error found in #10 byte of ...|ssProbe":9001,"name"|..., bigger context ...|"imagePullPolicy":"IfNotPresent","livenessProbe":9001,"name":"usagemetrics-helm","ports":[{"containe|...
Below is the deployment.yaml file i feel the issue in liveness and probeness configuration .
apiVersion: apps/v1
kind: Deployment
metadata:
name: release-name-helm
spec:
replicas: 1
selector:
matchLabels:
app: release-name-helm
release: release-name
template:
metadata:
labels:
app: release-name-helm
release: release-name
spec:
containers:
- name: release-name-helm
imagePullPolicy: IfNotPresent
image: hellworld
ports:
- name: "http"
containerPort: 9001
envFrom:
- configMapRef:
name: release-name-helm
- secretRef:
name: release-name-helm
livenessProbe:
9001
readinessProbe:
9001
The problem seems to be related to the livenessProbe and readynessProbe that are both wrong.
An example of livenessProbe of http from the documentation here is:
livenessProbe
httpGet:
path: /healthz
port: 8080
httpHeaders:
- name: Custom-Header
value: Awesome
initialDelaySeconds: 3
periodSeconds: 3
Your yamls if you only want to have a check of the port should be like:
apiVersion: apps/v1
kind: Deployment
metadata:
name: release-name-helm
spec:
replicas: 1
selector:
matchLabels:
app: release-name-helm
release: release-name
template:
metadata:
labels:
app: release-name-helm
release: release-name
spec:
containers:
- name: release-name-helm
imagePullPolicy: IfNotPresent
image: hellworld
ports:
- name: "http"
containerPort: 9001
envFrom:
- configMapRef:
name: release-name-helm
- secretRef:
name: release-name-helm
livenessProbe:
tcpSocket:
port: 9001
initialDelaySeconds: 5
periodSeconds: 10
readinessProbe:
tcpSocket:
port: 9001
initialDelaySeconds: 5
periodSeconds: 10

Accessing kubernetes headless service over ambassador

I have deployed my service as headless server and did follow the kubernetes configuration as mentioned in this link (http://vertx.io/docs/vertx-hazelcast/java/#_using_this_cluster_manager). My service is load balanced and proxied using ambassador. Everything was working fine as long as the service was not headless. Once the service changed to headless, ambassador is not able to discover my services. Which means it was looking for clusterIP and it is missing now as the services are headless. What is that I need to include in my deployment.yaml so these services are discovered by ambassador.
Error I see " upstream connect error or disconnect/reset before headers. reset reason: connection failure"
I need these services to be headless because that is the only way to create a cluster using hazelcast. And I am creating web socket connection and vertx eventbus.
apiVersion: v1
kind: Service
metadata:
name: abt-login-service
labels:
chart: "abt-login-service-0.1.0-SNAPSHOT"
annotations:
fabric8.io/expose: "true"
fabric8.io/ingress.annotations: 'kubernetes.io/ingress.class: nginx'
getambassador.io/config: |
---
apiVersion: ambassador/v1
name: login_mapping
ambassador_id: default
kind: Mapping
prefix: /login/
service: abt-login-service.default.svc.cluster.local
use_websocket: true
spec:
type: ClusterIP
clusterIP: None
selector:
app: RELEASE-NAME-abt-login-service
ports:
- port: 80
targetPort: 8080
protocol: TCP
name: http
- name: hz-port-name
port: 5701
protocol: TCP```
```Deployment.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: RELEASE-NAME-abt-login-service
labels:
draft: draft-app
chart: "abt-login-service-0.1.0-SNAPSHOT"
spec:
replicas: 2
selector:
matchLabels:
app: RELEASE-NAME-abt-login-service
minReadySeconds: 30
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
template:
metadata:
labels:
draft: draft-app
app: RELEASE-NAME-abt-login-service
component: abt-login-service
spec:
serviceAccountName: vault-auth
containers:
- name: abt-login-service
env:
- name: SPRING_PROFILES_ACTIVE
value: "dev"
- name: _JAVA_OPTIONS
value: "-XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:MaxRAMFraction=2 -Dsun.zip.disableMemoryMapping=true -XX:+UseParallelGC -XX:Min
HeapFreeRatio=5 -XX:MaxHeapFreeRatio=10 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -Dhazelcast.diagnostics.enabled=true
"
image: "draft:dev"
imagePullPolicy: IfNotPresent
ports:
- containerPort: 8080
ports:
- containerPort: 5701
livenessProbe:
httpGet:
path: /health
port: 8080
initialDelaySeconds: 60
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
readinessProbe:
httpGet:
path: /health
port: 8080
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
resources:
limits:
cpu: 500m
memory: 1024Mi
requests:
cpu: 400m
memory: 512Mi
terminationGracePeriodSeconds: 10```
How can I make these services discoverable by ambassador?