Kubernetes Statefulsets traffic issue - kubernetes

I am trying to deploy 3 replicas using statefulsets.
Now, I have three pods:
statefulset-test-0
statefulset-test-1
statefulset-test-2
Then, I need to use the following to rolling update:
kubectl rollout restart statefulsets/statefulset-test
it will stop statefulset-test-2 pod and create a new statefulset-test-2 pod, then it will stop statefulset-test-1.
At this point, statefulset-test-2 is running the new image, statefulset-test-1 has been stopped so it can not accept requests, and statefulset-test-0 is running the old image.
I was wondering how the k8s handles the request to those pods. Is the k8s send the request to test-0 and test-2 randomly or do they send the request to the new pod?
here is my yaml:
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: statefulset-test
labels:
app: statefulset-test
spec:
minReadySeconds: 20
serviceName: "statefulset-test"
updateStrategy:
type: RollingUpdate
replicas: 3
selector:
matchLabels:
app: statefulset-test
template:
metadata:
labels:
app: statefulset-test
spec:
containers:
- name: statefulset-test
image: ..
imagePullPolicy: Always
ports:
- containerPort: 123
- containerPort: 456
livenessProbe:
httpGet:
path: /api/Health
port: 123
initialDelaySeconds: 180
periodSeconds: 80
timeoutSeconds: 20
failureThreshold: 2
readinessProbe:
httpGet:
path: /api/Health
port: 123
initialDelaySeconds: 20
periodSeconds: 5
successThreshold: 1
---
apiVersion: v1
kind: Service
metadata:
name: statefulset-service
spec:
type: NodePort
selector:
app: statefulset-test
ports:
- name: statefulset-main
protocol: TCP
port: 123
targetPort: 123
nodePort: 678
- name : statefulset-grpc
protocol: TCP
port: 456
targetPort: 456
nodePort: 679

Related

What are some good commands to troubleshoot my helm deployment?

So I'm working on my first helm deployment. I'm working on deploying polr an URL shortener.
I'm having issues with my first deployment. Absolutely nothing starts up and I'm puzzled about where to go from here.
I'm using commands like..
kubectl describe deployment/polr
helm lint pre-polr
helm install polr pre-polr --dry-run --debug
However, it doesn't give me any good details and since there are no pods spinning up. I feel like I'm missing some commands that might help. Could anyone suggest any?
Here are my manifests:
Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
name: polr
labels:
app: polr-app
spec:
replicas: 1
selector:
matchLabels:
app: polr-app
template:
metadata:
labels:
app: polr-app
spec:
containers:
- name: polr
image: matthewspah/polr
ports:
- name: http
containerPort: 8080
protocol: TCP
readinessProbe:
tcpSocket:
port: 8080
initialDelaySeconds: 5
periodSeconds: 10
livenessProbe:
tcpSocket:
port: 8080
initialDelaySeconds: 15
periodSeconds: 20
- name: polr-db
image: bitnami/mysql
ports:
- name: mysql
containerPort: 3306
protocol: TCP
readinessProbe:
tcpSocket:
port: 3306
initialDelaySeconds: 5
periodSeconds: 10
livenessProbe:
tcpSocket:
port: 3306
initialDelaySeconds: 15
periodSeconds: 20
Service
apiVersion: v1
kind: Service
metadata:
name: polr
labels:
name: polr-app
spec:
selector:
app: polr-app
type: ClusterIP
ports:
- port: 8080
targetPort: 8080
protocol: TCP
name: http

How is it that my k8s service is accessible without explicitly exposing a port?

I am running a http server in k8s, I am able to access the server via it's svc name from another service by making a call to https://e2e-test-runner.hogwarts.svc.cluster.local However, I haven't explicitly configured service port. How does this work?
Does k8s map container port as service port when no service port is present?
This is my service definition:
apiVersion: v1
kind: Service
metadata:
name: e2e-test-runner
namespace: hogwarts
spec:
selector:
app: e2e-test-runner
This is my deployment definition:
apiVersion: apps/v1
kind: Deployment
metadata:
name: e2e-test-runner
namespace: hogwarts
spec:
strategy:
rollingUpdate:
maxSurge: 3
maxUnavailable: 0
revisionHistoryLimit: 3
selector:
matchLabels:
app: e2e-test-runner
template:
metadata:
labels:
app: e2e-test-runner
app.kubernetes.io/name: e2e-test-runner
spec:
containers:
- name: app
image: $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com/e2e-test-runner:$IMAGE_VERSION
ports:
- containerPort: 3000
resources:
requests:
memory: "250Mi"
cpu: "100m"
limits:
memory: "1Gi"
cpu: "1000m"
livenessProbe:
httpGet:
path: /health_check
port: 3000
scheme: "HTTPS"
initialDelaySeconds: 20
periodSeconds: 10
timeoutSeconds: 2
readinessProbe:
httpGet:
path: /health_check
port: 3000
scheme: "HTTPS"
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 5
# Gives pod 15 seconds to complete any outstanding requests before being force killed
terminationGracePeriodSeconds: 15

Readiness probe failing when second pod gets scheduled to the same node

I have a k8s service which maps to pod deployment with 2 replicas and is exposed as clusterIp service. I am seeing an issue when the 2nd pod gets scheduled to the same node the readiness probe (http call to an api in container port) is failing with "unable to connect error" . Is this due to some port conflict?
Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
name: demo-deployment
namespace: demo
labels:
app: demo
spec:
replicas: 2
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
selector:
matchLabels:
app: demo
template:
metadata:
annotations:
checksum/config: {{ include (print $.Template.BasePath "/app-configmap.yaml") . | sha256sum }}
labels:
app: demo
spec:
containers:
- name: demo
image: demo-app-image:1.0.1
ports:
- containerPort: 8081
livenessProbe:
httpGet:
path: /healthcheck
port: 8081
initialDelaySeconds: 30
periodSeconds: 60
failureThreshold: 3
successThreshold: 1
timeoutSeconds: 15
readinessProbe:
httpGet:
path: /healthcheck
port: 8081
initialDelaySeconds: 30
periodSeconds: 10
failureThreshold: 3
successThreshold: 1
timeoutSeconds: 15
volumeMounts:
- name: config-volume
mountPath: /config/app
volumes:
- name: config-volume
configMap:
name: demo-configmap
items:
- key: config
path: config.json
nodeSelector:
usage: demo-server
Service
apiVersion: v1
kind: Service
metadata:
name: demo-service
namespace: demo
labels:
app: demo-service
spec:
selector:
app: demo
ports:
- name: admin-port
protocol: TCP
port: 26001
targetPort: 8081

Readiness probe failing when second pod gets scheduled to the same container in k8s

I have a k8s service which maps to pod deployment with 2 replicas and is exposed as clusterIp service. I am seeing an issue when the 2nd pod gets scheduled to the same node the readiness probe (http call to an api in container port) is failing with "unable to connect error" . Is this due to some port conflict?
apiVersion: apps/v1
kind: Deployment
metadata:
name: demo-deployment
namespace: demo
labels:
app: demo
spec:
replicas: 2
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
selector:
matchLabels:
app: demo
template:
metadata:
annotations:
checksum/config: {{ include (print $.Template.BasePath "/app-configmap.yaml") . | sha256sum }}
labels:
app: demo
spec:
containers:
- name: demo
image: demo-app-image:1.0.1
ports:
- containerPort: 8081
livenessProbe:
httpGet:
path: /healthcheck
port: 8081
initialDelaySeconds: 30
periodSeconds: 60
failureThreshold: 3
successThreshold: 1
timeoutSeconds: 15
readinessProbe:
httpGet:
path: /healthcheck
port: 8081
initialDelaySeconds: 30
periodSeconds: 10
failureThreshold: 3
successThreshold: 1
timeoutSeconds: 15
volumeMounts:
- name: config-volume
mountPath: /config/app
volumes:
- name: config-volume
configMap:
name: demo-configmap
items:
- key: config
path: config.json
nodeSelector:
usage: demo-server

Accessing kubernetes headless service over ambassador

I have deployed my service as headless server and did follow the kubernetes configuration as mentioned in this link (http://vertx.io/docs/vertx-hazelcast/java/#_using_this_cluster_manager). My service is load balanced and proxied using ambassador. Everything was working fine as long as the service was not headless. Once the service changed to headless, ambassador is not able to discover my services. Which means it was looking for clusterIP and it is missing now as the services are headless. What is that I need to include in my deployment.yaml so these services are discovered by ambassador.
Error I see " upstream connect error or disconnect/reset before headers. reset reason: connection failure"
I need these services to be headless because that is the only way to create a cluster using hazelcast. And I am creating web socket connection and vertx eventbus.
apiVersion: v1
kind: Service
metadata:
name: abt-login-service
labels:
chart: "abt-login-service-0.1.0-SNAPSHOT"
annotations:
fabric8.io/expose: "true"
fabric8.io/ingress.annotations: 'kubernetes.io/ingress.class: nginx'
getambassador.io/config: |
---
apiVersion: ambassador/v1
name: login_mapping
ambassador_id: default
kind: Mapping
prefix: /login/
service: abt-login-service.default.svc.cluster.local
use_websocket: true
spec:
type: ClusterIP
clusterIP: None
selector:
app: RELEASE-NAME-abt-login-service
ports:
- port: 80
targetPort: 8080
protocol: TCP
name: http
- name: hz-port-name
port: 5701
protocol: TCP```
```Deployment.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: RELEASE-NAME-abt-login-service
labels:
draft: draft-app
chart: "abt-login-service-0.1.0-SNAPSHOT"
spec:
replicas: 2
selector:
matchLabels:
app: RELEASE-NAME-abt-login-service
minReadySeconds: 30
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
template:
metadata:
labels:
draft: draft-app
app: RELEASE-NAME-abt-login-service
component: abt-login-service
spec:
serviceAccountName: vault-auth
containers:
- name: abt-login-service
env:
- name: SPRING_PROFILES_ACTIVE
value: "dev"
- name: _JAVA_OPTIONS
value: "-XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:MaxRAMFraction=2 -Dsun.zip.disableMemoryMapping=true -XX:+UseParallelGC -XX:Min
HeapFreeRatio=5 -XX:MaxHeapFreeRatio=10 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -Dhazelcast.diagnostics.enabled=true
"
image: "draft:dev"
imagePullPolicy: IfNotPresent
ports:
- containerPort: 8080
ports:
- containerPort: 5701
livenessProbe:
httpGet:
path: /health
port: 8080
initialDelaySeconds: 60
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
readinessProbe:
httpGet:
path: /health
port: 8080
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
resources:
limits:
cpu: 500m
memory: 1024Mi
requests:
cpu: 400m
memory: 512Mi
terminationGracePeriodSeconds: 10```
How can I make these services discoverable by ambassador?