Hello I am writing a application which I want to run in GKE and here is my deployment
apiVersion: apps/v1
kind: Deployment
metadata:
name: chopcast-dev
spec:
selector:
matchLabels:
app: chopcast-dev
replicas: 3
template:
metadata:
labels:
app: chopcast-dev
spec:
containers:
- name: chopcast-dev
image: eu.gcr.io/valued-amp-998877/chopcast:latest
livenessProbe:
failureThreshold: 3
httpGet:
path: /api/v1/healthz
port: 5000
scheme: HTTP
initialDelaySeconds: 60
periodSeconds: 60
successThreshold: 1
timeoutSeconds: 10
readinessProbe:
failureThreshold: 3
httpGet:
path: /api/v1/healthz
port: 5000
scheme: HTTP
initialDelaySeconds: 60
periodSeconds: 60
successThreshold: 1
timeoutSeconds: 10
ports:
- containerPort: 5000
Here is the service
apiVersion: v1
kind: Service
metadata:
name: chopcast-dev-service
spec:
selector:
app: chopcast-dev
ports:
- protocol: "TCP"
port: 5000
targetPort: 5000
type: LoadBalancer
and here is the ingress
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: chopcast-dev-ingress
annotations:
kubernetes.io/ingress.global-static-ip-name: gke-tribetacttics-static-ip
networking.gke.io/managed-certificates: chopcast-dev-certificate
#kubernetes.io/ingress.allow-http: "false"
spec:
backend:
serviceName: "chopcast-dev-service"
servicePort: 5000
rules:
- http:
paths:
- backend:
serviceName: "chopcast-dev-service"
servicePort: 5000
host: "gke.abcd.com"
every time I get my service running successfully and when I run the exposed service with the assigned IP it works file ie
x.x.x.x:5000/api/v1/healthz
But whenever I put the ingress static IP I always get
Error: Server Error
The server encountered a temporary error and could not complete your request.
Please try again in 30 seconds.
I tried with BackendConfig too.
As per the documentation, the health should pick up from the readinessProbe but it seems it is not picking up as I can see in load balancer section it is 0/1 backend is up. Unfortunately, I don't want to serve 200 on the / route.
Please help.
Finally, I figured out what was going wrong. The readinessProbe path should have a trailing slash as my application is designed that way.
readinessProbe:
failureThreshold: 3
httpGet:
path: /api/v1/healthz/
port: 5000
scheme: HTTP
initialDelaySeconds: 60
periodSeconds: 60
successThreshold: 1
timeoutSeconds: 10
Here are few checklist
Check your application name, namespace and labels.
Use selectors and check all selectors are proper.
Use trailing slashes if your app router is designed such way.
Also check the container port in your deployment YAML.
Cheers
Which version of kubernetes are you using?
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: chopcast-dev-ingress
spec:
rules:
- host: "gke.abcd.com"
http:
paths:
- pathType: Prefix
path: "/"
backend:
service:
name: chopcast-dev-service
port:
number: 5000
Related
I am trying to deploy 3 replicas using statefulsets.
Now， I have three pods:
statefulset-test-0
statefulset-test-1
statefulset-test-2
Then, I need to use the following to rolling update:
kubectl rollout restart statefulsets/statefulset-test
it will stop statefulset-test-2 pod and create a new statefulset-test-2 pod, then it will stop statefulset-test-1.
At this point, statefulset-test-2 is running the new image, statefulset-test-1 has been stopped so it can not accept requests, and statefulset-test-0 is running the old image.
I was wondering how the k8s handles the request to those pods. Is the k8s send the request to test-0 and test-2 randomly or do they send the request to the new pod?
here is my yaml:
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: statefulset-test
labels:
app: statefulset-test
spec:
minReadySeconds: 20
serviceName: "statefulset-test"
updateStrategy:
type: RollingUpdate
replicas: 3
selector:
matchLabels:
app: statefulset-test
template:
metadata:
labels:
app: statefulset-test
spec:
containers:
- name: statefulset-test
image: ..
imagePullPolicy: Always
ports:
- containerPort: 123
- containerPort: 456
livenessProbe:
httpGet:
path: /api/Health
port: 123
initialDelaySeconds: 180
periodSeconds: 80
timeoutSeconds: 20
failureThreshold: 2
readinessProbe:
httpGet:
path: /api/Health
port: 123
initialDelaySeconds: 20
periodSeconds: 5
successThreshold: 1
---
apiVersion: v1
kind: Service
metadata:
name: statefulset-service
spec:
type: NodePort
selector:
app: statefulset-test
ports:
- name: statefulset-main
protocol: TCP
port: 123
targetPort: 123
nodePort: 678
- name : statefulset-grpc
protocol: TCP
port: 456
targetPort: 456
nodePort: 679
So I'm working on my first helm deployment. I'm working on deploying polr an URL shortener.
I'm having issues with my first deployment. Absolutely nothing starts up and I'm puzzled about where to go from here.
I'm using commands like..
kubectl describe deployment/polr
helm lint pre-polr
helm install polr pre-polr --dry-run --debug
However, it doesn't give me any good details and since there are no pods spinning up. I feel like I'm missing some commands that might help. Could anyone suggest any?
Here are my manifests:
Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
name: polr
labels:
app: polr-app
spec:
replicas: 1
selector:
matchLabels:
app: polr-app
template:
metadata:
labels:
app: polr-app
spec:
containers:
- name: polr
image: matthewspah/polr
ports:
- name: http
containerPort: 8080
protocol: TCP
readinessProbe:
tcpSocket:
port: 8080
initialDelaySeconds: 5
periodSeconds: 10
livenessProbe:
tcpSocket:
port: 8080
initialDelaySeconds: 15
periodSeconds: 20
- name: polr-db
image: bitnami/mysql
ports:
- name: mysql
containerPort: 3306
protocol: TCP
readinessProbe:
tcpSocket:
port: 3306
initialDelaySeconds: 5
periodSeconds: 10
livenessProbe:
tcpSocket:
port: 3306
initialDelaySeconds: 15
periodSeconds: 20
Service
apiVersion: v1
kind: Service
metadata:
name: polr
labels:
name: polr-app
spec:
selector:
app: polr-app
type: ClusterIP
ports:
- port: 8080
targetPort: 8080
protocol: TCP
name: http
I have hangfire dashboard working properly on local environment. Now, I'm trying to setup it as a container (pod) inside my cluster, deployed to azure so I can access hangfire dashboard through its url. However, I'm having issue access to it.
Below is my setup:
[UsedImplicitly]
public void Configure(IApplicationBuilder app)
{
var hangFireServerOptions = new BackgroundJobServerOptions
{
Activator = new ContainerJobActivator(app.ApplicationServices)
};
app.UseHealthChecks("/liveness");
app.UseHangfireServer(hangFireServerOptions);
app.UseHangfireDashboard("/hangfire", new DashboardOptions()
{
AppPath = null,
DashboardTitle = "Hangfire Dashboard",
Authorization = new[]
{
new HangfireCustomBasicAuthenticationFilter
{
User = Configuration.GetSection("HangfireCredentials:UserName").Value,
Pass = Configuration.GetSection("HangfireCredentials:Password").Value
}
}
});
app.UseHttpsRedirection();
app.UseRouting();
app.UseAuthorization();
app.UseEndpoints(endpoints =>
{
endpoints.MapControllers();
});
HangfireJobScheduler.ScheduleJobs(app.ApplicationServices.GetServices<IScheduledTask>()); //serviceProvider.GetServices<IScheduledTask>()
}
Service.yml
apiVersion: v1
kind: Service
metadata:
name: task-scheduler-api
spec:
ports:
- port: 80
targetPort: 80
name: http
- port: 443
targetPort: 443
name: https
selector:
app: task-scheduler-api
Deployment.yml
apiVersion: apps/v1
kind: Deployment
metadata:
name: task-scheduler
spec:
selector:
matchLabels:
app: task-scheduler
template:
metadata:
labels:
app: task-scheduler
spec:
containers:
- name: task-scheduler
image: <%image-name%>
# Resources and limit
resources:
requests:
cpu: <%cpu_request%>
memory: <%memory_request%>
limits:
cpu: <%cpu_limit%>
memory: <%memory_limit%>
ports:
- name: http
containerPort: 80
- name: https
containerPort: 443
readinessProbe:
httpGet:
path: /liveness
port: 80
initialDelaySeconds: 3
periodSeconds: 5
timeoutSeconds: 30
livenessProbe:
httpGet:
path: /liveness
port: 80
initialDelaySeconds: 15
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 7
Ingress.yml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: "nginx"
cert-manager.io/cluster-issuer: letsencrypt
nginx.ingress.kubernetes.io/rewrite-target: /
name: task-scheulder-api-ingress
namespace: default
spec:
rules:
- host: example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: task-scheduler-api
port:
number: 80
tls:
- hosts:
- example.com
secretName: task-scheduler-tls-production
I'm trying to access the dashboard by running: example.com/hangfire, but got 503 Service Temporarily Unavailable.
I'm checking logs on the pod. Every seem to be fine:
...
...
Content root path: /data
Now listening on: http://0.0.0.0:80
Now listening on: https://0.0.0.0:443
Application started. Press Ctrl+C to shut down.
....
Would anyone know what I'm missing and how to resolve it ? Thank you
I have figured out the issue. The main issue is that I did not have the match value for selector app in deployment.yml and service.yml. If I do kubectl get ep, it is showing me that I do not have any endpoint assign to the task scheduler pod, meaning that it is not really deployed yet.
As soon as I updated values in the deployment.yml and service.yml, url is accessible.
service.yml
apiVersion: v1
kind: Service
metadata:
name: task-scheduler-api
spec:
ports:
- port: 80
targetPort: 80
name: http
- port: 443
targetPort: 443
name: https
selector:
app: task-scheduler-api # This needs to match with value inside the deployment
deployment.yml
apiVersion: apps/v1
kind: Deployment
metadata:
name: task-scheduler
spec:
selector:
matchLabels:
app: task-scheduler # This needs to match with value in service
template:
metadata:
labels:
app: task-scheduler # This needs to match as well
spec:
containers:
- name: task-scheduler
image: <%image-name%>
# Resources and limit
resources:
requests:
cpu: <%cpu_request%>
memory: <%memory_request%>
limits:
cpu: <%cpu_limit%>
memory: <%memory_limit%>
ports:
- name: http
containerPort: 80
- name: https
containerPort: 443
readinessProbe:
httpGet:
path: /liveness
port: 80
initialDelaySeconds: 3
periodSeconds: 5
timeoutSeconds: 30
livenessProbe:
httpGet:
path: /liveness
port: 80
initialDelaySeconds: 15
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 7
Hopefully someone would find it useful. Thank you
This could be related to the ingress class, bcs this moved from annotation to an own field in networking.k8s.io/v1 :
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: task-scheulder-api-ingress
annotations:
cert-manager.io/cluster-issuer: letsencrypt
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
ingressClassName: nginx
tls:
- hosts:
- "example.com"
secretName: task-scheduler-tls-production
rules:
- host: "example.com"
http:
paths:
- path: /hangfire
pathType: ImplementationSpecific
backend:
service:
name: task-scheduler-api
port:
number: 8080
You also do not need to specify port 80 & 443 at the service as the ingress is responsible for implementing TLS:
apiVersion: v1
kind: Service
metadata:
name: task-scheduler-api
spec:
ports:
- port: 8080
targetPort: http
protocol: TCP
name: http
selector:
app: task-scheduler-api
For convenience you should also update the deployment:
- name: http
containerPort: 80
protocol: TCP
We are running several services in a Kubernetes cluster on GKE (Google Kubernetes Engine) and are having trouble configuring routing with Ingress.
Let's say that we have auth-service and user-service and would like to access them by the following urls: http://www.example.com/auth and http://www.example.com/user. All requests to these urls should be redirected to the correct services and routed internally (http://www.example.com/user/people -> http://user-service/people).
These is our configuration for the auth service:
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: api-auth
spec:
replicas: 1
template:
metadata:
labels:
app: api-auth
tier: backend
track: stable
spec:
containers:
- name: api-auth
image: "<our-image>"
ports:
- name: http
containerPort: 9000
livenessProbe:
httpGet:
path: /health
port: 9000
initialDelaySeconds: 180
timeoutSeconds: 5
readinessProbe:
httpGet:
path: /health
port: 9000
initialDelaySeconds: 180
timeoutSeconds: 5
---
kind: Service
apiVersion: v1
metadata:
name: auth-service
labels:
app: api-auth
spec:
type: NodePort
selector:
app: api-auth
tier: backend
ports:
- port: 80
targetPort: 9000
Internally, the service is running on Tomcat on port 9000, this part is working fine.
The problem is with our Ingress configuration:
kind: Ingress
apiVersion: extensions/v1beta1
metadata:
name: auth-ingress
annotations:
kubernetes.io/ingress.global-static-ip-name: <our-static-api>
kubernetes.io/ingress.class: "gce"
labels:
app: api-auth
spec:
rules:
- http:
paths:
- path: /auth
backend:
serviceName: auth-service
servicePort: 80
- path: /auth/*
backend:
serviceName: auth-service
servicePort: 80
- path: /user
backend:
serviceName: user-service
servicePort: 80
- path: /user/*
backend:
serviceName: user-service
servicePort: 80
Whenever I access our static api (let's call it example.com for now) in the following way: http://www.example.com/auth, I am getting 502 - Bad gateway. Running kubectl describe ingress says, that our services's health is unknown.
I am running our of ideas what might be causing this strange behavior. Could someone point me to the right direction?
You mentioned on Slack the services are Spring Boot apps. It's probably not related to that, but you need to make sure the ingress path matches the context of your Spring Boot app, i. e. if your ingress path is /user, your app context must be configured with server.context-path=/user. The service would then be reachable under http://user-service/user.
Your health check will reflect your readiness probes. The health check needs to use your nodePort port because the request is coming from a Load Balancer. If your health check is targeting port 9000, the request will not get through because that port on the node is not active.
Make sure your LB health check is targeting the correct port (in the 30000 range) and that the target path will respond with 200, otherwise your health checks will continue to fail and you will continue to get 502 errors
I am configuring a StatefulSet where I want the number of replicas (spec.replicas as shown below) available to somehow pass as a parameter into the application instance. My application needs spec.replicas to determine the numer of replicas so it knows what rows to load from a MySQL table. I don't want to hard-code the number of replicas in both spec.replicas and the application parameter as that will not work when scaling the number of replicas up or down, since the application parameter needs to adjust when scaling.
Here is my StatefulSet config:
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
labels:
run: my-app
name: my-app
namespace: my-ns
spec:
replicas: 3
selector:
matchLabels:
run: my-app
serviceName: my-app
podManagementPolicy: Parallel
template:
metadata:
labels:
run: my-app
spec:
containers:
- name: my-app
image: my-app:latest
command:
- /bin/sh
- /bin/start.sh
- dev
- 2000m
- "0"
- "3" **Needs to be replaced with # replicas**
- 127.0.0.1
- "32990"
imagePullPolicy: Always
livenessProbe:
httpGet:
path: /health
port: 8081
initialDelaySeconds: 180
periodSeconds: 10
timeoutSeconds: 3
readinessProbe:
failureThreshold: 10
httpGet:
path: /ready
port: 8081
scheme: HTTP
initialDelaySeconds: 30
periodSeconds: 15
successThreshold: 1
timeoutSeconds: 3
ports:
- containerPort: 8080
protocol: TCP
resources:
limits:
memory: 2500Mi
imagePullSecrets:
- name: snapshot-pull
restartPolicy: Always
I have read the Kubernetes docs and the spec.replicas field is scoped at the pod or container level, never the StatefulSet, at least as far as I have seen.
Thanks in advance.
You could use a yaml anchor to do this:
Check out:
https://helm.sh/docs/chart_template_guide/yaml_techniques/#yaml-anchors
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
labels:
run: my-app
name: my-app
namespace: my-ns
spec:
replicas: &numReplicas 3
selector:
matchLabels:
run: my-app
serviceName: my-app
podManagementPolicy: Parallel
template:
metadata:
labels:
run: my-app
spec:
containers:
- name: my-app
image: my-app:latest
command:
- /bin/sh
- /bin/start.sh
- dev
- 2000m
- "0"
- *numReplicas
- 127.0.0.1
- "32990"
imagePullPolicy: Always
livenessProbe:
httpGet:
path: /health
port: 8081
initialDelaySeconds: 180
periodSeconds: 10
timeoutSeconds: 3
readinessProbe:
failureThreshold: 10
httpGet:
path: /ready
port: 8081
scheme: HTTP
initialDelaySeconds: 30
periodSeconds: 15
successThreshold: 1
timeoutSeconds: 3
ports:
- containerPort: 8080
protocol: TCP
resources:
limits:
memory: 2500Mi
imagePullSecrets:
- name: snapshot-pull
restartPolicy: Always
Normally you would use the downward api for this kind of thing. https://kubernetes.io/docs/tasks/inject-data-application/downward-api-volume-expose-pod-information/
However it is currently not possible for kubernetes to propagate deployment/statefulset spec data into the pod spec with the downward api, nor should it be. If you are responsible for this software I'd set up some internal functionality so that it can find it's peers and determine their count periodically.