My application has a health check at the endpoint /service?cmd=watchdog and when I try to configure a HTTP liveness probe in kubernetes, the above endpoint is getting encoded to utf-8 as it gets applied to the pods (when I do a describe on the pod), the health check is being applies as /service%3fcmd=watchdog and that situation does not work for me.
in my deployment.yaml:
httpGet:
path: "/service/?cmd=watchdog"
port: 8080
Is there any way to avoid the encoding / work around this situation?
Related
My readiness probe specifies HTTPS and 8200 as the port to check my hashicorp vault pod.
readinessProbe:
httpGet:
scheme: HTTPS
path: '/v1/sys/health?activecode=200'
port: 8200
Once the pod is running kubectl describe pod shows this
Readiness probe failed: Error checking seal status: Error making API request.
URL: GET http://127.0.0.1:8200/v1/sys/seal-status
Code: 400. Raw Message:
Client sent an HTTP request to an HTTPS server.
I have found this similar problem. See the whole answer here.
According to this documentation:
The http/https scheme is controlled by the tlsDisable value.
When set to true, changes URLs from https to http (such as the VAULT_ADDR=http://127.0.0.1:8200 environment variable set on the Vault pods).
To turn it off:
global:
tlsDisable: false
NOTE:
Vault should always be used with TLS in production to provide secure communication between clients and the Vault server. It requires a certificate file and key file on each host where Vault is running.
See also this documentation. One can find there many examples of using readiness and liveness probes, f.e
readinessProbe:
exec:
command:
- /bin/sh
- -ec
- vault status
initialDelaySeconds: 5
periodSeconds: 5
I'm using janusgraph docker image - https://hub.docker.com/r/janusgraph/janusgraph
In my kubernetes deployment to initialise the remote graph using groovy script mounted to docker-entrypoint-initdb.d
This works as expected but in case if the remote host is not ready the janusgraph container throws exception and is still in the running mode.
Because of this kubernetes will not attempt to restart the container again. Is there any way so that I can configure this janusgraph container to terminate in case of any exception
As #Gavin has mentioned you can use probes to check if containers are working. Liveness Probes is used to know when containers are failed. If a container is unresponsive - it can restart the container.
Readiness probes inform when the container is available for accepting traffic. The readiness probe is used to control which pods are used as the backends for a service. A pod is considered ready when all of its containers are ready. If a pod is not ready, it is removed from service Endpoints.
Kubernetes supports three mechanisms for implementing liveness and readiness probes:
1) making an HTTP request against a container
This probes have additional fields that can be set on httpGet:
host: Host name to connect to, defaults to the pod IP. You probably want to set "Host" in httpHeaders instead.
scheme: Scheme to use for connecting to the host (HTTP or HTTPS). Defaults to HTTP.
path: Path to access on the HTTP server. Defaults to /.
httpHeaders: Custom headers to set in the request. HTTP allows repeated headers.
port: Name or number of the port to access on the container. Number must be in the range 1 to 65535.
Read more: http-probes.
livenessProbe:
httpGet:
path: /healthz
port: liveness-port
2) opening a TCP socket against a container
initialDelaySeconds: 15
livenessProbe: ~
periodSeconds: 20
port: 8080
tcpSocket: ~
3) running a command inside a container
livenessProbe:
exec:
command:
- sh
- /tmp/status_check.sh
initialDelaySeconds: 10
If you will get status code different than 0 this will mean that probe failed.
You can also add to probes additional params such as initialDelaySeconds: indicate number of seconds after the container has started before liveness or readiness probes are initiated. See: configuring-probes.
In every case add also restartPolicy: Never
to your pods definition. By default is always.
A readinessProbe could be employed here with a command like janusgraph show-config or something similar which will exit with code -1
spec:
containers:
- name: liveness
image: janusgraph/janusgraph:latest
readinessProbe:
exec:
command:
- janusgraph
- show-config
Kubernetes will terminate the pod if the readinessProbe fails. A livenessProbe could also be used here too, in case this pod needs to be terminated if the remote host ever becomes unavailable.
Consider enabling JanusGraph server metrics, which could then be used with Prometheus for additional monitoring or even with the livenessProbe itself.
I have set up liveness and readiness probes for an application, e.g.:
readinessProbe:
httpGet:
path: /actuator/health/readiness
port: 8080
...
livenessProbe:
httpGet:
path: /actuator/health/liveness
port: 8080
What I would like to do is whenever the probe is invoked by Kubernetes, its result is sent to an endpoint passing some additional information suchs as pod name, Docker image used, value from a secret, etc.
PROBE INVOKED -> RESULT COLLECTED -> INVOKE ENDPOINT (http://xxx - payload {"probe": "liveness", "result": "OK", "image" : "app:1.0.0", "key" : "VALUE_EXTRACTED_FROM_A_SECRET", ...}
Could you indicate a way to achieve this?
There are many ways to do this (e.g. depending on your pipeline).
You could call the probe endpoints yourself and react if the pod is not in the desired state.
You could could then call kubectl describe, log etc. and pass the result to your endpoint.
You could do so in with different approaches bash, python, etc...
I'm trying to use exec probes for readiness and liveness in GKE. This is because it is part of Kubernetes' recommended way to do health checks on gRPC back ends. However when I put the exec probe config into my deployment yaml and apply it, it doesn't take effect in GCP. This is my container yaml:
- name: rev79-uac-sandbox
image: gcr.io/rev79-232812/uac:latest
imagePullPolicy: Always
ports:
- containerPort: 3011
readinessProbe:
exec:
command: ["bin/grpc_health_probe", "-addr=:3011"]
initialDelaySeconds: 5
livenessProbe:
exec:
command: ["bin/grpc_health_probe", "-addr=:3011"]
initialDelaySeconds: 10
But still the health checks fail and when I look at the health check configuration in the GCP console I see a plain HTTP health check directed at '/'
When I edit a health check in GCP console there doesn't seem to be any way to choose an exec type. Also I can't see any mention of liveness checks as contrasted to readiness checks even though these are separate Kubernetes things.
Does Google cloud support using exec for health checks?
If so, how do I do it?
If not, how can I health check a gRPC server?
TCP probes are useful when we are using gRPC Services rather than using HTTP probes.
- containerPort: 3011
readinessProbe:
tcpSocket:
port: 3011
initialDelaySeconds: 5
periodSeconds: 10
livenessProbe:
tcpSocket:
port: 3011
initialDelaySeconds: 15
periodSeconds: 20
the kubelet will attempt to open a socket to your container on the specified port. If it can establish a connection, the container is considered healthy, if it can’t it is considered a failure
define-a-tcp-liveness-probe
Exec probes work in GKE just the same way they work everywhere. You can view liveness probe result in "kubectl describe pod". Or you can simply log in into pod, execute command and see its return code.
The server has to implement the grpc probe protocol as indicated here as indicated in this article
Both answers from Vasily Angapov and Suresh Vishnoi should in theory work, however in practice they don't (at least in my practice).
So my solution was to start another server on my backend container - an HTTP server that simply has the job of executing the health check whenever it gets a request and returning a 200 status if it passes and a 503 if it fails.
I also had to open a second port on my container for that server to listen on.
I searched for 'readi', 'ready', 'live' etc in the kub swagger. I only see
io.k8s.api.core.v1.PodReadinessGate
thank you
That's one thing you would define. For example the following yaml file:
apiVersion: v1
kind: Pod
metadata:
name: nginx
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx
ports:
- containerPort: 80
livenessProbe: #this block performs liveness probes
httpGet:
path: /healthz
port: 80
readinessProbe: #this block performs readiness probes
httpGet:
path: /
port: 80
So, a pod with nginx. I can simply add the blocks highlighted in the yaml file and there it is. kubelet will check on them. Of course you have to have something serving there (/healthz, in this example), otherwise you will get 404.
You can add some configuration to the probes, like the other answer suggests. There are some more options than those.
According to Configure Liveness and Readiness Probes services can be configured to use
liveness command
TCP liveness probe
liveness HTTP request
So if your service use HTTP requests for liveness and readiness you can see in pod definition section livenessProbe (same for readinessProbe)
livenessProbe:
httpGet:
path: /healthz
port: 8080
initialDelaySeconds: 3
periodSeconds: 3
See full example here
There is no way to check the state of Liveness and Readiness probes directly.
You can check the resulting state of the pod which reflects changes in the state of Liveness and Readiness probes, but with some delay caused by threshold values.
Using kubectl describe pod you can also see some events at the bottom, but you can only see them after they occur. You can’t have it as a reply to the request.
You can also look into REST requests that are running under the hood of kubectl commands. All you need to do is adding a verbose flag to kubectl command:
-v, --v=0: Set the level of log output to debug-level (0~4) or trace-level (5~10)