Kubernetes, configure startupProbe to read stdout logs - kubernetes

Question

I have Java application running on kube and wanted to set startupProbe to look for 'tomcat started' text in java logs that are thrown to stdout; any ideas how to make that happen? I saw the documentation but there are only references to checking/curling some endpoint or running a command. Question is how can pod check its own logs
Also - I see that stdout logs are temporarily stored on /var/logs/containers directory (that is on NODE, not POD) so kinda not usefull.

Answer 1

In your question you are focused a bit more on the solution that the actual problem. So let's see how we can tackle the problem from different angle. This is the tomcat echo command that you are trying to base your probe on:
https://github.com/apache/tomcat/blob/0a2ee9b1ba7ded327c2aa2361cccff6a16cdef84/bin/catalina.sh#L506
As you can see this indeed tell you that Tomcat has started but it is not validating anything for you as the code continue to run. You will also notice that this output is not from the tomcat itself but from the script that launches tomcat.
Opened port on the other side is much better option to validate that the web server is up and running. Here's an example how this can be checked:
If you curl the tomcat port that is opened,the exit code would be 0 which will tell you that the server has started:
curl -s localhost:8080 > /dev/null
Here we are using echo $? to check the output from the previous command to validate it:
/usr/local/tomcat# echo $?
0
Moving forward, let's now make a test with the port that is not opened:
We are using the same way of checking it as in previous steps, the difference is just providing the different port (not opened):
/usr/local/tomcat# curl -s localhost:8010 > /dev/null
And then use echo $? to check the exit code of the command:
/usr/local/tomcat# echo $?
7
Alternative way described in this answer would be to query the Kubernetes API for pod logs:
GET /api/v1/namespaces/{namespace}/pods/{name}/log
Having said that, the best way of building things would be to have actual health check endpoint from you application.