I've seen in the Kubernetes dashboard the tracking of some information in the form of:
X happened 14 times and the last occurence was at time T
Where is this data coming from? Can I query for it using the kubectl? Is there a K8s API for this information as well? Or is the dashboard running some kind of aggregation internally?
X happened 14 times and the last occurence was at time T
That is OOB dashboard functionality and most probably you should dive deep into the code to answer this question.
The thing is of course dashboard relies on open data that you can collect on your own using kubectl - the only question is what exactly you wanna see as output. Using kubectl in spike with greps, sorts, seds, etc will give you the same information you just asked. Maybe you wanna create new question and specify your exact task?
Related
I am using grafana to visualize some data from my InfluxDB-database, that collects data from some sensors. That said, it's the first time for my working with both grafana and InfluxDB, also I'm pretty new to coding so my knowledge is very limited.
As I scroll through threads and forums on the web trying to find guidance, I find a lot of tutorials mostly 2-4 years old that seem to use features in grafana that are simply not available vor me.
For example I tried to set an alert which tells me when my sensor is delivering flawed values (values that I my case cannot physically be true) too often. But when I'm using avg() from the classic condition operations, I can't select a time frame in which I want the average value monitored.
My expression part of the alert settings
Is it a problem that has to be configured via grafana.ini? Is it because these features cannot be used with InfluxDB?
For some background information, I'm using a Ubuntu Server via VirtualBox to run both the database and the grafana server. I'm using a little python script to distribute the sensor data into the database.
If someone could help me out soon that would be great!
So there is List Release Deployments API. It allows to get all the deployments started after the given timestamp. However, how to get those completed after it?
Rationale
Suppose I want to write a scheduled job that fetches information about completed deployments and pushes it to an Azure Application Insights bucket (for example for DORA metrics). I do not understand how it can be done easily without the ability to filter by the completion date. A relatively hard way would be to fetch by the started date, notice all the deployments inProgress or notDeployed and record them in a dedicated database. Then on the next polling cycle fetch new deployments and all those still recorded in the aforementioned database. This is much more complicated than it could have been with the ability to filter by the completion date.
Am I missing anything here? Maybe there is a simpler way (possibly using another API) that I just do not see?
EDIT 1
By the way, the hard way is even harder than I thought, since apparently there is no way to fetch release deployments by their Id.
EDIT 2
The plot thickens. If a stage has post deployment approvals, then the stage is reported as inProgress, even though de facto it has already been deployed. So an API that just filters by completion date would omit such deployments. It has to be with an option to include such deployments.
Currently I tried to fetch already rotated logs within the node using --since-time parameter.
Can anybody suggest what is the command/mechanism to fetch already rotated logs within kubernetes architecture using commands
You can't. Kubernetes does not store logs for you, it's just providing an API to access what's on disk. For long term storage look at things like Loki, ElasticSearch, Splunk, SumoLogic, etc etc.
I would like to be able to see all of the various things that happened to a kube cluster on a timeline, including when nodes were found to be dead, when new nodes were added, when pods crashed and when they were restarted.
So far the best that we have found is kubectl get event but that seems to have a few limitations:
it doesn't go back in time that far (I'm not sure how far it goes back. A day?)
it combines similar events and orders the resulting list by the time of the latest event in each group. This makes it impossible to know what happened during some time range since events in that range may have been combined with later events outside the range.
One idea that I have is to write a pod that will use the API to watch the stream of events and log them to a file. This would let us control retention and it seems that events that occur while we are watching will not be combined, solving the second problem as well.
What are other people doing about this?
My understanding is that Kubernetes itself dedups events, documented here:
https://github.com/kubernetes/kubernetes/blob/master/docs/design/event_compression.md
Once that happens, there is no way to get the individual events back.
See https://github.com/kubernetes/kubernetes/issues/36304 for complaints how that loses info. https://github.com/kubernetes/kubernetes/pull/46034 at least improved the message. See also https://github.com/kubernetes/enhancements/pull/1291 KEP for recent discussion and proposal to improve usability in kubectl.
How long events are retained? Their "time-to-live" is apparently controlled by kube-apiserver --event-ttl option, defaults to 1 hour:
https://github.com/kubernetes/kubernetes/blob/da53a247633/cmd/kube-apiserver/app/options/options.go#L71-L72
You can raise this. Might require more resources for etcd — from what I saw in some 2015 github discussions, event TTL used to be 2 days, and events were the main thing stressing etcd...
In a pinch, it might be possible to figure out what happened earlier from various log, especially the kubelet logs?
Saving events
Running kubectl get event -o yaml --watch into a persistent file sounds like a simple thing to do. I think when you watch events as they arrive, you see them pre-dedup.
Heapster can send events to some of the supported sinks:
https://github.com/kubernetes/heapster/blob/master/docs/sink-configuration.md
Eventrouter can send events to various sinks: https://github.com/heptiolabs/eventrouter/tree/master/sinks
Have you checked out the pod specific events tab in the Dashboard?
Some events from a cluster I have running in GKE:
kubernetes/heapster can persist event to gcl and influxdb, but for now there is no api to access stored data
Is there a way for me to programmatically get notified when Bluemix auto scaling has scaled up or down?
I'm reading streaming data from a queue and would like to make sure the number of instances that I have are balanced and data is partitioned correctly
At present this kind of notification service is not available, only you can do is query the instance scaling history in Web UI. I think this requirement is interesting and should be considered to provide to developer in the future.
This kind of alert isn't available yet but you can write a simple script monitoring output of
cf app (appname)
It returns the number of instances running and the state of each one, with the right combination of awk and grep (or a perl script for example) you could have your own alerter while waiting for this of functionality