Filter kubernetes file - kubernetes

I have the following file:
# HELP container_cpu_usage_seconds_total [ALPHA] Cumulative cpu time consumed by the container in core-seconds
# TYPE container_cpu_usage_seconds_total counter
container_cpu_usage_seconds_total{container="coredns",namespace="kube-system",pod="coredns-64897985d-qzvj8"} 1075.30302335 1641411355244
container_cpu_usage_seconds_total{container="etcd",namespace="kube-system",pod="etcd-minikube"} 7948.244422673 1641411341787
container_cpu_usage_seconds_total{container="kindnet-cni",namespace="kube-system",pod="kindnet-v9rn4"} 253.401092815 1641411342227
container_cpu_usage_seconds_total{container="kube-apiserver",namespace="kube-system",pod="kube-apiserver-minikube"} 21314.526032702 1641411341706
container_cpu_usage_seconds_total{container="kube-controller-manager",namespace="kube-system",pod="kube-controller-manager-minikube"} 9960.616171401 1641411346752
container_cpu_usage_seconds_total{container="kube-proxy",namespace="kube-system",pod="kube-proxy-ktclh"} 220.17024815 1641411352327
container_cpu_usage_seconds_total{container="kube-scheduler",namespace="kube-system",pod="kube-scheduler-minikube"} 1216.162832124 1641411355059
container_cpu_usage_seconds_total{container="metrics-server",namespace="kube-system",pod="metrics-server-6b76bd68b6-lpx4q"} 715.97119974 1641411344274
container_cpu_usage_seconds_total{container="storage-provisioner",namespace="kube-system",pod="storage-provisioner"} 47.685435216 1641411354429
# HELP container_memory_working_set_bytes [ALPHA] Current working set of the container in bytes
# TYPE container_memory_working_set_bytes gauge
container_memory_working_set_bytes{container="coredns",namespace="kube-system",pod="coredns-64897985d-qzvj8"} 1.5364096e+07 1641411355244
container_memory_working_set_bytes{container="etcd",namespace="kube-system",pod="etcd-minikube"} 5.9752448e+07 1641411341787
container_memory_working_set_bytes{container="kindnet-cni",namespace="kube-system",pod="kindnet-v9rn4"} 1.0326016e+07 1641411342227
container_memory_working_set_bytes{container="kube-apiserver",namespace="kube-system",pod="kube-apiserver-minikube"} 2.66002432e+08 1641411341706
container_memory_working_set_bytes{container="kube-controller-manager",namespace="kube-system",pod="kube-controller-manager-minikube"} 5.9129856e+07 1641411346752
container_memory_working_set_bytes{container="kube-proxy",namespace="kube-system",pod="kube-proxy-ktclh"} 2.00704e+07 1641411352327
container_memory_working_set_bytes{container="kube-scheduler",namespace="kube-system",pod="kube-scheduler-minikube"} 2.3130112e+07 1641411355059
container_memory_working_set_bytes{container="metrics-server",namespace="kube-system",pod="metrics-server-6b76bd68b6-lpx4q"} 2.6923008e+07 1641411344274
container_memory_working_set_bytes{container="storage-provisioner",namespace="kube-system",pod="storage-provisioner"} 1.4209024e+07 1641411354429
A few questions:
What format is this? I know it isn't JSON.
Can I use jq to parse/filter this data? I would like to get all metrics on the coredns container:
container_cpu_usage_seconds_total{container="coredns",namespace="kube-system",pod="coredns-64897985d-qzvj8"} 1075.30302335 1641411355244
container_memory_working_set_bytes{container="coredns",namespace="kube-system",pod="coredns-64897985d-qzvj8"} 1.5364096e+07 1641411355244

You could convert your file to JSON using https://github.com/prometheus/prom2json
Then it's jq all the way down, if you wish. E.g. with your input:
prom2json sample.prom | jq '
.[] | .metrics |= map(select(.labels.container=="coredns") )'
yields
{
"name": "container_memory_working_set_bytes",
"help": "[ALPHA] Current working set of the container in bytes",
"type": "GAUGE",
"metrics": [
{
"labels": {
"container": "coredns",
"namespace": "kube-system",
"pod": "coredns-64897985d-qzvj8"
},
"timestamp_ms": "1641411355244",
"value": "1.5364096e+07"
}
]
}
{
"name": "container_cpu_usage_seconds_total",
"help": "[ALPHA] Cumulative cpu time consumed by the container in core-seconds",
"type": "COUNTER",
"metrics": [
{
"labels": {
"container": "coredns",
"namespace": "kube-system",
"pod": "coredns-64897985d-qzvj8"
},
"timestamp_ms": "1641411355244",
"value": "1075.30302335"
}
]
}

Related

Extract status of Kubernetes CR created via ansible-operator

I am new to json query. Facing trouble extracting the status.conditions[ansibleResult].type
I have a CRD defined and created CR against the same, which is picked up by operator-sdk running ansible in the background. I am updating the CRD to provide relevant status once CR is accepted and processed by operator-sdk.
CR output in json appears like below.
{
"apiVersion": "vault.cpe.oraclecloud.com/v1alpha1",
"kind": "OciVaultKeys",
"metadata": {
"annotations": {
"kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"vault.cpe.oraclecloud.com/v1alpha1\",\"kind\":\"OciVaultKeys\",\"metadata\":{\"annotations\":{},\"name\":\"operator-key-broken\",\"namespace\":\"tms\"},\"spec\":{\"freeformTags\":[{\"key\":\"Type\",\"value\":\"Optional-Values-Added\"}],\"ociVaultKeyName\":\"operator-key-broken\",\"ociVaultKeyShapeAlgorithm\":\"RSA\",\"ociVaultKeyShapeLength\":32,\"ociVaultName\":\"ocivault-sample-12\"}}\n"
},
"creationTimestamp": "2022-03-18T07:43:03Z",
"finalizers": [
"vault.cpe.oraclecloud.com/finalizer"
],
"generation": 1,
"name": "operator-key-broken",
"namespace": "tms",
"resourceVersion": "717880023",
"selfLink": "/apis/vault.cpe.oraclecloud.com/v1alpha1/namespaces/tms/ocivaultkeys/operator-key-broken",
"uid": "0d634e72-f592-48e0-be9b-ebfa017b2dfe"
},
"spec": {
"freeformTags": [
{
"key": "Type",
"value": "Optional-Values-Added"
}
],
"ociVaultKeyName": "operator-key-broken",
"ociVaultKeyShapeAlgorithm": "RSA",
"ociVaultKeyShapeLength": 32,
"ociVaultName": "ocivault-sample-12"
},
"status": {
"conditions": [
{
"lastTransitionTime": "2022-03-18T07:43:27Z",
"message": "",
"reason": "",
"status": "False",
"type": "Successful"
},
{
"lastTransitionTime": "2022-03-18T08:26:08Z",
"message": "Running reconciliation",
"reason": "Running",
"status": "False",
"type": "Running"
},
{
"ansibleResult": {
"changed": 0,
"completion": "2022-03-18T08:26:24.217728",
"failures": 1,
"ok": 14,
"skipped": 1
},
"lastTransitionTime": "2022-03-18T08:26:25Z",
"message": "The task includes an option with an undefined variable. The error was: No first item, sequence was empty.\n\nThe error appears to be in '/home/opc/cpe-workstation/mr_folder/workspace-2/osvc-kubernetes-operators/oci-services/roles/ocivaultkeys/tasks/fetch_vault_details_oci.yml': line 12, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: DEBUG | Fetch Vault Details | Extract Vault OCID n service_endpoint in source region\n ^ here\n",
"reason": "Failed",
"status": "True",
"type": "Failure"
}
]
}
}
I wish to reliably extract the status.conditions[].type (for the element ansibleResult) in CRD.
CRD definition extract is as below
- name: v1alpha1
served: true
storage: true
additionalPrinterColumns:
- description: 'Status of the OCI Vault Key'
jsonPath: .status.conditions[-1].type
name: STATUS
type: string
priority: 0
CRD is looking for a jsonPath expression to extract.
Thanks
Please try following :
kubectl get ocivaultkeys operator-key-broken -o jsonpath='{.status.conditions[?(#.ansibleResult)].type}'
Expected output : Failure
jsonpath help

what is the meaning of 'window' in results from k8s metric server api

When I type this command on cli:
kubectl get --raw /apis/metrics.k8s.io/v1beta1/namespaces/<NAMESPACE>/pods/<POD_NAME> | jq
I can get these results as below:
{
"kind": "PodMetrics",
"apiVersion": "metrics.k8s.io/v1beta1",
"metadata": {
"name": "busybox",
"namespace": "default",
"selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/default/pods/busybox",
"creationTimestamp": "2019-12-10T18:23:20Z"
},
"timestamp": "2019-12-10T18:23:12Z",
"window": "30s",
"containers": [
{
"name": "busybox",
"usage": {
"cpu": "0",
"memory": "364Ki"
}
}
]
}
What is the meaning of that "window" item?
I am really want to know what it is exactly.
According to k8s source code:
// PodMetrics sets resource usage metrics of a pod.
type PodMetrics struct {
metav1.TypeMeta
metav1.ObjectMeta
// The following fields define time interval from which metrics were
// collected from the interval [Timestamp-Window, Timestamp].
Timestamp metav1.Time
Window metav1.Duration
// Metrics for all containers are collected within the same time window.
Containers []ContainerMetrics
}
You are most likely interested in this comment:
The following fields define time interval from which metrics were collected from the interval [Timestamp-Window, Timestamp].
So the usage result is an averaged data gathered over this window/interval.

Enable .NET Core collection in AppService with ARM or PowerShell

I would like to enable the .NET Core collection level with applicationInsight in Azure AppService as shown in below picture. From Azure Portal it works well. Keep in mind that by default value is set to disabled:
Now I would like to automate this using either ARM template or powershell.
I did a export template to see how it looks on ARM but there is no settings related to .NET Core collection
I check documentation on MS website but nothing about ARM template with enabling collection too
In PowerShell same problem
Is there anyone in the community who know how to enable the collection using ARM or PowerShell ?
Thanks a lot !
It's simply setting an app setting called XDT_MicrosoftApplicationInsights_Mode with value recommended (to enable) or default (to disable) as described here. You can do that both in ARM or PowerShell as below.
ARM (check appSettings part):
{
"resources": [
{
"name": "[parameters('name')]",
"type": "Microsoft.Web/sites",
"properties": {
"siteConfig": {
"appSettings": [
{
"name": "APPINSIGHTS_INSTRUMENTATIONKEY",
"value": "[reference('microsoft.insights/components/AppMonitoredSite', '2015-05-01').InstrumentationKey]"
},
{
"name": "APPLICATIONINSIGHTS_CONNECTION_STRING",
"value": "[reference('microsoft.insights/components/AppMonitoredSite', '2015-05-01').ConnectionString]"
},
{
"name": "ApplicationInsightsAgent_EXTENSION_VERSION",
"value": "~2"
},
{
"name": "XDT_MicrosoftApplicationInsights_Mode ",
"value": "recommended"
}
]
},
"name": "[parameters('name')]",
"serverFarmId": "[concat('/subscriptions/', parameters('subscriptionId'),'/resourcegroups/', parameters('serverFarmResourceGroup'), '/providers/Microsoft.Web/serverfarms/', parameters('hostingPlanName'))]",
"hostingEnvironment": "[parameters('hostingEnvironment')]"
},
"dependsOn": [
"[concat('Microsoft.Web/serverfarms/', parameters('hostingPlanName'))]",
"microsoft.insights/components/AppMonitoredSite"
],
"apiVersion": "2016-03-01",
"location": "[parameters('location')]"
},
{
"apiVersion": "2016-09-01",
"name": "[parameters('hostingPlanName')]",
"type": "Microsoft.Web/serverfarms",
"location": "[parameters('location')]",
"properties": {
"name": "[parameters('hostingPlanName')]",
"workerSizeId": "[parameters('workerSize')]",
"numberOfWorkers": "1",
"hostingEnvironment": "[parameters('hostingEnvironment')]"
},
"sku": {
"Tier": "[parameters('sku')]",
"Name": "[parameters('skuCode')]"
}
},
{
"apiVersion": "2015-05-01",
"name": "AppMonitoredSite",
"type": "microsoft.insights/components",
"location": "West US 2",
"properties": {
"ApplicationId": "[parameters('name')]",
"Request_Source": "IbizaWebAppExtensionCreate"
}
}
],
"parameters": {
"name": {
"type": "string"
},
"hostingPlanName": {
"type": "string"
},
"hostingEnvironment": {
"type": "string"
},
"location": {
"type": "string"
},
"sku": {
"type": "string"
},
"skuCode": {
"type": "string"
},
"workerSize": {
"type": "string"
},
"serverFarmResourceGroup": {
"type": "string"
},
"subscriptionId": {
"type": "string"
}
},
"$schema": "https://schema.management.azure.com/schemas/2014-04-01-preview/deploymentTemplate.json#",
"contentVersion": "1.0.0.0"
}
Powershell:
$app = Get-AzWebApp -ResourceGroupName "AppMonitoredRG" -Name "AppMonitoredSite" -ErrorAction Stop
$newAppSettings = #{} # case-insensitive hash map
$app.SiteConfig.AppSettings | %{$newAppSettings[$_.Name] = $_.Value} # preserve non Application Insights application settings.
$newAppSettings["APPINSIGHTS_INSTRUMENTATIONKEY"] = "012345678-abcd-ef01-2345-6789abcd"; # set the Application Insights instrumentation key
$newAppSettings["APPLICATIONINSIGHTS_CONNECTION_STRING"] = "InstrumentationKey=012345678-abcd-ef01-2345-6789abcd"; # set the Application Insights connection string
$newAppSettings["ApplicationInsightsAgent_EXTENSION_VERSION"] = "~2"; # enable the ApplicationInsightsAgent
$newAppSettings["XDT_MicrosoftApplicationInsights_Mode"] = "recommended"; # set the APM collection to recommended
$app = Set-AzWebApp -AppSettings $newAppSettings -ResourceGroupNamrecommendede $app.ResourceGroup -Name $app.Name -ErrorAction Stop

Extract LoadBalancer name from kubectl output with go-template

I'm trying to write a go template that extracts the value of the load balancer. Using --go-template={{status.loadBalancer.ingress}} returns [map[hostname:GUID.us-west-2.elb.amazonaws.com]]% When I add .hostname to the template I get an error saying, "can't evaluate field hostname in type interface {}". I've tried using the range keyword, but I can't seem to get the syntax right.
{
"apiVersion": "v1",
"kind": "Service",
"metadata": {
"creationTimestamp": "2018-07-30T17:22:12Z",
"labels": {
"run": "nginx"
},
"name": "nginx-http",
"namespace": "jx",
"resourceVersion": "495789",
"selfLink": "/api/v1/namespaces/jx/services/nginx-http",
"uid": "18aea6e2-941d-11e8-9c8a-0aae2cf24842"
},
"spec": {
"clusterIP": "10.100.92.49",
"externalTrafficPolicy": "Cluster",
"ports": [
{
"nodePort": 31032,
"port": 80,
"protocol": "TCP",
"targetPort": 8080
}
],
"selector": {
"run": "nginx"
},
"sessionAffinity": "None",
"type": "LoadBalancer"
},
"status": {
"loadBalancer": {
"ingress": [
{
"hostname": "GUID.us-west-2.elb.amazonaws.com"
}
]
}
}
}
As you can see from the JSON, the ingress element is an array. You can use the template function index to grab this array element.
Try:
kubectl get svc <name> -o=go-template --template='{{(index .status.loadBalancer.ingress 0 ).hostname}}'
This is assuming of course that you're only provisioning a single loadbalancer, if you have multiple, you'll have to use range
try this:
kubectl get svc <name> -o go-template='{{range .items}}{{range .status.loadBalancer.ingress}}{{.hostname}}{{printf "\n"}}{{end}}{{end}}'

What is the DNS Record logged by kubedns?

I'm using Google Container Engine and I'm noticing entries like the following in my logs
{
"insertId": "1qfzyonf2z1q0m",
"internalId": {
"projectNumber": "1009253435077"
},
"labels": {
"compute.googleapis.com/resource_id": "710923338689591312",
"compute.googleapis.com/resource_name": "fluentd-cloud-logging-gke-gas2016-4fe456307445d52d-worker-pool-",
"compute.googleapis.com/resource_type": "instance",
"container.googleapis.com/cluster_name": "gas2016-4fe456307445d52d",
"container.googleapis.com/container_name": "kubedns",
"container.googleapis.com/instance_id": "710923338689591312",
"container.googleapis.com/namespace_name": "kube-system",
"container.googleapis.com/pod_name": "kube-dns-v17-e4rr2",
"container.googleapis.com/stream": "stderr"
},
"logName": "projects/cml-236417448818/logs/kubedns",
"resource": {
"labels": {
"cluster_name": "gas2016-4fe456307445d52d",
"container_name": "kubedns",
"instance_id": "710923338689591312",
"namespace_id": "kube-system",
"pod_id": "kube-dns-v17-e4rr2",
"zone": "us-central1-f"
},
"type": "container"
},
"severity": "ERROR",
"textPayload": "I0718 17:05:20.552572 1 dns.go:660] DNS Record:&{worker-7.default.svc.cluster.local. 6000 10 10 false 30 0 }, hash:f97f8525\n",
"timestamp": "2016-07-18T17:05:20.000Z"
}
Is this an actual error or is the severity incorrect? Where can I find the definition for the struct that is being printed?
The severity is incorrect. This is some tracing/debugging that shouldn't have been left in the binary, and has been removed since 1.3 was cut. It will be removed in a future release.
See also: Google container engine cluster showing large number of dns errors in logs