How to ping DaemonSet in kubernetes - kubernetes

I've set up Kubernetes DaemonSet Manifest for handling metrics in my project, and having a little problem of pinging this DaemonSet, So my Eventual Question is, (If I have a daemonset in every pod of my project, how can I ping Specific One or Just the One In order to make sure that everything okay with it?) (Most Likely using curl, but would also consider some other ways too if it's possible:)
Example of DaemonSet I have
apiVersion: v1
kind: DaemonSet
metadata:
name: metrics-api-service
spec:
selector:
matchLabels:
app: api-metrics-app
template:
metadata:
labels:
app: api-metrics-app
spec:
terminationGracePeriodSeconds: 60
containers:
- name: api-metrics-service
image: image
livenessProbe:
exec:
< what I need to put here In order to ping the DaemonSet ? >
initialDelaySeconds: 60
resources:
limits:
cpu: "0.5"
memory: "500Mi"
requests:
cpu: "0.4"
memory: "200Mi"
Thanks

Healthcheck should be enough to making sure if its working or not, but if you still want to confirm from the external side then make sure your daemon set exposes the port and the security group allow internal traffic, you can ping/curl same node where the daemon set is running, every daemonset will get the node IP as an environment variable.
Pass the HOST_IP in the daemonset environment variable
env:
- name: HOST_IP
valueFrom:
fieldRef:
fieldPath: status.hostIP
and then update the liveness probe accordingly
livenessProbe:
exec:
command:
- sh
- -c
- curl -s $HOST_IP:PORT/healthcheck
I will recommend HTTP check over bash
healthcheck:
initialDelaySeconds: 100
periodSeconds: 10
timeoutSeconds: 5
httpGet:
path: /metrics
port: 3000
if /metrics seems working, just return 200 status code.

One way to do this would be to lookup for your Pods IP addresses, then query each of them.
for i in $(kubectl get pods -n logging get pods \
-l app=api-metrics-app \
-o jsonpath='{.items[*].status.podIP}')
do
curl http://$i:4242/metrics
done

Related

configMap volumes are not allowed to be used

I am using OKD4, and I am trying to mount /etc/php.ini in my pods using a ConfigMap.
In order to do so, I am creating the following K8S objects in my project.
Configmap (previously created to Deployment):
- apiVersion: v1
kind: ConfigMap
data:
php.ini: |-
[PHP]
;;;;;;;;;;;;;;;;;;;
; About php.ini ;
;;;;;;;;;;;;;;;;;;;
metadata:
name: php-ini
Deployment object:
- kind: Deployment
apiVersion: apps/v1
metadata:
name: privatebin
labels:
app: privatebin
spec:
replicas: 1
selector:
matchLabels:
app: privatebin
template:
metadata:
creationTimestamp: null
labels:
app: privatebin
deploymentconfig: privatebin
spec:
containers:
- name: privatebin
image: <my container registry>/privatebin:${IMAGE_TAG}
volumeMounts:
- name: config-volume
mountPath: php.ini
livenessProbe:
exec:
command:
- /bin/sh
- -c
- "[ -f /run/nginx.pid ] && ps -C nginx >/dev/null 2>&1 && ps -C php-fpm >/dev/null 2>&1"
initialDelaySeconds: 10
periodSeconds: 5
readinessProbe:
httpGet:
scheme: HTTP
path: /
port: 8080
initialDelaySeconds: 10
periodSeconds: 5
ports:
- containerPort: 8080
protocol: TCP
resources:
limits:
cpu: "250m" # parameterize in tekton pipeline
memory: "368Mi" # parameterize in tekton pipeline, maybe using template
requests:
cpu: "100m" # parameterize in tekton pipeline, maybe using template
memory: "256Mi" # parameterize in tekton pipeline, maybe using template
securityContext:
runAsUser: 1000
fsGroup: 1000
fsGroupChangePolicy: "OnRootMismatch"
imagePullPolicy: Always
restartPolicy: Always
terminationGracePeriodSeconds: 30
volumes:
- name: config-volume
configMap:
name: php-ini
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 25%
maxSurge: 25%
For some reason my pods are missing and there are the following errors:
lastTransitionTime: "2021-10-08T12:53:01Z"
lastUpdateTime: "2021-10-08T12:53:01Z"
message: 'pods "privatebin-55678c66c5-" is forbidden: unable to validate against
any security context constraint: [spec.containers[0].securityContext.runAsUser:
Invalid value: 1000: must be in the ranges: [1002460000, 1002469999] spec.volumes[0]:
Invalid value: "configMap": configMap volumes are not allowed to be used]'
ReplicaSet times out also with similar error:
status:
conditions:
lastTransitionTime: "2021-10-08T12:53:01Z"
message: 'pods "privatebin-55678c66c5-" is forbidden: unable to validate against
any security context constraint: [spec.containers[0].securityContext.runAsUser:
Invalid value: 1000: must be in the ranges: [1002460000, 1002469999] spec.volumes[0]:
Invalid value: "configMap": configMap volumes are not allowed to be used]'
Why can't I mount the ConfigMap? Is it because of the Securitycontext in the Deployment?
Thanks in advance,
(The error has nothing to do with configmaps, but when you get the error resolved you may need to tweak your configmap slightly to accurately drop the file into the directory you want it to land.)
OKD is OpenShift, so it's using SCC (not PSP).
By default you have access to the "restricted" SCC in your namespace. The UIDs being thrown out in the error are from the namespace annotation (oc get namespace FOO -o yaml) will show them.
To fix:
you can change your runAsUser to match the namespace annotation; or (better) just use "runAsNonRoot: true" which forces it to not run as root and takes the first uid in that annotation range. You may need to update the container image to leverage GROUP membership, not uid, for file access permissions.
you can allow your accounts to use the "nonroot" SCC to run as any uid, meeting your expectation to run as uid=1000. I would suggest you look at the first option as being the preferable option.
Your cluster is installed with a pod security policy that reject your spec. You can get the psp with kubectl get psp, then check the settings with kubectl describe psp <name>. Look at the settings volumes and runAsUser.

Influxdb 2.0 running in K8s gets data lost every time Statefulset pod is scaled down or rescheduled

I have deployed Influxdb 2.0.0 as Statefulset with EBS volume persistence. I've noticed that, if for some reason, pod gets rescheduled to other node or even if we scale down statefulset pod replicas = 0 and then scale up, the effect would be the same on persisted data: they will be lost.
Initially, in case of pod that gets rescheduled to other node, I would thought the problem is with EBS volume, it doesn't get unmounted and them mounted to another node where pod replica is running but that is NOT the case. EBS volume is present, same pv/pvc exists, but data is lost.
To figure out what might be the problem, I've purposely done influxdb setup and added data and then did this:
kubectl scale statefulsets influxdb --replicas=0
...
kubectl scale statefulsets influxdb --replicas=1
The effect was the same just like when influxdb pod got rescheduled. Data was lost.
Any specific reason why would something like that happen?
My environment:
I'm using EKS k8s environment with 1.15 k8s version of control plane/workers.
Fortunately, the problem was due to the big changes that happened between influxdb 1.x and 2.0.0 beta version in terms on where the actual data is persisted.
In 1.x version, data was persisted in:
/var/lib/influxdb
while on the 2.x version, data is persisted, by default, on:
/root/.influxdbv2
My EBS volume was mounted on the 1.x version location and with every restart of the pod (either caused by scaling down or by scheduling to other node), EBS volume was regularly attached but on the wrong location. That was the reason why there was no data.
Also, one difference that I see is that configuration params cannot be provided for 2.x version via configuration file (like it was on 1.x where I had configuration file mounted into the container as configmap). We have to provide additional configuration params inline. This link explains how: https://v2.docs.influxdata.com/v2.0/reference/config-options/
At the end this is the working version of Statefulset:
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
labels:
app: influxdb
name: influxdb
spec:
replicas: 1
selector:
matchLabels:
app: influxdb
serviceName: influxdb
template:
metadata:
labels:
app: influxdb
spec:
containers:
- image: quay.io/influxdb/influxdb:2.0.0-beta
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 3
httpGet:
path: /ping
port: api
scheme: HTTP
initialDelaySeconds: 30
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 5
name: influxdb
ports:
- containerPort: 9999
name: api
protocol: TCP
readinessProbe:
failureThreshold: 3
httpGet:
path: /ping
port: api
scheme: HTTP
initialDelaySeconds: 5
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
resources:
limits:
cpu: "800m"
memory: 1200Mi
requests:
cpu: 100m
memory: 256Mi
volumeMounts:
- mountPath: /root/.influxdbv2
name: influxdb-data
volumeClaimTemplates:
- metadata:
name: influxdb-data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 20Gi
volumeMode: Filesystem

How to change running pods limits in Kubernetes?

I have a self made Kubernetes cluster consisting of VMs. My problem is, that the coredns pods are always go in CrashLoopBackOff state, and after a while they go back to Running as nothing happened.. One solution that I found and could not try yet, is changing the default memory limit from 170Mi to something higher. As I'm not an expert in this, I thought this is not a hard thing, but I don't know how to change a running pod's configuration. It may be impossible, but there must be a way to recreate them with new configuration. I tried with kubectl patch, and looked up rolling-update too, but I just can't figure it out. How can I change the limit?
Here is the relevant part of the pod's data:
apiVersion: v1
kind: Pod
metadata:
annotations:
cni.projectcalico.org/podIP: 176.16.0.12/32
creationTimestamp: 2018-11-18T10:29:53Z
generateName: coredns-78fcdf6894-
labels:
k8s-app: kube-dns
pod-template-hash: "3497892450"
name: coredns-78fcdf6894-gnlqw
namespace: kube-system
ownerReferences:
- apiVersion: apps/v1
blockOwnerDeletion: true
controller: true
kind: ReplicaSet
name: coredns-78fcdf6894
uid: e3349719-eb1c-11e8-9000-080027bbdf83
resourceVersion: "73564"
selfLink: /api/v1/namespaces/kube-system/pods/coredns-78fcdf6894-gnlqw
uid: e34930db-eb1c-11e8-9000-080027bbdf83
spec:
containers:
- args:
- -conf
- /etc/coredns/Corefile
image: k8s.gcr.io/coredns:1.1.3
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 5
httpGet:
path: /health
port: 8080
scheme: HTTP
initialDelaySeconds: 60
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 5
name: coredns
ports:
- containerPort: 53
name: dns
protocol: UDP
- containerPort: 53
name: dns-tcp
protocol: TCP
- containerPort: 9153
name: metrics
protocol: TCP
resources:
limits:
memory: 170Mi
requests:
cpu: 100m
memory: 70Mi
EDIT:
It turned out, that in Ubuntu the Network Manager's dnsmasq drives the Corends pods crazy, so in /etc/NetworkManager/NetworkManager.conf I commented out the dnsmasq line, reboot and everything is okay.
You must edit coredns pod's template in coredns deployment definition:
kubectl edit deployment -n kube-system coredns
Once your default editor is opened with coredns deployment, in the templateSpec you will find part which is responsible for setting memory and cpu limits.

Istio Mixer container logs causing high disk space usage

I have an Istio-enabled EKS Cluster, and my nodes are constantly running out of disk space.
Calculating the overall disk usage lead me to the istio-mixer container, which has a log file using more than 50GB of disk space in only 12 days of uptime:
[root#ip-some-ip containers]# pwd
/var/lib/docker/containers
[root#ip-some-ip containers]# du -schx .[!.]* * | sort -h | tail -n 10
66M 8bf5e8ee5a03096c589ad8f53b9e1a3d3088ca67b0064f3796e406f00336b532
73M 657eca261461d10c5b1b81ab3078d2058b931a357395903808b0145b617c1662
101M bb338296ff06ef42ae6177c8a88e63438c26c398a457dc3f5301ffcb4ef2682b
127M 21f2da86055ad76882730abf65d4465386bb85598f797f451e7ad66726243613
134M 9c2be24e8b9345659b6f208c9f2d4650bb1ece11e0c4b0793aa01fdfebadb44e
383M 5d5fdbe6813ddc3ff2f6eb96f62f8317bd73e24730e2f44ebc537367d9987142
419M 475f8dfc74c3df2bc95c47df56e37d1dfb9181fae9aa783dafabba8283023115
592M 9193c50e586e0c7ecaeb87cecd8be13714a5d6ccd6ea63557c034ef56b07772f
52G 9c6b3e4f26603471d0aa9b6a61b3da5a69001e6b9be34432ffa62d577738c149
54G total
[root#ip-192-168-228-194 containers]# du -hs 9c6b3e4*/*.log
52G 9c6b3e4f26603471d0aa9b6a61b3da5a69001e6b9be34432ffa62d577738c149-json.log
[root#ip-ip-some-ip containers]# docker ps -a | grep 9c6b3e4f2660
9c6b3e4f2660 d559bdcd7a88 "/usr/local/bin/mi..." 12 days ago Up 12 days k8s_mixer_istio-telemetry-6b5579595f-fvm5x_istio-system_6324c262-f3b5-11e8-b615-0eccb0bb4724_0
My questions are:
This amount of log output is expected?
The mixer log level can be decreased? How? Changing it affects my telemetry metrics?
There is a way to configure a log "retention period"?
Additional info:
Istio v1.0.2 (deployed with the offical helm charts; no custom configs)
k8s v1.10.11-eks
The cluster has approximately 20 pods running in Istio-enabled namespaces
The default value of logging level in Mixer is info. And the logs provided by you, confirms that you have this settings. Therefore, a lot of redundant information gathered in logs and it is possible to decrease logging level for some sources.
You can change it in two ways:
On working pod without restart.
In your logs you can find the following line:
2018-12-12T17:54:55.461261Z info ControlZ available at 192.168.87.249:9876
It means, that in the mixer container on 9876 port you can find Istio ControlZ web-interface. To get an access to it from a computer with installed kubectl, you need to run the following command:
kubectl -n istio-system port-forward $(kubectl -n istio-system get pod -l istio=mixer,istio-mixer-type=telemetry -o jsonpath='{.items[0].metadata.name}') 9876:9876 &
After that, in your browser go to the link http://localhost:9876/scopez/, and you will see the following dashboard, where you can change log levels:
Add --log_output_level flag to the istio-telemetry deployment for the mixer container.
Here is the description for the flag from the mixer's documentation:
--log_output_level string
Comma-separated minimum per-scope logging level of messages to output, in the form of :,:,... where scope can be one of [adapters, api, attributes, default, grpcAdapter, loadshedding] and level can be one of [debug, info, warn, error, none] (default "default:info")
Note, that for key --log_output_level attributes:warn,api:error in yaml file you need to use one of the following:
value - --log_output_level=attributes:warn,api:error or
values - --log_output_level and - attributes:warn,api:error on different lines
The example of the deployment:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
annotations:
labels:
chart: mixer-1.0.4
istio: mixer
release: istio
name: istio-telemetry
namespace: istio-system
spec:
progressDeadlineSeconds: 600
replicas: 2
revisionHistoryLimit: 10
selector:
matchLabels:
app: telemetry
istio: mixer
istio-mixer-type: telemetry
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 1
type: RollingUpdate
template:
metadata:
annotations:
scheduler.alpha.kubernetes.io/critical-pod: ""
sidecar.istio.io/inject: "false"
creationTimestamp: null
labels:
app: telemetry
istio: mixer
istio-mixer-type: telemetry
spec:
containers:
- args: #Flags for the Mixer process
- --address #Flag on two different lines
- unix:///sock/mixer.socket
- --configStoreURL=k8s:// #Flag with '='
- --configDefaultNamespace=istio-system
- --trace_zipkin_url=http://zipkin:9411/api/v1/spans
- --log_output_level=attributes:warn,api:error # <------ THIS LINE IS WHAT YOU ARE LOOKING FOR
env:
- name: GODEBUG
value: gctrace=2
image: docker.io/istio/mixer:1.0.4
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 3
httpGet:
path: /version
port: 9093
scheme: HTTP
initialDelaySeconds: 5
periodSeconds: 5
successThreshold: 1
timeoutSeconds: 1
name: mixer
ports:
- containerPort: 9093
protocol: TCP
- containerPort: 42422
protocol: TCP
resources:
requests:
cpu: 10m
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /sock
name: uds-socket
- args:
- proxy
- --serviceCluster
- istio-telemetry
- --templateFile
- /etc/istio/proxy/envoy_telemetry.yaml.tmpl
- --controlPlaneAuthPolicy
- MUTUAL_TLS
env:
- name: POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
- name: INSTANCE_IP
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: status.podIP
image: docker.io/istio/proxyv2:1.0.4
imagePullPolicy: IfNotPresent
name: istio-proxy
ports:
- containerPort: 15090
name: http-envoy-prom
protocol: TCP
resources:
requests:
cpu: 10m
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /etc/certs
name: istio-certs
readOnly: true
- mountPath: /sock
name: uds-socket
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: istio-mixer-service-account
serviceAccountName: istio-mixer-service-account
terminationGracePeriodSeconds: 30
volumes:
- name: istio-certs
secret:
defaultMode: 420
optional: true
secretName: istio.istio-mixer-service-account
- emptyDir: {}
name: uds-socket
Additionally, you can configure log rotation for mixer process using the following flags:
--log_rotate string The path for the optional rotating log file
--log_rotate_max_age int The maximum age in days of a log file beyond which the file is rotated (0 indicates no limit) (default 30)
--log_rotate_max_backups int The maximum number of log file backups to keep before older files are deleted (0 indicates no limit) (default 1000)
--log_rotate_max_size int The maximum size in megabytes of a log file beyond which the file is rotated (default 104857600)
However, I have no possibility to generate a huge amount of such logs and test how it works.
Links:
Unfortunately, the official documentation is not good, but maybe it helps somehow.
And as a bonus, here is the list of all mixer server flags.
This is how I solved the problem and some useful information for new Istio versions.
Istio v1.0.2:
The huge amount of logs was being generated by the Stdio adapter:
The stdio adapter enables Istio to output logs and metrics to the
local machine. Logs and metrics can be directed to Mixer’s standard
output stream, standard error stream, or to any locally reachable
file.
In Istio v1.0.2 this adapter was enabled by default, streaming the logs to Mixer container stderr. To temporarily solve this, I deleted the following rules:
kubectl delete rule stdio --namespace=istio-system
kubectl delete rule stdio-tcp --namespace=istio-system
Deleting these rules does not affect the Prometheus metrics (which are handled by another adapter).
Istio v1.1.0+:
In this version, Istio introduced the mixer.adapters.stdio.enabled to helm installation options, disabling the stdio adapter by default, and including the following comment:
# stdio is a debug adapter in istio-telemetry, it is not recommended
for production use.
The changes were made in the following PRs:
Add adapter tuning install options to helm (#10525)
Turn policy off by default (#12114)

Save file to Kubernetes pod during deployment

I am trying to add a file to a pod's disk during initialization of the pod but without luck. Below is my deployment file which I use to deploy the pod. The file gets downloaded to the persistent volume, but the pod doesn't get into ready state. After a few seconds, the pods fail and get rebuilt. Which kicks off the whole process again.
Any help would be appreciated.
apiVersion: apps/v1beta2
kind: Deployment
metadata:
name: mapserver
spec:
selector:
matchLabels:
app: mapserver
template:
metadata:
labels:
app: mapserver
spec:
volumes:
- name: storage
persistentVolumeClaim:
claimName: mapserver-pv-claim
containers:
- name: maptiles
image: klokantech/tileserver-gl
command: ["/bin/sh"]
args:
- -c
- |
echo "[INFO] Startingcontainer"; if [ $(DOWNLOAD_MBTILES) = "true" ]; then
echo "[INFO] Download MBTILES_PLANET_URL";
rm /data/*
cd /data/
curl -k -sSL -X GET -u user:ww $(MBTILES_PLANET_URL) -O
echo "[INFO] Download finished";
fi;
env:
- name: MBTILES_PLANET_URL
value: 'https://abc-dev/nexus/repository/xyz-raw/2017-07-03_europe_netherlands.mbtiles'
- name: DOWNLOAD_MBTILES
value: 'true'
livenessProbe:
failureThreshold: 120
httpGet:
path: /health
port: 80
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 30
successThreshold: 1
timeoutSeconds: 5
ports:
- containerPort: 80
name: http
protocol: TCP
readinessProbe:
failureThreshold: 120
httpGet:
path: /health
port: 80
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 30
successThreshold: 1
timeoutSeconds: 5
resources:
limits:
cpu: 300m
memory: 3Gi
requests:
cpu: 100m
memory: 1Gi
volumeMounts:
- mountPath: "/data"
name: storage
am trying to add a file to a pod's disk during initialization of the pod but without luck.
In that case you might want to use InitContainers instead.
Judging from your manifest, your main command gets executed (copies the file and then exits) terminating the container (and accompanying pod) in the process. Deployment then restarts the exited pod and cycle repeats. If you use InitContainers instead (with the same definition and same PV as you are doing now for main container) you should then prepopulate data using InitContaienrs that runs to completion and then continue to use it in your normal container (that should have non-exiting main process as its command/entry point).
Note: if you don't want to use InitContainers or just as a quick test, you could append a regular non-exiting command after your copy statement, and also, check if you need to start container with tty, depending on your use case and ways to keep container up and running.