EventRateLimit on kube-apiserver is not being applied - kubernetes

EventRateLimit actual configuration which is not working
Version: K8s v1.23.10
kube-apiserver daemon
/usr/lib/systemd/system/kube-apiserver.service
...
--authorization-mode=Node,RBAC \
--client-ca-file=/var/lib/kubernetes/ca.pem \
--enable-admission-plugins=NamespaceLifecycle,NodeRestriction,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota,EventRateLimit \
--admission-control-config-file=/var/lib/kubernetes/admission-control-config.yml \
...
/var/lib/kubernetes/admission-control-config.yml
apiVersion: apiserver.config.k8s.io/v1
kind: AdmissionConfiguration
plugins:
- name: EventRateLimit
path: /var/lib/kubernetes/eventconfig.yml
/var/lib/kubernetes/eventconfig.yml
apiVersion: eventratelimit.admission.k8s.io/v1alpha1
kind: Configuration
limits:
- type: Server
qps: 2
burst: 2
- type: Namespace
qps: 2
burst: 2
- type: User
qps: 2
burst: 2
On kube-apiserver logs, clearly shows how is correctly loaded
Oct 04 11:16:09 master1_enc kube-apiserver[1560]: I1004 11:16:08.995589 1560 plugins.go:161] Loaded 12 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PodSecurity,Priority,EventRateLimit,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.
I'm testing my config with hey tool the following way:
creating many jobs
./hey -q 25 -c 2 -n 200 -H "Content-Type: application/json" -H "Authorization: Bearer XXX" -D ./job.json -m POST https://127.0.0.1:6443/apis/batch/v1/namespaces/default/jobs
-q 25 means 25 requests per second
-c 2 means 2 workers at the same time (basically 50 request per second)
-n 200 means 200 requests
hey tool result:
...
Summary:
Total: 4.0073 secs
Slowest: 0.0319 secs
Fastest: 0.0037 secs
Average: 0.0059 secs
Requests/sec: 49.9093
Status code distribution:
[201] 200 responses #<- 200 requests with http code 201 returned
...
THE PROBLEM: The test should fail, based on EventRateLimit config should accept only 2 requests per second, but is accepting much more, completely ignoring my config. What could i be doing wrong?
Kubernetes EventRateLimit documentation: https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#eventratelimit

Related

Can't connect to proxy from within the pod [ Connection reset by peer ]

I get "Connection reset by peer" every time I try to use proxy from the Kubernetes pod.
Here is the log when from the curl:
>>>> curl -x http://5.188.62.223:15624 -L http://google.com -vvv
* Trying 5.188.62.223:15624...
* Connected to 5.188.62.223 (5.188.62.223) port 15624 (#0)
> GET http://google.com/ HTTP/1.1
> Host: google.com
> User-Agent: curl/7.79.1
> Accept: */*
> Proxy-Connection: Keep-Alive
>
* Recv failure: Connection reset by peer
* Closing connection 0
curl: (56) Recv failure: Connection reset by peer
Interesting fact, that I have no issues when I use same proxy on local computer, in docker and on remote host. Apparently smith within the cluster doesn't let me communicate with it.
Currently I use Azure hosted Kubernetes. But the same error happens on Digital Ocean as well.
I would be grateful for every leading clue of how I can bypass this restrictions, because Im out of ideas.
Server Info:
{
Major:"1",
Minor:"20",
GitVersion:"v1.20.7",
GitCommit:"ca90e422dfe1e209df2a7b07f3d75b92910432b5",
GitTreeState:"clean",
BuildDate:"2021-10-09T04:59:48Z",
GoVersion:"go1.15.12", Compiler:"gc",
Platform:"linux/amd64"
}
The yaml file I use in order to start the pod is just super basic. But originally I use airflow with Kubernetes executor, which actually spawn pretty similar basic pods:
apiVersion: v1
kind: Pod
metadata:
name: scrapeevent.test
spec:
affinity: {}
containers:
- command:
- /bin/sh
- -ec
- while :; do echo '.'; sleep 5 ; done
image: jaklimoff/mooncrops-opensea:latest
imagePullPolicy: Always
name: base
restartPolicy: Never

I am experiencing a lot of CPU throttling on my k8s cluster -- do I have the high-throttling low-quota Linux kernel bug?

I am experiencing a lot of CPU throttling (see nginx graph below, other pods often 25% to 50%) in my Kubernetes cluster (k8s v1.18.12, running 4.9.0-11-amd64 #1 SMP Debian 4.9.189-3+deb9u2 (2019-11-11) x86_64 GNU/Linux).
Due to backports, I do not know whether my cluster contains the Linux kernel bug described in https://lkml.org/lkml/2019/5/17/581. How can I find out? Is there a simple way to check or measure?
If I have the bug, what is the best approach to get the fix? Or should I mitigate otherwise, e.g. not use CFS quota (--cpu-cfs-quota=false or no CPU limits) or reduce cfs_period_us and cfs_quota_us?
CPU Throttling Percentage for nginx (scaling horizontally around 15:00 and removing CPU limits around 19:30):
Recently, I'm working on debuging the cpu throttling issue, with the following 5 tests, I've tested out the bug in kernel (Linux version 4.18.0-041800rc4-generic)
This test case is intended to hit 100% throttling for the test 5000ms / 100 ms periods = 50 periods. A kernel without this bug should be able to have a CPU usage stats about 500ms.
Maybe you can try these tests to check whether your kernel will be throttlled.
[Multi Thread Test 1]
./runfibtest 1; ./runfibtest
From <https://github.com/indeedeng/fibtest>
[Result]
Throttled
./runfibtest 1
Iterations Completed(M): 465
Throttled for: 52
CPU Usage (msecs) = 508
./runfibtest 8
Iterations Completed(M): 283
Throttled for: 52
CPU Usage (msecs) = 327
[Multi Thread Test 2]
docker run -it --rm --cpu-quota 10000 --cpu-period 100000 hipeteryang/fibtest:latest /bin/sh -c "runfibtest 8 && cat /sys/fs/cgroup/cpu,cpuacct/cpu.stat && cat /sys/fs/cgroup/cpu,cpuacct/cpuacct.usage && cat /sys/fs/cgroup/cpu,cpuacct/cpuacct.usage_percpu"
[Result]
Throttled
Iterations Completed(M): 192
Throttled for: 53
CPU Usage (msecs) = 227
nr_periods 58
nr_throttled 56
throttled_time 10136239267
267434463
209397643 2871651 8044402 4226146 5891926 5532789 27939741 4104364
[Multi Thread Test 3]
docker run -it --rm --cpu-quota 10000 --cpu-period 100000 hipeteryang/stress-ng:cpu-delay /bin/sh -c "stress-ng --taskset 0 --cpu 1 --timeout 5s & stress-ng --taskset 1-7 --cpu 7 --cpu-load-slice -1 --cpu-delay 10 --cpu-method fibonacci --timeout 5s && cat /sys/fs/cgroup/cpu,cpuacct/cpu.stat && cat /sys/fs/cgroup/cpu,cpuacct/cpuacct.usage && cat /sys/fs/cgroup/cpu,cpuacct/cpuacct.usage_percpu"
Result
Throttled
nr_periods 56
nr_throttled 53
throttled_time 7893876370
379589091
330166879 3073914 6581265 724144 706787 5605273 29455102 3849694
For the following kubernetes test, we can use "kubectl logs pod-name" to get the result once the job is done
[Multi Thread Test 4]
apiVersion: batch/v1
kind: Job
metadata:
name: fibtest
namespace: default
spec:
template:
spec:
containers:
- name: fibtest
image: hipeteryang/fibtest
command: ["/bin/bash", "-c", "runfibtest 8 && cat /sys/fs/cgroup/cpu,cpuacct/cpu.stat && cat /sys/fs/cgroup/cpu,cpuacct/cpuacct.usage && cat /sys/fs/cgroup/cpu,cpuacct/cpuacct.usage_percpu"]
resources:
requests:
cpu: "50m"
limits:
cpu: "100m"
restartPolicy: Never
Result
Throttled
Iterations Completed(M): 195
Throttled for: 52
CPU Usage (msecs) = 230
nr_periods 56
nr_throttled 54
throttled_time 9667667360
255738571
213621103 4038989 2814638 15869649 4435168 5459186 4549675 5437010
[Multi Thread Test 5]
apiVersion: batch/v1
kind: Job
metadata:
name: stress-ng-test
namespace: default
spec:
template:
spec:
containers:
- name: stress-ng-test
image: hipeteryang/stress-ng:cpu-delay
command: ["/bin/bash", "-c", "stress-ng --taskset 0 --cpu 1 --timeout 5s & stress-ng --taskset 1-7 --cpu 7 --cpu-load-slice -1 --cpu-delay 10 --cpu-method fibonacci --timeout 5s && cat /sys/fs/cgroup/cpu,cpuacct/cpu.stat && cat /sys/fs/cgroup/cpu,cpuacct/cpuacct.usage && cat /sys/fs/cgroup/cpu,cpuacct/cpuacct.usage_percpu
"]
resources:
requests:
cpu: "50m"
limits:
cpu: "100m"
restartPolicy: Never
Result
Throttled
nr_periods 53
nr_throttled 50
throttled_time 6827221694
417331622
381601924 1267814 8332150 3933171 13654620 184120 6376208 2623172
Feel free to leave any comment, I’ll reply as soon as possible.
Since the fix was backported to many older Kernel versions, I do not know how to look up easily whether e.g. 4.9.0-11-amd64 #1 SMP Debian 4.9.189-3+deb9u2 (2019-11-11) x86_64 GNU/Linux has the fix.
But you can measure whether your CFS is working smoothly or is throttling too much, as described in https://gist.github.com/bobrik/2030ff040fad360327a5fab7a09c4ff1:
you run the given cfs.go with suitable settings for its sleeps and iterations as well as CFS settings, e.g. docker run --rm -it --cpu-quota 20000 --cpu-period 100000 -v $(pwd):$(pwd) -w $(pwd) golang:1.9.2 go run cfs.go -iterations 100 -sleep 1000ms
you check whether all burn took 5ms. If not, your CFS is throttling too much. This could be e.g. due to the original bug 198197 (see https://bugzilla.kernel.org/show_bug.cgi?id=198197) or the regression introduced by the fix for bug 198197 (details see https://lkml.org/lkml/2019/5/17/581).
This measurement approach is also taken in https://github.com/kubernetes/kops/issues/8954, showing that Linux kernel 4.9.0-11-amd64 is throttling too much (however, with an earlier Debian 4.9.189-3+deb9u1 (2019-09-20) than your Debian 4.9.189-3+deb9u2 (2019-11-11)).
The CFS bug was fixed in Linux 5.4, exec kubectl describe nodes | grep Kernel or go to any of your Kubernetes nodes and execuname -sr that will tell you the Kernel release you are running on.

Helm not adding timeout param in API server call

When we use helm to create Istio VirtualService or DestinationRules, it takes more than 30seconds. So we have set up a higher timeout of 5m.
Command used
helm upgrade --install --wait --timeout 5m --v 9 helloworld ./templateDir
However, I see that helm is not passing the new timeout value in APIserver call.
When we use kubectl to create the same, we set --request-timeout param for kubectl and everything is fine
Since it is a timeout from apiserver, is it possible to set a higher timeout value for all API server requests from helm? Is there any other workaround you can suggest for this problem ?
I0528 17:52:57.664992 11148 round_trippers.go:423] curl -k -v -XPOST -H "Content-Type: application/json" -H "Accept: application/json" 'https://aaaa.sk1.us-east-1.eks.amazonaws.com/apis/networking.istio.io/v1beta1/namespaces/default/destinationrules '
I0528 17:53:27.981691 11148 round_trippers.go:443] POST https://aaaa.sk1.us-east-1.eks.amazonaws.com/apis/networking.istio.io/v1beta1/namespaces/default/destinationrules 504 Gateway Timeout in 30316 milliseconds
I0528 17:53:27.981691 11148 round_trippers.go:449] Response Headers:
I0528 17:53:27.981691 11148 round_trippers.go:452] Audit-Id: cba23005-b8db-47f7-8801-4c89e6447cd3
I0528 17:53:27.981691 11148 round_trippers.go:452] Content-Type: application/json
I0528 17:53:27.981691 11148 round_trippers.go:452] Content-Length: 187
I0528 17:53:27.981691 11148 round_trippers.go:452] Date: Thu, 28 May 2020 12:23:27 GMT
I0528 17:53:27.981691 11148 request.go:1017] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Timeout: request did not complete within requested timeout 30s","reason":"Timeout","details":{},"code":504}
I0528 17:53:27.982759 11148 request.go:1017] Request Body: {"apiVersion":"networking.istio.io/v1beta1","kind":"VirtualService","metadata":{"name":"mbrsvc","namespace":"default"},"spec":{"hosts":["mbrsvc"],"http":[{"route":[{"destination":{"host":"mbrsvc","subset":"1.0.0"},"weight":100}]}]}}
Documentation states that the --timeout should be in seconds.
--timeout: A value in seconds to wait for Kubernetes commands to complete This defaults to 5m0s
--wait: Waits until all Pods are in a ready state, PVCs are bound, Deployments have minimum (Desired minus maxUnavailable) Pods in ready state and Services have an IP address (and Ingress if a LoadBalancer) before marking the release as successful. It will wait for as long as the --timeout value. If timeout is reached, the release will be marked as FAILED. Note: In scenario where Deployment has replicas set to 1 and maxUnavailable is not set to 0 as part of rolling update strategy, --wait will return as ready as it has satisfied the minimum Pod in ready condition.
Hope that helps.
This timeout comes from the server (Code 504) not from the client (Helm). Providing a --timeout is not going to have an influence here, I'm afraid

Setup Kubernetes HA cluster with kubeadm and F5 as load-balancer

I'm trying to setup a Kubernetes HA cluster using kubeadm as installer and F5 as load-balancer (cannot use HAproxy). I'm experiencing issues with the F5 configuration.
I'm using self-signed certificates and passed the apiserver.crt and apiserver.key to the load balancer.
For some reasons the kubeadm init script fails with the following error:
[apiclient] All control plane components are healthy after 33.083159 seconds
I0805 10:09:11.335063 1875 uploadconfig.go:109] [upload-config] Uploading the kubeadm ClusterConfiguration to a ConfigMap
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
I0805 10:09:11.340266 1875 request.go:947] Request Body: {"kind":"ConfigMap","apiVersion":"v1","metadata":{"name":"kubeadm-config","namespace":"kube-system","creationTimestamp":null},"data":{"ClusterConfiguration":"apiServer:\n certSANs:\n - $F5_LOAD_BALANCER_VIP\n extraArgs:\n authorization-mode: Node,RBAC\n timeoutForControlPlane: 4m0s\napiVersion: kubeadm.k8s.io/v1beta2\ncertificatesDir: /etc/kubernetes/pki\nclusterName: kubernetes\ncontrolPlaneEndpoint: $F5_LOAD_BALANCER_VIP:6443\ncontrollerManager: {}\ndns:\n type: CoreDNS\netcd:\n local:\n dataDir: /var/lib/etcd\nimageRepository: k8s.gcr.io\nkind: ClusterConfiguration\nkubernetesVersion: v1.15.1\nnetworking:\n dnsDomain: cluster.local\n podSubnet: 192.168.0.0/16\n serviceSubnet: 10.96.0.0/12\nscheduler: {}\n","ClusterStatus":"apiEndpoints:\n lnxkbmaster02:\n advertiseAddress: $MASTER01_IP\n bindPort: 6443\napiVersion: kubeadm.k8s.io/v1beta2\nkind: ClusterStatus\n"}}
I0805 10:09:11.340459 1875 round_trippers.go:419] curl -k -v -XPOST -H "Accept: application/json, */*" -H "Content-Type: application/json" -H "User-Agent: kubeadm/v1.15.1 (linux/amd64) kubernetes/4485c6f" 'https://$F5_LOAD_BALANCER_VIP:6443/api/v1/namespaces/kube-system/configmaps'
I0805 10:09:11.342399 1875 round_trippers.go:438] POST https://$F5_LOAD_BALANCER_VIP:6443/api/v1/namespaces/kube-system/configmaps 403 Forbidden in 1 milliseconds
I0805 10:09:11.342449 1875 round_trippers.go:444] Response Headers:
I0805 10:09:11.342479 1875 round_trippers.go:447] Content-Type: application/json
I0805 10:09:11.342507 1875 round_trippers.go:447] X-Content-Type-Options: nosniff
I0805 10:09:11.342535 1875 round_trippers.go:447] Date: Mon, 05 Aug 2019 08:09:11 GMT
I0805 10:09:11.342562 1875 round_trippers.go:447] Content-Length: 285
I0805 10:09:11.342672 1875 request.go:947] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"configmaps is forbidden: User \"system:anonymous\" cannot create resource \"configmaps\" in API group \"\" in the namespace \"kube-system\"","reason":"Forbidden","details":{"kind":"configmaps"},"code":403}
error execution phase upload-config/kubeadm: error uploading the kubeadm ClusterConfiguration: unable to create ConfigMap: configmaps is forbidden: User "system:anonymous" cannot create resource "configmaps" in API group "" in the namespace "kube-system"
The init is really basic:
kubeadm init --config=kubeadm-config.yaml --upload-certs
Here's the kubeadm-config.yaml:
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: stable
controlPlaneEndpoint: "$F5_LOAD_BALANCER_VIP:6443"
networking:
podSubnet: "192.168.0.0/16"
If I setup the cluster using a HAproxy the init runs smoothly:
#---------------------------------------------------------------------
# kubernetes
#---------------------------------------------------------------------
frontend kubernetes
bind $HAPROXY_LOAD_BALANCER_IP:6443
option tcplog
mode tcp
default_backend kubernetes-master-nodes
backend kubernetes-master-nodes
mode tcp
balance roundrobin
option tcp-check
server master01.my-domain $MASTER_01_IP:6443 check fall 3 rise 2
server master02.my-domain $MASTER_02_IP:6443 check fall 3 rise 2
server master03.my-domain $MASTER_03_IP:6443 check fall 3 rise 2
END
My solution has been to deploy the cluster without the proxy (F5) with a configuration as follows:
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: stable
controlPlaneEndpoint: "$MASTER_1_IP:6443"
networking:
podSubnet: "192.168.0.0/16"
Afterwards it was necessary to deploy on the cluster the F5 BIG-IP Controller for Kubernetes to manage the F5 device from Kubernetes.
Detailed guide can be found here:
https://clouddocs.f5.com/products/connectors/k8s-bigip-ctlr/v1.10/
Beware that it requires an additional F5 license and admin privileges.

Unable to install gitlab on Kubernetes

I am trying to install gitlab with helm on a kubernetes cluster which already have an ingress(cluster created by RKE). With gitlab, I want to deploy it into seperate namespace. For that, I ran the below command:
$ gitlab-config helm upgrade --install gitlab gitlab/gitlab \
--timeout 600 \
--set global.hosts.domain=asdsa.asdasd.net \
--set certmanager-issuer.email=sd#cloudssky.com \
--set global.edition=ce \
--namespace gitlab-ci \
--set gitlab.migrations.image.repository=registry.gitlab.com/gitlab-org/build/cng/gitlab-rails-ce \
--set gitlab.sidekiq.image.repository=registry.gitlab.com/gitlab-org/build/cng/gitlab-sidekiq-ce \
--set gitlab.unicorn.image.repository=registry.gitlab.com/gitlab-org/build/cng/gitlab-unicorn-ce \
--set gitlab.unicorn.workhorse.image=registry.gitlab.com/gitlab-org/build/cng/gitlab-workhorse-ce \
--set gitlab.task-runner.image.repository=registry.gitlab.com/gitlab-org/build/cng/gitlab-task-runner-ce
But the install fails while validating the domain with http01 test with cert-manager. For this, before running the above command, I've pointed my base domain to the existing Load Balancer in my cluster.
Is there something different which needs to be done for successful http01 validation?
Error:
Conditions:
Last Transition Time: 2018-11-18T15:22:00Z
Message: http-01 self check failed for domain "asdsa.asdasd.net"
Reason: ValidateError
Status: False
Type: Ready
More information:
The health checks for Load Balancer also keeps failing. So, even with using self-signed certificates, the installation is failing.
When tried to ssh into one of the nodes and check return status, here's what I saw:
$ curl -v localhost:32030/healthz
* Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 32030 (#0)
> GET /healthz HTTP/1.1
> Host: localhost:32030
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 503 Service Unavailable
< Content-Type: application/json
< Date: Mon, 19 Nov 2018 13:38:49 GMT
< Content-Length: 114
<
{
"service": {
"namespace": "gitlab-ci",
"name": "gitlab-nginx-ingress-controller"
},
"localEndpoints": 0
* Connection #0 to host localhost left intact
}
And, when I checked ingress controller service, it was up and running:
gitlab-nginx-ingress-controller LoadBalancer 10.43.168.81 XXXXXXXXXXXXXX.us-east-2.elb.amazonaws.com 80:32006/TCP,443:31402/TCP,22:31858/TCP
The issue was resolved here - https://gitlab.com/charts/gitlab/issues/939
Basically, the solution as mentioned in the thread is not formally documented because it needs confirmation.