I have a cluster on GKE and I want to set a limit for incoming requests, but I cannot find a way to do it using Kong Ingress Controller. I can't find any documentation or info about this specific topic.
Following the steps in this article, I achieved the desired results by adding the rate limit plugin in my kongo ingress. To do so, first, update / create your ingress definition and add the annotations defined below:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: func
namespace: default
annotations:
kubernetes.io/ingress.class: kong # <-- THIS
plugins.konghq.com: http-ratelimit # <-- THIS
spec:
...
After, to finally set the rate-limit, use this definition and apply it in your kubernetes cluster:
apiVersion: configuration.konghq.com/v1
kind: KongPlugin
metadata:
name: http-ratelimit
namespace: default
config:
policy: local
second: 1
plugin: rate-limiting
This will create a restriction of 1 request per second in your ingress. If you want anything different, just change the config section with your own configuration. Check the plugin's documentation for all possible configurations.
Related
I have Nginx Ingress Controller which is deployed via official Helm chart, in the doc I saw that I can set max_conn parameter, but I didn't get how to set it. I want to set it to 2, so that maximum of 2 clients could connect to my services. How do I set it? Should I set it in ingress controller values during helm install of Ingress Controller or in Ingress manifest?
From this document you can add these in the annotations
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: web-ingress
namespace: default
annotations:
nginx.org/max-conns: 2
Try this and let me know if this works
Found a similar stack question with a different approach which can help you to resolve your issues.
I have deployed some APIs in Azure Kubernetes Service and I have been experimenting with Kong to be able to use some of its features such as rate limiting and IP restriction but it doesn't always work as expected. Here is the plugin objects I use:
apiVersion: configuration.konghq.com/v1
kind: KongClusterPlugin
metadata:
name: kong-rate-limiting-plugin
annotations:
kubernetes.io/ingress.class: kong
labels:
global: 'true'
config:
minute: 10
policy: local
limit_by: ip
hide_client_headers: true
plugin: rate-limiting
---
apiVersion: configuration.konghq.com/v1
kind: KongClusterPlugin
metadata:
name: kong-ip-restriction-plugin
annotations:
kubernetes.io/ingress.class: kong
labels:
global: 'true'
config:
deny:
- {some IP}
plugin: ip-restriction
The first problem is when I tried to apply these plugins across the cluster by setting the global label to \"true\" as described here, I got this error when applying it with kubectl:
metadata.labels: Invalid value: "\\\"true\\\"": a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyValue', or 'my_value', or '12345', regex used for validation is '(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?')
The second problem is even though I used KongClusterPlugin and set global to 'true', I still had to add the plugins explicitly to the ingress object for them to work. Here is my ingress:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: my-ing
annotations:
konghq.com/plugins: kong-rate-limiting-plugin,kong-ip-restriction-plugin
konghq.com/protocols: https
konghq.com/https-redirect-status-code: "301"
namespace: default
spec:
ingressClassName: kong
...
And here is my service:
apiVersion: v1
kind: Service
metadata:
name: my-svc
namespace: default
spec:
externalTrafficPolicy: Local
type: LoadBalancer
...
The third problem is by setting limit_by to ip, I expected it to rate-limit per IP, but I noticed it would block all clients when the threshold was hit collectively by the clients. I tried to mitigate that by preserving the client IP and setting externalTrafficPolicy to Local in the service object as I thought maybe the Kubernetes objects weren't receiving the actual client's IP. Now the rate limiting behavior seems to be more reasonable, however sometimes it's as if it's back to its old state and returns HTTP 429 randomly. The other issue I see here is I can set externalTrafficPolicy to Local only when the service type has been set to LoadBalancer or NodePort. I set my service to be of type LoadBalancer which exposes it publicly and seems to be a problem. It would be ironic that using an ingress controller that's supposed to shield the service rather exposes it. Am I missing something here or does this make no sense?
The fourth problem is the IP restriction plugin doesn't seem to be working. I was able to successfully call the APIs from a machine with the IP I put in 'config.deny'.
The fifth problem is the number of times per minute I have to hit the APIs to get a HTTP 429 doesn't match the value I placed in 'config.minute'.
I am modifying a deployment which autoscales using a HorizontalPodAutoscaler (HPA). This deployment is part of a pipeline in which workers read messages from pubsub subscriptions, do some work and publish to the next topic. Right now I use a configmap to define the pipeline for the deployments (the configmap contains input subscription and output topics). The HPA autoscales based on the number of messages on the input subscription. I would like to be able to pull the subscription name for the HPA from a configmap if possible? Is there a way to do this?
example HPA:
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
name: my-deployment-hpa
namespace: default
labels:
name: my-deployment-hpa
spec:
minReplicas: 1
maxReplicas: 10
metrics:
- external:
metricName: pubsub.googleapis.com|subscription|num_undelivered_messages
metricSelector:
matchLabels:
resource.labels.subscription_id: "$INPUT_SUBSCRIPTION"
targetAverageValue: "2"
type: External
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: my-deployment
The value from the HPA currently $INPUT_SUBSCRIPTION could ideally come from a configmap.
Posting this answer as a community wiki for a better visibility as well as the answer was provided in the comments.
Answering the question from the post:
I would like to be able to pull the subscription name for the HPA from a configmap if possible? Is there a way to do this?
As pointed by user #Abdennour TOUMI there is no possibility to set the metric used by HPA with a ConfigMap:
Unfortunately, you cannot.. but you can using prometheus-adapter + HPA . Check this tuto: itnext.io/...
As for a manual workaround you could use a script that will extract needed metric name from the configMap and use a template to replace and apply new HPA.
With a configMap like:
apiVersion: v1
kind: ConfigMap
metadata:
name: example
data:
metric_name: "new_awesome_metric" # <-
not_needed: "only for example"
And following script:
#!/bin/bash
# variables
hpa_file_name="hpa.yaml"
configmap_name="example"
string_to_replace="PLACEHOLDER"
# extract the metric name used in a configmap
new_metric=$(kubectl get configmap $configmap_name -o json | jq '.data.metric_name')
# use the template to replace the $string_to_replace with your $new_metric and apply it
sed "s/$string_to_replace/$new_metric/g" $hpa_file_name | kubectl apply -f -
This script will need to have a hpa.yaml with the template to apply it as resource (example from question could be used with a change:
resource.labels.subscription_id: PLACEHOLDER
For more reference this HPA definition could be based on this guide:
Cloud.google.com: Kubernetes Engine: Tutorials: Autoscaling-metrics: PubSub
Dear members of stackoverflow,
It is possible to configure the maximum number of pods per node in the yaml configuration file of a kubernetes deployment? For example something as
---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: cdn-akamai-pipe
spec:
template:
metadata:
labels:
app: cdn-akamai-pipe
max-pods-per-node: 10
Thanks
This is a kubelet setting that can be set using the --max-pods flag, https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/#kubelet as such there is no way to set this using yaml configuration. If you are using a managed service this can generally be set during cluster creation
Maximum number of pods should be set on Kubelet and not in deployment yaml
--max-pods int32
Number of Pods that can run on this Kubelet. (default 110)
I'm trying to scale a Kubernetes Deployment using a HorizontalPodAutoscaler, which listens to a custom metrics through Stackdriver.
I'm having a GKE cluster, with a Stackdriver adapter enabled.
I'm able to publish the custom metric type to Stackdriver, and following is the way it's being displayed in Stackdriver's Metric Explorer.
This is how I have defined my HPA:
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
name: example-hpa
spec:
minReplicas: 1
maxReplicas: 10
metrics:
- type: External
external:
metricName: custom.googleapis.com|worker_pod_metrics|baz
targetValue: 400
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: test-app-group-1-1
After successfully creating example-hpa, executing kubectl get hpa example-hpa, always shows TARGETS as <unknown>, and never detects the value from custom metrics.
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
example-hpa Deployment/test-app-group-1-1 <unknown>/400 1 10 1 18m
I'm using a Java client which runs locally to publish my custom metrics.
I have given the appropriate resource labels as mentioned here (hard coded - so that it can run without a problem in local environment). I have followed this document to create the Java client.
private static MonitoredResource prepareMonitoredResourceDescriptor() {
Map<String, String> resourceLabels = new HashMap<>();
resourceLabels.put("project_id", "<<<my-project-id>>>);
resourceLabels.put("pod_id", "<my pod UID>");
resourceLabels.put("container_name", "");
resourceLabels.put("zone", "asia-southeast1-b");
resourceLabels.put("cluster_name", "my-cluster");
resourceLabels.put("namespace_id", "mynamespace");
resourceLabels.put("instance_id", "");
return MonitoredResource.newBuilder()
.setType("gke_container")
.putAllLabels(resourceLabels)
.build();
}
What am I doing wrong in the above-mentioned steps please? Thank you in advance for any answers provided!
EDIT [RESOLVED]:
I think I have had some misconfigurations, since kubectl describe hpa [NAME] --v=9 showed me some 403 status code, as well as I was using type: External instead of type: Pods (Thanks MWZ for your answer, pointing out this mistake).
I managed to fix it by creating a new project, a new service account, and a new GKE cluster (basically everything from the beginning again). Then I changed my yaml file as follows, exactly as this document explains.
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
name: test-app-group-1-1
namespace: default
spec:
scaleTargetRef:
apiVersion: apps/v1beta1
kind: Deployment
name: test-app-group-1-1
minReplicas: 1
maxReplicas: 5
metrics:
- type: Pods # Earlier this was type: External
pods: # Earlier this was external:
metricName: baz # metricName: custom.googleapis.com|worker_pod_metrics|baz
targetAverageValue: 20
I'm now exporting as custom.googleapis.com/baz, and NOT as custom.googleapis.com/worker_pod_metrics/baz. Also, now I'm explicitly specifying the namespace for my HPA in the yaml.
Since you can see your custom metric in Stackdriver GUI I'm guessing metrics are correctly exported. Based on Autoscaling Deployments with Custom Metrics I believe you wrongly defined metric to be used by HPA to scale the deployment.
Please try using this YAML:
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
name: example-hpa
spec:
minReplicas: 1
maxReplicas: 10
metrics:
- type: Pods
pods:
metricName: baz
targetAverageValue: 400
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: test-app-group-1-1
Please have in mind that:
The HPA uses the metrics to compute an average and compare it to the
target average value. In the application-to-Stackdriver export
example, a Deployment contains Pods that export metric. The following
manifest file describes a HorizontalPodAutoscaler object that scales a
Deployment based on the target average value for the metric.
Troubleshooting steps described on the page above can also be useful.
Side-note
Since above HPA is using beta API autoscaling/v2beta1 I got error when running kubectl describe hpa [DEPLOYMENT_NAME]. I ran kubectl describe hpa [DEPLOYMENT_NAME] --v=9 and got response in JSON.
It is a good practice to put some unique labels to target your metrics. Right now, based on metrics labelled in your java client, only pod_id looks unique which can't be used due to its stateless nature.
So, I would suggest you try introducing a deployment/metrics wide unqiue identifier.
resourceLabels.put("<identifier>", "<could-be-deployment-name>");
After this, you can try modifying your HPA with something similar to following:
kind: HorizontalPodAutoscaler
metadata:
name: example-hpa
spec:
minReplicas: 1
maxReplicas: 10
metrics:
- type: External
external:
metricName: custom.googleapis.com|worker_pod_metrics|baz
metricSelector:
matchLabels:
# define labels to target
metric.labels.identifier: <deployment-name>
# scale +1 whenever it crosses multiples of mentioned value
targetAverageValue: "400"
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: test-app-group-1-1
Apart from this, this setup has no issues and should work smooth.
Helper command to see what metrics are exposed to HPA :
kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1/namespaces/default/custom.googleapis.com|worker_pod_metrics|baz" | jq