Autoscaling Kubernetes based on log based metrics - kubernetes

Question

I use Autopilot on GKE. I've created some log based metrics that I'd like to use to scale up pods.
To begin with - I'm not sure if it's great idea - the metric is just number of records in DB to process... I have a feeling using logs to scale app might bring in some weird infinite loop or something....
Anyhow - I've tried entering logging.googleapis.com|user|celery-person-count as an external metric and got HPA cannot read metric value. Installed Stackdriver adapter but not too sure how to use it either.

Answer 1

GKE Autopilot clusters have Workload Identity enabled for consuming other GCP services, including Cloud Monitoring.
You'll want to follow the steps here in order to deploy the Custom Metrics Adapter on Autopilot clusters.
kubectl create clusterrolebinding cluster-admin-binding \
--clusterrole cluster-admin --user "$(gcloud config get-value account)"
kubectl create namespace custom-metrics
kubectl create serviceaccount --namespace custom-metrics \
custom-metrics-stackdriver-adapter
gcloud iam service-accounts create GSA_NAME
gcloud projects add-iam-policy-binding PROJECT_ID \
--member "serviceAccount:GSA_NAME#PROJECT_ID.iam.gserviceaccount.com" \
--role "roles/monitoring.viewer"
gcloud iam service-accounts add-iam-policy-binding \
--role roles/iam.workloadIdentityUser \
--member "serviceAccount:PROJECT_ID.svc.id.goog[custom-metrics/custom-metrics-stackdriver-adapter]" \
GSA_NAME#PROJECT_ID.iam.gserviceaccount.com
kubectl annotate serviceaccount \
--namespace custom-metrics custom-metrics-stackdriver-adapter \
iam.gke.io/gcp-service-account=GSA_NAME#PROJECT_ID.iam.gserviceaccount.com
kubectl apply -f manifests/adapter_new_resource_model.yaml
Given that you've already deployed the adapter, you'll want to delete the deployment first, although you might just be able to run the steps starting at gcloud iam ...
You'll need to replace GSA_NAME with a name of your choosing and PROJECT_ID with your Google Cloud project ID.

Answer 2

Yes , Horizontal Pod Autoscaling can be done using metric values. If your application is running in Kubernetes then you must use a custom metric for Autoscaling and if your application is not running in Kubernetes then you must use External metric for Autoscaling.
A custom metric can be selected for any of the following:
A particular node, Pod, or any Kubernetes object of any kind,
including a CustomResourceDefinition (CRD).
The average value for a metric reported by all Pods in a Deployment.
Refer this documentation for detailed explanation of using custom
metric for Horizontal Pod Autoscaling.

Answer 3

Custom metrics
GitHub link:
kubectl create clusterrolebinding cluster-admin-binding \
--clusterrole cluster-admin --user "$(gcloud config get-value account)"
resource : kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/k8s-stackdriver/master/custom-metrics-stackdriver-adapter/deploy/production/adapter_new_resource_model.yaml
Deploying application with metrics
git clone https://github.com/GoogleCloudPlatform/kubernetes-engine-samples.git
cd kubernetes-engine-samples/custom-metrics-autoscaling/direct-to-sd
HPA example
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
name: custom-metric-sd
namespace: default
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: custom-metric-sd
minReplicas: 1
maxReplicas: 5
metrics:
- type: Pods
pods:
metric:
name: custom-metric
target:
type: AverageValue
averageValue: 20
You can check out this link for more information.
if your workload running top of VM or not on K8s use the external metrics instead of custom.
Refer to the documentation on Custom and external metrics for autoscaling workloads .