Kubernetes HPA using metrics from another deployment - kubernetes

Question

Im currently trying to run an autoscaling demo using prometheus and the prometheus adapter, and i was wondering if there is a way to autoscale one of my deployments based on metrics that prometheus scrapes from another deployment.
What i have right now are 2 different deployments, kafka-consumer-application (which i want to scale) and kafka-exporter (which exposes the kafka metrics that I'll be using for scaling). I know that if I have both of them as containers in the same deployment the autoscaling works, but the issue is that the kafka-exporter also gets autoscaled and its not ideal, so i want to separate them. I tried with the following HPA but i could not get it to work:
kind: HorizontalPodAutoscaler
apiVersion: autoscaling/v2beta1
metadata:
name: consumer-hpa
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: kafka-consumer-application
minReplicas: 1
maxReplicas: 10
metrics:
- type: object
object:
target: kafka-exporter
metricName: "kafka_consumergroup_lag"
targetValue: 5
Im not sure if im doing something wrong or if this is just not something i can do, so any advice is appreciated.
Thanks!
Note: im running the adapter with this config:
rules:
default: false
resource: {}
custom:
- seriesQuery: 'kafka_consumergroup_lag'
resources:
overrides:
kubernetes_namespace: {resource: "namespace"}
kubernetes_pod_name: {resource: "pod"}
name:
matches: "kafka_consumergroup_lag"
as: "kafka_consumergroup_lag"
metricsQuery: 'avg_over_time(kafka_consumergroup_lag{topic="my-topic",consumergroup="we-consume"}[1m])'
``

Answer 1

In kubernetes documentation you can read:
Autoscaling on metrics not related to Kubernetes objects
Applications running on Kubernetes may need to autoscale based on metrics that don’t have an obvious relationship to any object in the Kubernetes cluster, such as metrics describing a hosted service with no direct correlation to Kubernetes namespaces. In Kubernetes 1.10 and later, you can address this use case with external metrics
So using external metrics, your HPA yaml could look like following:
kind: HorizontalPodAutoscaler
apiVersion: autoscaling/v2beta2
metadata:
name: consumer-hpa
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: kafka-consumer-application
minReplicas: 1
maxReplicas: 10
metrics:
- type: External
external:
metric:
name: kafka_consumergroup_lag
#selector:
# matchLabels:
# topic: "my-topic"
target:
type: AverageValue
averageValue: 5
If you have more than one kafka-exporter you can use selector to filter it (source):
selector is the string-encoded form of a standard kubernetes label selector for the given metric When set, it is passed as an additional parameter to the metrics server for more specific metrics scoping. When unset, just the metricName will be used to gather metrics
Also have a look at this Stack question.