Received 130 events in the last 3 days
I see that event on a new cluster, it functions properly, manages to pass successfully all health/liveness/functional requests
Is it a normal event that runs every X minutes?
I suspect it's related to the AWS load balancer controller but not sure how to proceed to explore that issue.
Here's the event object (Changed a bit the unique ids)
kind: Event
apiVersion: v1
metadata:
name: k8s-default-proxyngi-21j23klsu.16322252fc4d27866
namespace: default
selfLink: >-
/api/v1/namespaces/default/events/k8s-default-proxyngi-21j23klsu.16322252fc4d27866
uid: e6e56ba2-82b6-76aafb51c753
resourceVersion: '1578355'
creationTimestamp: '2021-02-21T12:52:52Z'
managedFields:
- manager: controller
operation: Update
apiVersion: v1
time: '2021-02-21T12:52:52Z'
fieldsType: FieldsV1
fieldsV1:
'f:count': {}
'f:firstTimestamp': {}
'f:involvedObject':
'f:apiVersion': {}
'f:kind': {}
'f:name': {}
'f:namespace': {}
'f:resourceVersion': {}
'f:uid': {}
'f:lastTimestamp': {}
'f:message': {}
'f:reason': {}
'f:source':
'f:component': {}
'f:type': {}
involvedObject:
kind: TargetGroupBinding
namespace: default
name: k8s-default-proxyngi-1c76e22ad3
uid: e6e56ba2-82b6-76aafb51c753-f4a4d9812632
apiVersion: elbv2.k8s.aws/v1beta1
resourceVersion: '238786'
reason: SuccessfullyReconciled
message: Successfully reconciled
source:
component: targetGroupBinding
firstTimestamp: '2021-02-16T15:50:37Z'
lastTimestamp: '2021-02-21T12:52:52Z'
count: 131
type: Normal
eventTime: null
reportingComponent: ''
reportingInstance: ''
Related
I am not sure where else to turn as all example I have seen I have pretty much copied and still cannot get it to work. The connector will not install and states empty password. I have validted each step and cannot get it to work. Here are the steps I have taken.
Container
FROM strimzi/kafka:0.16.1-kafka-2.4.0
USER root:root
RUN mkdir -p /opt/kafka/plugins/debezium
COPY ./debezium-connector-mysql/ /opt/kafka/plugins/debezium/
USER 1001
Next I create the secret to use with mySQL.
cat <<EOF | kubectl apply -n kafka-cloud -f -
apiVersion: v1
kind: Secret
metadata:
name: mysql-auth
type: Opaque
stringData:
mysql-auth.properties: |-
username: root
password: supersecret
EOF
Validate
% kubectl -n kafka-cloud get secrets | grep mysql-auth
mysql-auth Opaque 1 14m
Double check to make sure the user and password are not empty as the error in the connector state.
% kubectl -n kafka-cloud get secret mysql-auth -o yaml
apiVersion: v1
data:
mysql-auth.properties: dXNlcm5hbWU6IHJvb3QKcGFzc3dvcmQ6IHN1cGVyc2VjcmV0
kind: Secret
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"v1","kind":"Secret","metadata":{"annotations":{},"name":"mysql-auth","namespace":"kafka-cloud"},"stringData":{"mysql-auth.properties":"username: root\npassword: supersecret"},"type":"Opaque"}
creationTimestamp: "2022-03-02T23:48:55Z"
name: mysql-auth
namespace: kafka-cloud
resourceVersion: "4041"
uid: 14a7a878-d01f-4899-8dc7-81b515278f32
type: Opaque
Add Connect Cluster
cat <<EOF | kubectl apply -n kafka-cloud -f -
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
name: my-connect-cluster
annotations:
# # use-connector-resources configures this KafkaConnect
# # to use KafkaConnector resources to avoid
# # needing to call the Connect REST API directly
strimzi.io/use-connector-resources: "true"
spec:
version: 3.1.0
image: connect-debezium
replicas: 1
bootstrapServers: my-kafka-cluster-kafka-bootstrap:9092
config:
group.id: connect-cluster
offset.storage.topic: connect-cluster-offsets
config.storage.topic: connect-cluster-configs
status.storage.topic: connect-cluster-status
config.storage.replication.factor: 1
offset.storage.replication.factor: 1
status.storage.replication.factor: 1
config.providers: file
config.providers.file.class: org.apache.kafka.common.config.provider.FileConfigProvider
externalConfiguration:
volumes:
- name: mysql-auth-config
secret:
secretName: mysql-auth
EOF
Add Connector
cat <<EOF | kubectl apply -n kafka-cloud -f -
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnector
metadata:
name: mysql-test-connector
labels:
strimzi.io/cluster: my-connect-cluster
spec:
class: io.debezium.connector.mysql.MySqlConnector
tasksMax: 1
config:
database.hostname: 172.17.0.13
database.port: 3306
database.user: "${file:/opt/kafka/external-configuration/mysql-auth-config/mysql-auth.properties:username}"
database.password: "${file:/opt/kafka/external-configuration/mysql-auth-config/mysql-auth.properties:password}"
database.server.id: 184054
database.server.name: mysql-pod
database.whitelist: sample
database.history.kafka.bootstrap.servers: my-kafka-cluster-kafka-bootstrap:9092
database.history.kafka.topic: "schema-changes.sample"
key.converter: "org.apache.kafka.connect.storage.StringConverter"
value.converter: "org.apache.kafka.connect.storage.StringConverter"
EOF
Error
And no matter what I have tried to get this error. I have no idea what I am missing. I know it a simple config, but I cannot figure it out. I'm stuck.
% kubectl -n kafka-cloud describe kafkaconnector mysql-test-connector
Name: mysql-test-connector
Namespace: kafka-cloud
Labels: strimzi.io/cluster=my-connect-cluster
Annotations: <none>
API Version: kafka.strimzi.io/v1beta2
Kind: KafkaConnector
Metadata:
Creation Timestamp: 2022-03-02T23:44:20Z
Generation: 1
Managed Fields:
API Version: kafka.strimzi.io/v1beta2
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:kubectl.kubernetes.io/last-applied-configuration:
f:labels:
.:
f:strimzi.io/cluster:
f:spec:
.:
f:class:
f:config:
.:
f:database.history.kafka.bootstrap.servers:
f:database.history.kafka.topic:
f:database.hostname:
f:database.password:
f:database.port:
f:database.server.id:
f:database.server.name:
f:database.user:
f:database.whitelist:
f:key.converter:
f:value.converter:
f:tasksMax:
Manager: kubectl-client-side-apply
Operation: Update
Time: 2022-03-02T23:44:20Z
API Version: kafka.strimzi.io/v1beta2
Fields Type: FieldsV1
fieldsV1:
f:status:
.:
f:conditions:
f:observedGeneration:
f:tasksMax:
f:topics:
Manager: okhttp
Operation: Update
Subresource: status
Time: 2022-03-02T23:44:20Z
Resource Version: 3874
UID: c70ffe4e-3777-4524-af82-dad3a57ca25e
Spec:
Class: io.debezium.connector.mysql.MySqlConnector
Config:
database.history.kafka.bootstrap.servers: my-kafka-cluster-kafka-bootstrap:9092
database.history.kafka.topic: schema-changes.sample
database.hostname: 172.17.0.13
database.password:
database.port: 3306
database.server.id: 184054
database.server.name: mysql-pod
database.user:
database.whitelist: sample
key.converter: org.apache.kafka.connect.storage.StringConverter
value.converter: org.apache.kafka.connect.storage.StringConverter
Tasks Max: 1
Status:
Conditions:
Last Transition Time: 2022-03-02T23:45:00.097311Z
Message: PUT /connectors/mysql-test-connector/config returned 400 (Bad Request): Connector configuration is invalid and contains the following 1 error(s):
A value is required
You can also find the above list of errors at the endpoint `/{connectorType}/config/validate`
Reason: ConnectRestException
Status: True
Type: NotReady
Observed Generation: 1
Tasks Max: 1
Topics:
Events: <none>
The config param needed for the mySQL connector is:
database.allowPublicKeyRetrieval: true
That resolved the issue.
I have cassandra operator installed and I setup cassandra datacenter/cluster with 3 nodes.
I have created sample keyspace, table and inserted the data. I see it has created 3 PVC's in my storage section. When I deleting the dataceneter its delete associated PVC's as well ,So when I setup same configuration Datacenter/cluster , its completely new , No earlier keyspace or tables.
How can I make them persistence for future use? I am using sample yaml from below
https://github.com/datastax/cass-operator/tree/master/operator/example-cassdc-yaml/cassandra-3.11.x
I don't find any persistentVolumeClaim configuration in it , Its having storageConfig:
cassandraDataVolumeClaimSpec:
Is anyone came across such scenario?
Edit: Storage class details:
allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
annotations:
description: Provides RWO and RWX Filesystem volumes with Retain Policy
storageclass.kubernetes.io/is-default-class: "false"
name: ocs-storagecluster-cephfs-retain
parameters:
clusterID: openshift-storage
csi.storage.k8s.io/controller-expand-secret-name: rook-csi-cephfs-provisioner
csi.storage.k8s.io/controller-expand-secret-namespace: openshift-storage
csi.storage.k8s.io/node-stage-secret-name: rook-csi-cephfs-node
csi.storage.k8s.io/node-stage-secret-namespace: openshift-storage
csi.storage.k8s.io/provisioner-secret-name: rook-csi-cephfs-provisioner
csi.storage.k8s.io/provisioner-secret-namespace: openshift-storage
fsName: ocs-storagecluster-cephfilesystem
provisioner: openshift-storage.cephfs.csi.ceph.com
reclaimPolicy: Retain
volumeBindingMode: Immediate
Here is Cassandra cluster YAML:
apiVersion: cassandra.datastax.com/v1beta1
kind: CassandraDatacenter
metadata:
name: dc
generation: 2
spec:
size: 3
config:
cassandra-yaml:
authenticator: AllowAllAuthenticator
authorizer: AllowAllAuthorizer
role_manager: CassandraRoleManager
jvm-options:
additional-jvm-opts:
- '-Ddse.system_distributed_replication_dc_names=dc1'
- '-Ddse.system_distributed_replication_per_dc=1'
initial_heap_size: 800M
max_heap_size: 800M
resources: {}
clusterName: cassandra
systemLoggerResources: {}
configBuilderResources: {}
serverVersion: 3.11.7
serverType: cassandra
storageConfig:
cassandraDataVolumeClaimSpec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
storageClassName: ocs-storagecluster-cephfs-retain
managementApiAuth:
insecure: {}
EDIT: PV Details:
oc get pv pvc-15def0ca-6cbc-4569-a560-7b9e89a7b7a7 -o yaml
apiVersion: v1
kind: PersistentVolume
metadata:
annotations:
pv.kubernetes.io/provisioned-by: openshift-storage.cephfs.csi.ceph.com
creationTimestamp: "2022-02-23T20:52:54Z"
finalizers:
- kubernetes.io/pv-protection
managedFields:
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.: {}
f:pv.kubernetes.io/provisioned-by: {}
f:spec:
f:accessModes: {}
f:capacity:
.: {}
f:storage: {}
f:claimRef:
.: {}
f:apiVersion: {}
f:kind: {}
f:name: {}
f:namespace: {}
f:resourceVersion: {}
f:uid: {}
f:csi:
.: {}
f:controllerExpandSecretRef:
.: {}
f:name: {}
f:namespace: {}
f:driver: {}
f:nodeStageSecretRef:
.: {}
f:name: {}
f:namespace: {}
f:volumeAttributes:
.: {}
f:clusterID: {}
f:fsName: {}
f:storage.kubernetes.io/csiProvisionerIdentity: {}
f:subvolumeName: {}
f:volumeHandle: {}
f:persistentVolumeReclaimPolicy: {}
f:storageClassName: {}
f:volumeMode: {}
manager: csi-provisioner
operation: Update
time: "2022-02-23T20:52:54Z"
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:status:
f:phase: {}
manager: kube-controller-manager
operation: Update
time: "2022-02-23T20:52:54Z"
name: pvc-15def0ca-6cbc-4569-a560-7b9e89a7b7a7
resourceVersion: "51684941"
selfLink: /api/v1/persistentvolumes/pvc-15def0ca-6cbc-4569-a560-7b9e89a7b7a7
uid: 8ded2de5-6d4e-45a1-9b89-a385d74d6d4a
spec:
accessModes:
- ReadWriteOnce
capacity:
storage: 10Gi
claimRef:
apiVersion: v1
kind: PersistentVolumeClaim
name: server-data-cstone-cassandra-cstone-dc-default-sts-1
namespace: dv01-cornerstone
resourceVersion: "51684914"
uid: 15def0ca-6cbc-4569-a560-7b9e89a7b7a7
csi:
controllerExpandSecretRef:
name: rook-csi-cephfs-provisioner
namespace: openshift-storage
driver: openshift-storage.cephfs.csi.ceph.com
nodeStageSecretRef:
name: rook-csi-cephfs-node
namespace: openshift-storage
volumeAttributes:
clusterID: openshift-storage
fsName: ocs-storagecluster-cephfilesystem
storage.kubernetes.io/csiProvisionerIdentity: 1645064620191-8081-openshift-storage.cephfs.csi.ceph.com
subvolumeName: csi-vol-92d5e07d-94ea-11ec-92e8-0a580a20028c
volumeHandle: 0001-0011-openshift-storage-0000000000000001-92d5e07d-94ea-11ec-92e8-0a580a20028c
persistentVolumeReclaimPolicy: Retain
storageClassName: ocs-storagecluster-cephfs-retain
volumeMode: Filesystem
status:
phase: Bound
According to the spec:
The storage configuration. This sets up a 100GB volume at /var/lib/cassandra
on each server pod. The user is left to create the server-storage storage
class by following these directions...
https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/ssd-pd
Before you deploy the Cassandra spec, first ensure your cluster already have the CSI driver installed and working properly, then proceed to create the StorageClass as the spec required:
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: server-storage
provisioner: pd.csi.storage.gke.io
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
reclaimPolicy: Retain
parameters:
type: pd-ssd
Re-deploy your Cassandra now should have the data disk retain upon deletion.
I have multiple prometheusRules(rule a, rule b), and each rule defined different exp to constraint the alert; then, I have different AlertmanagerConfig(one receiver is slack, then other one's receiver is opsgenie); How can we make a connection between rules and alertmanagerconfig? for example: if rule a is triggered, I want to send message to slack; if rule b is triggered, I want to send message to opsgenie.
Here is what I tried, however, that does not work. Did I miss something?
This is prometheuisRule file
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
labels:
prometheus: service-prometheus
role: alert-rules
app: kube-prometheus-stack
release: monitoring-prom
name: rule_a
namespace: monitoring
spec:
groups:
- name: rule_a_alert
rules:
- alert: usage_exceed
expr: salesforce_api_usage > 100000
labels:
severity: urgent
This is alertManagerConfig
apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
labels:
alertmanagerConfig: slack
name: slack
namespace: monitoring
resourceVersion: "25842935"
selfLink: /apis/monitoring.coreos.com/v1alpha1/namespaces/monitoring/alertmanagerconfigs/opsgenie-and-slack
uid: fbb74924-5186-4929-b363-8c056e401921
spec:
receivers:
- name: slack-receiver
slackConfigs:
- apiURL:
key: apiURL
name: slack-config
route:
groupBy:
- job
groupInterval: 60s
groupWait: 60s
receiver: slack-receiver
repeatInterval: 1m
routes:
- matchers:
- name: job
value: service_a
receiver: slack-receiver
You need to match on a label of the alert, in your case you're trying to match on the label job with the value service_a which doesn't exist. You could either match on a label that does exist in the prometheuisRule file, eg severity, by changing the match in the alertManagerConfig file:
route:
routes:
- match:
severity: urgent
receiver: slack-receiver
or you could add another label to the prometheuisRule file:
spec:
groups:
- name: rule_a_alert
rules:
- alert: usage_exceed
expr: salesforce_api_usage > 100000
labels:
severity: urgent
job: service_a
I'm trying to install CockroachDB with Rancher and getting some problems, showing:
FailedBinding (5) 14 sec ago no persistent volumes available for this claim and no storage class is set
How can this be solved?
Here are the configurations in my local machine:
PersistentVolumeClaim: datadir-cockroachdb-0
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
creationTimestamp: "2021-01-07T23:50:42Z"
finalizers:
- kubernetes.io/pvc-protection
labels:
app.kubernetes.io/component: cockroachdb
app.kubernetes.io/instance: cockroachdb
app.kubernetes.io/name: cockroachdb
managedFields:
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:labels:
.: {}
f:app.kubernetes.io/component: {}
f:app.kubernetes.io/instance: {}
f:app.kubernetes.io/name: {}
f:spec:
f:accessModes: {}
f:resources:
f:requests:
.: {}
f:storage: {}
f:volumeMode: {}
f:status:
f:phase: {}
manager: k3s
operation: Update
time: "2021-01-07T23:50:41Z"
name: datadir-cockroachdb-0
namespace: default
resourceVersion: "188922"
selfLink: /api/v1/namespaces/default/persistentvolumeclaims/datadir-cockroachdb-0
uid: ef83d3c7-0309-44a8-b379-0134835d97a9
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 100Gi
volumeMode: Filesystem
status:
phase: Pending
CockroachDB
clusterDomain: cluster.local
conf:
attrs: []
cache: 25%
cluster-name: ''
disable-cluster-name-verification: false
http-port: 8080
join: []
locality: ''
logtostderr: INFO
max-disk-temp-storage: 0
max-offset: 500ms
max-sql-memory: 25%
port: 26257
single-node: false
sql-audit-dir: ''
image:
credentials: {}
pullPolicy: IfNotPresent
repository: cockroachdb/cockroach
tag: v20.1.3
ingress:
annotations: {}
enabled: false
hosts: []
labels: {}
paths:
- /
tls: []
init:
affinity: {}
annotations: {}
labels:
app.kubernetes.io/component: init
nodeSelector: {}
resources: {}
tolerations: []
labels: {}
networkPolicy:
enabled: false
ingress:
grpc: []
http: []
service:
discovery:
annotations: {}
labels:
app.kubernetes.io/component: cockroachdb
ports:
grpc:
external:
name: grpc
port: 26257
internal:
name: grpc-internal
port: 26257
http:
name: http
port: 8080
public:
annotations: {}
labels:
app.kubernetes.io/component: cockroachdb
type: ClusterIP
statefulset:
annotations: {}
args: []
budget:
maxUnavailable: 1
env: []
labels:
app.kubernetes.io/component: cockroachdb
nodeAffinity: {}
nodeSelector: {}
podAffinity: {}
podAntiAffinity:
type: soft
weight: 100
podManagementPolicy: Parallel
priorityClassName: ''
replicas: 3
resources: {}
secretMounts: []
tolerations: []
updateStrategy:
type: RollingUpdate
storage:
hostPath: ''
persistentVolume: volume1
annotations: {}
enabled: true
labels: {}
size: 1Gi
storageClass: local-storage ''
tls:
certs:
clientRootSecret: cockroachdb-root
nodeSecret: cockroachdb-node
provided: false
tlsSecret: false
enabled: false
init:
image:
credentials: {}
pullPolicy: IfNotPresent
repository: cockroachdb/cockroach-k8s-request-cert
tag: '0.4'
serviceAccount:
create: true
name: ''
Storage: 1Gi
PersistentVolume
apiVersion: v1
kind: PersistentVolume
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: '{"apiVersion":"v1","kind":"PersistentVolume","metadata":{"annotations":{},"labels":{"type":"local"},"name":"volume1"},"spec":{"accessModes":["ReadWriteOnce"],"capacity":{"storage":"10Gi"},"hostPath":{"path":"/data/volume1"}}}'
creationTimestamp: "2021-01-07T23:11:43Z"
finalizers:
- kubernetes.io/pv-protection
labels:
type: local
managedFields:
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:status:
f:phase: {}
manager: k3s
operation: Update
time: "2021-01-07T23:11:43Z"
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations: {}
f:labels:
.: {}
f:type: {}
f:spec:
f:accessModes: {}
f:capacity: {}
f:hostPath:
.: {}
f:path: {}
f:type: {}
f:persistentVolumeReclaimPolicy: {}
f:volumeMode: {}
manager: kubectl
operation: Update
time: "2021-01-07T23:11:43Z"
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
f:kubectl.kubernetes.io/last-applied-configuration: {}
f:spec:
f:capacity:
f:storage: {}
manager: Go-http-client
operation: Update
time: "2021-01-07T23:12:11Z"
name: volume1
resourceVersion: "173783"
selfLink: /api/v1/persistentvolumes/volume1
uid: 6e76984c-22cd-4219-9ff6-ba7f67c1ca72
spec:
accessModes:
- ReadWriteOnce
capacity:
storage: 4Gi
hostPath:
path: /data/volume1
type: ""
persistentVolumeReclaimPolicy: Retain
volumeMode: Filesystem
status:
phase: Available
StorageClass
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
creationTimestamp: "2021-01-07T23:29:17Z"
managedFields:
- apiVersion: storage.k8s.io/v1
fieldsType: FieldsV1
fieldsV1:
f:provisioner: {}
f:reclaimPolicy: {}
f:volumeBindingMode: {}
manager: rancher
operation: Update
time: "2021-01-07T23:29:17Z"
name: local-storage
resourceVersion: "180190"
selfLink: /apis/storage.k8s.io/v1/storageclasses/local-storage
uid: 0a5f8b75-7fb5-4965-91ee-91b0a087339a
provisioner: kubernetes.io/no-provisioner
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
With provided details looks like your storage class is missing on rancher.
Without storage class respective PVC won't get created so it's giving an error. Storage classes may change with cloud providers and also based on the requirement of the type of disk SSD, HDD.
You can get more idea : https://rancher.com/docs/rancher/v2.x/en/cluster-admin/volumes-and-storage/provisioning-new-storage/
check first your PV is available and after that check for storage class and PVC.
It looks like the issue was with Rancher this time (Thank you #Harsh Manvar for answering). If you have more questions about CockroachDB you can also join the CockroachDB community slack channel where you will find loads of experts who can answer your questions in a timely manner. (And be sure to join the #community channel also to have some FUN!) :) https://go.crdb.dev/p/slack
I'm trying to scale a deployment based on a custom metric coming from a custom metric server. I deployed my server and when I do
kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/services/kubernetes/test-metric"
I get back this JSON
{
"kind": "MetricValueList",
"apiVersion": "custom.metrics.k8s.io/v1beta1",
"metadata": {
"selfLink": "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/services/kubernetes/test-metric"
},
"items": [
{
"describedObject": {
"kind": "Service",
"namespace": "default",
"name": "kubernetes",
"apiVersion": "/v1"
},
"metricName": "test-metric",
"timestamp": "2019-01-26T02:36:19Z",
"value": "300m",
"selector": null
}
]
}
Then I created my hpa.yml using this
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
name: test-all-deployment
namespace: default
spec:
maxReplicas: 10
minReplicas: 1
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: test-all-deployment
metrics:
- type: Object
object:
target:
kind: Service
name: kubernetes
apiVersion: custom.metrics.k8s.io/v1beta1
metricName: test-metric
targetValue: 200m
but it doesn't scale and I'm not sure what is wrong. running get hpa returns
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
test-all-deployment Deployment/test-all-deployment <unknown>/200m 1 10 1 9m
The part I'm not sure about is the target object in the metrics collection in the hpa definition. Looking at the doc here https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/
It has
describedObject:
apiVersion: extensions/v1beta1
kind: Ingress
name: main-route
target:
kind: Value
value: 10k
but that gives me a validation error for API v2beta1. and looking at the actual object here https://github.com/kubernetes/api/blob/master/autoscaling/v2beta1/types.go#L296 it doesn't seem to match. I don't know how to specify that with the v2beta1 API.
It looks like there is a mistake in the documentation. In the same example two diffierent API version are used.
autoscaling/v2beta1 notation:
- type: Pods
pods:
metric:
name: packets-per-second
targetAverageValue: 1k
autoscaling/v2beta2 notation:
- type: Resource
resource:
name: cpu
target:
type: AverageUtilization
averageUtilization: 50
There is a difference between autoscaling/v2beta1 and autoscaling/v2beta2 APIs:
kubectl get hpa.v2beta1.autoscaling -o yaml --export > hpa2b1-export.yaml
kubectl get hpa.v2beta2.autoscaling -o yaml --export > hpa2b2-export.yaml
diff -y hpa2b1-export.yaml hpa2b2-export.yaml
#hpa.v2beta1.autoscaling hpa.v2beta2.autoscaling
#-----------------------------------------------------------------------------------
apiVersion: v1 apiVersion: v1
items: items:
- apiVersion: autoscaling/v2beta1 | - apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler kind: HorizontalPodAutoscaler
metadata: metadata:
creationTimestamp: "2019-03-21T13:17:47Z" creationTimestamp: "2019-03-21T13:17:47Z"
name: php-apache name: php-apache
namespace: default namespace: default
resourceVersion: "8441304" resourceVersion: "8441304"
selfLink: /apis/autoscaling/v2beta1/namespaces/default/ho | selfLink: /apis/autoscaling/v2beta2/namespaces/default/ho
uid: b8490a0a-4bdb-11e9-9043-42010a9c0003 uid: b8490a0a-4bdb-11e9-9043-42010a9c0003
spec: spec:
maxReplicas: 10 maxReplicas: 10
metrics: metrics:
- resource: - resource:
name: cpu name: cpu
targetAverageUtilization: 50 | target:
> averageUtilization: 50
> type: Utilization
type: Resource type: Resource
minReplicas: 1 minReplicas: 1
scaleTargetRef: scaleTargetRef:
apiVersion: extensions/v1beta1 apiVersion: extensions/v1beta1
kind: Deployment kind: Deployment
name: php-apache name: php-apache
status: status:
conditions: conditions:
- lastTransitionTime: "2019-03-21T13:18:02Z" - lastTransitionTime: "2019-03-21T13:18:02Z"
message: recommended size matches current size message: recommended size matches current size
reason: ReadyForNewScale reason: ReadyForNewScale
status: "True" status: "True"
type: AbleToScale type: AbleToScale
- lastTransitionTime: "2019-03-21T13:18:47Z" - lastTransitionTime: "2019-03-21T13:18:47Z"
message: the HPA was able to successfully calculate a r message: the HPA was able to successfully calculate a r
resource utilization (percentage of request) resource utilization (percentage of request)
reason: ValidMetricFound reason: ValidMetricFound
status: "True" status: "True"
type: ScalingActive type: ScalingActive
- lastTransitionTime: "2019-03-21T13:23:13Z" - lastTransitionTime: "2019-03-21T13:23:13Z"
message: the desired replica count is increasing faster message: the desired replica count is increasing faster
rate rate
reason: TooFewReplicas reason: TooFewReplicas
status: "True" status: "True"
type: ScalingLimited type: ScalingLimited
currentMetrics: currentMetrics:
- resource: - resource:
currentAverageUtilization: 0 | current:
currentAverageValue: 1m | averageUtilization: 0
> averageValue: 1m
name: cpu name: cpu
type: Resource type: Resource
currentReplicas: 1 currentReplicas: 1
desiredReplicas: 1 desiredReplicas: 1
kind: List kind: List
metadata: metadata:
resourceVersion: "" resourceVersion: ""
selfLink: "" selfLink: ""
Here is how the object definition is supposed to look like:
#hpa.v2beta1.autoscaling hpa.v2beta2.autoscaling
#-----------------------------------------------------------------------------------
type: Object type: Object
object: object:
metric: metric:
name: requests-per-second name: requests-per-second
describedObject: describedObject:
apiVersion: extensions/v1beta1 apiVersion: extensions/v1beta1
kind: Ingress kind: Ingress
name: main-route name: main-route
targetValue: 2k target:
type: Value
value: 2k