I have a question about a Helm upgrade. I'm working on a helm chart which deploys a pod with a docker image tag:2.190.1-alpine.
Release name is jenkins.
I want change the docker image in the helm chart to 2.240
I launch:
helm upgrade jenkins codecentric/jenkins --set image.tag=2.2
But I got error:
Error: UPGRADE FAILED: unable to decode "": resource.metadataOnlyObject.ObjectMeta: v1.ObjectMeta.Labels: ReadString: expects " or n, but found 2, error found in #10 byte of ...|version":2.240,"helm.|..., bigger context ...|s.io/name":"jenkins","app.kubernetes.io/version":2.240,"helm.sh/chart":"jenkins-1.7.0"},"name":"myReleases|...
Has anyone had experience with this?
Here is a deployment file:
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: "1"
creationTimestamp: "2020-02-06T22:39:51Z"
generation: 1
labels:
app.kubernetes.io/component: master
app.kubernetes.io/instance: jenkins
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: jenkins
helm.sh/chart: jenkins-1.5.1
name: jenkins
namespace: default
resourceVersion: "28697012"
selfLink: /apis/apps/v1/namespaces/default/deployments/jenkins
uid: 9934dea3-b3a1-4109-b725-da5410aacc5f
spec:
progressDeadlineSeconds: 600
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
app.kubernetes.io/component: master
app.kubernetes.io/instance: jenkins
app.kubernetes.io/name: jenkins
strategy:
type: Recreate
template:
metadata:
annotations:
checksum/init: e43addd28e27d8ab36052943e5586515d252a27c3ddd85e15c43e99501ab1c0b
checksum/ref: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
creationTimestamp: null
labels:
app.kubernetes.io/component: master
app.kubernetes.io/instance: jenkins
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: jenkins
helm.sh/chart: jenkins-1.5.1
spec:
containers:
- env:
- name: JENKINS_SLAVE_AGENT_PORT
value: "50000"
- name: JAVA_OPTS
value: -Dhudson.slaves.NodeProvisioner.initialDelay=0 -Dhudson.model.LoadStatistics.decay=0.7
-Dhudson.slaves.NodeProvisioner.MARGIN=30 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.6
-XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:MaxRAMFraction=2
-XshowSettings:vm
image: jenkins/jenkins:2.190.1-alpine
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 3
httpGet:
path: /login
port: http
scheme: HTTP
initialDelaySeconds: 90
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
name: jenkins-master
ports:
- containerPort: 8080
name: http
protocol: TCP
- containerPort: 50000
name: agent
protocol: TCP
readinessProbe:
failureThreshold: 3
httpGet:
path: /login
port: http
scheme: HTTP
initialDelaySeconds: 15
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /var/jenkins_home
name: jenkins-home
- mountPath: /usr/share/jenkins/ref/plugins
name: jenkins-plugins
dnsPolicy: ClusterFirst
initContainers:
- args:
- jenkins/jenkins
- 2.190.1-alpine
- "false"
command:
- /init/init.sh
image: jenkins/jenkins:2.190.1-alpine
imagePullPolicy: IfNotPresent
name: jenkins-init
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /init
name: init
- mountPath: /var/jenkins_home
name: jenkins-home
- mountPath: /usr/share/jenkins/ref/plugins
name: jenkins-plugins
restartPolicy: Always
schedulerName: default-scheduler
securityContext:
fsGroup: 1000
runAsNonRoot: true
runAsUser: 1000
serviceAccount: default
serviceAccountName: default
terminationGracePeriodSeconds: 30
volumes:
- name: jenkins-home
persistentVolumeClaim:
claimName: jenkins-pvc
- emptyDir: {}
name: jenkins-plugins
- configMap:
defaultMode: 365
name: jenkins-init
name: init
status:
availableReplicas: 1
conditions:
- lastTransitionTime: "2020-02-06T22:39:51Z"
lastUpdateTime: "2020-02-06T22:40:32Z"
message: ReplicaSet "jenkins-866f548f55" has successfully progressed.
reason: NewReplicaSetAvailable
status: "True"
type: Progressing
- lastTransitionTime: "2020-06-05T12:13:56Z"
lastUpdateTime: "2020-06-05T12:13:56Z"
message: Deployment has minimum availability.
reason: MinimumReplicasAvailable
status: "True"
type: Available
observedGeneration: 1
readyReplicas: 1
replicas: 1
updatedReplicas: 1
Is it safe to edit deployment file directly?
I noticed that the ERROR code listed version chart 1.7.0 even though mine is 1.5.1. Maybe I should state in the upgrade command that I stay with the old version?
Related
I have redis DB setup running on my minikube cluster. I have shutdown my minikube and started after 3 days and I can see my redis pod is failing to come up with below error from pod log
Bad file format reading the append only file: make a backup of your AOF file, then use ./redis-check-aof --fix <filename>.
Below is my Stateful Set yaml file for redis master deployed via a helm chart
apiVersion: apps/v1
kind: StatefulSet
metadata:
annotations:
meta.helm.sh/release-name: test-redis
meta.helm.sh/release-namespace: test
generation: 1
labels:
app.kubernetes.io/component: master
app.kubernetes.io/instance: test-redis
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: redis
helm.sh/chart: redis-14.8.11
name: test-redis-master
namespace: test
resourceVersion: "191902"
uid: 3a4e541f-154f-4c54-a379-63974d90089e
spec:
podManagementPolicy: OrderedReady
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
app.kubernetes.io/component: master
app.kubernetes.io/instance: test-redis
app.kubernetes.io/name: redis
serviceName: test-redis-headless
template:
metadata:
annotations:
checksum/configmap: dd1f90e0231e5f9ebd1f3f687d534d9ec53df571cba9c23274b749c01e5bc2bb
checksum/health: xxxxx
creationTimestamp: null
labels:
app.kubernetes.io/component: master
app.kubernetes.io/instance: test-redis
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: redis
helm.sh/chart: redis-14.8.11
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
labelSelector:
matchLabels:
app.kubernetes.io/component: master
app.kubernetes.io/instance: test-redis
app.kubernetes.io/name: redis
namespaces:
- tyk
topologyKey: kubernetes.io/hostname
weight: 1
containers:
- args:
- -c
- /opt/bitnami/scripts/start-scripts/start-master.sh
command:
- /bin/bash
env:
- name: BITNAMI_DEBUG
value: "false"
- name: REDIS_REPLICATION_MODE
value: master
- name: ALLOW_EMPTY_PASSWORD
value: "no"
- name: REDIS_PASSWORD
valueFrom:
secretKeyRef:
key: redis-password
name: test-redis
- name: REDIS_TLS_ENABLED
value: "no"
- name: REDIS_PORT
value: "6379"
image: docker.io/bitnami/redis:6.2.5-debian-10-r11
imagePullPolicy: IfNotPresent
livenessProbe:
exec:
command:
- sh
- -c
- /health/ping_liveness_local.sh 5
failureThreshold: 5
initialDelaySeconds: 20
periodSeconds: 5
successThreshold: 1
timeoutSeconds: 6
name: redis
ports:
- containerPort: 6379
name: redis
protocol: TCP
readinessProbe:
exec:
command:
- sh
- -c
- /health/ping_readiness_local.sh 1
failureThreshold: 5
initialDelaySeconds: 20
periodSeconds: 5
successThreshold: 1
timeoutSeconds: 2
resources: {}
securityContext:
runAsUser: 1001
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /opt/bitnami/scripts/start-scripts
name: start-scripts
- mountPath: /health
name: health
- mountPath: /data
name: redis-data
- mountPath: /opt/bitnami/redis/mounted-etc
name: config
- mountPath: /opt/bitnami/redis/etc/
name: redis-tmp-conf
- mountPath: /tmp
name: tmp
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext:
fsGroup: 1001
serviceAccount: test-redis
serviceAccountName: test-redis
terminationGracePeriodSeconds: 30
volumes:
- configMap:
defaultMode: 493
name: test-redis-scripts
name: start-scripts
- configMap:
defaultMode: 493
name: test-redis-health
name: health
- configMap:
defaultMode: 420
name: test-redis-configuration
name: config
- emptyDir: {}
name: redis-tmp-conf
- emptyDir: {}
name: tmp
updateStrategy:
rollingUpdate:
partition: 0
type: RollingUpdate
volumeClaimTemplates:
- apiVersion: v1
kind: PersistentVolumeClaim
metadata:
creationTimestamp: null
labels:
app.kubernetes.io/component: master
app.kubernetes.io/instance: test-redis
app.kubernetes.io/name: redis
name: redis-data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 8Gi
volumeMode: Filesystem
status:
phase: Pending
Please let me know your suggestions on how can I fix this.
I am not an Redis expert but from what I can see:
kubectl describe pod red3-redis-master-0
...
Bad file format reading the append only file: make a backup of your AOF file, then use ./redis-check-aof --fix <filename>
...
Means that your appendonly.aof file was corrupted with invalid byte sequences in the middle.
How we can proceed if redis-master is not working?:
Verify pvc attached to the redis-master-pod:
kubectl get pvc
NAME STATUS VOLUME
redis-data-red3-redis-master-0 Bound pvc-cf59a0b2-a3ee-4f7f-9f07-8f4922518359
Create new redis-client pod wit the same pvc redis-data-red3-redis-master-0:
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
name: redis-client
spec:
volumes:
- name: data
persistentVolumeClaim:
claimName: redis-data-red3-redis-master-0
containers:
- name: redis
image: docker.io/bitnami/redis:6.2.3-debian-10-r0
command: ["/bin/bash"]
args: ["-c", "sleep infinity"]
volumeMounts:
- mountPath: "/tmp"
name: data
EOF
Backup your files:
kubectl cp redis-client:/tmp .
Repair appendonly.aof file:
kubectl exec -it redis-client -- /bin/bash
cd /tmp
# make copy of appendonly.aof file:
cp appendonly.aof appendonly.aofbackup
# verify appendonly.aof file:
redis-check-aof appendonly.aof
...
0x 38: Expected prefix '*', got: '"'
AOF analyzed: size=62, ok_up_to=56, ok_up_to_line=13, diff=6
AOF is not valid. Use the --fix option to try fixing it.
...
# repair appendonly.aof file:
redis-check-aof --fix appendonly.aof
# compare files using diff:
diff appendonly.aof appendonly.aofbackup
Note:
As per docs:
The best thing to do is to run the redis-check-aof utility, initially without the --fix option, then understand the problem, jump at the given offset in the file, and see if it is possible to manually repair the file: the AOF uses the same format of the Redis protocol and is quite simple to fix manually. Otherwise it is possible to let the utility fix the file for us, but in that case all the AOF portion from the invalid part to the end of the file may be discarded, leading to a massive amount of data loss if the corruption happened to be in the initial part of the file.
In addition as described in the comments by #Miffa Young you can verify where your data is stored using k8s.io/minikube-hostpath provisioner:
kubectl get pv
...
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM
pvc-cf59a0b2-a3ee-4f7f-9f07-8f4922518359 8Gi RWO Delete Bound default/redis-data-red3-redis-master-0
...
kubectl describe pv pvc-cf59a0b2-a3ee-4f7f-9f07-8f4922518359
...
Source:
Type: HostPath (bare host directory volume)
Path: /tmp/hostpath-provisioner/default/redis-data-red3-redis-master-0
...
Your redis instance is failing down because your appendonly.aof is malformed and stored permanently under this location.
You can ssh into your vm:
minikube -p redis ssh
cd /tmp/hostpath-provisioner/default/redis-data-red3-redis-master-0
# from there you can backup/repair/remove your files:
Another solution is to install this chart using new name in this case new set of pv,pvc for redis StatefulSets will be created.
I think your redis is not quit Gracefully , so the AOF file is in a bad format What is AOF
you should repair aof file using a initcontainer by command (./redis-check-aof --fix .)
apiVersion: apps/v1
kind: StatefulSet
metadata:
annotations:
meta.helm.sh/release-name: test-redis
meta.helm.sh/release-namespace: test
generation: 1
labels:
app.kubernetes.io/component: master
app.kubernetes.io/instance: test-redis
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: redis
helm.sh/chart: redis-14.8.11
name: test-redis-master
namespace: test
resourceVersion: "191902"
uid: 3a4e541f-154f-4c54-a379-63974d90089e
spec:
podManagementPolicy: OrderedReady
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
app.kubernetes.io/component: master
app.kubernetes.io/instance: test-redis
app.kubernetes.io/name: redis
serviceName: test-redis-headless
template:
metadata:
annotations:
checksum/configmap: dd1f90e0231e5f9ebd1f3f687d534d9ec53df571cba9c23274b749c01e5bc2bb
checksum/health: xxxxx
creationTimestamp: null
labels:
app.kubernetes.io/component: master
app.kubernetes.io/instance: test-redis
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: redis
helm.sh/chart: redis-14.8.11
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
labelSelector:
matchLabels:
app.kubernetes.io/component: master
app.kubernetes.io/instance: test-redis
app.kubernetes.io/name: redis
namespaces:
- tyk
topologyKey: kubernetes.io/hostname
weight: 1
initContainers:
- name: repair-redis
image: docker.io/bitnami/redis:6.2.5-debian-10-r11
command: ['sh', '-c', "redis-check-aof --fix /data/appendonly.aof"]
containers:
- args:
- -c
- /opt/bitnami/scripts/start-scripts/start-master.sh
command:
- /bin/bash
env:
- name: BITNAMI_DEBUG
value: "false"
- name: REDIS_REPLICATION_MODE
value: master
- name: ALLOW_EMPTY_PASSWORD
value: "no"
- name: REDIS_PASSWORD
valueFrom:
secretKeyRef:
key: redis-password
name: test-redis
- name: REDIS_TLS_ENABLED
value: "no"
- name: REDIS_PORT
value: "6379"
image: docker.io/bitnami/redis:6.2.5-debian-10-r11
imagePullPolicy: IfNotPresent
livenessProbe:
exec:
command:
- sh
- -c
- /health/ping_liveness_local.sh 5
failureThreshold: 5
initialDelaySeconds: 20
periodSeconds: 5
successThreshold: 1
timeoutSeconds: 6
name: redis
ports:
- containerPort: 6379
name: redis
protocol: TCP
readinessProbe:
exec:
command:
- sh
- -c
- /health/ping_readiness_local.sh 1
failureThreshold: 5
initialDelaySeconds: 20
periodSeconds: 5
successThreshold: 1
timeoutSeconds: 2
resources: {}
securityContext:
runAsUser: 1001
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /opt/bitnami/scripts/start-scripts
name: start-scripts
- mountPath: /health
name: health
- mountPath: /data
name: redis-data
- mountPath: /opt/bitnami/redis/mounted-etc
name: config
- mountPath: /opt/bitnami/redis/etc/
name: redis-tmp-conf
- mountPath: /tmp
name: tmp
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext:
fsGroup: 1001
serviceAccount: test-redis
serviceAccountName: test-redis
terminationGracePeriodSeconds: 30
volumes:
- configMap:
defaultMode: 493
name: test-redis-scripts
name: start-scripts
- configMap:
defaultMode: 493
name: test-redis-health
name: health
- configMap:
defaultMode: 420
name: test-redis-configuration
name: config
- emptyDir: {}
name: redis-tmp-conf
- emptyDir: {}
name: tmp
updateStrategy:
rollingUpdate:
partition: 0
type: RollingUpdate
volumeClaimTemplates:
- apiVersion: v1
kind: PersistentVolumeClaim
metadata:
creationTimestamp: null
labels:
app.kubernetes.io/component: master
app.kubernetes.io/instance: test-redis
app.kubernetes.io/name: redis
name: redis-data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 8Gi
volumeMode: Filesystem
I want my prometheus server to scrape metrics from a pod.
I followed these steps:
Created a pod using deployment - kubectl apply -f sample-app.deploy.yaml
Exposed the same using kubectl apply -f sample-app.service.yaml
Deployed Prometheus server using helm upgrade -i prometheus prometheus-community/prometheus -f prometheus-values.yaml
created a serviceMonitor using kubectl apply -f service-monitor.yaml to add a target for prometheus.
All pods are running, but when I open prometheus dashboard, I don't see sample-app service as prometheus target, under status>targets in dashboard UI.
I've verified following:
I can see sample-app when I execute kubectl get servicemonitors
I can see sample-app exposes metrics in prometheus format under at /metrics
At this point I debugged further, entered into the prometheus pod using
kubectl exec -it pod/prometheus-server-65b759cb95-dxmkm -c prometheus-server sh
, and saw that proemetheus configuration (/etc/config/prometheus.yml) didn't have sample-app as one of the jobs so I edited the configmap using
kubectl edit cm prometheus-server -o yaml
Added
- job_name: sample-app
static_configs:
- targets:
- sample-app:8080
Assuming all other fields such as scraping interval, scrape_timeout stays default.
I can see the same has been reflected in /etc/config/prometheus.yml, but still prometheus dashboard doesn't show sample-app as targets under status>targets.
following are yamls for prometheus-server and service monitor.
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
autopilot.gke.io/resource-adjustment: '{"input":{"containers":[{"name":"prometheus-server-configmap-reload"},{"name":"prometheus-server"}]},"output":{"containers":[{"limits":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"2Gi"},"requests":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"2Gi"},"name":"prometheus-server-configmap-reload"},{"limits":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"2Gi"},"requests":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"2Gi"},"name":"prometheus-server"}]},"modified":true}'
deployment.kubernetes.io/revision: "1"
meta.helm.sh/release-name: prometheus
meta.helm.sh/release-namespace: prom
creationTimestamp: "2021-06-24T10:42:31Z"
generation: 1
labels:
app: prometheus
app.kubernetes.io/managed-by: Helm
chart: prometheus-14.2.1
component: server
heritage: Helm
release: prometheus
name: prometheus-server
namespace: prom
resourceVersion: "6983855"
selfLink: /apis/apps/v1/namespaces/prom/deployments/prometheus-server
uid: <some-uid>
spec:
progressDeadlineSeconds: 600
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
app: prometheus
component: server
release: prometheus
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
creationTimestamp: null
labels:
app: prometheus
chart: prometheus-14.2.1
component: server
heritage: Helm
release: prometheus
spec:
containers:
- args:
- --volume-dir=/etc/config
- --webhook-url=http://127.0.0.1:9090/-/reload
image: jimmidyson/configmap-reload:v0.5.0
imagePullPolicy: IfNotPresent
name: prometheus-server-configmap-reload
resources:
limits:
cpu: 500m
ephemeral-storage: 1Gi
memory: 2Gi
requests:
cpu: 500m
ephemeral-storage: 1Gi
memory: 2Gi
securityContext:
capabilities:
drop:
- NET_RAW
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /etc/config
name: config-volume
readOnly: true
- args:
- --storage.tsdb.retention.time=15d
- --config.file=/etc/config/prometheus.yml
- --storage.tsdb.path=/data
- --web.console.libraries=/etc/prometheus/console_libraries
- --web.console.templates=/etc/prometheus/consoles
- --web.enable-lifecycle
image: quay.io/prometheus/prometheus:v2.26.0
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 3
httpGet:
path: /-/healthy
port: 9090
scheme: HTTP
initialDelaySeconds: 30
periodSeconds: 15
successThreshold: 1
timeoutSeconds: 10
name: prometheus-server
ports:
- containerPort: 9090
protocol: TCP
readinessProbe:
failureThreshold: 3
httpGet:
path: /-/ready
port: 9090
scheme: HTTP
initialDelaySeconds: 30
periodSeconds: 5
successThreshold: 1
timeoutSeconds: 4
resources:
limits:
cpu: 500m
ephemeral-storage: 1Gi
memory: 2Gi
requests:
cpu: 500m
ephemeral-storage: 1Gi
memory: 2Gi
securityContext:
capabilities:
drop:
- NET_RAW
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /etc/config
name: config-volume
- mountPath: /data
name: storage-volume
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext:
fsGroup: 65534
runAsGroup: 65534
runAsNonRoot: true
runAsUser: 65534
seccompProfile:
type: RuntimeDefault
serviceAccount: prometheus-server
serviceAccountName: prometheus-server
terminationGracePeriodSeconds: 300
volumes:
- configMap:
defaultMode: 420
name: prometheus-server
name: config-volume
- name: storage-volume
persistentVolumeClaim:
claimName: prometheus-server
status:
availableReplicas: 1
conditions:
- lastTransitionTime: "2021-06-24T10:43:25Z"
lastUpdateTime: "2021-06-24T10:43:25Z"
message: Deployment has minimum availability.
reason: MinimumReplicasAvailable
status: "True"
type: Available
- lastTransitionTime: "2021-06-24T10:42:31Z"
lastUpdateTime: "2021-06-24T10:43:25Z"
message: ReplicaSet "prometheus-server-65b759cb95" has successfully progressed.
reason: NewReplicaSetAvailable
status: "True"
type: Progressing
observedGeneration: 1
readyReplicas: 1
replicas: 1
updatedReplicas: 1
yaml for service Monitor
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"monitoring.coreos.com/v1","kind":"ServiceMonitor","metadata":{"annotations":{},"creationTimestamp":"2021-06-24T07:55:58Z","generation":1,"labels":{"app":"sample-app","release":"prometheus"},"name":"sample-app","namespace":"prom","resourceVersion":"6884573","selfLink":"/apis/monitoring.coreos.com/v1/namespaces/prom/servicemonitors/sample-app","uid":"34644b62-eb4f-4ab1-b9df-b22811e40b4c"},"spec":{"endpoints":[{"port":"http"}],"selector":{"matchLabels":{"app":"sample-app","release":"prometheus"}}}}
creationTimestamp: "2021-06-24T07:55:58Z"
generation: 2
labels:
app: sample-app
release: prometheus
name: sample-app
namespace: prom
resourceVersion: "6904642"
selfLink: /apis/monitoring.coreos.com/v1/namespaces/prom/servicemonitors/sample-app
uid: <some-uid>
spec:
endpoints:
- port: http
selector:
matchLabels:
app: sample-app
release: prometheus
You need to use the prometheus-community/kube-prometheus-stack chart, which includes the Prometheus operator, in order to have Prometheus' configuration update automatically based on ServiceMonitor resources.
The prometheus-community/prometheus chart you used does not include the Prometheus operator that watches for ServiceMonitor resources in the Kubernetes API and updates the Prometheus server's ConfigMap accordingly.
It seems that you have the necessary CustomResourceDefinitions (CRDs) installed in your cluster, otherwise you would not have been able to create a ServiceMonitor resource. These are not included in the prometheus-community/prometheus chart so perhaps they were added to your cluster previously.
I am using helm 3 to install nexus in kubernetes v1.18:
helm install stable/sonatype-nexus --name=nexus
and then expose nexus by using traefik 2.x to outside by using domian: nexus.dolphin.com. But when I using domain to access nexus servcie it give me this tips:
Invalid host. To browse Nexus, click here/. To use the Docker registry, point your client
and I have read this question, but It seem not suite for my situation. And this is my nexus yaml config now:
kind: Deployment
apiVersion: apps/v1
metadata:
name: nexus-sonatype-nexus
namespace: infrastructure
selfLink: /apis/apps/v1/namespaces/infrastructure/deployments/nexus-sonatype-nexus
uid: 023de15b-19eb-442d-8375-11532825919d
resourceVersion: '1710210'
generation: 3
creationTimestamp: '2020-08-16T07:17:07Z'
labels:
app: sonatype-nexus
app.kubernetes.io/managed-by: Helm
chart: sonatype-nexus-1.23.1
fullname: nexus-sonatype-nexus
heritage: Helm
release: nexus
annotations:
deployment.kubernetes.io/revision: '1'
meta.helm.sh/release-name: nexus
meta.helm.sh/release-namespace: infrastructure
managedFields:
- manager: Go-http-client
operation: Update
apiVersion: apps/v1
time: '2020-08-16T07:17:07Z'
fieldsType: FieldsV1
- manager: kube-controller-manager
operation: Update
apiVersion: apps/v1
time: '2020-08-18T16:26:34Z'
fieldsType: FieldsV1
spec:
replicas: 1
selector:
matchLabels:
app: sonatype-nexus
release: nexus
template:
metadata:
creationTimestamp: null
labels:
app: sonatype-nexus
release: nexus
spec:
volumes:
- name: nexus-sonatype-nexus-data
persistentVolumeClaim:
claimName: nexus-sonatype-nexus-data
- name: nexus-sonatype-nexus-backup
emptyDir: {}
containers:
- name: nexus
image: 'sonatype/nexus3:3.20.1'
ports:
- name: nexus-docker-g
containerPort: 5003
protocol: TCP
- name: nexus-http
containerPort: 8081
protocol: TCP
env:
- name: install4jAddVmParams
value: >-
-Xms1200M -Xmx1200M -XX:MaxDirectMemorySize=2G
-XX:+UnlockExperimentalVMOptions
-XX:+UseCGroupMemoryLimitForHeap
- name: NEXUS_SECURITY_RANDOMPASSWORD
value: 'false'
resources: {}
volumeMounts:
- name: nexus-sonatype-nexus-data
mountPath: /nexus-data
- name: nexus-sonatype-nexus-backup
mountPath: /nexus-data/backup
livenessProbe:
httpGet:
path: /
port: 8081
scheme: HTTP
initialDelaySeconds: 30
timeoutSeconds: 1
periodSeconds: 30
successThreshold: 1
failureThreshold: 6
readinessProbe:
httpGet:
path: /
port: 8081
scheme: HTTP
initialDelaySeconds: 30
timeoutSeconds: 1
periodSeconds: 30
successThreshold: 1
failureThreshold: 6
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
imagePullPolicy: IfNotPresent
- name: nexus-proxy
image: 'quay.io/travelaudience/docker-nexus-proxy:2.5.0'
ports:
- name: nexus-proxy
containerPort: 8080
protocol: TCP
env:
- name: ALLOWED_USER_AGENTS_ON_ROOT_REGEX
value: GoogleHC
- name: CLOUD_IAM_AUTH_ENABLED
value: 'false'
- name: BIND_PORT
value: '8080'
- name: ENFORCE_HTTPS
value: 'false'
- name: NEXUS_DOCKER_HOST
- name: NEXUS_HTTP_HOST
- name: UPSTREAM_DOCKER_PORT
value: '5003'
- name: UPSTREAM_HTTP_PORT
value: '8081'
- name: UPSTREAM_HOST
value: localhost
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
imagePullPolicy: IfNotPresent
restartPolicy: Always
terminationGracePeriodSeconds: 30
dnsPolicy: ClusterFirst
serviceAccountName: nexus-sonatype-nexus
serviceAccount: nexus-sonatype-nexus
securityContext:
fsGroup: 2000
schedulerName: default-scheduler
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 25%
maxSurge: 25%
revisionHistoryLimit: 10
progressDeadlineSeconds: 600
status:
observedGeneration: 3
replicas: 1
updatedReplicas: 1
readyReplicas: 1
availableReplicas: 1
conditions:
- type: Progressing
status: 'True'
lastUpdateTime: '2020-08-18T16:23:54Z'
lastTransitionTime: '2020-08-18T16:23:54Z'
reason: NewReplicaSetAvailable
message: >-
ReplicaSet "nexus-sonatype-nexus-79fd4488d5" has successfully
progressed.
- type: Available
status: 'True'
lastUpdateTime: '2020-08-18T16:26:34Z'
lastTransitionTime: '2020-08-18T16:26:34Z'
reason: MinimumReplicasAvailable
message: Deployment has minimum availability.
why the domian could not access nexus by default? and what should I do to access nexus by domain?
From the documentation you should set a property of the helm chart: nexusProxy.env.nexusHttpHost to nexus.dolphin.com
The docker image used here has a proxy that allows you to access the Nexus HTTP and Nexus Docker services by different domains, if you don't specify either then you get the behaviour you're seeing.
I am using traefik 2.2.1 as my cluster's entrypoint using deployment type, this is my deployment config:
kind: Deployment
apiVersion: apps/v1
metadata:
name: traefik
namespace: kube-system
selfLink: /apis/apps/v1/namespaces/kube-system/deployments/traefik
uid: ddee327d-8570-44be-ab8d-06cb440187f4
resourceVersion: '335024'
generation: 12
creationTimestamp: '2020-06-04T07:37:20Z'
labels:
app.kubernetes.io/instance: traefik
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: traefik
helm.sh/chart: traefik-8.2.1
annotations:
deployment.kubernetes.io/revision: '7'
meta.helm.sh/release-name: traefik
meta.helm.sh/release-namespace: kube-system
spec:
replicas: 4
selector:
matchLabels:
app.kubernetes.io/instance: traefik
app.kubernetes.io/name: traefik
template:
metadata:
creationTimestamp: null
labels:
app.kubernetes.io/instance: traefik
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: traefik
helm.sh/chart: traefik-8.2.1
spec:
volumes:
- name: data
emptyDir: {}
containers:
- name: traefik
image: 'traefik:2.2.1'
args:
- '--global.checknewversion'
- '--global.sendanonymoususage'
- '--entryPoints.traefik.address=:9000'
- '--entryPoints.web.address=:80'
- '--entryPoints.websecure.address=:443'
- '--api.dashboard=true'
- '--ping=true'
- '--providers.kubernetescrd'
- '--providers.kubernetesingress'
ports:
- name: traefik
containerPort: 9000
protocol: TCP
- name: web
containerPort: 8000
protocol: TCP
- name: websecure
containerPort: 8443
protocol: TCP
resources: {}
volumeMounts:
- name: data
mountPath: /data
livenessProbe:
httpGet:
path: /ping
port: 9000
scheme: HTTP
initialDelaySeconds: 10
timeoutSeconds: 2
periodSeconds: 10
successThreshold: 1
failureThreshold: 3
readinessProbe:
httpGet:
path: /ping
port: 9000
scheme: HTTP
initialDelaySeconds: 10
timeoutSeconds: 2
periodSeconds: 10
successThreshold: 1
failureThreshold: 1
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
imagePullPolicy: IfNotPresent
securityContext:
capabilities:
drop:
- ALL
runAsUser: 65532
runAsGroup: 65532
runAsNonRoot: true
readOnlyRootFilesystem: true
restartPolicy: Always
terminationGracePeriodSeconds: 60
dnsPolicy: ClusterFirst
serviceAccountName: traefik
serviceAccount: traefik
securityContext:
fsGroup: 65532
schedulerName: default-scheduler
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
maxSurge: 1
revisionHistoryLimit: 10
progressDeadlineSeconds: 600
status:
observedGeneration: 12
replicas: 5
updatedReplicas: 2
readyReplicas: 3
availableReplicas: 3
unavailableReplicas: 2
conditions:
- type: Available
status: 'True'
lastUpdateTime: '2020-06-04T08:41:03Z'
lastTransitionTime: '2020-06-04T08:41:03Z'
reason: MinimumReplicasAvailable
message: Deployment has minimum availability.
- type: Progressing
status: 'True'
lastUpdateTime: '2020-06-04T10:57:35Z'
lastTransitionTime: '2020-06-04T10:48:40Z'
reason: ReplicaSetUpdated
message: ReplicaSet "traefik-dd74b59b" is progressing.
my question is: is it possible to make the treafik listening host's 80 and 443 port? If possible, how to make it? or should I change my deployment type to daemon set? if not, I have to deployment a nginx in each node to forward traffic.
Add hostNetwork: true in the spec. This makes the pod use host's network namespace.
...
spec:
hostNetwork: true
containers:
- name: traefik
...
I am trying to patch a deployment, using below command.I trying to set environment variable in the pod by passing existing value from the metadata label definition of the my yaml.
Patching Command
kubectl patch deployment deploytest3 --type=json -p='[{"op": "add", "path": "/spec/template/spec/containers/1/env/2","value": {"name": "CUSTOMERID","valueFrom": {"fieldRef": {"fieldPath": "metadata.labels['customerid']"}}}}]'
Below is my Yaml for reference.(This is for reference not the original)
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: "10"
creationTimestamp: 2019-01-22T11:24:17Z
generation: 10
labels:
app: my-cloud
name: deploytest3
namespace: default
resourceVersion: "60076984"
selfLink: /apis/extensions/v1beta1/namespaces/default/deployments/deploytest3
uid: 41950c24-1e38-11e9-ba0c-42010a8000d6
spec:
progressDeadlineSeconds: 600
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
app: my-cloud
customerid: "gx0388d"
customername: deploytest3
environment: dev
tier: backend
version: 11.0-latest
strategy:
rollingUpdate:
maxSurge: 0
maxUnavailable: 1
type: RollingUpdate
template:
metadata:
creationTimestamp: null
labels:
app: my-cloud
customerid: "gx0388d"
customername: deploytest3
environment: dev
tier: backend
version: 11.0-latest
spec:
containers:
- args:
- -c
- /bin/aSoftMS --simple-media-server;/bin/core_find.sh
command:
- /bin/bash
image: us.gcr.io/data/myms:11.0-latest
imagePullPolicy: IfNotPresent
name: ms
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /opt/latest/system_rw/
name: deploytest3
- args:
- -c
command:
- /bin/bash
image: us.gcr.io/latest/core:11.0-latest
imagePullPolicy: IfNotPresent
lifecycle:
preStop:
exec:
command:
- /bin/CoreDumpGen
livenessProbe:
failureThreshold: 3
httpGet:
httpHeaders:
- name: User-Agent
value: DeploymentAdmin
path: /livenessstatus
port: 8443
scheme: HTTPS
initialDelaySeconds: 60
periodSeconds: 15
successThreshold: 1
timeoutSeconds: 15
env:
- name: latest_CUSTOMER
value: abc-test
- name: latest
value: abc
image: us.gcr.io/latest/latest/core2:11.0-latest
imagePullPolicy: IfNotPresent
name: core2
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
dnsPolicy: ClusterFirst
hostname: deploytest3
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 60
volumes:
- name: bucket-auth
secret:
defaultMode: 256
secretName: bucket-auth
status:
availableReplicas: 1
conditions:
- lastTransitionTime: 2019-01-22T11:24:17Z
lastUpdateTime: 2019-01-22T11:24:17Z
message: Deployment has minimum availability.
reason: MinimumReplicasAvailable
status: "True"
type: Available
- lastTransitionTime: 2019-01-22T11:24:17Z
lastUpdateTime: 2019-05-31T13:49:03Z
message: ReplicaSet "deploytest3-7ddbc9b9c4" has successfully progressed.
reason: NewReplicaSetAvailable
status: "True"
type: Progressing
observedGeneration: 10
readyReplicas: 1
replicas: 1
updatedReplicas: 1
Below is error, i am getting while executing the above shared command
The Deployment "deploytest3" is invalid: spec.template.spec.containers[2].env[2].valueFrom.fieldRef.fieldPath: Invalid value: "metadata.labels[customerid]": error converting fieldPath:
field label not supported: metadata.labels[customerid]
I am manually add the environment variable in the deployment it works.
I think i am not setting correct with json format, Can someone help me.
"fieldPath": "metadata.labels['customerid']" - is incorrect
Apparently, it should be "fieldPath": "spec.template.metadata.labels['customerid']"
This is not exactly kubectl patch, but this will also set env variable from deployments label definition
kubectl set env deployment/deploytest3 CUSTOMERID=$(kubectl get deployments deploytest3 -o jsonpath='{$.spec.template.metadata.labels.customerid}')