Kuberntes/Prometheus - Unable to Annotate in service file - annotations

Question

My Kubernetes version is :
# kubectl --version
Kubernetes v1.4.0
I am planning to use Prometheus to monitor my Kube cluster. For this, I need to annotate the metrics URL.
My current metrics URL is like :
http://172.16.33.7:8080/metrics
But I want it to be like :
http://172.16.33.7:8080/websocket/metrics
First I tried to do this manually ::
kubectl annotate pods websocket-backend-controller-db83999c5b534b277b82badf6c152cb9m1 prometheus.io/path=/websocket/metrics
kubectl annotate pods websocket-backend-controller-db83999c5b534b277b82badf6c152cb9m1 prometheus.io/scrape='true'
kubectl annotate pods websocket-backend-controller-db83999c5b534b277b82badf6c152cb9m1 prometheus.io/port='8080'
All these commands work perfectly fine and I am able to see the annotations.
{
"metadata": {
"name": "websocket-backend-controller-v1krf",
"generateName": "websocket-backend-controller-",
"namespace": "default",
"selfLink": "/api/v1/namespaces/default/pods/websocket-backend-controller-v1krf",
"uid": "e323994b-4081-11e7-8bd0-0050569b6f44",
"resourceVersion": "27534379",
"creationTimestamp": "2017-05-24T13:07:06Z",
"labels": {
"name": "websocket-backend"
},
"annotations": {
"kubernetes.io/created-by": "{\"kind\":\"SerializedReference\",\"apiVersion\":\"v1\",\"reference\":{\"kind\":\"ReplicationController\",\"namespace\":\"default\",\"name\":\"websocket-backend-controller\",\"uid\":\"e321f1a8-4081-11e7-8bd0-0050569b6f44\",\"apiVersion\":\"v1\",\"resourceVersion\":\"27531840\"}}\n",
"prometheus.io/path": "/websocket/metrics",
"prometheus.io/port": "8080",
"prometheus.io/scrape": "true"
}
But since I want this configuration to remain permanent, I am setting the following annotations in my services files.
# cat websocket-service.yaml
apiVersion: v1
kind: Service
metadata:
name: websocket-service
labels:
baseApi: websocket
annotations:
prometheus.io/scrape: 'true'
prometheus.io/path: /websocket/metrics
prometheus.io/port: '8080'
spec:
selector:
name: websocket-backend
ports:
- port: 8080
targetPort: 8080
nodePort: 30800
protocol: TCP
type: NodePort
clusterIP: 10.100.10.45
I restarted my websocket service and the corresponding pods but these configs do not seem to be taking effect.
kubectl create -f websocket-service.yaml
kubectl create -f ../controllers/websocket-replication-controller.yaml
The result does not show the annotations configured.
{
"metadata": {
"name": "websocket-backend-controller-v1krf",
"generateName": "websocket-backend-controller-",
"namespace": "default",
"selfLink": "/api/v1/namespaces/default/pods/websocket-backend-controller-v1krf",
"uid": "e323994b-4081-11e7-8bd0-0050569b6f44",
"resourceVersion": "27531879",
"creationTimestamp": "2017-05-24T13:07:06Z",
"labels": {
"name": "websocket-backend"
},
"annotations": {
"kubernetes.io/created-by": "{\"kind\":\"SerializedReference\",\"apiVersion\":\"v1\",\"reference\":{\"kind\":\"ReplicationController\",\"namespace\":\"default\",\"name\":\"websocket-backend-controller\",\"uid\":\"e321f1a8-4081-11e7-8bd0-0050569b6f44\",\"apiVersion\":\"v1\",\"resourceVersion\":\"27531840\"}}\n"
}
All I am doing is rather than using a command line, I am setting the configs using services config but it does not seem to be working.

Answer 1

If you annotate the service, it doesn't take any effect on the possibly matched pods. Your pods are managed either by a ReplicationController, or over a ReplicaSet / Deployment. In that case, annotate these resources to make the annotations reach the pods. In example of deployments, you must use the template section, like:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
# Unique key of the Deployment instance
name: deployment-example
spec:
# 3 Pods should exist at all times.
replicas: 3
# Keep record of 2 revisions for rollback
revisionHistoryLimit: 2
template:
metadata:
annotations:
prometheus.io/scrape: 'true'
prometheus.io/path: /websocket/metrics
prometheus.io/port: '8080'