Azure Kubernetes - prometheus is deployed as a part of ISTIO not showing the deployments? - kubernetes

Question

I have used the following configuration to setup the Istio
cat << EOF | kubectl apply -f -
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
namespace: istio-system
name: istio-control-plane
spec:
# Use the default profile as the base
# More details at: https://istio.io/docs/setup/additional-setup/config-profiles/
profile: default
# Enable the addons that we will want to use
addonComponents:
grafana:
enabled: true
prometheus:
enabled: true
tracing:
enabled: true
kiali:
enabled: true
values:
global:
# Ensure that the Istio pods are only scheduled to run on Linux nodes
defaultNodeSelector:
beta.kubernetes.io/os: linux
kiali:
dashboard:
auth:
strategy: anonymous
components:
egressGateways:
- name: istio-egressgateway
enabled: true
EOF
and exposed the prometheus service as mentioned below
kubectl expose service prometheus --type=LoadBalancer --name=prometheus-svc --namespace istio-system
kubectl get svc prometheus-svc -n istio-system -o json
export PROMETHEUS_URL=$(kubectl get svc prometheus-svc -n istio-system -o jsonpath="{.status.loadBalancer.ingress[0]['hostname','ip']}"):$(kubectl get svc prometheus-svc -n istio-system -o 'jsonpath={.spec.ports[0].port}')
echo http://${PROMETHEUS_URL}
curl http://${PROMETHEUS_URL}
I have deployed an application however couldn't see the below deployments in prometheus

Answer 1

The standard prometheus installation by istio does not configure your pods to send metrics to prometheus. It just collects data from the istio resouces.
To add your pods to being scraped add the following annotations in the deployment.yml of your application:
apiVersion: apps/v1
kind: Deployment
[...]
spec:
template:
metadata:
annotations:
prometheus.io/scrape: true # determines if a pod should be scraped. Set to true to enable scraping.
prometheus.io/path: /metrics # determines the path to scrape metrics at. Defaults to /metrics.
prometheus.io/port: 80 # determines the port to scrape metrics at. Defaults to 80.
[...]
By the way: The prometheus instance installed with istioctl should not be used for production. From docs:
[...] pass --set values.prometheus.enabled=true during installation. This built-in deployment of Prometheus is intended for new users to help them quickly getting started. However, it does not offer advanced customization, like persistence or authentication and as such should not be considered production ready.
You should setup your own prometheus and configure istio to report to it. See:
Reference: https://istio.io/latest/docs/ops/integrations/prometheus/#option-1-metrics-merging
The following yaml provided by istio can be used as reference for setup of prometheus:
https://raw.githubusercontent.com/istio/istio/release-1.7/samples/addons/prometheus.yaml
Furthermore, if I remember correctly, installation of addons like kiali, prometheus, ... with istioctl will be removed with istio 1.8 (release date december 2020). So you might want to setup your own instances with helm anyway.