I've been learning about how to deploy ambassador on kubernetes on minikube by this tutorial, and that works as I can see the page for successfully installed ambassador. The main problem is, when I try to change the image of the UI such that it should open other app in the link, it opens the same successfull page of ambassador.
Previous tour.yaml
---
apiVersion: v1
kind: Service
metadata:
name: tour
annotations:
getambassador.io/config: |
---
apiVersion: ambassador/v1
kind: Mapping
name: tour-ui_mapping
prefix: /
service: tour:5000
---
apiVersion: ambassador/v1
kind: Mapping
name: tour-backend_mapping
prefix: /backend/
service: tour:8080
labels:
ambassador:
- request_label:
- backend
spec:
ports:
- name: ui
port: 5000
targetPort: 5000
- name: backend
port: 8080
targetPort: 8080
selector:
app: tour
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: tour
spec:
replicas: 1
selector:
matchLabels:
app: tour
strategy:
type: RollingUpdate
template:
metadata:
labels:
app: tour
spec:
containers:
- name: tour-ui
image: quay.io/datawire/tour:ui-0.2.1
ports:
- name: http
containerPort: 5000
- name: quote
image: quay.io/datawire/tour:backend-0.2.1
ports:
- name: http
containerPort: 8080
resources:
limits:
cpu: "0.1"
memory: 100Mi
modified tour.yaml(removed backend and changed the image)
---
apiVersion: v1
kind: Service
metadata:
name: tour
annotations:
getambassador.io/config: |
---
apiVersion: ambassador/v1
kind: Mapping
name: tour-ui_mapping
prefix: /
service: tour:5000
spec:
ports:
- name: ui
port: 5000
targetPort: 5000
selector:
app: tour
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: tour
spec:
replicas: 1
selector:
matchLabels:
app: tour
strategy:
type: RollingUpdate
template:
metadata:
labels:
app: tour
spec:
containers:
- name: tour-ui
image: quay.io/integreatly/tutorial-web-app:2.10.5
ports:
- name: http
containerPort: 5000
resources:
limits:
cpu: "0.1"
memory: 100Mi
ambassador-service.yaml
---
apiVersion: v1
kind: Service
metadata:
name: ambassador
spec:
type: NodePort
externalTrafficPolicy: Local
ports:
- port: 80
targetPort: 8080
selector:
service: ambassador
Please help, I'm really confused what is the cause behind it and how I can resolve it.
What you're doing above is replacing the tour Kubernetes service and deployment with your own alternative. This is a bit of an unusual pattern; I'd suspect that there's probably a typo somewhere which means Kubernetes isn't picking up on your change.
I'd suggest creating a unique test Kubernetes service and deployment, and pointing the image in your deployment to your new image. Then you can register a new prefix with Ambassador.
You can also look at the Ambassador diagnostics (see https://www.getambassador.io/reference/diagnostics/) which will tell you which routes are registered with Ambassador.
Related
I'm new with Prometheus and I have simple flask app in Kubernetes cluster also I have Prometheus-Monitoring-Grafana services in cluster too in namespace calles prometheus-monitoring. But the problem is when I create ServiceMonitor via .yaml file to connect my app to monitor with Prometheus I see that targets is not added but in config i see that job was added. But status in Prometheus - Service Discovery is Dropped.
A have no idea why my service is not connect to serviceMonitor
serviceMonitor/default/monitoring-webapp/0 (0 / 2 active targets)
app.py
app = Flask(__name__)
metrics = PrometheusMetrics(app)
#app.route('/api')
def index():
return 'ok'
deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: webapp-deployment
labels:
app: webapp
spec:
replicas: 1
selector:
matchLabels:
app: webapp
template:
metadata:
labels:
app: webapp
spec:
containers:
- name: webapp
image: dmitriy83/flask_one:latest
imagePullPolicy: Always
resources:
requests:
memory: "64Mi"
cpu: "250m"
limits:
memory: "128Mi"
cpu: "500m"
ports:
- containerPort: 5000
env:
- name: flask_url
value: http://flasktwo-service:5003
imagePullSecrets:
- name: dockersecret
---
apiVersion: v1
kind: Service
metadata:
name: webapp-service
spec:
selector:
app: webapp
ports:
- name: service
protocol: TCP
port: 5000
targetPort: 5000
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: monitoring-webapp
labels:
release: prometheus-monitoring
app: webapp
spec:
endpoints:
- path: /metrics
port: service
targetPort: 5000
namespaceSelector:
matchNames:
- default
selector:
matchLabels:
app: webapp
Finally i figured it out. The issue was port name. Please find workable solution below
deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: webapp
labels:
component: backend
instance: app
name: containers-my-app
namespace: default
spec:
selector:
matchLabels:
component: backend
instance: app
name: containers-my-app
template:
metadata:
labels:
component: backend
instance: app
name: containers-my-app
spec:
containers:
- name: app
image: dmitriy83/flask_one:latest
imagePullPolicy: IfNotPresent
ports:
- containerPort: 5000
name: webapp
imagePullSecrets:
- name: myregistrykey
service.yaml
apiVersion: v1
kind: Service
metadata:
name: webapp
labels:
component: backend
instance: app
name: containers-my-app
namespace: default
spec:
type: ClusterIP
ports:
- name: http
port: 5000
protocol: TCP
targetPort: webapp # one of the major thing w/o it you could not have active targets in Prometheus
selector:
component: backend
instance: app
name: containers-my-app
finally monitor.yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: webapp-super
labels:
component: backend
instance: app
name: containers-my-app
release: kube-prometheus-stack # You need to verify what is your realease name pf prometheus
namespace: prometheus-monitoring # choose in what name space your prometheus is
spec:
namespaceSelector:
matchNames:
- default
selector:
matchLabels:
component: backend
instance: app
name: containers-my-app
endpoints:
- port: http # http - is a port name which was put in service.yaml
I have a golang webapp pod running in kubernetes cluster, and I tried to deploy a prometheus pod to monitor the golang webapp pod.
I specified prometheus.io/port: to 2112 in the service.yaml file, which is the port that the golang webapp is listening on, but when I go to the Prometheus dashboard, it says that the 2112 endpoint is down.
I'm following this guide, tried this thread's solution thread, but still getting result saying 2112 endpoint is down.
Below is the my service.yaml and deployment.yaml
apiVersion: v1
kind: Service
metadata:
name: prometheus-service
namespace: monitoring
annotations:
prometheus.io/scrape: 'true'
prometheus.io/path: '/metrics'
prometheus.io/port: '2112'
spec:
selector:
app: prometheus-server
type: NodePort
ports:
- port: 8080
targetPort: 9090
nodePort: 30000
---
apiVersion: v1
kind: Service
metadata:
namespace: monitoring
name: goapp
spec:
type: NodePort
selector:
app: golang
ports:
- name: main
protocol: TCP
port: 80
targetPort: 2112
nodePort: 30001
apiVersion: apps/v1
kind: Deployment
metadata:
name: prometheus-deployment
namespace: monitoring
labels:
app: prometheus-server
spec:
replicas: 1
selector:
matchLabels:
app: prometheus-server
template:
metadata:
labels:
app: prometheus-server
spec:
containers:
- name: prometheus
image: prom/prometheus
args:
- "--config.file=/etc/prometheus/prometheus.yml"
- "--storage.tsdb.path=/prometheus/"
ports:
- containerPort: 9090
volumeMounts:
- name: prometheus-config-volume
mountPath: /etc/prometheus/
- name: prometheus-storage-volume
mountPath: /prometheus/
volumes:
- name: prometheus-config-volume
configMap:
defaultMode: 420
name: prometheus-server-conf
- name: prometheus-storage-volume
emptyDir: {}
---
apiVersion: apps/v1
kind: Deployment
metadata:
namespace: monitoring
name: golang
spec:
replicas: 1
template:
metadata:
labels:
app: golang
spec:
containers:
- name: gogo
image: effy77/gogo2
ports:
- containerPort: 2112
selector:
matchLabels:
app: golang
I will try add prometheus.io/port: 2112 to the prometheus deployment part, as I suspect that might be the cause.
I was confused with where to put the annotations,got my clarifications from this thread, I needed to put it under the service's metadata that needs to be scraped by prothemeus. So in my case it needs to be in goapp's metadata.
apiVersion: v1
kind: Service
metadata:
namespace: monitoring
name: goapp
annotations:
prometheus.io/scrape: 'true'
prometheus.io/path: '/metrics'
prometheus.io/port: '2112'
I am trying to expose some custom metrics of a kubernetes application at prometheus.
I successfully deploy deploy my app at kubernetes. The ServiceMonitor is also added but no targets are discovered (0/0 up) .
The application is an nginx server with the relevant nginx-prometheus-exporter sidecar.
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-example-v3
labels:
app: nginx-example-v3
spec:
selector:
matchLabels:
app: nginx-example-v3
template:
metadata:
labels:
app: nginx-example-v3
spec:
containers:
- name: nginx
image: nginx
resources:
limits:
memory: "128Mi"
cpu: "100m"
ports:
- name: http
containerPort: 8080
volumeMounts:
- name: "config"
mountPath: "/etc/nginx/nginx.conf"
subPath: "nginx.conf"
- name: exporter
image: nginx/nginx-prometheus-exporter:0.8.0
ports:
- containerPort: 9113
volumes:
- name: "config"
configMap:
name: "nginx-example-v2-config"
---
apiVersion: v1
kind: Service
metadata:
labels:
name: nginx-example-v3
name: nginx-example-v3
spec:
type: LoadBalancer
selector:
app: nginx-example-v3
ports:
- name: http
port: 8080
targetPort: 8080
- name: http-exporter
port: 9113
targetPort: 9113
After that i can see the nginx custom metrics exposed at the /metrics API:
Then i apply the monitoring service:
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: nginx-example-v3
spec:
endpoints:
- bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
interval: 15s
port: web
selector:
matchLabels:
app: nginx-example-v3
i can see that the service is successfully added at prometheus "Service Discovery" section:
BUT no targets are discovered (0/0 up) on behalf of prometheus
What i am missing???
Any help is really apreciated since i have been stuck on this many days!
Thank you very much in advance. :-)
Replies
#efotopoulou
I slightly changed the service manifest and i am able to get the metrics now.
sorry for the new discussion. i hope my manifests help other persons to configure their services.
This is the updated manifest.
:-)
apiVersion: v1
kind: Service
metadata:
labels:
app: nginx-example-v3
name: nginx-example-v3
spec:
type: LoadBalancer
selector:
app: nginx-example-v3
ports:
- name: http
port: 8080
targetPort: 8080
- name: web
port: 9113
targetPort: 9113
I have a cluster with istio injection enabled and cockroach db stateful set defined:
apiVersion: v1
kind: ServiceAccount
metadata:
name: cockroachdb-serviceaccount
---
apiVersion: v1
kind: Service
metadata:
# This service is meant to be used by clients of the database. It exposes a ClusterIP that will
# automatically load balance connections to the different database pods.
name: cockroachdb-public
labels:
app: cockroachdb
spec:
ports:
# The main port, served by gRPC, serves Postgres-flavor SQL, internode
# traffic and the cli.
- port: 26257
targetPort: 26257
name: tcp
# The secondary port serves the UI as well as health and debug endpoints.
- port: 8080
targetPort: 8080
name: http
selector:
app: cockroachdb
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: cockroachdb-statefulset
labels:
version: v20.1.2
spec:
serviceName: cockroachdb
replicas: 3
selector:
matchLabels:
app: cockroachdb
template:
metadata:
labels:
app: cockroachdb
version: v20.1.2
spec:
serviceAccountName: cockroachdb-serviceaccount
containers:
- name: cockroachdb
image: cockroachdb/cockroach:v20.1.2
ports:
- containerPort: 26257
name: tcp
- containerPort: 8080
name: http
volumeMounts:
- name: datadir
mountPath: /cockroach/cockroach-data
env:
- name: COCKROACH_CHANNEL
value: kubernetes-insecure
command:
- "/bin/bash"
- "-ecx"
# The use of qualified `hostname -f` is crucial:
# Other nodes aren't able to look up the unqualified hostname.
- "exec /cockroach/cockroach start --logtostderr --insecure --advertise-host $(hostname -f) --http-addr 0.0.0.0 --join cockroachdb-statefulset-0.cockroachdb,cockroachdb-statefulset-1.cockroachdb,cockroachdb-statefulset-2.cockroachdb --cache 25% --max-sql-memory 25%"
# No pre-stop hook is required, a SIGTERM plus some time is all that's
# needed for graceful shutdown of a node.
terminationGracePeriodSeconds: 5
volumes:
- name: datadir
persistentVolumeClaim:
claimName: datadir
podManagementPolicy: Parallel
updateStrategy:
type: RollingUpdate
volumeClaimTemplates:
- metadata:
name: datadir
spec:
accessModes:
- "ReadWriteOnce"
resources:
requests:
storage: 4Gi
---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
name: cockroachdb-public
spec:
host: cockroachdb-public
trafficPolicy:
tls:
mode: ISTIO_MUTUAL
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: cockroachdb-public
spec:
hosts:
- cockroachdb-public
http:
- match:
- port: 8080
route:
- destination:
host: cockroachdb-public
port:
number: 8080
tcp:
- match:
- port: 26257
route:
- destination:
host: cockroachdb-public
port:
number: 26257
and a service that accesses it:
apiVersion: v1
kind: ServiceAccount
metadata:
name: downstream-serviceaccount
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: downstream-deployment-v1
labels:
app: downstream
version: v1
spec:
replicas: 1
selector:
matchLabels:
app: downstream
version: v1
template:
metadata:
labels:
app: downstream
version: v1
spec:
serviceAccountName: downstream-serviceaccount
containers:
- name: downstream
image: downstream:0.1
ports:
- containerPort: 80
env:
- name: DATABASE_URL
value: postgres://roach#cockroachdb-public:26257/roach?sslmode=disable
---
apiVersion: v1
kind: Service
metadata:
name: downstream-service
labels:
app: downstream
spec:
type: ClusterIP
selector:
app: downstream
ports:
- port: 80
targetPort: 80
name: http
protocol: TCP
---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
name: downstream-service
spec:
host: downstream-service
trafficPolicy:
tls:
mode: ISTIO_MUTUAL
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: downstream-service
spec:
hosts:
- downstream-service
http:
- name: "downstream-service-routes"
match:
- port: 80
route:
- destination:
host: downstream-service
port:
number: 80
Now I'd like to restrict access to cockroach db only to downstream-service and to cockroachdb itself (since nodes need intercommunication between each other).
I'm trying to restrict the traffic with something like this:
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: default-deny-all
namespace: default
spec:
{}
---
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: cockroachdb-authorizationpolicy-allow-from-downstream
namespace: default
spec:
selector:
matchLabels:
app: cockroachdb
action: ALLOW
rules:
- from:
- source:
principals: ["cluster.local/ns/default/sa/downstream-serviceaccount"]
- to:
- operation:
ports: ["26257"]
---
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: cockroachdb-authorizationpolicy-allow-from-cockroachdb
namespace: default
spec:
selector:
matchLabels:
app: cockroachdb
action: ALLOW
rules:
- from:
- source:
principals: ["cluster.local/ns/default/sa/cockroachdb-serviceaccount"]
- to:
- operation:
ports: ["26257"]
but doesn't seem to do anything. I can still e.g. access cockroachdb-public:8080 cluster HTTP UI from downstream-service.
Now when I add the following:
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: default-deny-all-to-cockroachdb
namespace: default
spec:
selector:
matchLabels:
app: cockroachdb
action: DENY
rules:
- to:
- operation:
ports: ["26257"]
then all the traffic is blocked (including the traffic between cockroachdb nodes).
What am I doing wrong here?
You are having the same problem that a guy couple of days ago. In your authorization policy you have two policies:
service account downstream-serviceaccount (and cockroachdb-serviceaccount for the other authorization policy) from default namespace can access the service with labels app: cockroachdb on any port on default namespace.
Any service account, from any namespace can access the service with labels app: cockroachdb, on port 26257.
In order to make it an AND, you would do this:
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: cockroachdb-authorizationpolicy-allow-from-cockroachdb
namespace: default
spec:
selector:
matchLabels:
app: cockroachdb
action: ALLOW
rules:
- from:
- source:
principals: ["cluster.local/ns/default/sa/cockroachdb-serviceaccount"]
to: <- remove the dash from here
- operation:
ports: ["26257"]
Same with the other AuthorizationPolicy object. Also note that you don't need to explicitly create a DENY policy. When you create an ALLOW one, it automatically denies everything else.
I have deployed JFrog Container Registry to my Kubernetes cluster, which all comes up fine but when I try to access it via browser, it redirects to /ui which returns a 404 but nothing seems to show in the logs.
I have not used the Helm chart as I do not need the nginx or Postgres etc just to try it out.
My deployment is this:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: jcr
namespace: <REDACTED>
spec:
replicas: 1
template:
metadata:
labels:
app: jcr
spec:
containers:
- name: jcr
image: docker.bintray.io/jfrog/artifactory-jcr:latest
ports:
- containerPort: 8081
volumeMounts:
- name: jcr-data
mountPath: /jcr-data
volumes:
- name: jcr-data
persistentVolumeClaim:
claimName: jcr-data
securityContext:
fsGroup: 2000
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: jcr-data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
---
apiVersion: v1
kind: Service
metadata:
name: jcr
namespace: <REDACTED>
annotations:
prometheus.io/scrape: 'true'
prometheus.io/path: /
prometheus.io/port: '8081'
spec:
selector:
app: jcr
ports:
- port: 80
targetPort: 8081
sessionAffinity: None
type: ClusterIP
---
apiVersion: contour.heptio.com/v1beta1
kind: IngressRoute
metadata:
labels:
app: jcr
name: jcr
namespace: <REDACTED>
spec:
virtualhost:
fqdn: <REDACTED>
tls:
secretName: jcr-live
routes:
- match: /
services:
- name: jcr
port: 80
Looks like your port configuration is missing some changes.
You need to expose port 8082 in the jcr container, which is now the main UI port
Once port is exposed, you should add this port to your service.
So your revised yaml should look something like (Deployment and Service):
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: jcr
namespace: <REDACTED>
spec:
replicas: 1
template:
metadata:
labels:
app: jcr
spec:
containers:
- name: jcr
image: docker.bintray.io/jfrog/artifactory-jcr:latest
ports:
- containerPort: 8081
- containerPort: 8082
volumeMounts:
- name: jcr-data
mountPath: /jcr-data
volumes:
- name: jcr-data
persistentVolumeClaim:
claimName: jcr-data
securityContext:
fsGroup: 2000
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: jcr-data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
---
apiVersion: v1
kind: Service
metadata:
name: jcr
namespace: <REDACTED>
annotations:
prometheus.io/scrape: 'true'
prometheus.io/path: /
prometheus.io/port: '8081'
spec:
selector:
app: jcr
ports:
- port: 80
targetPort: 8082
- port: 8081
targetPort: 8081
sessionAffinity: None
type: ClusterIP
Notice I left 8081 open, which allows for direct access to Artifactory if needed for better performance (Artifactory is now running behind a router service).
NOTE - I recommend using the official JFrog Container Registry Helm chart, which greatly simplifies the process of configuring and managing your JCR deployment lifecycle.