Zipkin tracing not working for docker-compose and Dapr - docker-compose

Traces that should have been sent by dapr runtime to zipkin server somehow fails to reach it.
The situation is the following:
I'm using Docker Desktop on my Windows PC. I have downloaded the sample from dapr repository (https://github.com/dapr/samples/tree/master/hello-docker-compose) which runs perfectly out of the box with docker-compose up.
Then I've added Zipkin support as per dapr documentation:
added this service in the bottom of docker-compose.yml
zipkin:
image: "openzipkin/zipkin"
ports:
- "9411:9411"
networks:
- hello-dapr
added config.yaml in components folder
apiVersion: dapr.io/v1alpha1
kind: Configuration
metadata:
name: daprsystem
spec:
mtls:
enabled: false
tracing:
enabled: true
exporterType: zipkin
samplingRate: "1"
expandParams: true
includeBody: true
zipkin:
endpointAddress: "http://zipkin:9411/api/v2/spans"
When application runs, it should send traces to the server, but nothing is found in zipkin UI and logs.
Strange thing start to appear in the logs from nodeapp-dapr_1 service: error while reading spiffe id from client cert
pythonapp-dapr_1 | time="2021-03-15T19:14:17.9654602Z" level=debug msg="found mDNS IPv4 address in cache: 172.19.0.7:34549" app_id=pythonapp instance=ce32220407e2 scope=dapr.contrib type=log ver=edge
nodeapp-dapr_1 | time="2021-03-15T19:14:17.9661792Z" level=debug msg="error while reading spiffe id from client cert: unable to retrieve peer auth info. applying default global policy action" app_id=nodeapp instance=773c486b5aac scope=dapr.runtime.grpc.api type=log ver=edge
nodeapp_1 | Got a new order! Order ID: 947
nodeapp_1 | Successfully persisted state.
Additional info - current dapr version used is 1.0.1. I made sure that security (mtls) is disabled in config file.

Configuration file is supposed to be in different folder then components.
Create new folder e.g. dapr next to the components folder.
Move components folder into newly created dapr folder.
Then create config.yaml in dapr folder.
Update docker-compose accordingly.
docker-compose
services:
nodeapp-dapr:
image: "daprio/daprd:edge"
command: ["./daprd",
"-app-id", "nodeapp",
"-app-port", "3000",
"-placement-host-address", "placement:50006",
"-dapr-grpc-port", "50002",
"-components-path", "/dapr/components",
"-config", "/dapr/config.yaml"]
volumes:
- "./dapr/components/:/dapr"
depends_on:
- nodeapp
network_mode: "service:nodeapp"
config.yaml
apiVersion: dapr.io/v1alpha1
kind: Configuration
metadata:
name: daprConfig
spec:
mtls:
enabled: false
tracing:
enabled: true
samplingRate: "1"
expandParams: true
includeBody: true
zipkin:
endpointAddress: http://host.docker.internal:9411/api/v2/spans
I had issue with localhost and 127.0.0.1 in URL which I resolved using host.docker.internal as hostname.
PS: Don't forget to kill all *-dapr_1 containers so it can load new configuration.

Related

Serving MLFlow artifacts through `--serve-artifacts` without passing credentials

A new version of MLFlow (1.23) provided a --serve-artifacts option (via this pull request) along with some example code. This should allow me to simplify the rollout of a server for data scientists by only needing to give them one URL for the tracking server, rather than a URI for the tracking server, URI for the artifacts server, and a username/password for the artifacts server. At least, that's how I understand it.
A complication that I have is that I need to use podman instead of docker for my containers (and without relying on podman-compose). I ask that you keep those requirements in mind; I'm aware that this is an odd situation.
What I did before this update (for MLFlow 1.22) was to create a kubernetes play yaml config, and I was successfully able to issue a podman play kube ... command to start a pod and from a different machine successfully run an experiment and save artifacts after setting the appropriate four env variables. I've been struggling with getting things working with the newest version.
I am following the docker-compose example provided here. I am trying a (hopefully) simpler approach. The following is my kubernetes play file defining a pod.
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: "2022-01-14T19:07:15Z"
labels:
app: mlflowpod
name: mlflowpod
spec:
containers:
- name: minio
image: quay.io/minio/minio:latest
ports:
- containerPort: 9001
hostPort: 9001
- containerPort: 9000
hostPort: 9000
resources: {}
tty: true
volumeMounts:
- mountPath: /data
name: minio-data
args:
- server
- /data
- --console-address
- :9001
- name: mlflow-tracking
image: localhost/mlflow:latest
ports:
- containerPort: 80
hostPort: 8090
resources: {}
tty: true
env:
- name: MLFLOW_S3_ENDPOINT_URL
value: http://127.0.0.1:9000
- name: AWS_ACCESS_KEY_ID
value: minioadmin
- name: AWS_SECRET_ACCESS_KEY
value: minioadmin
command: ["mlflow"]
args:
- server
- -p
- 80
- --host
- 0.0.0.0
- --backend-store-uri
- sqlite:///root/store.db
- --serve-artifacts
- --artifacts-destination
- s3://mlflow
- --default-artifact-root
- mlflow-artifacts:/
# - http://127.0.0.1:80/api/2.0/mlflow-artifacts/artifacts/experiments
- --gunicorn-opts
- "--log-level debug"
volumeMounts:
- mountPath: /root
name: mlflow-data
volumes:
- hostPath:
path: ./minio
type: Directory
name: minio-data
- hostPath:
path: ./mlflow
type: Directory
name: mlflow-data
status: {}
I start this with podman play kube mlflowpod.yaml. On the same machine (or a different one, it doesn't matter), I have cloned and installed mlflow into a virtual environment. From that virtual environment, I set an environmental variable MLFLOW_TRACKING_URI to <name-of-server>:8090. I then run the example.py file in the mlflow_artifacts example directory. I get the following response:
....
botocore.exceptions.NoCredentialsError: Unable to locate credentials
Which seems like the client needs the server credentials to minIO, which I thought the proxy was supposed to take care of.
If I also provide the env variables
$env:MLFLOW_S3_ENDPOINT_URL="http://<name-of-server>:9000/"
$env:AWS_ACCESS_KEY_ID="minioadmin"
$env:AWS_SECRET_ACCESS_KEY="minioadmin"
Then things work. But that kind of defeats the purpose of the proxy...
What is it about the proxy setup via kubernates play yaml and podman that is going wrong?
Just in case anyone stumbles upon this, I had same issue based on your description. However the problem on my side was that I was that I tried to test this with a preexisting experiment (default), and I did not create new one, so the old setting carried over, thus resulting in MLFlow trying to use s3 trough credentials and not https.
Hope this helps at least some of you out there.

Port forwarding with traefik and docker-compose

I would like to serve a docker-compose service through traefik with port-forwarding. I had many tries and the best I could achieve from now is described below:
First I create two networks:
docker network create frontend # To expose traefik
docker network create backend # To bind backend services
The traefik configuration is about (development, dashboard enabled at :8080):
version: "3.6"
services:
proxy:
image: traefik:latest
restart: unless-stopped
command:
- "--log.level=DEBUG"
- "--api"
- "--api.dashboard"
- "--api.insecure"
- "--providers.docker"
- "--providers.docker.exposedbydefault=false"
# Service entrypoint:
- "--entrypoints.lora-server.address=:8090"
volumes:
- "/var/run/docker.sock:/var/run/docker.sock:ro"
labels:
- "traefik.enable=true"
- "traefik.docker.network=frontend"
- "traefik.http.routers.traefik.entrypoints=traefik"
- "traefik.http.routers.traefik.service=api#internal"
networks:
- backend
- frontend
ports:
- "8080:8080"
- "8090:8090"
networks:
frontend:
external: true
backend:
external: true
The backend service is described here (a fork from ChripStack but it could be anything else):
version: "3"
services:
# [...]
chirpstack-application-server:
image: chirpstack/chirpstack-application-server:3
networks:
- backend
volumes:
- ./configuration/chirpstack-application-server:/etc/chirpstack-application-server
labels:
- "traefik.enable=true"
- "traefik.docker.network=backend"
- "traefik.http.routers.chirpstack.entrypoints=lora-server"
- "traefik.http.routers.chirpstack.rule=Host(`{host:}`)"
- "traefik.http.routers.chirpstack.service=chirpstack-application-server#docker"
- "traefik.http.services.chirpstack-application-server.loadbalancer.server.port=8080"
# [...]
networks:
backend:
external: true
The service also natively run on :8080 and I would like to access it on :8090 through traefik.
When I run both applications, traefik registers the new service and does not complain (no errors, no warning, the flow seems complete at least from the dashboard interface).
time="2020-07-30T11:47:47Z" level=debug msg="Creating middleware" middlewareType=Pipelining serviceName=chirpstack-application-server#docker entryPointName=lora-server routerName=chirpstack#docker middlewareName=pipelining
time="2020-07-30T11:47:47Z" level=debug msg="Creating load-balancer" entryPointName=lora-server routerName=chirpstack#docker serviceName=chirpstack-application-server#docker
time="2020-07-30T11:47:47Z" level=debug msg="Creating server 0 http://192.168.112.9:8080" entryPointName=lora-server routerName=chirpstack#docker serviceName=chirpstack-application-server#docker serverName=0
time="2020-07-30T11:47:47Z" level=debug msg="Added outgoing tracing middleware chirpstack-application-server#docker" middlewareName=tracing entryPointName=lora-server routerName=chirpstack#docker middlewareType=TracingForwarder
But I could not access the service, I am geting 404 errors when I try to connect http://host:8090.
To my understanding, it seems traefik does not know how it should complete the flow between the two networks: http://frontend:8090 -> http://backend:8080 (because I haven't referenced it anywhere).
What should I change in my configuration to make it work? How can I specify to traefik that it must route the HTTP traffic from frontend:8090 to backend:8080? Your help is much appreciated.
Traefik listens on some port, that's the only thing defined with entrypoints. You don't specify network for this incoming requests, because it's unrelated. traefik.docker.network is only used for routing after Traefik handles the incoming requests.
So the correct flow is not http://frontend:8090 -> http://backend:8080, but http://host:8090 -> http://backend:8080.
I think your configuration is correct. The only thing which seems to be odd is usage of Host rule. Maybe try to change it to something like this:
"traefik.http.routers.chirpstack.rule=hostregexp(`{host:.+}`)"

Not able to start apache-nifi in aks

Hi all I am working on Nifi and I am trying to install it in AKS (Azure kubernetes service).
Using nifi 1.9.2 version. While installing it in AKS gives me an error
replacing target file /opt/nifi/nifi-current/conf/nifi.properties
sed: preserving permissions for ‘/opt/nifi/nifi-current/conf/sedSFiVwC’: Operation not permitted
replacing target file /opt/nifi/nifi-current/conf/nifi.properties
sed: preserving permissions for ‘/opt/nifi/nifi-current/conf/sedK3S1JJ’: Operation not permitted
replacing target file /opt/nifi/nifi-current/conf/nifi.properties
sed: preserving permissions for ‘/opt/nifi/nifi-current/conf/sedbcm91T’: Operation not permitted
replacing target file /opt/nifi/nifi-current/conf/nifi.properties
sed: preserving permissions for ‘/opt/nifi/nifi-current/conf/sedIuYSe1’: Operation not permitted
NiFi running with PID 28.
The specified run.as user nifi
does not exist. Exiting.
Received trapped signal, beginning shutdown...
Below is my nifi.yml file
apiVersion: apps/v1
kind: Deployment
metadata:
name: nifi-core
spec:
replicas: 1
selector:
matchLabels:
app: nifi-core
template:
metadata:
labels:
app: nifi-core
spec:
containers:
- name: nifi-core
image: my-azurecr.io/nifi-core-prod:1.9.2
env:
- name: NIFI_WEB_HTTP_PORT
value: "8080"
- name: NIFI_VARIABLE_REGISTRY_PROPERTIES
value: "./conf/custom.properties"
resources:
requests:
cpu: "6"
memory: 12Gi
limits:
cpu: "6"
memory: 12Gi
ports:
- containerPort: 8080
volumeMounts:
- name: my-nifi-core-conf
mountPath: /opt/nifi/nifi-current/conf
volumes:
- name: my-nifi-core-conf
azureFile:
shareName: my-file-nifi-core/nifi/conf
secretName: my-nifi-secret
readOnly: false
I have some customization in nifi Dockerfile, which copies some config files related to my configuration. When I ran my-azurecr.io/nifi-core-prod:1.9.2 docker image on my local it works as expected
But when I try to run it on AKS its giving above error. since its related to permissions I have tried with both user nifi and root in Dockerfile.
All the required configuration files are provided in volume my-nifi-core-conf running in same resourse group.
Since I am starting nifi with docker my exception is, it will behave same regardless of environment. Either on my local or in AKS.
But error also say user nifi does not exist. The official nifi-image setup the user requirement.
Can anyone help, I cant event start container in interaction mode as pods in not in running mode. Thanks in advance.
I think your missing the Security Context definition for your Kubernetes Pod. The user that Nifi runs under within a Docker has a specific UID and GID, and with the error message you getting, I would suspect that because that user is not defined in the Pod's security context it's not launching as expected.
Have a look at section on the Kubernetes documentation about security contexts, and that should be enough get you started.
I would also have a look at using something like Minikube when testing Kubernetes deployments as Kubernetes adds a large number of controls around a container engine like Docker.
Security Contexts Docs: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/
Minikube: https://kubernetes.io/docs/setup/learning-environment/minikube/
If you never figured this out, I was able to do this by running an initContainer before the main container, and changing the directory perms there.
initContainers:
- name: init1
image: busybox:1.28
volumeMounts:
- name: nifi-pvc
mountPath: "/opt/nifi/nifi-current"
command: ["sh", "-c", "chown -R 1000:1000 /opt/nifi/nifi-current"] #or whatever you want to do as root
update: does not work with nifi 1.14.0 - works with 1.13.2

Debugging Traefik when the Site Cannot Be Reached from outside Company's Intranet

Using docker-compose I have deployed a web application that uses Traefik as the reverse proxy, listening on port 80. This works without problem when I'm inside my company's intranet. Outside of the intranet, however, I get a 'site cannot be reached' response. Pinging the address from outside shows that the address is reachable and port 80 is open.
I've also tried to use segments in my Traefik configuration to route both the internal and external hostname I have been provided but this has no effect:
version: "3.5"
services:
test:
image: emilevauge/whoami
deploy:
labels:
traefik.enable: "true"
traefik.foo.frontend.rule: "Host:${HOSTNAME};PathPrefixStrip:/test"
traefik.bar.frontend.rule: "Host:${EXTERNAL_HOSTNAME};PathPrefixStrip:/test"
traefik.port: 80
networks:
- frontend
...
I have configured the access logs to see if my requests are reaching Traefik, can anyone advise me what I should be looking for and how to filter the huge amount of text produced to find it? This is my Traefik setup configuration:
version: '3.5'
services:
traefik:
image: traefik:alpine
command: |-
--entryPoints="Name:http Address::80"
--entryPoints="Name:https Address::443 TLS"
--defaultentrypoints="http,https"
--acme
--acme.acmelogging="true"
--acme.domains="${HOSTNAME}"
--acme.domains="${EXTERNAL_HOSTNAME}"
--acme.email="${ACME_EMAIL}"
--acme.entrypoint="https"
--acme.httpchallenge
--acme.httpchallenge.entrypoint="http"
--acme.storage="/opt/traefik/acme.json"
--acme.onhostrule="true"
--docker
--docker.swarmmode
--docker.domain="${HOSTNAME}"
--docker.network="frontend"
--docker.watch
--api
--api.statistics
--logLevel="DEBUG"
networks:
- frontend

Keycloak behind Kong and strange redirect

Setup:
minikube version: v0.27.0
Kong (helm install stable/kong) / version 1.0.2
Keycloak (helm install stable/keycloak) / version 4.8.3.Final
I have a self signed SSL certificate for my "hello.local".
What I need to achieve: Keycloak behind Kong at "https://hello.local/".
My steps:
1) fresh minikube
2) Install Keycloak with helm, following values.yaml:
keycloak:
basepath: ""
replicas: 1
...
extraEnv: |
- name: PROXY_ADDRESS_FORWARDING
value: "true"
(that would create service auth-keycloak-http)
3) Install Kong with helm, following values.yaml:
replicaCount: 1
admin:
ingress:
enabled: true
hosts: ['hello.local']
proxy:
type: LoadBalancer
ingress:
enabled: true
hosts: ['hello.local']
tls:
- hosts:
- hello.local
secretName: tls-certificate
annotations:
kubernetes.io/ingress.class: "nginx"
nginx.ingress.kubernetes.io/rewrite-target: /
nginx.ingress.kubernetes.io/ssl-passthrough: "true"
tls:
enabled: true
postgresql:
...
4) I setup service and route for Kong
Service:
Protocol: http
Host: auth-keycloak-http
Port: 80
Route:
Hosts: hello.local
After that I can open "https://hello.local" and can see welcome page from Keycloak where I can click Administration Console and after that I have redirect to "https://hello.local:8443/admin/master/console/" in my browser. So we should not have redirect with another port at this point.
Setup with 2 docker images (Keycloak + Kong) is working if PROXY_ADDRESS_FORWARDING is true.
How can I make Keycloak (helm chart) to work behind Kong (helm chart) in kubernetes cluster as expected, without redirect?
This is being discussed in github issue 1, github issue 2 and github issue 3. Also, Similar questions on stackoverflow
Original answer:
Seems, it is necessary to setup following environment variables in values.yaml of keycloak helm chart:
...
extraEnv: |
- name: KEYCLOAK_HTTP_PORT
value: "80"
- name: KEYCLOAK_HTTPS_PORT
value: "443"
- name: KEYCLOAK_HOSTNAME
value: example.com
...
All of them are required, after that, redirect would work correctly.
Added 2021 Sep:
Issue with weird behavior with redirect to port 8443 for some action (like go to Account management with the link on the top right of admin console).
In fact we do not need to set any KEYCLOAK_HTTP_PORT or KEYCLOAK_HTTPS_PORT.
Some changes are required on proxy side. On proxy we need to set x-forwarded-port to 443 for this route.
In my case we use Kong:
On the route, where Keycloak is exposed, we need to add (this one worked for me):
serverless > post function with following content:
ngx.var.upstream_x_forwarded_port=443
More info on KONG and x_forwarded_*