I have a docker-compose setup with service 1 (grafana) sending requests to service 2 (prometheus).
I've put a Keycloak Gatekeeper in front of service 2, asking for authentication.
That works perfectly fine.
Now my idea was to also place a second Keycloak Gatekeeper in front of service 1, injecting said authentication.
Unfortunately, that doesnt work.
Observing my traffic via wireshark, it seems my second Keycloak Gatekeeper is not even involved in any communication.
My docker compose file for service 1 and the forwarding proxy looks roughly like this:
keycloak-forwarder:
image: quay.io/keycloak/keycloak-gatekeeper
command:
- --enable-forwarding=true
- --forwarding-username=<my_username>
- --forwarding-password=<my_password>
- --forwarding-domains=${BASE_DOMAIN}/grafana
- --listen=:3001
- --client-id=<my_keycloak_client_id>
- --client-secret=<my_keycloak_client_secret>
- --discovery-url=${DOMAIN_PROTOCOL}://${KEYCLOAK_DOMAIN}/auth/realms/<my_keycloak_realm>
networks:
- webgateway
grafana:
image: grafana/grafana
networks:
- webgateway
labels:
traefik.enable: true
traefik.backend: grafana
traefik.frontend.rule: Host:${BASE_DOMAIN};PathPrefix:/grafana;PathPrefixStrip:/grafana
traefik.port: 3000
traefik.docker.network: webgateway
Ok, I've found the solution.
There are several things, that I did wrong.
The forwarding-domains flag describes the domains the request goes to,
not comes from. So if grafana talks to prometheus, the prometheus
domain needs to be entered here.
The Keycloak Gatekeeper doesn't automatically intercept communication. So grafana
needs to use it explicitly as proxy.
So the fixed docker-compose looks the following:
keycloak-forwarder:
image: quay.io/keycloak/keycloak-gatekeeper
command:
- --enable-forwarding=true
- --forwarding-username=<my_username>
- --forwarding-password=<my_password>
- --forwarding-domains=${BASE_DOMAIN}/prometheus
- --listen=:3001
- --client-id=<my_keycloak_client_id>
- --client-secret=<my_keycloak_client_secret>
- --discovery-url=${DOMAIN_PROTOCOL}://${KEYCLOAK_DOMAIN}/auth/realms/<my_keycloak_realm>
networks:
- webgateway
grafana:
image: grafana/grafana
networks:
- webgateway
environment:
- HTTP_PROXY=http://keycloak-forwarder:3001
- HTTPS_PROXY=http://keycloak-forwarder:3001
labels:
traefik.enable: true
traefik.backend: grafana
traefik.frontend.rule: Host:${BASE_DOMAIN};PathPrefix:/grafana;PathPrefixStrip:/grafana
traefik.port: 3000
traefik.docker.network: webgateway
Related
I want the next port forwarding:
http://traefik.service.localhost/ -> Traefik UI
http://api.service.localhost/ -> 'Hello-Word' page from api-service
Here is my attempt to create appropriate docker-compose.yml file:
version: '3.8'
services:
reverse-proxy:
image: traefik:v2.4
container_name: reverse-proxy
command:
- "--api.insecure=true"
- "--providers.docker"
ports:
- "80:80"
- "8080:8080"
labels:
- traefik.enable=true
- traefik.docker.network=pred-network
volumes:
- /var/run/docker.sock:/var/run/docker.sock
networks:
- pred-network
api-service:
image: x86_64/prediction-service:0.8.1
container_name: api-service
environment:
SERVING_SERVICE: model-service
expose:
- 80
labels:
- traefik.enable=true
- traefik.http.routers.api-service.rule=Host(`api.service.localhost`)
networks:
- pred-network
networks:
pred-network:
I am getting the following:
http://traefik.service.localhost/ -> HTTP Error 404. The requested resource is not found.
http://api.service.localhost/ -> HTTP Error 404. The requested resource is not found.
The only link that actually works:
http://api.service.localhost:8080/ -> Traefik UI
If I would include:
ports:
- "8070:80"
into api-service part of docker-compose.yml I could access my 'Hello-Word':
http://localhost:8070/ -> "Hello-Word"
In this case it not get routed through Traefik but directly through api-service. Is it possible to do it over Traefik?
To have a service routed by traefik with docker, you should have labels. Traefik will fetch those labels in order to know how to do routing.
Thus, to have request of api.service.localhost routed to your api service, you should have the label that define the corresponding rule:
traefik.http.routers.service.rule=Host(`api.service.localhost`)
As for the label of Traefik, you should have two labels:
traefik.http.routers.traefik.rule=Host(`traefik.service.localhost`)
traefik.http.routers.traefik.service=api#internal
Beside those traefik labels, you should tell traefik to expose the dashboard using a specific service by adding this configuration: --api.dashboard (More help to expose the dashboard here).
Note, if your api service uses another port than the one by default, you could add this label:
traefik.http.services.service.loadbalancer.server.port=80
Ps, do not expose twice the port 80, you only need to expose it for the traefik container.
I'm trying to deploy a React + FastApi + Postgres application on docker compose with Traefik as the reverse proxy. I'm running into issues with Bad Gateway errors. Running my FastAPI locally runs it on port 8888 and exposes the path /docs to view the api documentation. I'd like to eventually have the application running on example.local with the docs available on example.local/api/docs. My docker-compose.yaml is as follows (loosely based on this one):
version: '3.8'
services:
proxy:
image: traefik:v2.4
networks:
- web
volumes:
- /var/run/docker.sock:/var/run/docker.sock
ports:
- '80:80'
- '8080:8080'
- '443:443'
command:
- --providers.docker
- --api.insecure=true
- --providers.docker.exposedbydefault=false
- --providers.docker.network=web
- --entrypoints.web.address=:80
labels:
- traefik.enable=true
- traefik.http.routers.example-proxy-http.rule=Host(`example.local`)
- traefik.http.routers.example-proxy-http.entrypoints=web
- traefik.http.services.example-proxy.loadbalancer.server.port=80
backend:
build:
context: ./backend
dockerfile: Dockerfile
command: python app/main.py
volumes:
- ./backend/app:/app
env_file:
- .env
networks:
- web
- backend
labels:
- traefik.enable=true
- traefik.http.routers.example-backend-http.rule=PathPrefix(`api/docs`)
- traefik.http.routers.example-backend-http.entrypoints=web
- traefik.http.services.example-backend.loadbalancer.server.port=8888
networks:
web:
external: true
backend:
external: false
I've added 127.0.0.1 example.local to my /etc/hosts file.
From reading around it seems like Bad Gateway errors tend to occur from traefik and related services not being on the same network, or traefik routing traffic to the wrong port on the service container. However if I set ports: - '8888:8888' in my backend service I can access the docs from localhost:8888/docs so I'm pretty sure 8888 is the correct port for the backend loadbalancer. From what I can see traefik and the backend service are on the same network too and I've set it as the default traefik network with --providers.docker.network=web. Interestingly if I visit localhost/api/docs in my browser I'm served up a page from FastAPI. So it could be an issue with my traefik http router labels? I'm quite new to traefik and proxies so would appreciate any help or guidance, thanks!
UPDATE
If I specify the host for the backend by adding
- traefik.http.routers.infilmation-backend-http.rule=Host(`example.local`) && PathPrefix(`/docs`)
to the backend service labels, then visiting example.local/docs does serve up page from FastApi. So I guess my question would be what is the best way of setting up a host for this application? Is there a way I can specify a default host for all services then any PathPrefix rules would be in relation to that host?
I would like to serve a docker-compose service through traefik with port-forwarding. I had many tries and the best I could achieve from now is described below:
First I create two networks:
docker network create frontend # To expose traefik
docker network create backend # To bind backend services
The traefik configuration is about (development, dashboard enabled at :8080):
version: "3.6"
services:
proxy:
image: traefik:latest
restart: unless-stopped
command:
- "--log.level=DEBUG"
- "--api"
- "--api.dashboard"
- "--api.insecure"
- "--providers.docker"
- "--providers.docker.exposedbydefault=false"
# Service entrypoint:
- "--entrypoints.lora-server.address=:8090"
volumes:
- "/var/run/docker.sock:/var/run/docker.sock:ro"
labels:
- "traefik.enable=true"
- "traefik.docker.network=frontend"
- "traefik.http.routers.traefik.entrypoints=traefik"
- "traefik.http.routers.traefik.service=api#internal"
networks:
- backend
- frontend
ports:
- "8080:8080"
- "8090:8090"
networks:
frontend:
external: true
backend:
external: true
The backend service is described here (a fork from ChripStack but it could be anything else):
version: "3"
services:
# [...]
chirpstack-application-server:
image: chirpstack/chirpstack-application-server:3
networks:
- backend
volumes:
- ./configuration/chirpstack-application-server:/etc/chirpstack-application-server
labels:
- "traefik.enable=true"
- "traefik.docker.network=backend"
- "traefik.http.routers.chirpstack.entrypoints=lora-server"
- "traefik.http.routers.chirpstack.rule=Host(`{host:}`)"
- "traefik.http.routers.chirpstack.service=chirpstack-application-server#docker"
- "traefik.http.services.chirpstack-application-server.loadbalancer.server.port=8080"
# [...]
networks:
backend:
external: true
The service also natively run on :8080 and I would like to access it on :8090 through traefik.
When I run both applications, traefik registers the new service and does not complain (no errors, no warning, the flow seems complete at least from the dashboard interface).
time="2020-07-30T11:47:47Z" level=debug msg="Creating middleware" middlewareType=Pipelining serviceName=chirpstack-application-server#docker entryPointName=lora-server routerName=chirpstack#docker middlewareName=pipelining
time="2020-07-30T11:47:47Z" level=debug msg="Creating load-balancer" entryPointName=lora-server routerName=chirpstack#docker serviceName=chirpstack-application-server#docker
time="2020-07-30T11:47:47Z" level=debug msg="Creating server 0 http://192.168.112.9:8080" entryPointName=lora-server routerName=chirpstack#docker serviceName=chirpstack-application-server#docker serverName=0
time="2020-07-30T11:47:47Z" level=debug msg="Added outgoing tracing middleware chirpstack-application-server#docker" middlewareName=tracing entryPointName=lora-server routerName=chirpstack#docker middlewareType=TracingForwarder
But I could not access the service, I am geting 404 errors when I try to connect http://host:8090.
To my understanding, it seems traefik does not know how it should complete the flow between the two networks: http://frontend:8090 -> http://backend:8080 (because I haven't referenced it anywhere).
What should I change in my configuration to make it work? How can I specify to traefik that it must route the HTTP traffic from frontend:8090 to backend:8080? Your help is much appreciated.
Traefik listens on some port, that's the only thing defined with entrypoints. You don't specify network for this incoming requests, because it's unrelated. traefik.docker.network is only used for routing after Traefik handles the incoming requests.
So the correct flow is not http://frontend:8090 -> http://backend:8080, but http://host:8090 -> http://backend:8080.
I think your configuration is correct. The only thing which seems to be odd is usage of Host rule. Maybe try to change it to something like this:
"traefik.http.routers.chirpstack.rule=hostregexp(`{host:.+}`)"
So I have the following docker-compose.yml
version: "3.7"
services:
roundclinic-mysql:
image: mysql:5.7
networks:
- spring-boot-mysql-network
environment:
- MYSQL_DATABASE=
- MYSQL_USER=
- MYSQL_PASSWORD=
- MYSQL_ROOT_PASSWORD=
volumes:
- ./mysqldata:/var/lib/mysql:rw,delegated
ports:
- "3306:3306"
web-service:
image: roundclinic/roundclinic:latest
networks:
- spring-boot-mysql-network
- traefik-network
depends_on:
- roundclinic-mysql
ports:
- 8080:8080
environment:
- "SPRING_PROFILES_ACTIVE=dev"
links:
- roundclinic-mysql
labels:
- "--providers.docker.network=traefik_default"
- "traefik.enable=true"
- "traefik.http.routers.roundclinic.rule=Host(`api-dev.roundclinic.app`)"
- "traefik.http.routers.roundclinic.entrypoints=web"
- "traefik.http.services.cal.loadbalancer.server.port=8080"
traefik:
image: "traefik:v2.2"
container_name: "traefik"
command:
- "--log.level=DEBUG"
- "--api.insecure=true"
- "--providers.docker=true"
- "--providers.docker.exposedbydefault=false"
- "--entrypoints.web.address=:80"
- "traefik.docker.network=traefik-network"
ports:
- "80:80"
volumes:
- "/var/run/docker.sock:/var/run/docker.sock:ro"
networks:
traefik-network:
driver: bridge
external: true
spring-boot-mysql-network:
driver: bridge
volumes:
my-db:
Spring boot starts up fine and can connect to mysql.
When I connect to http://api-dev.roundclinic.app:8080/../ I can hit my application just fine
When I connect to http://api-dev.roundclinic.app/../ I get a gateway timeout. I can see in the traefik logs that it's forwarding the request to what seems to be the correct IP and port, but nothing hits the actual application. I'm not sure what's going on here. Any help?
When accessing port 8080 you are bypassing Traefik and directly access your application, correct?
Generally speaking the Traefik labels look good. Entrypoint, Port and Host are defined, router and service port are present. These are usually all the requirements for Docker-based setups.
One thing that I noticed is that the traefik container uses "traefik.docker.network=traefik-network", but your web app uses:
"--providers.docker.network=traefik_default".
I am not sure if traefik_default is something that traefik provides but that mismatch in network names might be the issue.
I can't test if that is the problem but that would be the first thing to check.
One way would be to simplify your config but just always using the networks key from docker compose instead of mixing it with labels and arguments.
I have tried my best to follow the documentation for both Grafana and Traefik to reverse proxy incoming requests to https://metrics.mydomain/grafana through Traefik to a Grafana instance running as a container, but I get 404 responses from Grafana and I can't figure out why.
Here's what I've done after reading docs and trying various things in my docker-compose.yml file:
version: '3'
services:
traefik:
image: traefik
restart: always
ports:
- 80:80
- 443:443
- 8080:8080
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- /home/ben/traefik.toml:/etc/traefik/traefik.toml
- /etc/letsencrypt/live/mydomain/fullchain.pem:/mydomain_fullchain.pem
- /etc/letsencrypt/live/mydomain/privkey.pem:/mydomain_privkey.pem
grafana:
image: grafana/grafana
restart: always
ports:
- 3000:3000
volumes:
- /srv/grafana:/var/lib/grafana
environment:
- GF_SERVER_ROOT_URL=%(protocol)s://%(domain)s:%(http_port)s/grafana
labels:
- traefik.enable=true
- 'traefik.frontend.rule=Host:metrics.mydomain;PathPrefix:/grafana;'
- 'traefik.frontend.redirect.entryPoint=https'
Traefik does pick up on this. Here's what it shows on the dashboard:
However, when I actually request https://metrics.mydomain/grafana, what I get back is 404 and an error document, and then subsequent 404s for the resources requested by the custom error document:
Finally, and most confusingly, it appears that the requests as received by Grafana itself are correct, but still result in 404s:
method=GET path=/grafana status=404 remote_addr=192.168.0.1
method=GET path=/grafana/public/build/grafana.dark.css status=404 remote_addr=192.168.0.1
...etc
I don't understand why this is so fussy, and there doesn't appear (to me) to be any evidence that would suggest what the actual problem is.
p.s. - I did leave a comment on a Grafana issue that looked relevant. https://github.com/grafana/grafana/issues/2066#issuecomment-380647174
I was able to fix this by using PathPrefixStrip instead of PathPrefix
This issue is resolved as I write this in the latest image. I was experiencing the same issue with grafana/grafana:5.1.0.
My working configuration for reference:
version: '2'
services:
grafana:
image: grafana/grafana
environment:
- GF_SERVER_ROOT_URL=%(protocol)s://%(domain)s:%(http_port)s/grafana
- GF_SERVER_DOMAIN=devserver1.mydomain
- GF_SERVER_SERVE_FROM_SUB_PATH=true
labels:
- "traefik.http.routers.grafana.rule=Host(`devserver1.mydomain`) && PathPrefix(`/grafana`)"
- "traefik.http.services.grafana.loadbalancer.server.port=3000"
networks:
- traefik_default
volumes:
- grafana-storage:/var/lib/grafana
networks:
traefik_default:
external: true
volumes:
grafana-storage:
Grafana's documentation on running behind a reverse proxy Run Grafana behind a reverse proxy. An explanation on how to Configure with environment variables.