Using Traefik to reverse proxy Grafana at a suburl; 404 responses - grafana

Question

I have tried my best to follow the documentation for both Grafana and Traefik to reverse proxy incoming requests to https://metrics.mydomain/grafana through Traefik to a Grafana instance running as a container, but I get 404 responses from Grafana and I can't figure out why.
Here's what I've done after reading docs and trying various things in my docker-compose.yml file:
version: '3'
services:
traefik:
image: traefik
restart: always
ports:
- 80:80
- 443:443
- 8080:8080
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- /home/ben/traefik.toml:/etc/traefik/traefik.toml
- /etc/letsencrypt/live/mydomain/fullchain.pem:/mydomain_fullchain.pem
- /etc/letsencrypt/live/mydomain/privkey.pem:/mydomain_privkey.pem
grafana:
image: grafana/grafana
restart: always
ports:
- 3000:3000
volumes:
- /srv/grafana:/var/lib/grafana
environment:
- GF_SERVER_ROOT_URL=%(protocol)s://%(domain)s:%(http_port)s/grafana
labels:
- traefik.enable=true
- 'traefik.frontend.rule=Host:metrics.mydomain;PathPrefix:/grafana;'
- 'traefik.frontend.redirect.entryPoint=https'
Traefik does pick up on this. Here's what it shows on the dashboard:
However, when I actually request https://metrics.mydomain/grafana, what I get back is 404 and an error document, and then subsequent 404s for the resources requested by the custom error document:
Finally, and most confusingly, it appears that the requests as received by Grafana itself are correct, but still result in 404s:
method=GET path=/grafana status=404 remote_addr=192.168.0.1
method=GET path=/grafana/public/build/grafana.dark.css status=404 remote_addr=192.168.0.1
...etc
I don't understand why this is so fussy, and there doesn't appear (to me) to be any evidence that would suggest what the actual problem is.
p.s. - I did leave a comment on a Grafana issue that looked relevant. https://github.com/grafana/grafana/issues/2066#issuecomment-380647174

Answer 1

I was able to fix this by using PathPrefixStrip instead of PathPrefix

Answer 2

This issue is resolved as I write this in the latest image. I was experiencing the same issue with grafana/grafana:5.1.0.
My working configuration for reference:
version: '2'
services:
grafana:
image: grafana/grafana
environment:
- GF_SERVER_ROOT_URL=%(protocol)s://%(domain)s:%(http_port)s/grafana
- GF_SERVER_DOMAIN=devserver1.mydomain
- GF_SERVER_SERVE_FROM_SUB_PATH=true
labels:
- "traefik.http.routers.grafana.rule=Host(`devserver1.mydomain`) && PathPrefix(`/grafana`)"
- "traefik.http.services.grafana.loadbalancer.server.port=3000"
networks:
- traefik_default
volumes:
- grafana-storage:/var/lib/grafana
networks:
traefik_default:
external: true
volumes:
grafana-storage:
Grafana's documentation on running behind a reverse proxy Run Grafana behind a reverse proxy. An explanation on how to Configure with environment variables.