I've been using Traefik for a while now to get and renew an ACME wildcard certificate. It uses AWS Route 53 for a DNS Challenge. The cert recently expired, having failed to renew with an error I can't find a solution anywhere. "Endpoint" doesn't seem to exist in any documentation on Traefik I can find.
route53: MissingEndpoint: 'Endpoint' configuration is required for this service
FULL ERROR:
time="2022-05-09T20:23:49Z" level=error msg="Error renewing certificate from LE: {*.internal.<DOMAIN REMOVED> []}, error: one or more domains had a problem:\n[*.internal.<DOMAIN REMOVED>] [*.internal.<DOMAIN REMOVED>] acme: error presenting token: route53: MissingEndpoint: 'Endpoint' configuration is required for this service\n" providerName=dns-route53.acme ACME CA="https://acme-v02.api.letsencrypt.org/directory" here
Here's a cut down version of my docker-compose file to define Traefik:
services:
traefik:
image: "traefik:latest"
environment:
- AWS_ACCESS_KEY_ID=${TRAEFIK_AWS_ACCESS_KEY_ID}
- AWS_SECRET_ACCESS_KEY=${TRAEFIK_AWS_SECRET_ACCESS_KEY}
- AWS_REGION=${AWS_REGION}
- AWS_HOSTED_ZONE_ID=${ROUTE53_HOSTED_ZONE_ID}
command:
- "--providers.docker=true"
- "--providers.docker.exposedbydefault=false"
- "--entryPoints.http.address=:80"
- "--entryPoints.https.address=:443"
- "--entrypoints.https.http.tls.certResolver=dns-route53"
- "--entrypoints.public.http.tls.certResolver=dns-route53"
- "--entrypoints.https.http.tls.domains[0].main=*.${DOMAIN}"
- "--certificatesresolvers.dns-route53.acme.dnsChallenge=true"
- "--certificatesResolvers.dns-route53.acme.dnsChallenge.provider=route53"
- "--certificatesResolvers.dns-route53.acme.email=dns#${DOMAIN}"
- "--certificatesResolvers.dns-route53.acme.storage=/letsencrypt/acme.json"
ports:
- "80:80"
- "443:443"
volumes:
- "/var/run/docker.sock:/var/run/docker.sock:ro"
restart: always
I'm using:
Traefik version 2.6.6 built on 2022-05-03T16:58:48Z"
Docker version 20.10.14, build a224086 (all services as a single docker-compose)
AWS Route 53
This likely won't help anyone, but this specific problem was caused because AWS_REGION was set incorrectly.
Related
I'm trying to understand how to implement https with grafana/caddy in docker compose without a domain name.
Currently, I access grafana via http://xx.xxx.xx.xx:3000/
I would like this to be https, but am struggling to understand how to generate the cert and have it work as expected. I think letsencrypt requires a domain which I don't have.
version: "3"
networks:
monitor-net:
driver: bridge
volumes:
grafana_data: {}
services:
grafana:
image: grafana/grafana:8.4.4
container_name: grafana
volumes:
- grafana_data:/var/lib/grafana
- ./grafana/provisioning/dashboards:/etc/grafana/provisioning/dashboards
- ./grafana/provisioning/datasources:/etc/grafana/provisioning/datasources
environment:
- GF_SECURITY_ADMIN_USER=${GF_ADMIN_USER}
- GF_SECURITY_ADMIN_PASSWORD=${GF_ADMIN_PASS}
- GF_USERS_ALLOW_SIGN_UP=false
restart: unless-stopped
expose:
- 3000
networks:
- monitor-net
labels:
org.label-schema.group: "monitoring"
caddy:
image: caddy:2.3.0
container_name: caddy
ports:
- "3000:3000"
- "9090:9090"
- "9093:9093"
- "9091:9091"
volumes:
- ./caddy:/etc/caddy
environment:
- ADMIN_USER=${GF_ADMIN_USER}
- ADMIN_PASSWORD=${GF_ADMIN_PASS}
- ADMIN_PASSWORD_HASH=${ADMIN_PASS_HASH}
restart: unless-stopped
networks:
- monitor-net
labels:
org.label-schema.group: "monitoring"
I'm assuming I would create a volume on /etc/caddy/certs where I'd store the certificates, but don't know how to generate it for IP only or how it gets recognized by caddy.
Caddy for IP with SSL
By default, Caddy serves all sites over HTTPS.
Caddy serves IP addresses and local/internal hostnames over HTTPS using self-signed certificates that are automatically trusted locally (if permitted).
Examples: localhost, 127.0.0.1
Offical Docs Here
in your Caddyfile you have to add something like this
http://192.168.1.25:3000 {
reverse_proxy grafana_ip:3000
}
It looks like Caddy does not support generating HTTPS certificates for IP addresses. Additionally, Let's Encrypt does not currently support issuing certificates for bare IP addresses.
However, it does appear that ZeroSSL supports generating certificates for IPs. You could try using these instructions to change one or all of your sites to use ZeroSSL, but I wasn't able to get this to work on my test server.
The best option is probably to get a domain that you can point at your server, and then serve it from there.
I had the previous version (from last yr) of minio running well behind traefik and on portainer with no issues. I attempted an upgrade 4 days ago to the new version and it's been downhill since then. I get the login page but it won't accept credentials; Error:
{"code":500,"detailedMessage":"Post \"https://storage.example.com/\": dial tcp <ip>:443: i/o timeout","message":"invalid Login"}
I suspect it has something to do with the TLS certificate.
Below is my compose file, I've generated the certs as required; any assistance is is welcomed:
minio:
image: minio/minio
container_name: minio
restart: unless-stopped
command: server /data --certs-dir "./minio-data/certs" --address ":9000" --console-address ":9001"
networks:
- traefik-proxy2
expose:
- "9000"
- "9001"
volumes:
- ./minio-data:/data
environment:
- "MINIO_ROOT_USER=love"
- "MINIO_ROOT_PASSWORD=love1234"
- "MINIO_BROWSER_REDIRECT_URL=https://stash.example.com"
- "MINIO_SERVER_URL=https://storage.example.com"
labels:
- "traefik.enable=true"
- "traefik.http.services.minio.loadbalancer.server.port=9000"
- "traefik.http.routers.minio.rule=Host(`storage.example.com`)"
- "traefik.http.middlewares.minio-https-redirect.redirectscheme.scheme=https"
- "traefik.http.routers.minio.middlewares=minio-https-redirect"
- "traefik.http.routers.minio.entrypoints=https"
- "traefik.http.routers.minio.service=minio"
- "traefik.http.routers.minio.tls=true"
- "traefik.http.routers.minio.tls.certresolver=http"
- "traefik.http.services.minio-console.loadbalancer.server.port=9001"
- "traefik.http.routers.minio-console.rule=Host(`stash.example.com`)"
I was getting the same Invalid Login error: Post "http://minio.localhost/": dial tcp: lookup minio.localhost on 127.0.0.11:53: no such host.
The following docker-compose.yml works. The solution is NOT to use MINIO_DOMAIN or MINIO_SERVER_URL. See # comments at environment: variables.
version: "3.3"
services:
minio:
# Please use fixed versions :D
image: minio/minio:RELEASE.2021-10-06T23-36-31Z
networks:
- traefik-proxy2
volumes:
- minio-data:/data
command:
- server
- /data
- --console-address
- ":9001"
environment:
- MINIO_ROOT_USER=love
- MINIO_ROOT_PASSWORD=love1234
# Do NOT use MINIO_DOMAIN or MINIO_SERVER_URL with Traefik.
# All Routing is done by Traefik, just tell minio where to redirect to.
- MINIO_BROWSER_REDIRECT_URL=http://stash.localhost
deploy:
labels:
- traefik.enable=true
- traefik.docker.network=traefik-proxy2
- traefik.constraint-label=traefik-proxy2
- traefik.http.routers.minio.service=minio
- traefik.http.routers.minio.rule=Host(`storage.localhost`)
- traefik.http.services.minio.loadbalancer.server.port=9000
- traefik.http.routers.minio-console.service=minio-console
- traefik.http.routers.minio-console.rule=Host(`stash.localhost`)
- traefik.http.services.minio-console.loadbalancer.server.port=9001
volumes:
minio-data:
networks:
traefik-proxy2:
external: true
I'm trying to deploy a React + FastApi + Postgres application on docker compose with Traefik as the reverse proxy. I'm running into issues with Bad Gateway errors. Running my FastAPI locally runs it on port 8888 and exposes the path /docs to view the api documentation. I'd like to eventually have the application running on example.local with the docs available on example.local/api/docs. My docker-compose.yaml is as follows (loosely based on this one):
version: '3.8'
services:
proxy:
image: traefik:v2.4
networks:
- web
volumes:
- /var/run/docker.sock:/var/run/docker.sock
ports:
- '80:80'
- '8080:8080'
- '443:443'
command:
- --providers.docker
- --api.insecure=true
- --providers.docker.exposedbydefault=false
- --providers.docker.network=web
- --entrypoints.web.address=:80
labels:
- traefik.enable=true
- traefik.http.routers.example-proxy-http.rule=Host(`example.local`)
- traefik.http.routers.example-proxy-http.entrypoints=web
- traefik.http.services.example-proxy.loadbalancer.server.port=80
backend:
build:
context: ./backend
dockerfile: Dockerfile
command: python app/main.py
volumes:
- ./backend/app:/app
env_file:
- .env
networks:
- web
- backend
labels:
- traefik.enable=true
- traefik.http.routers.example-backend-http.rule=PathPrefix(`api/docs`)
- traefik.http.routers.example-backend-http.entrypoints=web
- traefik.http.services.example-backend.loadbalancer.server.port=8888
networks:
web:
external: true
backend:
external: false
I've added 127.0.0.1 example.local to my /etc/hosts file.
From reading around it seems like Bad Gateway errors tend to occur from traefik and related services not being on the same network, or traefik routing traffic to the wrong port on the service container. However if I set ports: - '8888:8888' in my backend service I can access the docs from localhost:8888/docs so I'm pretty sure 8888 is the correct port for the backend loadbalancer. From what I can see traefik and the backend service are on the same network too and I've set it as the default traefik network with --providers.docker.network=web. Interestingly if I visit localhost/api/docs in my browser I'm served up a page from FastAPI. So it could be an issue with my traefik http router labels? I'm quite new to traefik and proxies so would appreciate any help or guidance, thanks!
UPDATE
If I specify the host for the backend by adding
- traefik.http.routers.infilmation-backend-http.rule=Host(`example.local`) && PathPrefix(`/docs`)
to the backend service labels, then visiting example.local/docs does serve up page from FastApi. So I guess my question would be what is the best way of setting up a host for this application? Is there a way I can specify a default host for all services then any PathPrefix rules would be in relation to that host?
I'm running the basic TLS challenge docker example:
version: "3.3"
services:
traefik:
image: "traefik:v2.4"
restart: always
command:
- "--log.level=DEBUG"
- "--api.insecure=true"
- "--providers.docker=true"
- "--providers.docker.exposedbydefault=false"
- "--entrypoints.websecure.address=:443"
- "--certificatesresolvers.mytlschallenge.acme.tlschallenge=true"
#- "--certificatesresolvers.myresolver.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory"
- "--certificatesresolvers.mytlschallenge.acme.email=${SSL_EMAIL}"
- "--certificatesresolvers.mytlschallenge.acme.storage=/letsencrypt/acme.json"
ports:
- "443:443"
- "8080:8080"
volumes:
- ${DATA_FOLDER}/letsencrypt:/letsencrypt
- /var/run/docker.sock:/var/run/docker.sock:ro
n8n:
image: mjysci/n8n:latest-rpi
restart: always
ports:
- "127.0.0.1:5678:5678"
labels:
- traefik.enable=true
- traefik.http.routers.n8n.rule=Host(`${SUBDOMAIN}.${DOMAIN_NAME}`)
- traefik.http.routers.n8n.tls=true
- traefik.http.routers.n8n.entrypoints=websecure
- traefik.http.routers.n8n.tls.certresolver=mytlschallenge
- traefik.http.middlewares.n8n.headers.SSLRedirect=true
- traefik.http.middlewares.n8n.headers.STSSeconds=315360000
- traefik.http.middlewares.n8n.headers.browserXSSFilter=true
- traefik.http.middlewares.n8n.headers.contentTypeNosniff=true
- traefik.http.middlewares.n8n.headers.forceSTSHeader=true
- traefik.http.middlewares.n8n.headers.SSLHost=${DOMAIN_NAME}
- traefik.http.middlewares.n8n.headers.STSIncludeSubdomains=true
- traefik.http.middlewares.n8n.headers.STSPreload=true
environment:
- N8N_BASIC_AUTH_ACTIVE=true
- N8N_BASIC_AUTH_USER
- N8N_BASIC_AUTH_PASSWORD
- N8N_HOST=${SUBDOMAIN}.${DOMAIN_NAME}
- N8N_PORT=5678
- N8N_PROTOCOL=https
- NODE_ENV=production
- WEBHOOK_TUNNEL_URL=https://${SUBDOMAIN}.${DOMAIN_NAME}/
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- ${DATA_FOLDER}/.n8n:/home/node/.n8n
But when I'm running it says:
level=error msg="Unable to obtain ACME certificate for domains \"<MY_DOMAIN>\": unable to generate a certificate for the domains [<MY_DOMAIN>]: error: one or more domains had a problem:\n[<MY_DOMAIN>] [<MY_DOMAIN>] acme: error presenting token: timeout 2021-02-01 10:09:04.491784271 +0000 UTC m=+378.657940910\n" providerName=mytlschallenge.acme routerName=n8n#docker rule="Host(`<MY_DOMAIN>`)"
In the browser, the application is available and works well with HTTPS but it says the certificate isn't valid (obviously). What could be wrong here?
i'm having the same problem on some domains.
It looks like its a timeout.
i tried to restart traefik now. same issue again.
ps: add traefik in the title of the thread, you may gain more visibility and answers.
EDIT: I reverted to version 2.3.7 and it works again.
there must be a bug in 2.4 (latest)
i created an issue :
https://github.com/traefik/traefik/issues/7848
I would like to serve a docker-compose service through traefik with port-forwarding. I had many tries and the best I could achieve from now is described below:
First I create two networks:
docker network create frontend # To expose traefik
docker network create backend # To bind backend services
The traefik configuration is about (development, dashboard enabled at :8080):
version: "3.6"
services:
proxy:
image: traefik:latest
restart: unless-stopped
command:
- "--log.level=DEBUG"
- "--api"
- "--api.dashboard"
- "--api.insecure"
- "--providers.docker"
- "--providers.docker.exposedbydefault=false"
# Service entrypoint:
- "--entrypoints.lora-server.address=:8090"
volumes:
- "/var/run/docker.sock:/var/run/docker.sock:ro"
labels:
- "traefik.enable=true"
- "traefik.docker.network=frontend"
- "traefik.http.routers.traefik.entrypoints=traefik"
- "traefik.http.routers.traefik.service=api#internal"
networks:
- backend
- frontend
ports:
- "8080:8080"
- "8090:8090"
networks:
frontend:
external: true
backend:
external: true
The backend service is described here (a fork from ChripStack but it could be anything else):
version: "3"
services:
# [...]
chirpstack-application-server:
image: chirpstack/chirpstack-application-server:3
networks:
- backend
volumes:
- ./configuration/chirpstack-application-server:/etc/chirpstack-application-server
labels:
- "traefik.enable=true"
- "traefik.docker.network=backend"
- "traefik.http.routers.chirpstack.entrypoints=lora-server"
- "traefik.http.routers.chirpstack.rule=Host(`{host:}`)"
- "traefik.http.routers.chirpstack.service=chirpstack-application-server#docker"
- "traefik.http.services.chirpstack-application-server.loadbalancer.server.port=8080"
# [...]
networks:
backend:
external: true
The service also natively run on :8080 and I would like to access it on :8090 through traefik.
When I run both applications, traefik registers the new service and does not complain (no errors, no warning, the flow seems complete at least from the dashboard interface).
time="2020-07-30T11:47:47Z" level=debug msg="Creating middleware" middlewareType=Pipelining serviceName=chirpstack-application-server#docker entryPointName=lora-server routerName=chirpstack#docker middlewareName=pipelining
time="2020-07-30T11:47:47Z" level=debug msg="Creating load-balancer" entryPointName=lora-server routerName=chirpstack#docker serviceName=chirpstack-application-server#docker
time="2020-07-30T11:47:47Z" level=debug msg="Creating server 0 http://192.168.112.9:8080" entryPointName=lora-server routerName=chirpstack#docker serviceName=chirpstack-application-server#docker serverName=0
time="2020-07-30T11:47:47Z" level=debug msg="Added outgoing tracing middleware chirpstack-application-server#docker" middlewareName=tracing entryPointName=lora-server routerName=chirpstack#docker middlewareType=TracingForwarder
But I could not access the service, I am geting 404 errors when I try to connect http://host:8090.
To my understanding, it seems traefik does not know how it should complete the flow between the two networks: http://frontend:8090 -> http://backend:8080 (because I haven't referenced it anywhere).
What should I change in my configuration to make it work? How can I specify to traefik that it must route the HTTP traffic from frontend:8090 to backend:8080? Your help is much appreciated.
Traefik listens on some port, that's the only thing defined with entrypoints. You don't specify network for this incoming requests, because it's unrelated. traefik.docker.network is only used for routing after Traefik handles the incoming requests.
So the correct flow is not http://frontend:8090 -> http://backend:8080, but http://host:8090 -> http://backend:8080.
I think your configuration is correct. The only thing which seems to be odd is usage of Host rule. Maybe try to change it to something like this:
"traefik.http.routers.chirpstack.rule=hostregexp(`{host:.+}`)"