TLS challenge with docker-compose: acme: error presenting token: timeout - docker-compose

I'm running the basic TLS challenge docker example:
version: "3.3"
services:
traefik:
image: "traefik:v2.4"
restart: always
command:
- "--log.level=DEBUG"
- "--api.insecure=true"
- "--providers.docker=true"
- "--providers.docker.exposedbydefault=false"
- "--entrypoints.websecure.address=:443"
- "--certificatesresolvers.mytlschallenge.acme.tlschallenge=true"
#- "--certificatesresolvers.myresolver.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory"
- "--certificatesresolvers.mytlschallenge.acme.email=${SSL_EMAIL}"
- "--certificatesresolvers.mytlschallenge.acme.storage=/letsencrypt/acme.json"
ports:
- "443:443"
- "8080:8080"
volumes:
- ${DATA_FOLDER}/letsencrypt:/letsencrypt
- /var/run/docker.sock:/var/run/docker.sock:ro
n8n:
image: mjysci/n8n:latest-rpi
restart: always
ports:
- "127.0.0.1:5678:5678"
labels:
- traefik.enable=true
- traefik.http.routers.n8n.rule=Host(`${SUBDOMAIN}.${DOMAIN_NAME}`)
- traefik.http.routers.n8n.tls=true
- traefik.http.routers.n8n.entrypoints=websecure
- traefik.http.routers.n8n.tls.certresolver=mytlschallenge
- traefik.http.middlewares.n8n.headers.SSLRedirect=true
- traefik.http.middlewares.n8n.headers.STSSeconds=315360000
- traefik.http.middlewares.n8n.headers.browserXSSFilter=true
- traefik.http.middlewares.n8n.headers.contentTypeNosniff=true
- traefik.http.middlewares.n8n.headers.forceSTSHeader=true
- traefik.http.middlewares.n8n.headers.SSLHost=${DOMAIN_NAME}
- traefik.http.middlewares.n8n.headers.STSIncludeSubdomains=true
- traefik.http.middlewares.n8n.headers.STSPreload=true
environment:
- N8N_BASIC_AUTH_ACTIVE=true
- N8N_BASIC_AUTH_USER
- N8N_BASIC_AUTH_PASSWORD
- N8N_HOST=${SUBDOMAIN}.${DOMAIN_NAME}
- N8N_PORT=5678
- N8N_PROTOCOL=https
- NODE_ENV=production
- WEBHOOK_TUNNEL_URL=https://${SUBDOMAIN}.${DOMAIN_NAME}/
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- ${DATA_FOLDER}/.n8n:/home/node/.n8n
But when I'm running it says:
level=error msg="Unable to obtain ACME certificate for domains \"<MY_DOMAIN>\": unable to generate a certificate for the domains [<MY_DOMAIN>]: error: one or more domains had a problem:\n[<MY_DOMAIN>] [<MY_DOMAIN>] acme: error presenting token: timeout 2021-02-01 10:09:04.491784271 +0000 UTC m=+378.657940910\n" providerName=mytlschallenge.acme routerName=n8n#docker rule="Host(`<MY_DOMAIN>`)"
In the browser, the application is available and works well with HTTPS but it says the certificate isn't valid (obviously). What could be wrong here?

i'm having the same problem on some domains.
It looks like its a timeout.
i tried to restart traefik now. same issue again.
ps: add traefik in the title of the thread, you may gain more visibility and answers.
EDIT: I reverted to version 2.3.7 and it works again.
there must be a bug in 2.4 (latest)
i created an issue :
https://github.com/traefik/traefik/issues/7848

Related

Minio Buckets not working behind Traefik reverse-proxy

I have a minio docker service running, which is connectable on storage/console.
My traefik also works for this.
But I suspect the connections to use the pattern BUCKET.backup.lo.domain.com which leads to a 404 from traefik.
I clearly see this pattern, for example, when using Cyberduck to connect (in the traefik logs and Cyberduck itself). The connection itself is possible (backup.lo.domain.com). I also get the buckets listed. But as soon as I click on the bucket it shows a modal with the bucket.lo.domain.com pattern and the traefik default certificate.
version: "3.8"
volumes:
minio-data:
services:
minio:
container_name: minio-backup
image: quay.io/minio/minio:RELEASE.2022-01-08T03-11-54Z
networks:
- traefik
volumes:
- minio-data:/data
command:
- server
- /data
- --console-address
- ":9001"
environment:
- TZ=${TIME_ZONE}
- MINIO_ROOT_USER=root
- MINIO_ROOT_PASSWORD=password
- MINIO_BROWSER_REDIRECT_URL=https://backup-console.lo.domain.com
- MINIO_DOMAIN=https://backup.lo.domain.com
labels:
- traefik.enable=true
- traefik.docker.network=traefik
- traefik.http.routers.minio.service=minio
- traefik.http.routers.minio.rule=Host(`backup.lo.domain.com`)
- traefik.http.routers.minio.tls.certresolver=letsenc
- traefik.http.routers.minio.entrypoints=websecure
- traefik.http.services.minio.loadbalancer.server.port=9000
- "traefik.http.routers.minio-console.service=minio-console"
- "traefik.http.routers.minio-console.rule=Host(`backup-console.lo.domain.com`)"
- "traefik.http.routers.minio-console.entrypoints=websecure"
- "traefik.http.routers.minio-console.tls.certresolver=letsenc"
- "traefik.http.services.minio-console.loadbalancer.server.port=9001"
restart: unless-stopped
traefik:
image: traefik
container_name: traefik
command:
- --providers.docker=true
- --entryPoints.web.address=:80
- --entryPoints.websecure.address=:443
labels:
- "traefik.http.routers.traefik.tls=true"
- "traefik.http.routers.traefik.entrypoints=websecure"
- "traefik.http.routers.traefik.tls.certresolver=letsenc"
- "traefik.http.routers.traefik.service=api#internal"
- "traefik.http.routers.traefik.tls.domains[0].main=lo.domain.com"
- "traefik.http.routers.traefik.tls.domains[0].sans=*.lo.domain.com"
volumes:
- "/var/run/docker.sock:/var/run/docker.sock:ro"
restart: unless-stopped
networks:
traefik:
networks:
traefik_public:
Any ideas? Could I use something like wildcards for subdomains?
The problem here is that buckets do not have sub-domain DNS entries in your setup. If you disable this and use path-style requests things should work fine: https://docs.cyberduck.io/protocols/s3/#disable-use-of-virtual-host-style-requests
Specifically, you need to set s3.bucket.virtualhost.disable to true in Cyberbuck.

Minio install behind Traefik

I had the previous version (from last yr) of minio running well behind traefik and on portainer with no issues. I attempted an upgrade 4 days ago to the new version and it's been downhill since then. I get the login page but it won't accept credentials; Error:
{"code":500,"detailedMessage":"Post \"https://storage.example.com/\": dial tcp <ip>:443: i/o timeout","message":"invalid Login"}
I suspect it has something to do with the TLS certificate.
Below is my compose file, I've generated the certs as required; any assistance is is welcomed:
minio:
image: minio/minio
container_name: minio
restart: unless-stopped
command: server /data --certs-dir "./minio-data/certs" --address ":9000" --console-address ":9001"
networks:
- traefik-proxy2
expose:
- "9000"
- "9001"
volumes:
- ./minio-data:/data
environment:
- "MINIO_ROOT_USER=love"
- "MINIO_ROOT_PASSWORD=love1234"
- "MINIO_BROWSER_REDIRECT_URL=https://stash.example.com"
- "MINIO_SERVER_URL=https://storage.example.com"
labels:
- "traefik.enable=true"
- "traefik.http.services.minio.loadbalancer.server.port=9000"
- "traefik.http.routers.minio.rule=Host(`storage.example.com`)"
- "traefik.http.middlewares.minio-https-redirect.redirectscheme.scheme=https"
- "traefik.http.routers.minio.middlewares=minio-https-redirect"
- "traefik.http.routers.minio.entrypoints=https"
- "traefik.http.routers.minio.service=minio"
- "traefik.http.routers.minio.tls=true"
- "traefik.http.routers.minio.tls.certresolver=http"
- "traefik.http.services.minio-console.loadbalancer.server.port=9001"
- "traefik.http.routers.minio-console.rule=Host(`stash.example.com`)"
I was getting the same Invalid Login error: Post "http://minio.localhost/": dial tcp: lookup minio.localhost on 127.0.0.11:53: no such host.
The following docker-compose.yml works. The solution is NOT to use MINIO_DOMAIN or MINIO_SERVER_URL. See # comments at environment: variables.
version: "3.3"
services:
minio:
# Please use fixed versions :D
image: minio/minio:RELEASE.2021-10-06T23-36-31Z
networks:
- traefik-proxy2
volumes:
- minio-data:/data
command:
- server
- /data
- --console-address
- ":9001"
environment:
- MINIO_ROOT_USER=love
- MINIO_ROOT_PASSWORD=love1234
# Do NOT use MINIO_DOMAIN or MINIO_SERVER_URL with Traefik.
# All Routing is done by Traefik, just tell minio where to redirect to.
- MINIO_BROWSER_REDIRECT_URL=http://stash.localhost
deploy:
labels:
- traefik.enable=true
- traefik.docker.network=traefik-proxy2
- traefik.constraint-label=traefik-proxy2
- traefik.http.routers.minio.service=minio
- traefik.http.routers.minio.rule=Host(`storage.localhost`)
- traefik.http.services.minio.loadbalancer.server.port=9000
- traefik.http.routers.minio-console.service=minio-console
- traefik.http.routers.minio-console.rule=Host(`stash.localhost`)
- traefik.http.services.minio-console.loadbalancer.server.port=9001
volumes:
minio-data:
networks:
traefik-proxy2:
external: true

Let's Encrypt with Traefik - JWS has an invalid anti-replay nonce

I have this composer file, it was working properly, but 2 days ago I added a new domain and it fails to get the Let's encrypt certficate,
version: '3.4'
services:
traefik:
image: "traefik:v2.4"
container_name: "traefik"
command:
- "--log.level=DEBUG"
- "--api.insecure=true"
- "--providers.docker=true"
- "--providers.docker.exposedbydefault=false"
- "--entrypoints.web.address=:80"
- "--entrypoints.web.http.redirections.entryPoint.to=websecure"
- "--entrypoints.web.http.redirections.entryPoint.scheme=https"
- "--entrypoints.web.http.redirections.entrypoint.permanent=true"
- "--entrypoints.websecure.address=:443"
- "--certificatesresolvers.myresolver.acme.dnschallenge=true"
- "--certificatesresolvers.myresolver.acme.dnschallenge.provider=cloudns"
- "--certificatesresolvers.myresolver.acme.email=myemail#gmail.com"
- "--certificatesresolvers.myresolver.acme.storage=/letsencrypt/acme.json"
labels:
- "com.centurylinklabs.watchtower.enable=false"
ports:
- "80:80"
- "443:443"
environment:
- "CLOUDNS_AUTH_ID=xxxx"
- "CLOUDNS_AUTH_PASSWORD=xxxx"
volumes:
- type: volume
source: traefik
target: /letsencrypt
- "/var/run/docker.sock:/var/run/docker.sock:ro"
networks:
- "organic"
php-apache:
image: "php7.4"
restart: "unless-stopped"
container_name: "php-apache"
labels:
- "traefik.enable=true"
- "traefik.http.routers.exampledomain.rule=Host(`example.com`)"
- "traefik.http.routers.exampledomain.entrypoints=websecure"
- "traefik.http.routers.exampledomain.tls.certresolver=myresolver"
- "com.centurylinklabs.watchtower.enable=false"
networks:
- "organic"
networks:
organic:
driver: bridge
Here are the errors I'm getting
traefik | time="2021-04-17T03:31:02Z" level=debug msg="legolog: [INFO] retry due to: acme: error: 400 :: POST :: :: urn:ietf:params:acme:error:badNonce :: JWS has an invalid anti-replay nonce:
traefik | time="2021-04-17T03:31:02Z" level=error msg="Unable to obtain ACME certificate for domains \"example.com\": unable to generate a certificate for thedomains [example.com]: error: one or more domains had a problem:\n[example.com] [example.com] acme: error presenting token: time limit exceeded: last error: %!w(<nil>)\n" providerName=myresolver.acme rule="Host(`example.com`)" routerName=exampledomain#docker
The tx is generated successfully in ClouDNS is normal, I have spent two days trying to figure out what might be the problem with no luck.
I had the same problem recently and after many configuration tries I deleted the traefik container (not recreate) as well as the letsencrypt volume. Then I redeployed with docker-compose up -d. On startup of traefik new certificates there generated immediately and without any complains. Hope this helps solves your issue as well.
traefik:
image: "traefik:v2.4"
container_name: "traefik"
restart: always
command:
#- "--log.level=DEBUG"
- "--providers.docker=true"
- "--providers.docker.exposedbydefault=false"
- "--providers.docker.network=traefik_proxy"
- "--providers.docker.endpoint=tcp://docker-socket-proxy:2375"
- "--ping=true"
- "--ping.entryPoint=ping"
- "--certificatesresolvers.letsencrypt.acme.dnschallenge=true"
- "--certificatesresolvers.letsencrypt.acme.dnschallenge.provider=cloudflare"
- "--certificatesresolvers.letsencrypt.acme.email=${CERTIFICATE_EMAIL}"
- "--certificatesresolvers.letsencrypt.acme.storage=/letsencrypt/acme.json"
- "--certificatesResolvers.letsencrypt.acme.dnsChallenge.resolvers=1.1.1.1:53,8.8.8.8:53"
- "--entryPoints.ping.address=:8081"
- "--entrypoints.http.address=:80"
- "--entrypoints.https.address=:443"
- "--entrypoints.https.http.tls.certresolver=letsencrypt"
- "--entrypoints.https.http.tls.domains[0].main=${DOMAINNAME}"
- "--entrypoints.https.http.tls.domains[0].sans=*.${DOMAINNAME}"
environment:
- CF_DNS_API_TOKEN=${CLOUDFLARE_DNS_API_TOKEN}
healthcheck:
test: ["CMD", "wget", "-c", "http://localhost:8081/ping"]
retries: 3
interval: "15s"
timeout: "3s"
start_period: "15s"
networks:
- traefik_proxy
- socket-proxy
security_opt:
- no-new-privileges:true
ports:
- target: 80
published: 80
protocol: tcp
mode: host
- target: 443
published: 443
protocol: tcp
mode: host
volumes:
- "letsencrypt:/letsencrypt"
#- "/var/run/docker.sock:/var/run/docker.sock:ro"
labels:
- "com.centurylinklabs.watchtower.enable=true"
depends_on:
- docker-socket-proxy

Identity Server4: Unable to obtain configuration from 'http://localhost:44338/.well-known/openid-configuration

I'm working on microservices in .Net core 3.1 with docker-compose. we are using Linux based Docker containers. MVC Client application is perfectly running in IIS express along with Identity 4 API, when I run microservices in docker-compose then I am getting error in the client application.
{"StatusCode":500,"Message":"Internal Server Error."}
In the MVC client container, I checked the container's log. I got the below error.
Unable to obtain configuration from 'http://localhost:44338/.well-known/openid-configuration'
this above URL is representing Identity Server 4 that is also running in a container.
The sequence of my docker-compose is
version: '3.4'
services:
identity.api:
environment:
- ASPNETCORE_URLS=https://+:443;http://+:80
- MvcClient=http://localhost:44353
ports:
- "44338:80"
- "443"
volumes:
- ${APPDATA}/Microsoft/UserSecrets:/root/.microsoft/usersecrets:ro
- ${APPDATA}/ASP.NET/Https:/root/.aspnet/https:ro
rinmvc:
environment:
- ASPNETCORE_ENVIRONMENT=Development
- ASPNETCORE_URLS=https://+:443;http://+:80;
- ASPNETCORE_URLS=http://0.0.0.0:80
- IdentityUrl=http://localhost:44338
ports:
- "44353:80"
- "443"
volumes:
- ${APPDATA}/Microsoft/UserSecrets:/root/.microsoft/usersecrets:ro
- ${APPDATA}/ASP.NET/Https:/root/.aspnet/https:ro
After spending few days on this issue, I got the correct solution.
I've modified my docker-compose.override file as below:
version: '3.4'
services:
rabbitmq:
ports:
- "15672:15672"
- "5672:5672"
identity.api:
environment:
- ASPNETCORE_ENVIRONMENT=Development
- ASPNETCORE_URLS=http://+:80
- MvcClient=http://rinmvc
ports:
- "44338:80"
volumes:
- ${APPDATA}/Microsoft/UserSecrets:/root/.microsoft/usersecrets:ro
- ${APPDATA}/ASP.NET/Https:/root/.aspnet/https:ro
rinmvc:
environment:
- ASPNETCORE_ENVIRONMENT=Development
- ASPNETCORE_URLS=http://+:80
- IdentityUrl=http://identity.api
ports:
- "44353:80"
volumes:
- ${APPDATA}/Microsoft/UserSecrets:/root/.microsoft/usersecrets:ro
- ${APPDATA}/ASP.NET/Https:/root/.aspnet/https:ro

Traefik: Simple Letsencrypt HTTPS redirect to whoami service throws "404 page not found"

I've tried to get this up and running for two days now and some simple HTTP -> HTTPs redirect does not work! :(
Pretty simple use case:
whoami.my-example-domain.com:80 => redirect to whoami.my-example-domain.com:443 and then traefik internally redirects to :80 of my whoami service docker container.
Here's the docker-compose.yml
version: "3"
services:
reverse-proxy:
image: traefik:alpine
command:
- --logLevel=WARN
- --defaultentrypoints=http,https
- --entrypoints=Name:http Address::80 Redirect.EntryPoint:https
- --entrypoints=Name:https Address::443 TLS
- --acme
- --acme.email=myemail#gmail.com
- --acme.storage=acme.json
- --acme.entryPoint=https
- --acme.httpChallenge.entryPoint=http
- --acme.OnHostRule=true
- --acme.onDemand=false
- --acme.acmeLogging=true
- --docker
- --docker.watch
- --docker.exposedbydefault=false
- --docker.domain=docker.localhost
restart: always
networks:
- web
ports:
- "80:80" # The HTTP port
- "443:443" # The HTTPS port
volumes:
- /var/run/docker.sock:/var/run/docker.sock # So that Traefik can listen to the Docker events
- /opt/data/traefik/acme.json:/acme.json
whoami:
image: containous/whoami # A container that exposes an API to show its IP address
labels:
- "traefik.enable=true"
- "traefik.frontend.rule=Host:whoami.some-example-domain.com"
- "traefik.port=80"
- "traefik.frontend.entryPoints=http"
networks:
web:
external: true
When I now call http://whoami.some-example-domain.com (this is just a demo domain and won't work) => it redirects to HTTPs... which is cool, but then it throws the famous "404 page not found" traefik standard error.
If already tried to set the following labels to the container:
"traefik.port=80"
"traefik.frontend.entryPoints=http"
That didn't work either.
Any help would be appreciated! Thanks in advance!
Regards,
Sascha
You have to remove traefik.frontend.entryPoints (linked to defaultentrypoints) or use traefik.frontend.entryPoints=http,https
version: "3"
services:
reverse-proxy:
image: traefik:v1.7.8
command:
- --logLevel=WARN
- --defaultentrypoints=http,https
- --entrypoints=Name:http Address::80 Redirect.EntryPoint:https
- --entrypoints=Name:https Address::443 TLS
- --acme
- --acme.email=myemail#gmail.com
- --acme.storage=acme.json
- --acme.entryPoint=https
- --acme.httpChallenge.entryPoint=http
- --acme.OnHostRule=true
- --acme.onDemand=false
- --acme.acmeLogging=true
- --docker
- --docker.exposedbydefault=false
- --docker.domain=some-example-domain.com
restart: always
networks:
- web
ports:
- "80:80" # The HTTP port
- "443:443" # The HTTPS port
volumes:
- /var/run/docker.sock:/var/run/docker.sock # So that Traefik can listen to the Docker events
- /opt/data/traefik/acme.json:/acme.json
whoami:
image: containous/whoami # A container that exposes an API to show its IP address
labels:
- "traefik.enable=true"
- "traefik.frontend.rule=Host:some-example-domain.com"
networks:
- web
networks:
web:
external: true