Problem getting certificate from let's encrypt using Traefik with docker - docker-compose

I've set up Traefik with Docker and a service behind it. The basic setup works. I can browse to port 80 using the domain name I'm redirected to https and then see "invalid certificate" - since the let's encrypt part is broken.
[ router ] <-:80/:443-> [linux/docker [Traefik:80/:443][Service:8080]]
Here is the entry in the log (edited domain.)
Logs:
acme: Error -> One or more domains had a problem:\n[xyz.example.net] acme: error: 400 :: urn:ietf:params:acme:error:connection :: Fetching
http://xyz.example.net/.well-known/acme-challenge/eIAFZqaGMHMWaBjINjzk4m8PuWiYfuCHCTnSU9M:
Error getting validation data, url: \n"
The error message is accurate, I can not browse to that URL. I have noticed that I can go to that URL using the internal IP http://10.0.0.21/.well-known/acme-challenge/key and Traefik responds with this in the log:
traefik | time="2019-05-28T21:20:52Z" level=error msg="Error getting challenge for token retrying in 542.914495ms"
I suspect the problem is the domain name redirect setup. My service is at xyz.example.net (and so is Traefik.) I suspect the problem is that Traefik is redirecting all traffic coming in on xyz.example.net:80/:443 to the service, and not handling the ./well-known/acme-challenge itself. Do I need to give the gateway itself a name? (E.g. zzz.example.net is Traefik and xyz.example.net is the service?)
How can I fix this?
My TOML file:
debug = false
logLevel = "ERROR" #DEBUG, INFO, WARN, ERROR, FATAL, PANIC
InsecureSkipVerify = true
defaultEntryPoints = ["https", "http"]
[entryPoints]
[entryPoints.http]
[entryPoints.http.redirect]
entryPoint = "https"
[entryPoints.https]
address = ":443"
[entryPoints.https.tls]
[retry]
[docker]
endpoint = "unix:///var/run/docker.sock"
domain = "example.net"
watch = true
exposedbydefault = false
[acme]
email = "me#example.net"
storage = "acme.json"
entryPoint = "https"
onDemand = false
onHostRule = true
[acme.httpChallenge]
entryPoint = "http"

I noticed that although the internal IP 192.xxx worked - the external IP did not. Of course, I would say, this seems like a firewall problem - BUT - the firewall lets thru traffic just fine for the services that I was testing, so I was confused.
The solution? port 80 was not being forwarded on the firewall, 443 was. So when I tried testing with curl/browser I was typing in https://xyz.example.com - and that was working.

Related

Nginx ingress with grpc, missing :te header

Trying to setup nginx grpc application in GKE following this (https://kubernetes.github.io/ingress-nginx/examples/grpc) tutorial but getting the following error when I add the nginx.ingress.kubernetes.io/backend-protocol: "GRPC" annotation.
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNKNOWN
details = "Missing :te header"
debug_error_string = "UNKNOWN:Error received from peer ipv4:34.149.11.89:443 {created_time:"2023-01-28T12:32:39.285586-08:00", grpc_status:2, grpc_message:"Missing :te header"}"

Traefik 2.4: redirect http to https based on IP / Header

Im using traefik with a global redirect from http to https, using this commands in docker-compose file
- "--entrypoints.web.http.redirections.entryPoint.to=websecure"
- "--entrypoints.web.http.redirections.entryPoint.scheme=https"
- "--entrypoints.web.http.redirections.entrypoint.permanent=true"
Which works just fine.
Now, is there a possiblity to make this global redirect rule-based? I want to redirect all traffic from http to https, but if a specific header is presented or the request comes from a specific IP, the redirect should not take place.
I had a similar need to redirect based on source IP. The solution has 3 parts:
Rule to match the X-Real-IP Header,
Set a higher priority and
Optionally use middleware to exclude IPs that should not match.
I use YAML format with the file provider. The priority is based on the rule length so you'll want to make it larger to override long rules. The private-ips middleware should not be reached and is there mostly for testing.
# Private service
[http.routers.example-private]
priority = 100
entrypoints = [ "web-secure" ]
service = "example-private"
rule = "Host(`example.com`) && (Headers(`X-Real-Ip`, `127.0.0.1`) || Headers(`My-Header`, `some-value`))"
middlewares = [ "private-ips#file" ]
# Public service
[http.routers.example]
priority = 10
entrypoints = [ "web-secure" ]
service = "example"
rule = "Host(`example.com`)"
middlewares = [ "redirect-https#file" ]
# Private IPs
[http.middlewares.private-ips.ipWhiteList]
sourceRange = [
"127.0.0.1/32", # localhost
]
# Permanent redirect to HTTPS.
[http.middlewares.redirect-https.redirectScheme]
scheme = "https"
permanent = true

failed to run jupyterhub when changing default port

I'm trying to set up jupyterhub. The 8000 is used for a different program, so I have to use a different port.
I change the file /etc/jupyterhub/jupyterhub_config.py add/uncomments:
c.JupyterHub.hub_port = 9003
c.JupyterHub.ip = '111.111.11.1'
c.JupyterHub.port = 9002
c.ConfigurableHTTPProxy.api_url = 'http://127.0.0.1:9000'
when I tried to running jupyterhub, I got the error:
[W 2020-06-03 14:48:48.930 JupyterHub proxy:554] Stopped proxy at pid=47639
[W 2020-06-03 14:48:48.932 JupyterHub proxy:643] Running JupyterHub without SSL. I hope there is SSL termination happening somewhere else...
[I 2020-06-03 14:48:48.932 JupyterHub proxy:646] Starting proxy # http://111.111.11.1:9002/
14:48:49.301 [ConfigProxy] info: Proxying http://111.111.11.1:9002 to (no default)
14:48:49.307 [ConfigProxy] info: Proxy API at http://127.0.0.1:9000/api/routes
14:48:49.315 [ConfigProxy] error: Uncaught Exception
[E 2020-06-03 14:48:49.437 JupyterHub app:2718]
Traceback (most recent call last):
File "/home/user/miniconda/2020.02/python/3.7/lib/python3.7/site-packages/jupyterhub/app.py", line 2716, in launch_instance_async
await self.start()
File "/home/user/miniconda/2020.02/python/3.7/lib/python3.7/site-packages/jupyterhub/app.py", line 2524, in start
await self.proxy.get_all_routes()
File "/home/user/miniconda/2020.02/python/3.7/lib/python3.7/site-pack#c.JupyterHub.hub_ip = '127.0.0.1'
ages/jupyterhub/proxy.py", line 806, in get_all_routes
resp = await self.api_request('', client=client)
File "/home/user/miniconda/2020.02/python/3.7/lib/python3.7/site-packages/jupyterhub/proxy.py", line 774, in api_request
result = await client.fetch(req)
tornado.httpclient.HTTPClientError: HTTP 403: Forbidden
What is the correct way to install jupyterhub on a port other than 8000?
Thanks.
I think some of these parameters are now obsolete, so it may depend which version you are running, but I'll assume JupyterHub 1.0+.
There are a few different services that make up JupyterHub, and the 'hub' service, confusingly, as not actually the one you are concerned with. The proxy is the main entrypoint to the application, and it proxies traffic to the hub by default, and to specific user Jupyter servers if the traffic is to a /user/ URL.
In addition, the 'hub' service also has an API endpoint that user servers can access directly (this doesn't go through the proxy). And the proxy has an extra API endpoint too, for direct access from the hub...
It is the proxy service that defaults to port 8000. To change to 80, for example try this:
## The public facing URL of the whole JupyterHub application.
#
# This is the address on which the proxy will bind. Sets protocol, ip, base_url
c.JupyterHub.bind_url = 'https://0.0.0.0:80'

Error creating TLS config after updating Traefik to v1.3.6

I'm attempting to update from Traefik v1.2.3 to v1.3.6 on Kubernetes. I have my TLS certificates mounted inside of the pods from secrets. Under v1.2.3, everything works as expected. When I try to apply my v1.3.6 deployment (only change being the new docker image), the pods fail to start with the following message:
time="2017-08-22T20:27:44Z" level=error msg="Error creating TLS config: tls: failed to find any PEM data in key input"
time="2017-08-22T20:27:44Z" level=fatal msg="Error preparing server: tls: failed to find any PEM data in key input"
Below is my traefik.toml file:
defaultEntryPoints = ["http","https"]
[entryPoints]
[entryPoints.http]
address = ":80"
[entryPoints.http.redirect]
entryPoint = "https"
[entryPoints.https]
address = ":443"
[entryPoints.https.tls]
[[entryPoints.https.tls.certificates]]
CertFile = "/ssl/wildcard.foo.mydomain.com.crt"
KeyFile = "/ssl/wildcard.foo.mydomain.com.key"
[[entryPoints.https.tls.certificates]]
CertFile = "/ssl/wildcard.mydomain.com.crt"
KeyFile = "/ssl/wildcard.mydomain.com.key"
[[entryPoints.https.tls.certificates]]
CertFile = "/ssl/wildcard.local.crt"
KeyFile = "/ssl/wildcard.local.key"
[kubernetes]
labelselector = "expose=internal"
My initial impression of the errors produced by the pods are that the keys in the secret are not valid. However, I am able to base64 decode the contents of the secret and see that the values match those of the certificate files I have stored locally. Additionally, I would expect to see this error on any version of Traefik if these were in fact, invalid. In reviewing the change log for Traefik, I see that the SSL library was updated but the related PR indicates that this only added ciphers and did not remove any previously supported.
:Edit w/ additional info:
Running with --logLevel=DEBUG provides this additional information (provided in full below in case it's helpful):
[cluster-traefik-2693375319-w67hf] time="2017-08-22T21:41:19Z" level=debug msg="Global configuration loaded {"GraceTimeOut":10000000000,"Debug":false,"CheckNewVersion":true,"AccessLogsFile":"","TraefikLogsFile":"","LogLevel":"DEBUG","EntryPoints":{"http":{"Network":"","Address":":80","TLS":null,"Redirect":{"EntryPoint":"https","Regex":"","Replacement":""},"Auth":null,"Compress":false},"https":{"Network":"","Address":":443","TLS":{"MinVersion":"","CipherSuites":null,"Certificates":[{"CertFile":"/ssl/wildcard.foo.mydomain.com.crt","KeyFile":"/ssl/wildcard.foo.mydomain.com.key"},{"CertFile":"/ssl/wildcard.mydomain.com.crt","KeyFile":"/ssl/wildcard.mydomain.com.key"},{"CertFile":"/ssl/wildcard.local.crt","KeyFile":"/ssl/wildcard.local.key"}],"ClientCAFiles":null},"Redirect":null,"Auth":null,"Compress":false}},"Cluster":null,"Constraints":[],"ACME":null,"DefaultEntryPoints":["http","https"],"ProvidersThrottleDuration":2000000000,"MaxIdleConnsPerHost":200,"IdleTimeout":180000000000,"InsecureSkipVerify":false,"Retry":null,"HealthCheck":{"Interval":30000000000},"Docker":null,"File":null,"Web":{"Address":":8080","CertFile":"","KeyFile":"","ReadOnly":false,"Statistics":null,"Metrics":{"Prometheus":{"Buckets":[0.1,0.3,1.2,5]}},"Path":"","Auth":null},"Marathon":null,"Consul":null,"ConsulCatalog":null,"Etcd":null,"Zookeeper":null,"Boltdb":null,"Kubernetes":{"Watch":true,"Filename":"","Constraints":[],"Endpoint":"","Token":"","CertAuthFilePath":"","DisablePassHostHeaders":false,"Namespaces":null,"LabelSelector":"expose=internal"},"Mesos":null,"Eureka":null,"ECS":null,"Rancher":null,"DynamoDB":null}"
[cluster-traefik-2693375319-w67hf] time="2017-08-22T21:41:19Z" level=info msg="Preparing server https &{Network: Address::443 TLS:0xc42060d800 Redirect:<nil> Auth:<nil> Compress:false}"
[cluster-traefik-2693375319-w67hf] time="2017-08-22T21:41:19Z" level=error msg="Error creating TLS config: tls: failed to find any PEM data in key input"
[cluster-traefik-2693375319-w67hf] time="2017-08-22T21:41:19Z" level=fatal msg="Error preparing server: tls: failed to find any PEM data in key input"
This issue turned out to be new validation logic in the crypto/tls library in Go 1.8. They are now validating the certificate blocks end in ----- where as before they did not. The private key for one of my certificate files ended in ---- (missing a hyphen). Adding the missing character fixed this issue.

Nginx can't access a uWSGI unix socket on CentOS 7

I have configured uWSGI to serve my Django app on a unix socket, and Nginx as a proxy to this socket. The server is running CentOS 7. I think I have configured Nginx so that it has permission to read and write to uWSGI's socket, but I'm still getting a permission denied error. Why can't Nginx access the uWSGI socket on CentOS 7?
[uwsgi]
socket=/socket/uwsgi.sock
virtualenv=/home/site/virtsite/
chdir=/home/site/wsgitest/
module=wsgitest.wsgi:application
vhost = true
master=True
workers=8
chmod-socket=666
pidfile=/home/site/wsgitest/uwsgi-master.pid
max-requests=5000
chown-socket=nginx:nginx
uid = nginx
gid = nginx
listen.owner = nginx
listen.group = nginx
server {
listen 80;
location / {
uwsgi_pass unix:///home/site/wsgitest/uwsgi.sock;
include uwsgi_params;
}
}
uwsgi --ini uwsgi.ini (as root)
ls -l /home/site/wsgitest/uwsgi.sock
srwxrwxrwx. 1 nginx nginx 0 Oct 13 10:05 uwsgi.sock
2014/10/12 19:01:44 [crit] 19365#0: *10 connect() to unix:///socket/uwsgi.sock failed (13: Permission denied) while connecting to upstream, client: 2.191.102.217, server: , request: "GET / HTTP/1.1", upstream: "uwsgi://unix:///socket/uwsgi.sock:", host: "179.227.126.222"
The Nginx and uWSGI configurations are correct. The problem is that SELinux denied Nginx access to the socket. This results in a generic access denied error in Nginx's log. The important messages are actually in SELinux's audit log.
# show the new rules to be generated
grep nginx /var/log/audit/audit.log | audit2allow
# show the full rules to be applied
grep nginx /var/log/audit/audit.log | audit2allow -m nginx
# generate the rules to be applied
grep nginx /var/log/audit/audit.log | audit2allow -M nginx
# apply the rules
semodule -i nginx.pp
You may need to generate the rules multiple times, trying to access the site after each pass, since the first SELinux error might not be the only one that can be generated. Always inspect the policy that audit2allow suggests creating.
These steps were taken from this blog post which contains more details about how to investigate and what output you'll get.
Configure your uwsgi.ini with uid and gid user.
#uwsgi.ini
uid = nginx
gid = nginx
Regards,
I wished I could comment :(
Everything looks fine from here except unix socket path
unix:///socket/uwsgi.sock failed (2: No such file or directory)
Docs says it has just one slash
uwsgi_pass unix:/tmp/uwsgi.socket;