Ingress controller cannot find the certificate. I get the error failed to reconcile listeners due to failed to create listener due to CertificateNotFound
Its a imported LetsEncrypt certificate and describing the certificate using aws cli works.
Here is the error in controller logs:
1 controller.go:217] kubebuilder/controller "msg"="Reconciler error" "error"="failed to reconcile listeners due to failed to create listener due to CertificateNotFound: Certificate 'arn:aws:acm:ap-south-1:1234:certificate/f9a3a88e-481c-4c91-9d55-1234' not found\n\tstatus code: 400, request id: 811dddad-532c-4846-ae10-c69c2f4b4c3b" "controller"="alb-ingress-controller" "request"={"Namespace":"test-web-dev","Name":"test-ingress-dev"}
I've set up Traefik with Docker and a service behind it. The basic setup works. I can browse to port 80 using the domain name I'm redirected to https and then see "invalid certificate" - since the let's encrypt part is broken.
[ router ] <-:80/:443-> [linux/docker [Traefik:80/:443][Service:8080]]
Here is the entry in the log (edited domain.)
Logs:
acme: Error -> One or more domains had a problem:\n[xyz.example.net] acme: error: 400 :: urn:ietf:params:acme:error:connection :: Fetching
http://xyz.example.net/.well-known/acme-challenge/eIAFZqaGMHMWaBjINjzk4m8PuWiYfuCHCTnSU9M:
Error getting validation data, url: \n"
The error message is accurate, I can not browse to that URL. I have noticed that I can go to that URL using the internal IP http://10.0.0.21/.well-known/acme-challenge/key and Traefik responds with this in the log:
traefik | time="2019-05-28T21:20:52Z" level=error msg="Error getting challenge for token retrying in 542.914495ms"
I suspect the problem is the domain name redirect setup. My service is at xyz.example.net (and so is Traefik.) I suspect the problem is that Traefik is redirecting all traffic coming in on xyz.example.net:80/:443 to the service, and not handling the ./well-known/acme-challenge itself. Do I need to give the gateway itself a name? (E.g. zzz.example.net is Traefik and xyz.example.net is the service?)
How can I fix this?
My TOML file:
debug = false
logLevel = "ERROR" #DEBUG, INFO, WARN, ERROR, FATAL, PANIC
InsecureSkipVerify = true
defaultEntryPoints = ["https", "http"]
[entryPoints]
[entryPoints.http]
[entryPoints.http.redirect]
entryPoint = "https"
[entryPoints.https]
address = ":443"
[entryPoints.https.tls]
[retry]
[docker]
endpoint = "unix:///var/run/docker.sock"
domain = "example.net"
watch = true
exposedbydefault = false
[acme]
email = "me#example.net"
storage = "acme.json"
entryPoint = "https"
onDemand = false
onHostRule = true
[acme.httpChallenge]
entryPoint = "http"
I noticed that although the internal IP 192.xxx worked - the external IP did not. Of course, I would say, this seems like a firewall problem - BUT - the firewall lets thru traffic just fine for the services that I was testing, so I was confused.
The solution? port 80 was not being forwarded on the firewall, 443 was. So when I tried testing with curl/browser I was typing in https://xyz.example.com - and that was working.
I am running prometheus as a kubernetes pod and wants prometheus to write data to the inflluxDB I have added the entries to the prometheus.yml , below entries been added
remote_read:
- url: "http://localhost:8086/api/v1/prom/write?u=xxxxxx&p=ids3pr0m&db=xxxxxx"
remote_write:
- url: "http://localhost:8086/api/v1/prom/read?u=xxxxxx&p=ids3pr0m&db=xxxxxx"
the pod is running file and able to read it , but keep on giving me below error .
time="2018-05-03T17:38:31Z" level=warning msg="Error sending 100 samples to remote storage: server returned HTTP status 400 Bad Request: {"error":"proto: wrong wireType = 2 for field StartTimestampMs"}" source="queue_manager.go:500"
time="2018-05-03T17:38:31Z" level=warning msg="Error sending 100 samples to remote storage: server returned HTTP status 400 Bad Request: {"error":"proto: wrong wireType = 2 for field StartTimestampMs"}" source="queue_manager.go:500"
time="2018-05-03T17:38:31Z" level=warning msg="Error sending 100 samples to remote storage: server returned HTTP status 400 Bad Request: {"error":"proto: wrong wireType = 2 for field StartTimestampMs"}" source="queue_manager.go:500"
Can someone help me on this
Ran into this as well and for me it was using a Prometheus version 2.x.
Looks InfluxDB only supports version 1.8
Eureka + Gateway + BackendServerA + BackendServerB. After BackendServerB is down, it throws exception:
com.netflix.zuul.exception.ZuulException: Forwarding error
Caused by: com.netflix.client.ClientException: Load balancer does not have available server for client: service-B
Event after I manually reboot serverB, it still throws the same error with a 500 http code returned. However, if I turn on the other 3 servers before Gateway, it works properly. I used application.properties file to configure routes.
I'm attempting to update from Traefik v1.2.3 to v1.3.6 on Kubernetes. I have my TLS certificates mounted inside of the pods from secrets. Under v1.2.3, everything works as expected. When I try to apply my v1.3.6 deployment (only change being the new docker image), the pods fail to start with the following message:
time="2017-08-22T20:27:44Z" level=error msg="Error creating TLS config: tls: failed to find any PEM data in key input"
time="2017-08-22T20:27:44Z" level=fatal msg="Error preparing server: tls: failed to find any PEM data in key input"
Below is my traefik.toml file:
defaultEntryPoints = ["http","https"]
[entryPoints]
[entryPoints.http]
address = ":80"
[entryPoints.http.redirect]
entryPoint = "https"
[entryPoints.https]
address = ":443"
[entryPoints.https.tls]
[[entryPoints.https.tls.certificates]]
CertFile = "/ssl/wildcard.foo.mydomain.com.crt"
KeyFile = "/ssl/wildcard.foo.mydomain.com.key"
[[entryPoints.https.tls.certificates]]
CertFile = "/ssl/wildcard.mydomain.com.crt"
KeyFile = "/ssl/wildcard.mydomain.com.key"
[[entryPoints.https.tls.certificates]]
CertFile = "/ssl/wildcard.local.crt"
KeyFile = "/ssl/wildcard.local.key"
[kubernetes]
labelselector = "expose=internal"
My initial impression of the errors produced by the pods are that the keys in the secret are not valid. However, I am able to base64 decode the contents of the secret and see that the values match those of the certificate files I have stored locally. Additionally, I would expect to see this error on any version of Traefik if these were in fact, invalid. In reviewing the change log for Traefik, I see that the SSL library was updated but the related PR indicates that this only added ciphers and did not remove any previously supported.
:Edit w/ additional info:
Running with --logLevel=DEBUG provides this additional information (provided in full below in case it's helpful):
[cluster-traefik-2693375319-w67hf] time="2017-08-22T21:41:19Z" level=debug msg="Global configuration loaded {"GraceTimeOut":10000000000,"Debug":false,"CheckNewVersion":true,"AccessLogsFile":"","TraefikLogsFile":"","LogLevel":"DEBUG","EntryPoints":{"http":{"Network":"","Address":":80","TLS":null,"Redirect":{"EntryPoint":"https","Regex":"","Replacement":""},"Auth":null,"Compress":false},"https":{"Network":"","Address":":443","TLS":{"MinVersion":"","CipherSuites":null,"Certificates":[{"CertFile":"/ssl/wildcard.foo.mydomain.com.crt","KeyFile":"/ssl/wildcard.foo.mydomain.com.key"},{"CertFile":"/ssl/wildcard.mydomain.com.crt","KeyFile":"/ssl/wildcard.mydomain.com.key"},{"CertFile":"/ssl/wildcard.local.crt","KeyFile":"/ssl/wildcard.local.key"}],"ClientCAFiles":null},"Redirect":null,"Auth":null,"Compress":false}},"Cluster":null,"Constraints":[],"ACME":null,"DefaultEntryPoints":["http","https"],"ProvidersThrottleDuration":2000000000,"MaxIdleConnsPerHost":200,"IdleTimeout":180000000000,"InsecureSkipVerify":false,"Retry":null,"HealthCheck":{"Interval":30000000000},"Docker":null,"File":null,"Web":{"Address":":8080","CertFile":"","KeyFile":"","ReadOnly":false,"Statistics":null,"Metrics":{"Prometheus":{"Buckets":[0.1,0.3,1.2,5]}},"Path":"","Auth":null},"Marathon":null,"Consul":null,"ConsulCatalog":null,"Etcd":null,"Zookeeper":null,"Boltdb":null,"Kubernetes":{"Watch":true,"Filename":"","Constraints":[],"Endpoint":"","Token":"","CertAuthFilePath":"","DisablePassHostHeaders":false,"Namespaces":null,"LabelSelector":"expose=internal"},"Mesos":null,"Eureka":null,"ECS":null,"Rancher":null,"DynamoDB":null}"
[cluster-traefik-2693375319-w67hf] time="2017-08-22T21:41:19Z" level=info msg="Preparing server https &{Network: Address::443 TLS:0xc42060d800 Redirect:<nil> Auth:<nil> Compress:false}"
[cluster-traefik-2693375319-w67hf] time="2017-08-22T21:41:19Z" level=error msg="Error creating TLS config: tls: failed to find any PEM data in key input"
[cluster-traefik-2693375319-w67hf] time="2017-08-22T21:41:19Z" level=fatal msg="Error preparing server: tls: failed to find any PEM data in key input"
This issue turned out to be new validation logic in the crypto/tls library in Go 1.8. They are now validating the certificate blocks end in ----- where as before they did not. The private key for one of my certificate files ended in ---- (missing a hyphen). Adding the missing character fixed this issue.