traefik failed external connectivity - 443 already in use - docker-compose

I am running a server and I have a pointed my domain via cloudflare to my server IP and have a signed SSL certificate via LetsEncrypt for my domain. My server is running an apache webserver using porto 443 for the ssl traffic.
I installed docker and a run a couple of containers. My goal is to get traefik up and running using port 443 as well and route all docker traffic through it. Is that even possible?
I used this here: https://www.linuxserver.io/2018/02/03/using-traefik-as-a-reverse-proxy-with-docker/ to write my traefik.toml file and my docker-compose file.
However, whenever I start up the docker-compose all services are up except traefik.
I receive following error:
ERROR: for traefik Cannot start service traefik: driver failed programming external connectivity on endpoint traefik (2d10b64b47e62e7dcb5f94265529fb647e4ba62dbeeb43c201ea02d39f60b381): Error starting userland proxy: listen tcp 0.0.0.0:443: bind: address already in use
ERROR: Encountered errors while bringing up the project.
I wonder if the reason is that I already use port 443 for my domain?!
How can I fix this?
Thanks for your help!

You are using docker in Linux:
Some of these commands should give you a clue that you are using them:
sudo lsof -i -P -n | grep LISTEN
netstat -tulpn | grep LISTEN
example:
docker-pr 2405 root 4u IPv6 28930 0t0 TCP
*:443 (LISTEN)
if 443 is occupied by docker ... it means that you have in some YML an exposed port 443, besides that of Traefik (if it is some other application proceeds to change port or close it [pkill])
You can try to separate the "Services" of the YML in different YML and turn them on one by one, in order to find the image that is causing you conflicts
(If you separate them remember to create the appropriate external "Networks".)
(by the way ... I recommend that the first image that starts be the one of traefik)
(You can also copy and paste your YML files for better help.)
Edit
RewriteEngine on
RewriteCond %{HTTPS} off
# RewriteCond %{SERVER_PORT} ^9000$
RewriteRule ^(.*)$ https://%{HTTP_HOST}:9443%{REQUEST_URI}
edit2 in toml config traefik: (I have no idea what works, try it)
# Entrypoints, http and https
[entryPoints]
# http should be redirected to https
[entryPoints.http]
address = ":80"
[entryPoints.http.redirect]
entryPoint = "https"
# https is the default
[entryPoints.https]
address = ":9443"
[entryPoints.https.tls]
The other solution that occurs to me is to make your main apache as a Proxy Tunnel, BUT, then you do not need to bring traefik :P

I've got the same issue.
I've tried everything I found on stackoverflow and github.
Only this worked for me:
sudo lsof -i -P -n | grep LISTEN
And I've got somethink like this:
And I decided to kill the first PID (related to 80 port)
sudo kill -9 1876
And then I've started the service with docker on network and everything worked fine. Hooray!!!

Related

Kubernetes Port Forwarding - Error listen tcp4 127.0.0.1:88: bind: permission denied

I am using minikube on my local machine. Getting this error while using kubernetes port forwarding. Can anyone help?
mjafary$ kubectl port-forward sa-frontend 88:80
Unable to listen on port 88: All listeners failed to create with the following errors:
Unable to create listener: Error listen tcp4 127.0.0.1:88: bind: permission denied, Unable to create listener: Error listen tcp6 [::1]:88: bind: permission denied
error: Unable to listen on any of the requested ports: [{88 80}]
kubectl fails to open the port 88 because it is a privileged port. All ports <1024 require special permissions.
There are many ways to solve your problem.
You can stick to ports >= 1024, and use for example the port 8888 instead of 88: kubectl port-forward sa-frontend 8888:80
You could use kubectl as root: sudo kubectl port-forward sa-frontend 88:80 (not recommended, kubectl would then look for its config as root)
You could grant the kubectl binary the capability to open privileged ports. This answer explains in depth how to do this.
If you want to go for the 3rd option, here is a short way of doing it:
sudo setcap CAP_NET_BIND_SERVICE=+eip /usr/bin/kubectl
This will let kubectl open any port while still running with the privileges of a regular user. You can check if this worked by using
sudo getcap /usr/bin/kubectl
/usr/bin/kubectl = cap_net_bind_service+eip
Be aware that this grants the permission to whoever uses the binary. If you want finer grained permissions, use authbind.
Note: as ng-sek-long commented, kubectl is not necessarily installed as /usr/bin/kubectl. You should replace it with the path to the kubectl binary on your machine.
As mentioned by user48678 you can bypass the limitation using sudo.
Add -E flag to pass environment.
mjafary$ sudo -E kubectl port-forward sa-frontend 88:80
If you don't pass -E, the KUBECONFIG environment variable will not be set for example.
I tried sudo as follows and it got me past the permission denied issue.
sudo kubectl port-forward sa-frontend 88:80
I am now getting a different issue but will create a new tracker for that to keep things straight
You can receive the permission denied error when port forwarding with kubectl if the target port on the local host is already in use.
Ensure that you do not have a docker container or another application running on port 88.
Simply use other port to redirect
sudo -E kubectl port-forward --address 0.0.0.0 nginx-yaml 9000:80
and access : http://localhost:9000
Probably your localhost - the local machine is using ipv6 when ipv4 is expected.
Error listen tcp4 127.0.0.1:88: bind: permission denied, Unable to
create listener: Error listen tcp6 [::1]:88: bind: permission
denied
Please disable IPv6.
Can you show the output of?:
cat /proc/sys/net/ipv6/conf/all/disable_ipv6

Is there a way to determine to which etcd host the kubernetes apiserver is talking to?

Only apiserver talks directly to etcd. In the etcd cluster there are many hosts. I would like to see to which etcd host the apiserver is talking to. This may be different for each api resource like Pod or Node. I prefer to see etcd host information for each request.
Specifically, kubernetes 1.6.13 and etcd 3.1.14 using v3 store.
I have tried:
Enable etcd client and grpc logging on the kubernetnes api server.
I think grpc only logs in unexpected events. Similarly for etcd
clientv3. I was not able to get information about the etcd side of the
connection.
Enable http2 debug logging with GODEBUG=http2debug=2 on api server
To my surprise http2 debug logs print a lot of information about each request but I could not find the remote endpoint information. I am still skeptical about this I may be missing a mention in the log files. Not completely sure.
Debug logs on the etcd side.
Enabling debug logs with Enabling Debug Logging prints only about v2 store accesses. For v3 store one could use the http://<host>2379/debug/requests endpoint but that is not available in my version of etcd 3.1.14.
I have not tried yet to use GODEBUG=http2debug=2 on the etcd side. Maybe the http2 logs on the etcd have the info I need.
tcpdump or tcpflow
The apiserver <-> etcd connection is encrypted. Would these show me the request url ? I think I did not see that information in the dumps.
Man in the middle attack the apiserver <-> etcd connection with mitmproxy. I do not think this should be that complicated.
I hope, I have missed a super obvious and simple way to accomplish this.
Update:
About using lsof based approaches:
Using lsof, we can list the connections with endpoints information at one time. I do not think there is enough information in lsof output to arrive at endpoint information per request. Apiserver opens a lot of connections to etcd. Looking at the code that observation looks reasonable to me. See NewStorage in here
$ sudo lsof -p 20816 | grep :2379 | wc -l
130
The connections looks like this
$ sudo lsof -
p 20816 | grep :2379 | head -n 5
hyperkube 20816 root 3u IPv4 58093240 0t0 TCP compute-master7001.dsv31.boxdc.net:36360->compute-etcd7001.dsv31.boxdc.net:2379 (ESTABLISHED)
hyperkube 20816 root 5u IPv4 58085987 0t0 TCP compute-master7001.dsv31.boxdc.net:26005->compute-etcd7002.dsv31.boxdc.net:2379 (ESTABLISHED)
hyperkube 20816 root 6u IPv4 58085988 0t0 TCP compute-master7001.dsv31.boxdc.net:55650->compute-etcd7003.dsv31.boxdc.net:2379 (ESTABLISHED)
hyperkube 20816 root 7u IPv4 58102030 0t0 TCP compute-master7001.dsv31.boxdc.net:36366->compute-etcd7001.dsv31.boxdc.net:2379 (ESTABLISHED)
hyperkube 20816 root 8u IPv4 58085990 0t0 TCP compute-master7001.dsv31.boxdc.net:55654->compute-etcd7003.dsv31.boxdc.net:2379 (ESTABLISHED)
........
Looking at this, I cannot know which etcd is used for each request between the apiserver and etcd.
Update:
I think at the etcdv3 client code that ships with kubernetes 1.6.13, the grpc.Balancer.Get function returns the endpoint address used for each grpc request. I think one could add a log print here and make apiserver log the etcd address per request.
Find the pid of apiserver
ps aux | grep apiserver
Then use lsof to see the open socket connections
lsof -p $PID | grep :2379

Fix IP with port to IP without port

I have a website.example.com The website is hosted on OVH I would like to point a sub domain shop.example.com to another website hosted on another server
(95.110.189.135:8069) the problem is that I can't c name to an IP with a port.
I used Ubuntu for my odoo server
I've got odoo on my vps server with database. Now, It's working on IP with port (example: 55.55.55.55:8069). So now,
How can I change it to IP without port?
If I want a domain name - how can I do this?
I found the solution it's easy to redirect to port 80
to do that add a line of code in the file
etc/rc.local
iptables -t nat -A PREROUTING -p tcp --dport 80 -j REDIRECT --to-port 8069
then the file will become like this
#!/bin/sh -e
#
# rc.local
#
# This script is executed at the end of each multiuser runlevel.
# Make sure that the script will "exit 0" on success or any other
# value on error.
#
# In order to enable or disable this script just change the execution
# bits.
#
# By default this script does nothing.
iptables -t nat -A PREROUTING -p tcp --dport 80 -j REDIRECT --to-port 8069
exit 0
save and then restart the server
You cannot use plain DNS to transfer traffic to another port. This is not possible with either canonical name (CNAME record) or address (A record). These DNS services are only used for address resolution.
To solve your configuration issue you can use reverse proxy, e.g. Nginx. You can find example configurations from the Odoo.com site at https://www.odoo.com/documentation/11.0/setup/deploy.html#https. This is describing how to use https in port 443 to proxy Odoo in upstream service at port 8069. For public services you should use encrypted https, not http. Point your show.example.com in DNS to your "another" server ip address and on that server have Odoo and Nginx running. Your Odoo can run on port 8069 and your Nginx would run on https 443 and proxy connections to Odoo upstream service on localhost 8069.
Hope this helps you forward. Please check your configuration with someone who have experience with this kind of setups before you go production. This will make sure your configuration is secure.

How to change the Rundeck web interface port from 4440 to 80

I'm trying to use Rundeck on Ubuntu 14.04.
How do I change the web interface port from 4440 (default) to 80?
The port number seems to be referred to in various config locations (including /etc/rundeck/profile and /etc/rundeck/framework.properties) but changing these had no effect for me and I can't find a specific section in the documentation on changing the web port.
Use a proxy pass with either Apache or Nginx. Its a solution..
You needed to modify these files from 4440 to 80
framework.properties
profile
rundeck-config.properties
But you will get the following java exception when trying to run in 80.
java.net.SocketException: Permission denied
Hence apache or nginx proxy pass is the working solution
Do you can use two rules iptables
iptables -t nat -I PREROUTING -p tcp --dport 80 -j REDIRECT --to-port 4440
iptables -I INPUT -p tcp --dport 4440 -j ACCEPT

Cannot access an application hosted on jBoss remotely

I have hosted an application in a machine running Red Hat Enterprise Linux 5. I started the jBoss AS using the command.
./run.sh -b 0.0.0.0
and
./run.sh -Djboss.bind.address=<<server_address>>
and
./run.sh --host=<<ipaddress>>
but using any of these commands i cannot access the application remotely. Using the the above commands I cannot even access the application on the host machine itself, using localhost as well as ip address. I am not able to figure out the problem here. I can ping the linux machine from the other windows machines.
Check iptables rules are not blocking firstly
Also are you running as a user? If so, you will not have permission to bind to a port number less than 1024.
try telneting the port from the server itself to check the service is responding e.g.
telnet localhost 8080
presuming that you are running on 8080 in the example above.
you can drop your iptables temporarily for testing if it is safe to do so by:
/etc/init.d/iptables stop
and restart them when you've finished with
/etc/init.d/iptables start
you can make a permanent change to your iptables config by adding the following line to /etc/sysconfig/iptables
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 8080 -j ACCEPT