I am trying to open up some ports on my compute VM.
For example, I have this in firewall-rules
$ gcloud compute firewall-rules list
NAME NETWORK SRC_RANGES RULES SRC_TAGS TARGET_TAGS
default-allow-http default 0.0.0.0/0 tcp:80 http-server
default-allow-https default 0.0.0.0/0 tcp:443 https-server
default-allow-icmp default 0.0.0.0/0 icmp
default-allow-internal default 10.128.0.0/9 tcp:0-65535,udp:0-65535,icmp
default-allow-rdp default 0.0.0.0/0 tcp:3389
default-allow-ssh default 0.0.0.0/0 tcp:22
test-24284 default 0.0.0.0/0 tcp:24284 test-tcp-open-24284
I have created a centos 7 instance to which I have attached the tags
$ gcloud compute instances describe test-network-opened
...
...
items:
- http-server
- https-server
- test-tcp-open-24284
...
...
Now when I try to check from my dev box to see whether the port is opened or not using nmap on the public IP showed in the console for the VM
$ nmap -p 24284 35.193.xxx.xxx
Nmap scan report for 169.110.xxx.xx.bc.googleusercontent.com (35.193.xxx.xxx)
Host is up (0.25s latency).
PORT STATE SERVICE
24284/tcp closed unknown
Nmap done: 1 IP address (1 host up) scanned in 1.15 seconds
Now it's hitting the external NAT IP for my VM which would be 169.110.xxx.xx
I tried checking using the iptables rules, but that didn't show anything
[root#test-network-opened ~]# iptables -S | grep 24284
[root#test-network-opened ~]#
So I enabled firewalld and tried opening the port with it
[root#test-network-opened ~]# firewall-cmd --zone=public --add-port=24284/tcp --permanent
success
[root#test-network-opened ~]# firewall-cmd --reload
success
[root#test-network-opened ~]# iptables -S | grep 24284
[root#test-network-opened ~]#
I am not sure where I am doing it wrong with this. I referred these relevant questions on SO about this
How to open a specific port such as 9090 in Google Compute Engine
Can't open port 8080 on Google Compute Engine running Debian
How to open a port on google compute engine
https://cloud.google.com/compute/docs/vpc/using-firewalls
https://cloud.google.com/sdk/gcloud/reference/compute/instances/describe
The ports were opened by the firewall but since I didn't have an application using the port already, nmap was showing the closed port which meant it was able to reach to the server and not firewalled
If it was it would have showed filtered.
I didn't have any application running on it so, didn't know this as a possibility. Careless of me.
Thanks for pointing this out.
Related
I have a local cluster up and running with Kubernetes.
With COVID, I work now in 2 places, home, and at office.
When I start my local network at home, it will work only at home.
When changing location , I get:
Unable to connect to the server: dial tcp 192.168.0.78:8443: connect: no route to host
I tried to update context with
minikube update-context
But it doesn't work.
The only solution that I've found is to drop Minikube and deploy it again
Any idea how to fix it without dropping minikube?
On your host PC, get the IP address of the minikube VM, execute command:
$ minikube ip
Sample result - xxx.yyy.zzz.qqq
Then create a rich firewall rule to allow all traffic from this VM to your Host:
$ firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="xxx.yyy.zzz.qqq" accept'
If you create and delete your minikube VM frequently, you can also allow the whole subnet:
$ firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="xxx.yyy.zzz.qqq/24" accept'
Take a look: minikube-cannot-connect.
Is there a corresponding command with kubectl to:
ssh -L8888:rds.aws.com:5432 example.com
kubectl has port-forward you can also specify --address but that strictly requires an IP address.
The older answer is valid.
Still, a workaround would be to use something like
https://hub.docker.com/r/marcnuri/port-forward
kubectl run --env REMOTE_HOST=your.service.com --env REMOTE_PORT=8080 --env LOCAL_PORT=8080 --port 8080 --image marcnuri/port-forward test-port-forward
Run it on the cluster and then port forward to it.
kubectl port-forward test-port-forward 8080:8080
Short answer, No.
In OpenSSH, local port forwarding is configured using the -L option:
ssh -L 80:intra.example.com:80 gw.example.com
This example opens a connection to the gw.example.com jump server, and forwards any connection to port 80 on the local machine to port 80 on intra.example.com.
By default, anyone (even on different machines) can connect to the specified port on the SSH client machine. However, this can be restricted to programs on the same host by supplying a bind address:
ssh -L 127.0.0.1:80:intra.example.com:80 gw.example.com
You can read the docs here.
The port-forward in Kubernetes works only within the cluster, you can forward traffic that will hit specified port to Deployment or Service or a Pod
kubectl port-forward TYPE/NAME [options] [LOCAL_PORT:]REMOTE_PORT [...[LOCAL_PORT_N:]REMOTE_PORT_N]
--address flag is to specify what to listen on 0.0.0.0 means everything localhost is as name and you can set an IP on which it can be listening on.
Documentation is available here, you can also read Use Port Forwarding to Access Applications in a Cluster.
One workaround you can use if you have an SSH server somewhere on the Internet is to SSH to your server from your pod, port-forwarding in reverse:
# Suppose a web console is being served at
# http://my-service-8f6717ab-e.default:8888/
# inside your cluster:
kubectl exec -it my-job-f523b248-7htj6 -- ssh -R8888:my-service-8f6717ab-e.default:8888 user#34.23.1.2
Then you can connect to the service inside Kubernetes from outside of it. If the SSH server is not local to you, you can SSH to it from your local machine with a normal port forward:
me#my-macbook-pro:$ ssh -L8888:localhost:8888 user#34.23.1.2
Then point your browser to http://localhost:8888/
I just installed Kubernetes with minkube on my desktop(running Ubuntu 18.10) and was then trying to install Postgresql on the desktop machine using Helm.
After installing helm, I did:
helm install stable/postgresql
When this completed successfully, I forwarded postgres port with:
kubectl port-forward --namespace default svc/wise-beetle-postgresql 5432:5432 &
and then I tested connecting to it locally from my desktop with:
psql --host 127.0.0.1 -U postgres
which succeeds.
I attempted to connect to postgres from my laptop and that fails with:
psql -h $MY_DESKTOP_LAN_IP -p 5432 -U postgres
psql: could not connect to the server: Connection refused
Is the server running on host $MY_DESKTOP_LAN_IP and accepting TCP/IP connections on port 5432?
To ensure that my desktop was indeed listening on 5432, I did:
netstat -natp | grep 5432
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp 0 0 127.0.0.1:5432 0.0.0.0:* LISTEN 17993/kubectl
tcp6 0 0 ::1:5432 :::* LISTEN 17993/kubectl
Any help anyone? I'm lost.
you need to configure postgresql.conf to allow external client connections look for listen parameter and set it to *, it is under your postgres data directory, and then add your laptop's ip in pg_hba.conf. It controls the client access to your postgresql server, more on this here - https://www.postgresql.org/docs/9.3/auth-pg-hba-conf.html
In my case the solution was a little bit of deeper understanding of networking.
For clarity, let's call the machine on which minikube is installed "A".
The IP of this machine as it is visible from other computers on my Wifi maybe be say: 192.100.200.300.1
Since Postgres was being exposed on port 5432, my expectation was that postgres should be visible externally on: 192.100.200.300.1:5432.
But this understanding is wrong which is what was leading to unexpected behavior.
The problem was that minikube runs in a VM and it gets its own IP address. It doesn't simply use the IP of the machine on which it is running. Minikube's IP is different from the IP
of the machine on which it is running. To find out the IP of minikube, run: minikube ip. Let's call this IP $MINIKUBE_IP.
And then I had to setup port forwarding like:
kubectl port-forward --address "192.100.200.300" --namespace default svc/wise-beetle-postgresql 5000:5432 &
So now, if you called a service on: 192.100.200.300:5000 it would be forwarded to port 5432 on the machine which is running minikube and then 5432 would be received by your postgres instance.
Hope this untangles or clarifies this problem that others might encounter.
I opened a Kubernetes NodePort on a machine and blocked all traffic to this port with the following rule:
sudo ufw deny 30001
But I can still access that port via browser. Is it common? I can't find any information on that in the docs.
Finally found the issue: kube-proxy is writing iptables rules (https://kubernetes.io/docs/tasks/debug-application-cluster/debug-service/#is-kube-proxy-writing-iptables-rules) which are catched before the ufw rules one added manually. This can be confirmed by checking the order in the output of iptables -S -v.
I have searched for this everywhere but after an hour and a half of searching I've not found anything relevant.
How do I connect to a database on my google compute engine? i.e I want to connect to the postgres server running on my google compute engine using pgadmin3 from my laptop.
Is this even possible? If so how do I go about it?
Thanks in advance!
You need to:
Ensure Postgres is listening for TCP traffic (you can check that by connecting to your instance and running netstat -ntpl). Usually, Postgres will be listening on port 5432.
Ensure there is no local firewall blocking traffic to Postgres' port on the instance (you can run iptables -L)
Ensure there is no GCE firewall blocking traffic to your instance on Postgres' port from your IP. You should read this documentation page, and specifically the "firewalls" section
PostgreSQL must also be configured to allow remote connections, otherwise the connection request will fail, even if all firewalls rules are correct and PostgreSQL server is listening on the right port.
Steps
Outline
Couldn't create links, but this is a rather long answer so this may helps.
Tools to check ports during any step
0.1 nc or netcat
0.2 nmap
0.3 netstat
0.4 lsof
IP addresses
1.1 Your laptop's public IP address
1.2 GCE instance's IP address
Firewall rules
2.1 Check existing
2.2 Add new firewall rules
Configure PostgreSQL to accept remote connections
3.1 Finding the above configuration files
3.2 postgresql.conf
3.3 pg_hba.conf
0. Tools to check ports during any step
0.1 nc or netcat
$ nc -zv 4.3.2.1 5432
Where
-v Produce more verbose output.
-z Only scan for listening daemons, without sending any data to
them. Cannot be used together with -l.
Possible outcomes:
Connection to 4.3.2.1port [tcp/postgresql] succeeded!
Yay.
nc: connect to 4.3.2.1 port 8000 (tcp) failed: Connection refused
Port open by firewall, but service either not listening or refusing connection.
command just hangs
Firewall is blocking.
0.2 nmap
$ nmap 4.3.2.1
Starting Nmap 7.70 ( https://nmap.org ) at 2019-09-09 18:28 PDT
Nmap scan report for 1.2.3.4.bc.googleusercontent.com (4.3.2.1)
Host is up (0.12s latency).
Not shown: 993 filtered ports
PORT STATE SERVICE
22/tcp open ssh
80/tcp closed http
443/tcp closed https
3389/tcp closed ms-wbt-server
4000/tcp closed remoteanything
5432/tcp open postgresql # firewall open, service up and listening
8000/tcp closed http-alt # firewall open, is service up or listening?
0.3 netstat
$ netstat -tuplen
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State User Inode PID/Program name
tcp 0 0 0.0.0.0:4000 0.0.0.0:* LISTEN 1000 4223185 29432/beam.smp
tcp 0 0 127.0.0.1:5432 0.0.0.0:* LISTEN 1000 4020942 15020/postgres
tcp 0 0 127.0.0.1:5433 0.0.0.0:* LISTEN 1000 3246566 20553/postgres
tcp6 0 0 ::1:5432 :::* LISTEN 1000 4020941 15020/postgres
tcp6 0 0 ::1:5433 :::* LISTEN 1000 3246565 20553/postgres
udp 0 0 224.0.0.251:5353 0.0.0.0:* 1000 4624644 6311/chrome --type=
udp 0 0 224.0.0.251:5353 0.0.0.0:* 1000 4624643 6311/chrome --type=
udp 0 0 224.0.0.251:5353 0.0.0.0:* 1000 4625649 6230/chrome
udp 0 0 0.0.0.0:68 0.0.0.0:* 0 20911 -
udp6 0 0 :::546 :::* 0 4621237 -
where
-t | --tcp
-u | --udp
-p, --program
Show the PID and name of the program to which each socket belongs.
-l, --listening
Show only listening sockets. (These are omitted by default.)
-e, --extend
Display additional information. Use this option twice for maximum
detail.
--numeric, -n
Show numerical addresses instead of trying to determine symbolic host,
port or user names.
When issued on the instance where PostgreSQL is running, and you don't see the lines below, it means that PostgreSQL is not configured for remote connections:
tcp 0 0 0.0.0.0:5432 0.0.0.0:* LISTEN 1001 238400 30826/postgres
tcp6 0 0 :::5432 :::* LISTEN 1001 238401 30826/postgres
0.4 lsof
To check on instance whether service is running at all.
$ sudo lsof -i -P -n | grep LISTEN
systemd-r 457 systemd-resolve 13u IPv4 14870 0t0 TCP 127.0.0.53:53 (LISTEN)
sshd 733 root 3u IPv4 19233 0t0 TCP *:22 (LISTEN)
sshd 733 root 4u IPv6 19244 0t0 TCP *:22 (LISTEN)
postgres 2733 postgres 3u IPv4 23655 0t0 TCP 127.0.0.1:5432 (LISTEN)
python3 26083 a_user 4u IPv4 392307 0t0 TCP *:8000 (LISTEN)
1. IP addresses
To connect from your laptop, you will need the public IP address of your laptop, and that of the Google Compute Engine (GCE) instance.
1.1 Your laptop's public IP address
(From this article.)
$ dig +short myip.opendns.com #resolver1.opendns.com
4.3.2.1
1.2 GCE instance's IP address
$ gcloud compute instances list
NAME ZONE MACHINE_TYPE PREEMPTIBLE INTERNAL_IP EXTERNAL_IP STATUS
access-news us-east1-d n1-standard-2 10.142.0.5 34.73.156.19 RUNNING
lynx-dev us-east1-d n1-standard-1 10.142.0.2 35.231.66.229 RUNNING
tr2 us-east1-d n1-standard-1 10.142.0.3 35.196.195.199 RUNNING
If you also need the network-tags of the instances:
$ gcloud compute instances list --format='table(name,status,tags.list())'
NAME STATUS TAGS
access-news RUNNING fingerprint=mdTPd8rXoQM=,items=[u'access-news', u'http-server', u'https-server']
lynx-dev RUNNING fingerprint=CpSmrCTD0LE=,items=[u'http-server', u'https-server', u'lynx-dev']
tr2 RUNNING fingerprint=84JxACwWD7U=,items=[u'http-server', u'https-server', u'tr2']
2. Firewall rules
Dealing only with GCE firewall rules below, but make sure that iptables doesn't inadvertently blocks traffic.
See also
"Firewall rules overview" (official docs)
GCE firewall rules vs. iptables
Summary of GCE firewall terms
Behaviour of GCE firewall rules on instances (external vs internal IP addresses)
2.1 Check existing
$ gcloud compute firewall-rules list
NAME NETWORK DIRECTION PRIORITY ALLOW DENY DISABLED
default-allow-http default INGRESS 1000 tcp:80 False
default-allow-https default INGRESS 1000 tcp:443 False
default-allow-icmp default INGRESS 65534 icmp False
default-allow-internal default INGRESS 65534 tcp:0-65535,udp:0-65535,icmp False
default-allow-rdp default INGRESS 65534 tcp:3389 False
default-allow-ssh default INGRESS 65534 tcp:22 False
pg-from-tag1-to-tag2 default INGRESS 1000 tcp:5432 False
To show all fields of the firewall, please show in JSON format: --format=json
To show all fields in table format, please see the examples in --help.
A more comprehensive list that includes network-tags as well (from gcloud compute firewall-rules list --help):
$ gcloud compute firewall-rules list --format="table( \
name, \
network, \
direction, \
priority, \
sourceRanges.list():label=SRC_RANGES, \
destinationRanges.list():label=DEST_RANGES, \
allowed[].map().firewall_rule().list():label=ALLOW, \
denied[].map().firewall_rule().list():label=DENY, \
sourceTags.list():label=SRC_TAGS, \
sourceServiceAccounts.list():label=SRC_SVC_ACCT, \
targetTags.list():label=TARGET_TAGS, \
targetServiceAccounts.list():label=TARGET_SVC_ACCT, \
disabled \
)"
NAME NETWORK DIRECTION PRIORITY SRC_RANGES DEST_RANGES ALLOW DENY SRC_TAGS SRC_SVC_ACCT TARGET_TAGS TARGET_SVC_ACCT DISABLED
default-allow-http default INGRESS 1000 0.0.0.0/0 tcp:80 http-server False
default-allow-https default INGRESS 1000 0.0.0.0/0 tcp:443 https-server False
default-allow-icmp default INGRESS 65534 0.0.0.0/0 icmp False
default-allow-internal default INGRESS 65534 10.128.0.0/9 tcp:0-65535,udp:0-65535,icmp False
default-allow-rdp default INGRESS 65534 0.0.0.0/0 tcp:3389 False
default-allow-ssh default INGRESS 65534 0.0.0.0/0 tcp:22 False
pg-from-tag1-to-tag2 default INGRESS 1000 4.3.2.1 tcp:5432 tag1 tag2 False
2.2 Add new firewall rules
To open the default PostgreSQL port (5432) from every source to every instance:
$ gcloud compute firewall-rules create \
postgres-all \
--network default \
--priority 1000 \
--direction ingress \
--action allow \
--rules tcp:5432 \
To restrict it between your computer (source: YOUR_IP) and the GCE instance (destination: INSTANCE_IP):
$ gcloud compute firewall-rules create \
postgres-from-you-to-instance \
--network default \
--priority 1000 \
--direction ingress \
--action allow \
--rules tcp:5432 \
--destination-ranges INSTANCES_IP \
--source-ranges YOUR_IP \
Instead of --source-ranges and --destination-ranges one could use source and target network tags or service accounts as well. See the "Source or destination" section in the firewall docs.
3. Configure PostgreSQL to accept remote connections
This is an update to Neeraj Singh's post.
By default PostgreSQL is configured to be bound to “localhost”, therefore the below configuration files will need to be updated:
postgresql.conf, and
pg_hba.conf
3.1 Finding the above configuration files
The location of both files can be queried from PostgreSQL itself (trick taken from this Stackoverflow thread):
$ sudo -u postgres psql -c "SHOW hba_file" -c "SHOW config_file"
3.2 postgresql.conf
The configuration file comes with helpful hints to get this working:
listen_addresses = 'localhost' # what IP address(es) to listen on;
# comma-separated list of addresses;
# defaults to 'localhost'; use '*' for all
# (change requires restart)
For a quick and dirty solution just change it to
listen_addresses = '*'
Restart the server (see here how). Once PostgreSQL is restarted, it will start listening on all IP addresses (see netstat -tuplen).
To restart PostgreSQL:
$ sudo systemctl restart postgresql#11-main
# or
$ pg_ctl restart
The listen_addresses documentation says that it "Specifies the TCP/IP address(es) on which the server is to listen for connections from client applications.", but that's all. It specifies the sockets the packets are accepted from, but if the incoming connections are not authenticated (configured via pg_hba.conf), then the packets will be rejected (dropped?) regardless.
3.3 pg_hba.conf
From 20.1. The pg_hba.conf File: "_Client authentication is controlled by a configuration file, which traditionally is named pg_hba.conf and is stored in the database cluster's data directory. (HBA stands for host-based authentication.) _"
This is a complex topic so reading the documentation is crucial, but this will suffice for development on trusted networks:
host all all 0.0.0.0/0 trust
host all all ::/0 trust
Another restart is required at this point.