Can Spark be configured so that instead of binding to address 127.0.1.1 for port 7077, can
instead be bound to 0.0.0.0 . In same way as port 8080 is bound :
netstat -pln
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 127.0.1.1:7077 0.0.0.0:* LISTEN 2864/java
tcp 0 0 0.0.0.0:8080 0.0.0.0:* LISTEN 2864/java
tcp 0 0 127.0.1.1:6066 0.0.0.0:* LISTEN 2864/java
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN -
udp 0 0 0.0.0.0:68 0.0.0.0:* -
udp 0 0 192.168.192.22:123 0.0.0.0:* -
udp 0 0 127.0.0.1:123 0.0.0.0:* -
udp 0 0 0.0.0.0:123 0.0.0.0:* -
udp 0 0 0.0.0.0:21415 0.0.0.0:* -
Active UNIX domain sockets (only servers)
Proto RefCnt Flags Type State I-Node PID/Program name Path
unix 2 [ ACC ] STREAM LISTENING 7195 - /var/run/dbus/system_bus_socket
unix 2 [ ACC ] SEQPACKET LISTENING 405 - /run/udev/control
Reason I'm asking this is that I'm unable to connect workers to master node and I think the issue is that the master ip is not discoverable.
Error when try to connect slave to master :
15/04/02 21:58:18 WARN Remoting: Tried to associate with unreachable remote address [akka.tcp://sparkMaster#raspberrypi:7077]. Address is now gated for 5000 ms, all messages to this address will be delivered to dead letters. Reason: Connection refused: raspberrypi/192.168.192.22:7077
15/04/02 21:58:18 INFO RemoteActorRefProvider$RemoteDeadLetterActorRef: Message [org.apache.spark.deploy.DeployMessages$RegisterWorker] from Actor[akka://sparkWorker/user/Worker#1677101765] to Actor[akka://sparkWorker/deadLetters] was not delivered. [10] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
In spark-env.sh you can set SPARK_MASTER_IP=<ip>.
A hostname would also work fine (via SPARK_STANDALONE_MASTER=<hostname>), just make sure the workers connect to exactly the same hostname as the master binds to (i.e. the spark:// address that is shown in Spark master UI).
Related
I can connect to my DigitalOcean Ubuntu 20LTS VM instant that has PostgreSQL 14 installed without issue, but I'm trying to make it more secure with only specific IPs that can connect to the database.
I heard the way to do this is to modify the /etc/postgresql/14/main/postgresql.conf file.
When I have this line, I can connect to my database without issue.
listen_addresses='0.0.0.0'
However, if I modify this line with:
listen_addresses='123.123.123.123'
I get this DataGrip error message: [08001] Connection to 111.111.111.111:12345 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
111.111.111.111:12345 is my (fake) VM's IP and port that I already set up.
123.123.123.123 is my (fake) computer's external IP that I get from here or here
Any suggestions? Is there a log I can search from that will give me a better understanding of what is going on?
Also to note, with listen_addresses='0.0.0.0', running ss -ptl gives an output of
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
LISTEN 0 4096 127.0.0.53%lo:domain 0.0.0.0:*
LISTEN 0 128 0.0.0.0:ssh 0.0.0.0:*
LISTEN 0 244 0.0.0.0:12345 0.0.0.0:*
LISTEN 0 128 [::]:ssh [::]:*
with listen_addresses='123.123.123.123', running ss -ptl gives an output of
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
LISTEN 0 4096 127.0.0.53%lo:domain 0.0.0.0:*
LISTEN 0 128 0.0.0.0:ssh 0.0.0.0:*
LISTEN 0 128 [::]:ssh [::]:*
Documentation that I used so far:
https://www.postgresql.org/docs/current/runtime-config-connection.html
https://www.postgresql.org/docs/current/auth-pg-hba-conf.html
I have created an asio server with acceptor:
m_acceptor(m_ios, asio::ip::tcp::endpoint(asio::ip::address_v4::any(), port_num)
where port number is 3333
At this point, the netstat -antup command shows :
13:tcp 0 0 0.0.0.0:3333 0.0.0.0:* LISTEN 26566/./test
So, I believe this means that local address 0 0.0.0.0:3333 is ready to listen to any connection on port 3333
After this, I start the client which creates the endpoint to ip : 127.0.0.1 and port 3333
After this, the netstat output is:
tcp 0 0 0.0.0.0:3333 0.0.0.0:* LISTEN 26566/./test
tcp 0 0 127.0.0.1:3333 127.0.0.1:46675 ESTABLISHED 26566/./test
tcp 0 0 127.0.0.1:46675 127.0.0.1:3333 ESTABLISHED 26685/./test
Process 26566 is master process
Process 26685 is slave process
What I do not understand is what does the the port 46675 mean in the address shown above? This definitely represents the client side, but from where was this port number allocated to the client?
Does this mean that client has connected to port 3333 but the port from which it itself connects is 46675?
Does this mean that client has connected to port 3333 but the port from which it itself connects is 46675?
Basically. It describes the client endpoint. This is BSD/Posix sockets jargon.
What I do not understand is what does the the port 46675 mean in the address shown above? This definitely represents the client side, but from where was this port number allocated to the client?
It gets automatically chosen (by the TCP stack, usually in the kernel) from the local port range. E.g. on linux you can manipulate that range (if you have permission):
sudo sysctl -w net.ipv4.ip_local_port_range="60000 61000"
(Warning: don't do this unless you know what you're doing). See also https://en.wikipedia.org/wiki/Ephemeral_port
I have installed and configured RabbitMQ on Ubuntu 16.04 server using reference. Since the default user that is guest is only allowed to connect locally by default, I added a new user with the administrator tag and set its permission so that it can access / virtual host. I enabled RabbitMQ management console. I am successfully able to login with the user I created. I am also able to connect with RabbitMQ when I am connecting to it via localhost using my created user. But when I am trying to connect with the RabbitMQ server through other servers using following code:
import pika
credentials = pika.PlainCredentials('new_user', 'new_pass')
parameters = pika.ConnectionParameters('<server's Public IP>', 5672,'/',credentials)
connection = pika.BlockingConnection(parameters)
It throws an error:
Traceback (most recent call last):
File "", line 1, in
File "/Library/Python/2.7/site-packages/pika/adapters/blocking_connection.py", line 339, in init
self._process_io_for_connection_setup()
File "/Library/Python/2.7/site-packages/pika/adapters/blocking_connection.py", line 374, in _process_io_for_connection_setup
self._open_error_result.is_ready)
File "/Library/Python/2.7/site-packages/pika/adapters/blocking_connection.py", line 395, in _flush_output
raise exceptions.ConnectionClosed()
pika.exceptions.ConnectionClosed
The same code works fine when I run this code on server, on which RabbitMQ is installed and by replacing <server's Public IP> with 0.0.0.0.
Output of sudo netstat -nltp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:25672 0.0.0.0:* LISTEN 18021/beam
tcp 0 0 0.0.0.0:4369 0.0.0.0:* LISTEN 18110/epmd
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1230/sshd
tcp 0 0 0.0.0.0:15672 0.0.0.0:* LISTEN 18021/beam
tcp6 0 0 :::5672 :::* LISTEN 18021/beam
tcp6 0 0 :::4369 :::* LISTEN 18110/epmd
tcp6 0 0 :::22 :::* LISTEN 1230/sshd
What could be causing this error?
this usually happens with a very low connection timeout. adjust your connection string to include a larger connection timeout, such as 30 or 60 seconds, and you should be good to go.
looks like pika uses this setting https://pika.readthedocs.io/en/latest/modules/parameters.html#pika.connection.ConnectionParameters.blocked_connection_timeout
As syslog uses the predefined socket port number of 514, is there any way to rebind this socket port number to any other port number specifically between 49152 and 65535. I am using Unix C 'gcc' compiler.
bash-3.2$ netstat -anp | grep udp
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
udp 0 0 0.0.0.0:2049 0.0.0.0:* -
udp 0 0 0.0.0.0:514 <-- needs to be changed 0.0.0.0:* -
udp 0 0 127.1.1.1:6688 0.0.0.0:* -
udp 0 0 0.0.0.0:4785 0.0.0.0:* -
udp 0 0 0.0.0.0:69 0.0.0.0:* -
udp 0 0 0.0.0.0:47451 0.0.0.0:* -
udp 0 0 0.0.0.0:613 0.0.0.0:* -
udp 0 0 0.0.0.0:111 0.0.0.0:* -
udp 0 0 0.0.0.0:1009 0.0.0.0:* -
udp 0 0 0.0.0.0:1012 0.0.0.0:* -
I need to change the 514 to the specified value.
gcc is simply a compiler and so we cannot change the port number of syslog application during compilation. Having said that, it is likely that one can configure the port, usually in the configuration files, and you should be able to set it to 514. Here is an example that talks about configuring both the client side and the server side: http://itvomit.com/2012/06/01/linux-sending-log-files-to-a-remote-server/ . Depending upon your system, I would explore how to change the port number.
We should keep in mind that if we change the port from 514 to X, then if remote clients send information to syslog, they they would also need to send it to the new port X and not 514. Here is a link on that explains how we need to change the client side config to redirect logging messages to a port number different from 514: http://docs.splunk.com/Documentation/Storm/Storm/User/Howtosetupsyslog
Whenever I open some help within eclipse I get a page saying:
Server Error. The following error occurred: [code=CANT_CONNECT_LOOPBACK] Cannot connect due to potential loopback problems
I'm running Ubuntu 10.04.
Any ideas what this can be?
UPDATE
Some commands outputs (some private info replaced):
$ ifconfig -a
eth0 Link encap:Ethernet HWaddr 00:xx:xx:xx:xx:xx
inet addr:123.12.123.235 Bcast:123.12.456.255 Mask:255.255.254.0
inet6 addr: fe80::xxx:eff:xxxx:xxxx/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:1343040 errors:0 dropped:0 overruns:0 frame:0
TX packets:1133672 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:829265876 (829.2 MB) TX bytes:242912202 (242.9 MB)
Memory:f3200000-f3220000
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:125 errors:0 dropped:0 overruns:0 frame:0
TX packets:125 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:15910 (15.9 KB) TX bytes:15910 (15.9 KB)
$ route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
123.12.123.0 0.0.0.0 255.255.254.0 U 1 0 0 eth0
169.254.0.0 0.0.0.0 255.255.0.0 U 1000 0 0 eth0
0.0.0.0 123.12.456.254 0.0.0.0 UG 0 0 0 eth0
$ sudo netstat -anp
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 765/portmap
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 871/sshd
tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN 1181/cupsd
tcp 0 0 0.0.0.0:52068 0.0.0.0:* LISTEN 786/rpc.statd
tcp 0 0 127.0.0.1:3306 0.0.0.0:* LISTEN 1186/mysqld
tcp 0 0 0.0.0.0:53709 0.0.0.0:* LISTEN -
tcp 0 0 123.12.123.235:755 123.12.5.48:2049 ESTABLISHED -
tcp 0 0 123.12.123.235:60793 123.12.5.129:8080 ESTABLISHED 2264/firefox-bin
tcp 0 0 123.12.123.235:57940 123.12.5.43:8080 ESTABLISHED 2264/firefox-bin
tcp 0 0 123.12.123.235:57928 123.12.5.43:8080 CLOSE_WAIT 2247/google-chrome
tcp 0 0 123.12.123.235:35767 123.12.5.129:8080 ESTABLISHED 2247/google-chrome
tcp 0 0 123.12.123.235:57930 123.12.5.43:8080 ESTABLISHED 2247/google-chrome
tcp 0 0 123.12.123.235:57931 123.12.5.43:8080 CLOSE_WAIT 2247/google-chrome
tcp6 0 0 :::80 :::* LISTEN 1278/apache2
tcp6 0 0 :::22 :::* LISTEN 871/sshd
tcp6 0 0 ::1:631 :::* LISTEN 1181/cupsd
tcp6 0 0 :::55934 :::* LISTEN 1956/eclipse
tcp6 0 0 :::5900 :::* LISTEN 1792/vino-server
udp 0 0 0.0.0.0:35631 0.0.0.0:* 912/avahi-daemon: r
udp 0 0 0.0.0.0:962 0.0.0.0:* 786/rpc.statd
udp 0 0 0.0.0.0:68 0.0.0.0:* 1575/dhclient
udp 0 0 0.0.0.0:46149 0.0.0.0:* -
udp 0 0 0.0.0.0:5353 0.0.0.0:* 912/avahi-daemon: r
udp 0 0 0.0.0.0:111 0.0.0.0:* 765/portmap
udp 0 0 0.0.0.0:36211 0.0.0.0:* 786/rpc.statd
udp 0 0 123.12.123.235:123 0.0.0.0:* 1689/ntpd
udp 0 0 127.0.0.1:123 0.0.0.0:* 1689/ntpd
udp 0 0 0.0.0.0:123 0.0.0.0:* 1689/ntpd
udp6 0 0 fe80::227:eff:fe07::123 :::* 1689/ntpd
udp6 0 0 ::1:123 :::* 1689/ntpd
udp6 0 0 :::123 :::* 1689/ntpd
Active UNIX domain sockets (servers and established) due to post size limit.
UPDATE 2
My proxy bypass settings:
I know this is a late answer, but I had the same problem and resolved it, so to tie up this one...
This is a combination of two bugs:
(i) Eclipse's internal help browser doesn't use the Eclipse proxy settings! See:
https://bugs.eclipse.org/bugs/show_bug.cgi?id=318969
(and the bugs referenced in comment #7 therein)
(ii) Ubuntu's proxy support is horribly broken in certain subtle ways. See:
https://bugs.launchpad.net/ubuntu/+bug/300271
The fix/workaround is to manually set the no_proxy environment variable before running eclipse (as reported in the Eclipse #308035 bug referenced from the 318969 one) e.g.
export no_proxy=127.0.0.1,localhost
eclipse &
Help then launches correctly within Eclipse. Of course, once Eclipse is launched (thus running its own internal HTTP server), you can also access the local help manually from another browser (or, if within the 'can't access 127.0.0.1' screen, there's an icon at the top to launch in an external window --> default system browser).
This may well apply on other Linux distros using Gnome.
[Couldn't post the 308035 bug link because my low reputation means I can only post 2 hyperlinks :-( Getting excited at this privilege come 10 reputation points :-)]
Basically, this error means that Eclipse is failing to establish a TCP/IP connection to your localhost using 127.0.0.1 (Eclipse starts a server for the Help).
If you are using some specific proxy settings (either global at the OS level or local at the Eclipse level), double check that you are bypassing the proxy for localhost and 127.0.0.1.
If this doesn't help, try setting the hostname that help uses to localhost when starting eclipse (either on the command line or in the eclipse.ini):
eclipse -vmargs -Dserver_host=localhost
Eclipse help is actually an HTTP server.
This is probably a permissions problem with your installation of Eclipse.
I have no suggestions except to check the permissions on your Eclipse folder, or delete and reinstall Eclipse.
had the same problem recently installing and running Eclipse on 9.10. Found that the default settings for Eclipse were fine but 9.10 had no proxy bypass set for 127.0.0.1 in its system settings. Also had to install Apache2 via Synaptic. I installed Apache2, did not change any settings for Apache2 and then went System > Preferences > Network Proxy Preferences clicked on Ignored Hosts and added "127.0.0.1". Reset the Eclipse Network Preferences back to default, restarted Eclipse and help worked perfectly. Hope this works for others.
David, thanks for the netstat output; you'll notice that Eclipse is listening on an IPv6 port:
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp6 0 0 :::55934 :::* LISTEN 1956/eclipse
Is your proxy configuration set to bypass both 127.0.0.1 and ::1?
Make sure your /etc/hosts file is properly done. Usually
if the line containing 127.0.0.1 has your host name remove it and just leave 'localhost'
if the opposite were true, try adding your hostname to it :)
such things happen because gnome is trying to match hostname and sockets to handle UI things. Might be worth asking on superuser..