How to debug "ERROR: Could not reach the worker node."? - bluedata-3.7

Question

I am trying to set up a gateway and a worker node on a BlueData 3.7 controller using SSH credentials.
I believe that I have met all the pre-requisites in the docs, however, I get an error trying to install the gateway and the worker in the Installation section of the controller UI:
The error I get for both hosts are:
ERROR: Could not reach the worker node.
The URL for the errors are:
http://x.x.x.x/bdswebui/logworker/?id=/api/v1/workers/4
http://x.x.x.x/bdswebui/logworker/?id=/api/v1/workers/5
I have checked the logs on the gateway and the worker. Both show:
# tree /var/log/bluedata/
/var/log/bluedata/
└── install
1 directory, 0 files
All hosts can ssh to each other without a password prompt.
No firewall is running:
# iptables --list
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
I've deleted the worker while listening in with tcpdump on the worker:
# tcpdump -i eth0 -ttttnnvvS src host x.x.x.x and tcp
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
2019-08-17 00:08:41.570940 IP (tos 0x0, ttl 64, id 3977, offset 0, flags [DF], proto TCP (6), length 48)
x.x.x.x.39039 > y.y.y.y.22: Flags [S], cksum 0x6215 (correct), seq 1453535125, win 29200, options [mss 1460,nop,wscale 9], length 0
...
x.x.x.x.46064 > y.y.y.y.22: Flags [F.], cksum 0x564b (correct), seq 1997087540, ack 4031219947, win 238, length 0
2019-08-17 00:14:54.710739 IP (tos 0x0, ttl 64, id 15525, offset 0, flags [DF], proto TCP (6), length 40)
x.x.x.x.46064 > y.y.y.y.22: Flags [.], cksum 0x564a (correct), seq 1997087541, ack 4031219948, win 238, length 0
Checking port 46064 on the controller, I can see that it is coming from a beam process which gives me confidence the network connectivity is ok between the two machines:
# lsof -i -P -n | grep 46064
beam.smp 12714 root 16u IPv4 498735 0t0 TCP x.x.x.x:46064->y.y.y.y:22 (ESTABLISHED)
What else can I do to debug?

Answer 1

The debug information I needed could be found in the /tmp folder on the worker and gateway hosts, e.g.
/tmp/bd_prechecks.nnnnn.log
/tmp/bd_prechecks.nnnnn.log.xtrace
/tmp/bds-nnnnnnnnnnnnnn.log
/tmp/bds-nnnnnnnnnnnnnn.log.xtrace
/tmp/worker_setup_x.x.x.x-n-n-n-n-n-n
/tmp/worker_setup_x.x.x.x-n-n-n-n-n-n.xtrace
For more information, see http://docs.bluedata.com/37_step-1-troubleshooting