Raspberry pi cluster: control_cb (pmiserv_cb.c:200): assert (!closed) failed - raspberry-pi

I have 2 PIs with ip 192.168.0.2 and 192.168.0.3. I can password-less ssh to each pi. (I used to have 3001 for ssh port, but now I change it back to 22)
I follow tinkernut's pi cluster tutorial: https://www.tinkernut.com/2014/05/make-cluster-computer-part-2/
I stuck in this part of tutorial: "mpiexec -f machinefile -n 4 hostname"
My command is simply "mpiexec -f machinefile -n 2 hostname"
After running my simple command, the output is:
pi01 (works!)
when it tries to reach pi02, the errors are
host: 192.168.0.2
host: 192.168.0.3
==================================================================================================
mpiexec options:
----------------
Base path: /home/rpimpi/mpi-install/bin/
Launcher: (null)
Debug level: 1
Enable X: -1
Global environment:
-------------------
MAIL=/var/mail/pi
USER=pi
SHLVL=1
HOME=/home/pi
OLDPWD=/home/pi/misc
HUSHLOGIN=FALSE
L1=tty1
LOGNAME=pi
_=./test.sh
TERM=linux
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/games:/usr/games:/home/rpimpi/mpi-install/bin
LANG=en_GB.UTF-8
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.axa=00;36:*.oga=00;36:*.spx=00;36:*.xspf=00;36:
SHELL=/bin/bash
PWD=/home/pi/misc/test
Hydra internal environment:
---------------------------
GFORTRAN_UNBUFFERED_PRECONNECTED=y
Proxy information:
*********************
[1] proxy: 192.168.0.2 (1 cores)
Exec list: hostname (1 processes);
[2] proxy: 192.168.0.3 (1 cores)
Exec list: hostname (1 processes);
==================================================================================================
[mpiexec#pi01] Timeout set to -1 (-1 means infinite)
[mpiexec#pi01] Got a control port string of 192.168.0.2:39681
Proxy launch args: /home/rpimpi/mpi-install/bin/hydra_pmi_proxy --control-port 192.168.0.2:39681 --debug --rmk user --launcher ssh --demux poll --pgid 0 --retries 10 --usize -2 --proxy-id
Arguments being passed to proxy 0:
--version 3.1.4 --iface-ip-env-name MPIR_CVAR_CH3_INTERFACE_HOSTNAME --hostname 192.168.0.2 --global-core-map 0,1,2 --pmi-id-map 0,0 --global-process-count 2 --auto-cleanup 1 --pmi-kvsname kvs_2638_0 --pmi-process-mapping (vector,(0,2,1)) --ckpoint-num -1 --global-inherited-env 15 'MAIL=/var/mail/pi' 'USER=pi' 'SHLVL=1' 'HOME=/home/pi' 'OLDPWD=/home/pi/misc' 'HUSHLOGIN=FALSE' 'L1=tty1' 'LOGNAME=pi' '_=./test.sh' 'TERM=linux' 'PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/games:/usr/games:/home/rpimpi/mpi-install/bin' 'LANG=en_GB.UTF-8' 'LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.axa=00;36:*.oga=00;36:*.spx=00;36:*.xspf=00;36:' 'SHELL=/bin/bash' 'PWD=/home/pi/misc/test' --global-user-env 0 --global-system-env 1 'GFORTRAN_UNBUFFERED_PRECONNECTED=y' --proxy-core-count 1 --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /home/pi/misc/test --exec-args 1 hostname
Arguments being passed to proxy 1:
--version 3.1.4 --iface-ip-env-name MPIR_CVAR_CH3_INTERFACE_HOSTNAME --hostname 192.168.0.3 --global-core-map 0,1,2 --pmi-id-map 0,1 --global-process-count 2 --auto-cleanup 1 --pmi-kvsname kvs_2638_0 --pmi-process-mapping (vector,(0,2,1)) --ckpoint-num -1 --global-inherited-env 15 'MAIL=/var/mail/pi' 'USER=pi' 'SHLVL=1' 'HOME=/home/pi' 'OLDPWD=/home/pi/misc' 'HUSHLOGIN=FALSE' 'L1=tty1' 'LOGNAME=pi' '_=./test.sh' 'TERM=linux' 'PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/games:/usr/games:/home/rpimpi/mpi-install/bin' 'LANG=en_GB.UTF-8' 'LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.axa=00;36:*.oga=00;36:*.spx=00;36:*.xspf=00;36:' 'SHELL=/bin/bash' 'PWD=/home/pi/misc/test' --global-user-env 0 --global-system-env 1 'GFORTRAN_UNBUFFERED_PRECONNECTED=y' --proxy-core-count 1 --exec --exec-appnum 0 --exec-proc-count 1 --exec-local-env 0 --exec-wdir /home/pi/misc/test --exec-args 1 hostname
[mpiexec#pi01] Launch arguments: /home/rpimpi/mpi-install/bin/hydra_pmi_proxy --control-port 192.168.0.2:39681 --debug --rmk user --launcher ssh --demux poll --pgid 0 --retries 10 --usize -2 --proxy-id 0
[mpiexec#pi01] Launch arguments: /usr/bin/ssh -x 192.168.0.3 "/home/rpimpi/mpi-install/bin/hydra_pmi_proxy" --control-port 192.168.0.2:39681 --debug --rmk user --launcher ssh --demux poll --pgid 0 --retries 10 --usize -2 --proxy-id 1
pi01
[mpiexec#pi01] control_cb (/home/pi/Downloads/mpich3/mpich-3.1.4/src/pm/hydra/pm/pmiserv/pmiserv_cb.c:200): assert (!closed) failed
[mpiexec#pi01] HYDT_dmxu_poll_wait_for_event (/home/pi/Downloads/mpich3/mpich-3.1.4/src/pm/hydra/tools/demux/demux_poll.c:76): callback returned error status
[mpiexec#pi01] HYD_pmci_wait_for_completion (/home/pi/Downloads/mpich3/mpich-3.1.4/src/pm/hydra/pm/pmiserv/pmiserv_pmci.c:198): error waiting for event
[mpiexec#pi01] main (/home/pi/Downloads/mpich3/mpich-3.1.4/src/pm/hydra/ui/mpich/mpiexec.c:344): process manager error waiting for completion
Research I done so far:
https://stackoverflow.com/questions/27419782/error-while-executing-mpi
mpiexec checkpointing error (RPi)
MPICH example cpi generates error when it runs on multiple fresh installed vps


I realized that I don’t use switch to connect all PIs, instead of I use wifi dongle for each PI. I ssh to the PI1 itself, this forces to add to know_hosts under .ssh. I did this to PI2 as well. Now I am able exeucte this command: “mpiexec -f machinefile -n 2 hostname”. It outputs “pi01” and “pi02”. If it is still not working, I recommend you back up your .ssh and regenerate a new one.
http://kenpeter.github.io/raspberry/pi/2015/07/13/control-cb-assert-failed.html

Related

VSCode Remote-SSH extension through SSH gateway?

I need to access several hosts that are behind an SSH gateway.
The way the gateway works is:
I upload my public key to a web portal
I get an FQDN and port for SSH access
My client connects with the gateway, which then connects me to the specific host I'm trying to connect to based on the port I came in on
My SSH key is never actually deployed to the remote host itself. It's held by the gateway which has its own keypair that is distributed to the remote host--I do not have access to this private key.
With default settings, here's the logs:
[07:25:44.335] Log Level: 2
[07:25:44.337] remote-ssh#0.65.7
[07:25:44.337] darwin arm64
[07:25:44.338] SSH Resolver called for "ssh-remote+HOSTNAME", attempt 1
[07:25:44.338] "remote.SSH.useLocalServer": true
[07:25:44.338] "remote.SSH.path": undefined
[07:25:44.338] "remote.SSH.configFile": undefined
[07:25:44.338] "remote.SSH.useFlock": true
[07:25:44.338] "remote.SSH.lockfilesInTmp": false
[07:25:44.338] "remote.SSH.localServerDownload": auto
[07:25:44.339] "remote.SSH.remoteServerListenOnSocket": false
[07:25:44.339] "remote.SSH.showLoginTerminal": false
[07:25:44.339] "remote.SSH.defaultExtensions": []
[07:25:44.339] "remote.SSH.loglevel": 2
[07:25:44.339] SSH Resolver called for host: HOSTNAME
[07:25:44.339] Setting up SSH remote "HOSTNAME"
[07:25:44.341] Acquiring local install lock: /var/folders/_9/vfmchjgd5gl2q6s8p3y1pmg80000gn/T/vscode-remote-ssh-HOSTNAME-install.lock
[07:25:44.349] Looking for existing server data file at /Users/josh/Library/Application Support/Code/User/globalStorage/ms-vscode-remote.remote-ssh/vscode-ssh-host-HOSTNAME-379476f0e13988d90fab105c5c19e7abc8b1dea8-0.65.7/data.json
[07:25:44.349] Using commit id "379476f0e13988d90fab105c5c19e7abc8b1dea8" and quality "stable" for server
[07:25:44.351] Install and start server if needed
[07:25:44.354] PATH: /opt/homebrew/bin:/opt/homebrew/sbin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Applications/Wireshark.app/Contents/MacOS
[07:25:44.354] Checking ssh with "ssh -V"
[07:25:44.358] > OpenSSH_8.1p1, LibreSSL 2.7.3
[07:25:44.359] askpass server listening on /var/folders/_9/vfmchjgd5gl2q6s8p3y1pmg80000gn/T/vscode-ssh-askpass-a4f2481dc32265d4aa0d1997d55a93523914f69d.sock
[07:25:44.360] Spawning local server with {"serverId":1,"ipcHandlePath":"/var/folders/_9/vfmchjgd5gl2q6s8p3y1pmg80000gn/T/vscode-ssh-askpass-04d1b772731e5c58d6a1e3212e8ab358ab99d39c.sock","sshCommand":"ssh","sshArgs":["-v","-T","-D","52496","-o","ConnectTimeout=15","HOSTNAME","bash"],"dataFilePath":"/Users/josh/Library/Application Support/Code/User/globalStorage/ms-vscode-remote.remote-ssh/vscode-ssh-host-HOSTNAME-379476f0e13988d90fab105c5c19e7abc8b1dea8-0.65.7/data.json"}
[07:25:44.360] Local server env: {"DISPLAY":"1","ELECTRON_RUN_AS_NODE":"1","SSH_ASKPASS":"/Users/josh/.vscode/extensions/ms-vscode-remote.remote-ssh-0.65.7/out/local-server/askpass.sh","VSCODE_SSH_ASKPASS_NODE":"/Applications/Visual Studio Code.app/Contents/Frameworks/Code Helper (Renderer).app/Contents/MacOS/Code Helper (Renderer)","VSCODE_SSH_ASKPASS_MAIN":"/Users/josh/.vscode/extensions/ms-vscode-remote.remote-ssh-0.65.7/out/askpass-main.js","VSCODE_SSH_ASKPASS_HANDLE":"/var/folders/_9/vfmchjgd5gl2q6s8p3y1pmg80000gn/T/vscode-ssh-askpass-a4f2481dc32265d4aa0d1997d55a93523914f69d.sock"}
[07:25:44.360] Spawned 6028
[07:25:44.411] > local-server-1> Spawned ssh, pid=6034
[07:25:44.413] stderr> OpenSSH_8.1p1, LibreSSL 2.7.3
[07:25:44.615] stderr> debug1: Server host key: ssh-rsa SHA256:uvobnEougyevd5GyZzHzqTJ2C/vx2RgT1agdqvlXZRc
[07:25:44.743] stderr> This is a private system; unauthorized access is prohibited. Disconnect immediately if you have not been explicitly authorized to access this system.
[07:25:45.714] stderr> Authenticated to 54.189.187.155 ([54.189.187.155]:47000).
[07:25:46.440] > ready: 9df4e9b85f9c
[07:25:46.496] > Linux 5.3.0-1017-aws #18~18.04.1-Ubuntu SMP Wed Apr 8 15:12:16 UTC 2020
[07:25:46.496] Platform: linux
[07:25:46.564] > 9df4e9b85f9c: running
> Acquiring lock on /home/ubuntu/.vscode-server/bin/379476f0e13988d90fab105c5c19e7abc8b1dea8/vscode-remote-lock.ubuntu.379476f0e13988d90fab105c5c19e7abc8b1dea8
[07:25:46.588] > Installing to /home/ubuntu/.vscode-server/bin/379476f0e13988d90fab105c5c19e7abc8b1dea8...
> 9df4e9b85f9c%%1%%
> Downloading with wget
[07:25:48.464] > Download complete
> 9df4e9b85f9c%%2%%
> tar --version:
> tar (GNU tar) 1.29
> Copyright (C) 2015 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.
>
> Written by John Gilmore and Jay Fenlason.
[07:25:49.883] > Checking /home/ubuntu/.vscode-server/.379476f0e13988d90fab105c5c19e7abc8b1dea8.log and /home/ubuntu/.vscode-server/.379476f0e13988d90fab105c5c19e7abc8b1dea8.pid for a running server
> WARNING: VS Code Server is running but its logfile is missing. Don't delete the VS Code Server manually, run the command 'Uninstall VS Code Server'.
> Starting server with command... /home/ubuntu/.vscode-server/bin/379476f0e13988d90fab105c5c19e7abc8b1dea8/server.sh --start-server --host=127.0.0.1 --enable-remote-auto-shutdown --port=0 &> "/home/ubuntu/.vscode-server/.379476f0e13988d90fab105c5c19e7abc8b1dea8.log" < /dev/null
> printenv:
> SSH_CONNECTION=10.1.1.254 45382 10.1.1.4 22
> LANG=C.UTF-8
> XDG_SESSION_ID=74
> USER=ubuntu
> PWD=/home/ubuntu
> HOME=/home/ubuntu
> SSH_CLIENT=10.1.1.254 45382 22
> MAIL=/var/mail/ubuntu
> SHELL=/bin/bash
> SHLVL=3
> VSCODE_AGENT_FOLDER=/home/ubuntu/.vscode-server
> LOGNAME=ubuntu
> XDG_RUNTIME_DIR=/run/user/1000
> PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
> _=/usr/bin/printenv
> OLDPWD=/home/ubuntu/.vscode-server/bin/379476f0e13988d90fab105c5c19e7abc8b1dea8
> Spawned remote server: 19082
[07:25:49.917] > Waiting for server log...
[07:25:49.941] > Waiting for server log...
[07:25:49.989] >
> *
> * Reminder: You may only use this software with Visual Studio family products,
> * as described in the license (https://go.microsoft.com/fwlink/?linkid=2077057)
> *
>
> Checking server status on port 43643 with wget
[07:25:50.158] > 9df4e9b85f9c: start
> SSH_AUTH_SOCK====
> DISPLAY====
> webUiAccessToken====
> listeningOn==43643==
> osReleaseId==ubuntu==
> arch==x86_64==
> tmpDir==/run/user/1000==
> platform==linux==
> unpackResult==success==
> didLocalDownload==0==
> downloadTime==1868==
> installTime==1402==
> extInstallTime====
> serverStartTime==88==
> connectionToken==1111aaa1-11a1-111a-11a1-1a111a111a11==
> 9df4e9b85f9c: end
[07:25:50.158] Received install output:
SSH_AUTH_SOCK====
DISPLAY====
webUiAccessToken====
listeningOn==43643==
osReleaseId==ubuntu==
arch==x86_64==
tmpDir==/run/user/1000==
platform==linux==
unpackResult==success==
didLocalDownload==0==
downloadTime==1868==
installTime==1402==
extInstallTime====
serverStartTime==88==
connectionToken==1111aaa1-11a1-111a-11a1-1a111a111a11==
[07:25:50.159] Remote server is listening on 43643
[07:25:50.159] Parsed server configuration: {"serverConfiguration":{"remoteListeningOn":{"port":43643},"osReleaseId":"ubuntu","arch":"x86_64","webUiAccessToken":"","sshAuthSock":"","display":"","tmpDir":"/run/user/1000","platform":"linux","connectionToken":"1111aaa1-11a1-111a-11a1-1a111a111a11"},"downloadTime":1868,"installTime":1402,"serverStartTime":88,"installUnpackCode":"success"}
[07:25:50.160] Persisting server connection details to /Users/josh/Library/Application Support/Code/User/globalStorage/ms-vscode-remote.remote-ssh/vscode-ssh-host-HOSTNAME-379476f0e13988d90fab105c5c19e7abc8b1dea8-0.65.7/data.json
[07:25:50.162] Starting forwarding server. localPort 52500 -> socksPort 52496 -> remotePort 43643
[07:25:50.162] Forwarding server listening on 52500
[07:25:50.163] Waiting for ssh tunnel to be ready
[07:25:50.164] Tunneled 43643 to local port 52500
[07:25:50.164] Resolved "ssh-remote+HOSTNAME" to "127.0.0.1:52500"
[07:25:50.165] [Forwarding server 52500] Got connection 0
[07:25:50.171] ------
[07:25:50.198] [Forwarding server 52500] Got connection 1
[07:25:50.220] Failed to set up socket for dynamic port forward to remote port 43643: Socket closed. Is the remote port correct?
[07:25:50.245] Failed to set up socket for dynamic port forward to remote port 43643: Socket closed. Is the remote port correct?
I ran "rm -rf ~/.vscode-server", then I then followed the instructions under "Connecting to systems that dynamically assign machines per connection" on this page: https://code.visualstudio.com/docs/remote/troubleshooting
Here's my ~/.ssh/config:
Host ubuntu
HostName <HOSTNAME>
Port 47000
ControlMaster auto
ControlPath ~/.ssh/sockets/%r#%h:%p
ControlPersist 600
And here's the remote server's sshd_config (I'm using a Mac so ControlMaster should work):
# $OpenBSD: sshd_config,v 1.101 2017/03/14 07:19:07 djm Exp $
# This is the sshd server system-wide configuration file. See
# sshd_config(5) for more information.
# This sshd was compiled with PATH=/usr/bin:/bin:/usr/sbin:/sbin
# The strategy used for options in the default sshd_config shipped with
# OpenSSH is to specify options with their default value where
# possible, but leave them commented. Uncommented options override the
# default value.
#Port 22
#AddressFamily any
#ListenAddress 0.0.0.0
#ListenAddress ::
#HostKey /etc/ssh/ssh_host_rsa_key
#HostKey /etc/ssh/ssh_host_ecdsa_key
#HostKey /etc/ssh/ssh_host_ed25519_key
# Ciphers and keying
#RekeyLimit default none
# Logging
#SyslogFacility AUTH
#LogLevel INFO
# Authentication:
#LoginGraceTime 2m
#PermitRootLogin prohibit-password
#StrictModes yes
#MaxAuthTries 6
#MaxSessions 10
#PubkeyAuthentication yes
# Expect .ssh/authorized_keys2 to be disregarded by default in future.
#AuthorizedKeysFile .ssh/authorized_keys .ssh/authorized_keys2
#AuthorizedPrincipalsFile none
#AuthorizedKeysCommand none
#AuthorizedKeysCommandUser nobody
# For this to work you will also need host keys in /etc/ssh/ssh_known_hosts
#HostbasedAuthentication no
# Change to yes if you don't trust ~/.ssh/known_hosts for
# HostbasedAuthentication
#IgnoreUserKnownHosts no
# Don't read the user's ~/.rhosts and ~/.shosts files
#IgnoreRhosts yes
# To disable tunneled clear text passwords, change to no here!
PasswordAuthentication no
#PermitEmptyPasswords no
# Change to yes to enable challenge-response passwords (beware issues with
# some PAM modules and threads)
ChallengeResponseAuthentication no
# Kerberos options
#KerberosAuthentication no
#KerberosOrLocalPasswd yes
#KerberosTicketCleanup yes
#KerberosGetAFSToken no
# GSSAPI options
#GSSAPIAuthentication no
#GSSAPICleanupCredentials yes
#GSSAPIStrictAcceptorCheck yes
#GSSAPIKeyExchange no
# Set this to 'yes' to enable PAM authentication, account processing,
# and session processing. If this is enabled, PAM authentication will
# be allowed through the ChallengeResponseAuthentication and
# PasswordAuthentication. Depending on your PAM configuration,
# PAM authentication via ChallengeResponseAuthentication may bypass
# the setting of "PermitRootLogin without-password".
# If you just want the PAM account and session checks to run without
# PAM authentication, then enable this but set PasswordAuthentication
# and ChallengeResponseAuthentication to 'no'.
UsePAM yes
AllowStreamLocalForwarding yes
#AllowAgentForwarding yes
AllowTcpForwarding yes
#GatewayPorts no
X11Forwarding yes
#X11DisplayOffset 10
#X11UseLocalhost yes
#PermitTTY yes
PrintMotd no
#PrintLastLog yes
#TCPKeepAlive yes
#UseLogin no
#PermitUserEnvironment no
#Compression delayed
#ClientAliveInterval 0
#ClientAliveCountMax 3
#UseDNS no
#PidFile /var/run/sshd.pid
#MaxStartups 10:30:100
#PermitTunnel no
#ChrootDirectory none
#VersionAddendum none
# no default banner path
#Banner none
# Allow client to pass locale environment variables
AcceptEnv LANG LC_*
# override default of no subsystems
Subsystem sftp /usr/lib/openssh/sftp-server
# Example of overriding settings on a per-user basis
#Match User anoncvs
# X11Forwarding no
# AllowTcpForwarding no
# PermitTTY no
# ForceCommand cvs server
After attempting to reconnect, these are the logs:
[07:35:52.712] Log Level: 2
[07:35:52.714] remote-ssh#0.65.7
[07:35:52.714] darwin arm64
[07:35:52.715] SSH Resolver called for "ssh-remote+ubuntu", attempt 1
[07:35:52.715] "remote.SSH.useLocalServer": true
[07:35:52.715] "remote.SSH.path": undefined
[07:35:52.715] "remote.SSH.configFile": undefined
[07:35:52.715] "remote.SSH.useFlock": true
[07:35:52.715] "remote.SSH.lockfilesInTmp": false
[07:35:52.715] "remote.SSH.localServerDownload": auto
[07:35:52.715] "remote.SSH.remoteServerListenOnSocket": false
[07:35:52.716] "remote.SSH.showLoginTerminal": false
[07:35:52.716] "remote.SSH.defaultExtensions": []
[07:35:52.716] "remote.SSH.loglevel": 2
[07:35:52.716] SSH Resolver called for host: ubuntu
[07:35:52.716] Setting up SSH remote "ubuntu"
[07:35:52.717] Acquiring local install lock: /var/folders/_9/vfmchjgd5gl2q6s8p3y1pmg80000gn/T/vscode-remote-ssh-ubuntu-install.lock
[07:35:52.726] Looking for existing server data file at /Users/josh/Library/Application Support/Code/User/globalStorage/ms-vscode-remote.remote-ssh/vscode-ssh-host-ubuntu-379476f0e13988d90fab105c5c19e7abc8b1dea8-0.65.7/data.json
[07:35:52.726] Using commit id "379476f0e13988d90fab105c5c19e7abc8b1dea8" and quality "stable" for server
[07:35:52.728] Install and start server if needed
[07:35:52.730] PATH: /opt/homebrew/bin:/opt/homebrew/sbin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Applications/Wireshark.app/Contents/MacOS
[07:35:52.730] Checking ssh with "ssh -V"
[07:35:52.733] > OpenSSH_8.1p1, LibreSSL 2.7.3
[07:35:52.734] askpass server listening on /var/folders/_9/vfmchjgd5gl2q6s8p3y1pmg80000gn/T/vscode-ssh-askpass-53beece063b2e0e2b4c364baee37f2170d54995c.sock
[07:35:52.735] Spawning local server with {"serverId":1,"ipcHandlePath":"/var/folders/_9/vfmchjgd5gl2q6s8p3y1pmg80000gn/T/vscode-ssh-askpass-ef52c1a9145be571fa987d52436563f7df923b30.sock","sshCommand":"ssh","sshArgs":["-v","-T","-D","52611","-o","ConnectTimeout=15","ubuntu"],"dataFilePath":"/Users/josh/Library/Application Support/Code/User/globalStorage/ms-vscode-remote.remote-ssh/vscode-ssh-host-ubuntu-379476f0e13988d90fab105c5c19e7abc8b1dea8-0.65.7/data.json"}
[07:35:52.735] Local server env: {"DISPLAY":"1","ELECTRON_RUN_AS_NODE":"1","SSH_ASKPASS":"/Users/josh/.vscode/extensions/ms-vscode-remote.remote-ssh-0.65.7/out/local-server/askpass.sh","VSCODE_SSH_ASKPASS_NODE":"/Applications/Visual Studio Code.app/Contents/Frameworks/Code Helper (Renderer).app/Contents/MacOS/Code Helper (Renderer)","VSCODE_SSH_ASKPASS_MAIN":"/Users/josh/.vscode/extensions/ms-vscode-remote.remote-ssh-0.65.7/out/askpass-main.js","VSCODE_SSH_ASKPASS_HANDLE":"/var/folders/_9/vfmchjgd5gl2q6s8p3y1pmg80000gn/T/vscode-ssh-askpass-53beece063b2e0e2b4c364baee37f2170d54995c.sock"}
[07:35:52.735] Spawned 6261
[07:35:52.786] > local-server-1> Spawned ssh, pid=6267
[07:35:52.788] stderr> OpenSSH_8.1p1, LibreSSL 2.7.3
[07:35:52.973] stderr> debug1: Server host key: ssh-rsa SHA256:uvobnEougyevd5GyZzHzqTJ2C/vx2RgT1agdqvlXZRc
[07:35:53.108] stderr> This is a private system; unauthorized access is prohibited. Disconnect immediately if you have not been explicitly authorized to access this system.
[07:35:53.879] stderr> Authenticated to HOSTNAME ([HOSTNAME]:47000).
[07:35:54.565] > -- UDF SSH ---------------------
> Deployment: f5 CIS for k8s w calico (w lab guide)
> Component: kube-master
> --------------------------------
> Welcome to Ubuntu 18.04.4 LTS (GNU/Linux 5.3.0-1017-aws x86_64)
>
> * Documentation: https://help.ubuntu.com
> * Management: https://landscape.canonical.com
> * Support: https://ubuntu.com/advantage
>
> System information as of Wed Aug 11 14:35:54 UTC 2021
>
> System load: 0.16 IP address for eth0: 10.1.1.4
> Usage of /: 31.3% of 19.32GB IP address for eth1: 10.1.20.20
> Memory usage: 17% IP address for docker0: 172.17.0.1
> Swap usage: 0% IP address for tunl0: 192.168.106.0
> Processes: 232 IP address for tailscale0: 100.71.141.61
> Users logged in: 1
>
> * Super-optimized for small spaces - read how we shrank the memory
> footprint of MicroK8s to make it the smallest full K8s around.
>
> https://ubuntu.com/blog/microk8s-memory-optimisation
>
> Get cloud support with Ubuntu Advantage Cloud Guest:
> http://www.ubuntu.com/business/services/cloud
>
> * Canonical Livepatch is available for installation.
> - Reduce system reboots and improve kernel security. Activate at:
> https://ubuntu.com/livepatch
>
> 187 packages can be updated.
> 135 updates are security updates.
>
> New release '20.04.2 LTS' available.
> Run 'do-release-upgrade' to upgrade to it.
>
>
> Last login: Wed Aug 11 14:24:27 2021 from 10.1.1.254
[07:35:54.588] > echo 'ready: c81f6e315e44'
[07:35:54.613] > ubuntu#ip-10-1-1-4:~$ echo 'ready: c81f6e315e44'
> ready: c81f6e315e44
> ubuntu#ip-10-1-1-4:~$
[07:35:54.638] > uname -rsv
> Linux 5.3.0-1017-aws #18~18.04.1-Ubuntu SMP Wed Apr 8 15:12:16 UTC 2020
> ubuntu#ip-10-1-1-4:~$
[07:35:54.638] Platform: linux
[07:35:54.690] > bash
> ubuntu#ip-10-1-1-4:~$
[07:35:54.691] > function mainScriptWrapper {
> >
> > echo "c81f6e315e44: running"
> >
> v> #
> > # Setup
> sion"
> f> #
> > VSC_TMP="${XDG_RUNTIME_DIR:-`echo /tmp`}"
> > COMMIT_ID="379476f0e13988d90fab105c5c19e7abc8b1dea8"
> > EXTENSIONS=""
> > TELEMETRY=""
> > export VSCODE_AGENT_FOLDER=$HOME/.vscode-server
> >
[07:35:54.694] > ALLOW_CLIENT_DOWNLOAD="1"
> > FORCE_CLIENT_DOWNLOAD="0"
> > now=`date +%s`
> > SERVER_LISTEN_FLAG=--port=0
> > UNPACK_RESULT=
> > DID_LOCAL_DOWNLOAD=0
> > DOWNLOAD_TIME=
> ed > INSTALL_TIME=
> > EXT_INSTALL_TIME=
> icen> SERVER_START_TIME=
> > CONNECTION_TOKEN=
[07:35:54.713] > >
> > VSCH_DIR="$VSCODE_AGENT_FOLDER/bin/$COMMIT_ID"
> > VSCH_LOGFILE="$VSCODE_AGENT_FOLDER/.$COMMIT_ID.log"
> > VSCH_PIDFILE="$VSCODE_AGENT_FOLDER/.$COMMIT_ID.pid"
> > VSCH_TOKENFILE="$VSCODE_AGENT_FOLDER/.$COMMIT_ID.token"
> >
> > fail_with_exitcode() {
>
> fi
> >
[07:35:54.736] > echo "c81f6e315e44: start"
> > echo "exitCode==$1=="
> >
[07:35:54.761] > echo_common_results
> [[>
[07:35:54.785] > echo "c81f6e315e44: end"
> >
[07:35:54.786] Received install output: "
> echo "exitCode==$1=="
> echo_common_results
[[> echo "
[07:35:54.788] Failed to parse remote port from server output
[07:35:54.789] Terminating local server
[07:35:54.791] Resolver error: Error:
at Function.Create (/Users/josh/.vscode/extensions/ms-vscode-remote.remote-ssh-0.65.7/out/extension.js:1:64597)
at Object.t.handleInstallOutput (/Users/josh/.vscode/extensions/ms-vscode-remote.remote-ssh-0.65.7/out/extension.js:1:63240)
at Object.e [as tryInstallWithLocalServer] (/Users/josh/.vscode/extensions/ms-vscode-remote.remote-ssh-0.65.7/out/extension.js:1:387511)
at processTicksAndRejections (internal/process/task_queues.js:93:5)
at async /Users/josh/.vscode/extensions/ms-vscode-remote.remote-ssh-0.65.7/out/extension.js:1:294411
at async Object.t.withShowDetailsEvent (/Users/josh/.vscode/extensions/ms-vscode-remote.remote-ssh-0.65.7/out/extension.js:1:406401)
at async /Users/josh/.vscode/extensions/ms-vscode-remote.remote-ssh-0.65.7/out/extension.js:1:386050
at async E (/Users/josh/.vscode/extensions/ms-vscode-remote.remote-ssh-0.65.7/out/extension.js:1:382648)
at async Object.t.resolveWithLocalServer (/Users/josh/.vscode/extensions/ms-vscode-remote.remote-ssh-0.65.7/out/extension.js:1:385666)
at async Object.t.resolve (/Users/josh/.vscode/extensions/ms-vscode-remote.remote-ssh-0.65.7/out/extension.js:1:295808)
at async /Users/josh/.vscode/extensions/ms-vscode-remote.remote-ssh-0.65.7/out/extension.js:127:110656
[07:35:54.798] ------
The final thing I attempted was setting "remote.SSH.remoteServerListenOnSocket": true in the VScode Remote SSH settings. I also already had "AllowStreamLocalForwarding yes" in the sshd_config.
[07:40:37.993] Log Level: 2
[07:40:37.994] remote-ssh#0.65.7
[07:40:37.995] darwin arm64
[07:40:37.995] SSH Resolver called for "ssh-remote+ubuntu", attempt 1
[07:40:37.995] "remote.SSH.useLocalServer": false
[07:40:37.995] "remote.SSH.showLoginTerminal": false
[07:40:37.996] "remote.SSH.remotePlatform": {"10.10.20.201":"macOS","100.127.23.80":"linux","4b7cb23c-b5a6-4459-b9e7-940a9c7fe875.access.udf.f5.com":"linux","ds918":"linux"}
[07:40:37.996] "remote.SSH.path": undefined
[07:40:37.996] "remote.SSH.configFile": undefined
[07:40:37.996] "remote.SSH.useFlock": true
[07:40:37.996] "remote.SSH.lockfilesInTmp": false
[07:40:37.996] "remote.SSH.localServerDownload": auto
[07:40:37.996] "remote.SSH.remoteServerListenOnSocket": true
[07:40:37.996] "remote.SSH.showLoginTerminal": false
[07:40:37.996] "remote.SSH.defaultExtensions": []
[07:40:37.996] "remote.SSH.loglevel": 2
[07:40:37.996] SSH Resolver called for host: ubuntu
[07:40:37.996] Setting up SSH remote "ubuntu"
[07:40:38.005] Using commit id "379476f0e13988d90fab105c5c19e7abc8b1dea8" and quality "stable" for server
[07:40:38.007] Install and start server if needed
[07:40:39.167] PATH: /opt/homebrew/bin:/opt/homebrew/sbin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Applications/Wireshark.app/Contents/MacOS
[07:40:39.167] Checking ssh with "ssh -V"
[07:40:39.171] > OpenSSH_8.1p1, LibreSSL 2.7.3
[07:40:39.173] Running script with connection command: ssh -T -D 52660 -o ConnectTimeout=15 ubuntu bash
[07:40:40.134] > 786dc09e271f: running
> Acquiring lock on /home/ubuntu/.vscode-server/bin/379476f0e13988d90fab105c5c19e7abc8b1dea8/vscode-remote-lock.ubuntu.379476f0e13988d90fab105c5c19e7abc8b1dea8
> Installing to /home/ubuntu/.vscode-server/bin/379476f0e13988d90fab105c5c19e7abc8b1dea8...
> 786dc09e271f%%1%%
> Downloading with wget
[07:40:40.135] Got some output, clearing connection timeout
[07:40:42.399] > Download complete
> 786dc09e271f%%2%%
> tar --version:
> tar (GNU tar) 1.29
> Copyright (C) 2015 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.
>
> Written by John Gilmore and Jay Fenlason.
[07:40:43.749] > Checking /home/ubuntu/.vscode-server/.379476f0e13988d90fab105c5c19e7abc8b1dea8.log and /home/ubuntu/.vscode-server/.379476f0e13988d90fab105c5c19e7abc8b1dea8.pid for a running server
> WARNING: VS Code Server is running but its logfile is missing. Don't delete the VS Code Server manually, run the command 'Uninstall VS Code Server'.
> Starting server with command... /home/ubuntu/.vscode-server/bin/379476f0e13988d90fab105c5c19e7abc8b1dea8/server.sh --start-server --host=127.0.0.1 --enable-remote-auto-shutdown --socket-path=/run/user/1000/vscode-ssh-remote-server-sock-1628692840 &> "/home/ubuntu/.vscode-server/.379476f0e13988d90fab105c5c19e7abc8b1dea8.log" < /dev/null
> printenv:
> SSH_CONNECTION=10.1.1.254 10190 10.1.1.4 22
> LANG=C.UTF-8
> XDG_SESSION_ID=76
> USER=ubuntu
> PWD=/home/ubuntu
> HOME=/home/ubuntu
> SSH_CLIENT=10.1.1.254 10190 22
> MAIL=/var/mail/ubuntu
> SHELL=/bin/bash
> SHLVL=2
> VSCODE_AGENT_FOLDER=/home/ubuntu/.vscode-server
> LOGNAME=ubuntu
> XDG_RUNTIME_DIR=/run/user/1000
> PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
> _=/usr/bin/printenv
> OLDPWD=/home/ubuntu/.vscode-server/bin/379476f0e13988d90fab105c5c19e7abc8b1dea8
> Spawned remote server: 4972
[07:40:43.773] > Waiting for server log...
[07:40:43.798] > Waiting for server log...
[07:40:43.860] >
> *
> * Reminder: You may only use this software with Visual Studio family products,
> * as described in the license (https://go.microsoft.com/fwlink/?linkid=2077057)
> *
>
> Checking server status on socket /run/user/1000/vscode-ssh-remote-server-sock-1628692840 with curl
[07:40:44.016] > 786dc09e271f: start
> SSH_AUTH_SOCK====
> DISPLAY====
> webUiAccessToken====
> listeningOn==/run/user/1000/vscode-ssh-remote-server-sock-1628692840==
> osReleaseId==ubuntu==
> arch==x86_64==
> tmpDir==/run/user/1000==
> platform==linux==
> unpackResult==success==
> didLocalDownload==0==
> downloadTime==2266==
> installTime==1329==
> extInstallTime====
> serverStartTime==84==
> connectionToken==111a1111-a111-1a11-1111-11111aaa111a==
> 786dc09e271f: end
[07:40:44.016] Received install output:
SSH_AUTH_SOCK====
DISPLAY====
webUiAccessToken====
listeningOn==/run/user/1000/vscode-ssh-remote-server-sock-1628692840==
osReleaseId==ubuntu==
arch==x86_64==
tmpDir==/run/user/1000==
platform==linux==
unpackResult==success==
didLocalDownload==0==
downloadTime==2266==
installTime==1329==
extInstallTime====
serverStartTime==84==
connectionToken==111a1111-a111-1a11-1111-11111aaa111a==
[07:40:44.018] Remote server is listening on /run/user/1000/vscode-ssh-remote-server-sock-1628692840
[07:40:44.018] Parsed server configuration: {"serverConfiguration":{"remoteListeningOn":{"socketPath":"/run/user/1000/vscode-ssh-remote-server-sock-1628692840"},"osReleaseId":"ubuntu","arch":"x86_64","webUiAccessToken":"","sshAuthSock":"","display":"","tmpDir":"/run/user/1000","platform":"linux","connectionToken":"111a1111-a111-1a11-1111-11111aaa111a"},"downloadTime":2266,"installTime":1329,"serverStartTime":84,"installUnpackCode":"success"}
[07:40:44.022] Running script with connection command: ssh -T -L 127.0.0.1:52664:/run/user/1000/vscode-ssh-remote-server-sock-1628692840 -o ConnectTimeout=15 ubuntu bash
[07:40:44.022] Spawning tunnel with: ssh -T -L 127.0.0.1:52664:/run/user/1000/vscode-ssh-remote-server-sock-1628692840 -o ConnectTimeout=15 ubuntu bash << 'EOSSH'
echo SSH_AUTH_SOCK=$SSH_AUTH_SOCK && echo -e 'Connected to SSH Host - Please do not close this terminal' && while true; do sleep 180; echo -n ' '; done
EOSSH
[07:40:44.925] > SSH_AUTH_SOCK=
> Connected to SSH Host - Please do not close this terminal
[07:40:44.925] Spawned SSH tunnel between local port 52664 and remote target /run/user/1000/vscode-ssh-remote-server-sock-1628692840
[07:40:44.926] Waiting for ssh tunnel to be ready
[07:40:44.928] Tunneled /run/user/1000/vscode-ssh-remote-server-sock-1628692840 to local port 52664
[07:40:44.928] Resolved "ssh-remote+ubuntu" to "127.0.0.1:52664"
[07:40:44.935] ------
I didn't see any errors in the above and the status bar in VScode says I'm connected. But I can't open a terminal window nor open any folders on the remote host. I got these toast notifications:
Could not fetch remote environment
Failed to connect to the remote extension host server (Error: WebSocket close with status code 1006)
Anything else I haven't tried that I should? Is there a way I can get more verbose logs here in case this info isn't enough to determine what the problem is?

Snakemake: Singularity parameters --home and --bind set by default but disallowed on HPC

I have already posted this as an issue on Github at https://github.com/snakemake/snakemake/issues/279 but haven't got any response yet. I hope to find help here.
Version
I am using the following versions on our HPC cluster:
Snakemake c5.4.4
singularity version 3.5.3
Minimal example
singularity: "docker://bash"
rule test:
shell: "echo test"
Describe the bug
snakemake --use-singularity --debug
returns this message:
Building DAG of jobs...
Pulling singularity image docker://bash.
Using shell: /bin/bash
Provided cores: 1
Rules claiming more threads will be scaled down.
Job counts:
count jobs
1 test
1
[Fri Mar 13 15:59:30 2020]
rule test:
jobid: 0
Activating singularity image /data/nanopore/test/.snakemake/singularity/36b22e49e8a03fd08160e9345dd1034e.simg
FATAL: container creation failed: not mounting user requested home: user bind control is disallowed
[Fri Mar 13 15:59:30 2020]
Error in rule test:
jobid: 0
RuleException:
CalledProcessError in line 4 of /data/nanopore/test/Snakefile:
Command ' singularity exec --home /data/nanopore/test --bind /opt/snakemake/v5.4.4/lib/python3.5/site-packages/snakemake-5.4.4-py3.5.egg:/mnt/snakemake /data/nanopore/test/.snakemake/singularity/36b22e49e8a03fd08160e9345dd1034e.simg bash -c 'set -euo pipefail; echo test'' returned non-zero exit status 255
File "/data/nanopore/test/Snakefile", line 4, in __rule_test
File "/usr/lib/python3.5/concurrent/futures/thread.py", line 55, in run
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: /data/nanopore/test/.snakemake/log/2020-03-13T155917.601627.snakemake.log
Apparently, snakemake runs singularity with default values for --home and --bind. These were disallowed by the administrator, however.
Executing
singularity exec --home /data/nanopore/test --bind /opt/snakemake/v5.4.4/lib/python3.5/site-packages/snakemake-5.4.4-py3.5.egg:/mnt/snakemake /data/nanopore/test/.snakemake/singularity/36b22e49e8a03fd08160e9345dd1034e.simg bash -c 'set -euo pipefail;'
returns:
FATAL: container creation failed: not mounting user requested home: user bind control is disallowed
Additional context
Is there a way to disable the Singularity default parameter setting in snakemake? Inside the singularity container the /data directory is still writeable and readable anyway.
Thanks a lot

Concourse CI Windows Worker

I'm trying to setup a Concourse CI environment with a Windows 7 worker.
I have one machine under Ubuntu Server (16.04) hosting my TSA server and one worker (for the support of git resources), and a second one under Windows 7 hosting a worker.
Everything seems to work fine as:
I can login into the web ui
the fly -t my_concourseci workers command returns :
name containers platform tags team state version
ubuntu 1 linux none none running 1.1
windows7 0 windows none none running 1.1
the fly -t my_concourseci execute -c test.yml command returns:
executing build 146
initializing
running echo Hello World!
Hello World!
with the following content in test.yml file:
platform: windows
run:
path: echo
args: [ "Hello World!" ]
Nevertheless when I add an input to my task:
platform: windows
> inputs:
> - name: concourse
run:
path: echo
args: [ "Hello World!" ]
I get the following error:
executing build 148
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 5698k 0 5698k 0 0 1948k 0 --:--:-- 0:00:02 --:--:-- 1949k
initializing
failed to stream in to volume
errored
If I look at the windows worker log I got this error that comes up:
{"timestamp":"1500642862.643555164",
"source":"baggageclaim",
"message":"baggageclaim.api.volume-server.stream-in.bad-stream-payload",
"log_level":1,
"data":{"error":"tar extract failed (exit status 2). output: \"\\ngzip: stdin: not in gzip format\\n/usr/bin/tar: Child returned status 1\\n/usr/bin/tar: Error is not recoverable: exiting now\\n\"",
"session":"2.1.8730",
"volume":"15bf1fc6-0727-4962-6c84-18446e54ab96"}
}
Any ideas about what can cause a not in gzip format error ? Knowing that if I run the exact same task on a linux platform every works fine.
platform: linux
image_resource:
type: docker-image
source: {repository: busybox}
inputs:
- name: concourse
run:
path: echo
args: [ "Hello World!" ]
----- STDOUT
executing build 149
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 5699k 0 5699k 0 0 1917k 0 --:--:-- 0:00:02 --:--:-- 1918k
initializing
Pulling busybox#sha256:2605a2c4875ce5eb27a9f7403263190cd1af31e48a2044d400320548356251c4...
sha256:2605a2c4875ce5eb27a9f7403263190cd1af31e48a2044d400320548356251c4: Pulling from library/busybox
9e87eff13613: Pulling fs layer
9e87eff13613: Verifying Checksum
9e87eff13613: Download complete
9e87eff13613: Pull complete
Digest: sha256:2605a2c4875ce5eb27a9f7403263190cd1af31e48a2044d400320548356251c4
Status: Downloaded newer image for busybox#sha256:2605a2c4875ce5eb27a9f7403263190cd1af31e48a2044d400320548356251c4
Successfully pulled busybox#sha256:2605a2c4875ce5eb27a9f7403263190cd1af31e48a2044d400320548356251c4.
running echo Hello World!
succeeded
Thanks.

Launch powershell as Administrator, and run the concourse_worker.exe from there. That worked for me.
I am running a Windows Server 2016 Base on AWS based on this AMI ami-e1876a98

Thanks for your answers, it helps me solve my problem.
I were starting my concourse worker within a MSYS 1.0 environment. The thing is that MSYS contains tar and gunzip binaries within the $PATH. When I started concourse worker inside a Power Shell or cmd.exe without any MSYS unix-like binaries in the $PATH it worked like a charm !
Note: Be sure to have no MSYS binaries in the $PATH Windows environment variable for this to work, especially check that Git-Bash environment tools are not added to your windows $PATH environment variable.
Thanks again.

Raspberry pi GPIO value is always 0

I'm trying to change the value of GPIO port manually.
To change the value of gpio18 (physical port 12), I did as follow:
# gpio unexportall
# echo 18 > /sys/class/gpio/export
# echo out > /sys/class/gpio/gpio18/direction
# cat /sys/class/gpio/gpio18/value #output is 0
# echo 1 > /sys/class/gpio/gpio18/value
# cat /sys/class/gpio/gpio18/value #output is 1
For gpio7 (physical port 26) I perform the exact commands as above. But the value remains 0:
# echo 7 > /sys/class/gpio/export
# echo out > /sys/class/gpio/gpio7/direction
# cat /sys/class/gpio/gpio7/value #output is 0
# echo 1 > /sys/class/gpio/gpio7/value
# cat /sys/class/gpio/gpio7/value #output is 0 again
I've tried with all other gpio ports. All ports function properly except gpio7 and gpio11 (the value is always 0).
By the way before using this raspberry pi, I was working with another raspberry pi2. Strangely the gpio7 in the previous raspberry pi had the same problem too. Is it a hardware problem or I am doing something wrong?
EDIt: I have also used Wiringpi and J4Pi, the result is the same.

fast scala compiler: Unable to establish connection to compilation daemon

I have a ubuntu machine under OpenVZ virtualization. When I run fsc test.scala I get:
Unable to establish connection to compilation daemon
Then ps aux shows me lots of processes that look like this
ren 17482 0.0 0.0 4908 1400 pts/0 S 00:29 0:00 /bin/bash --posix /usr/bin/scala -Djava.net.preferIPv4Stack=true scala.tools
ren 17490 0.2 0.5 1246676 24268 pts/0 Sl 00:29 0:00 java -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Stack=true -Xbootc
I have run out of ideas how to make it work, version is Scala code runner version 2.9.2 -- Copyright 2002-2011, LAMP/EPFL. Any ideas?
Edit:
Found this, so:
selinux is not installed
export | grep SCALA doesn't return
anything, however scalac and scala compiling/running fine so I assume
this step is ok
ping localhost - works
fsc -reset - same error
fsc -verbose >> logfile.log 2>&1 produces this:
Fast Scala compiler version 2.9.2 -- Copyright 2002-2011, LAMP/EPFL
[Given arguments: -verbose]
[Transformed arguments: -verbose -current-dir /home/ren]
[VM arguments: ]
[Temp directory: /tmp/scala-devel/ren]
[Port number: 36737]
java.net.SocketException: Invalid argument or cannot assign requested address
[Connecting to compilation daemon at port 36737 failed; re-trying...]
No compile server running: starting one with args ''
[Executing command: scala scala.tools.nsc.CompileServer -v]
Starting CompileServer on port 34962
Redirect dir is /tmp/scala-devel/ren/output-redirects
[Port number: 34962]
java.net.SocketException: Invalid argument or cannot assign requested address
[Connecting to compilation daemon at port 34962 failed; re-trying...]
[Port number: 34962]
java.net.SocketException: Invalid argument or cannot assign requested address
[Connecting to compilation daemon at port 34962 failed; re-trying...]
No compile server running: starting one with args '' and so on ...
Edit 2:
Well, starting compilation server like this scala scala.tools.nsc.CompileServer -v actually works! And it seems fsc did start all those servers, but for some reason it decided it didn't. fsc -server localhost:port test.scala works. Which is good enough for now.

It seems fsc uses hostname to connect to compilation server. My hostname didn't response to ping. Changing it to localhost in /etc/hostname fixed the problem.