I want to run simulations in parallel in a remote Cluster calling them from matlab.
I manage to run them in my local Ubuntu machine using.
unix('parallel -j4 flow > /dev/null :::: Pool.txt');
But when I want it to run it in a remote cluster I really did not mange to make the parallel command to work.
The first problem was to avoid entering the password.
For that I used the sshpass as this
unix('sshpass -p password ssh user#cluster.example.com')
That get me in to the server but it does not continue to the next command line.
I try so many commands that I do not want to reference here.
But basically can some one that understands well the parallel GNU command usage tell me how can I connect to a cluster. and run the simulations there. is it better just to make a script at the server and run it from matlab?
Any expert advice is highly appreciated.
Your problem is not with GNU Parallel but with configuring ssh. First, you must get ssh set up, then the rest is easy.
So, on your local Ubuntu machine, you need to create your keys:
ssh-keygen -t rsa -b 2048
That will make some files in $HOME/.ssh. You now need to copy the public part of those keys to each and every node of the remote cluster where you want to run your parallel jobs:
ssh-copy-id -i $HOME/.ssh/id_rsa.pub CLUSTERUSERNAME#NODE-0
...
ssh-copy-id -i $HOME/.ssh/id_rsa.pub CLUSTERUSERNAME#NODE-15
e.g.
ssh-copy-id -i $HOME/.ssh/id_rsa.pub fred#192.168.0.100
Now, test you can ssh into each node:
ssh fred#node2
Then, on your local Ubuntu box, set up your config file for ssh, so it will be $HOME/.ssh/config
Host node0
Hostname 192.168.0.100
User fred
...
...
Host node15
Hostname 192.168.0.115
User fred
Now you can just use:
ssh node0
and it will know that means fred#192.168.0.100
Now GNU Parallel will work with:
parallel -S node0,node1,node2
Related
I'm reading these docs on sharing SSH keys with a dev container, but I can't get it to work.
My setup is as follows:
Windows 10 with Docker Desktop 4.2.0 using the WSL2 backend
A WSL2 distro running Ubuntu 20.04
In WSL2, I have ssh-agent running and aware of my key:
λ ssh-add -l
4096 SHA256:wDqVYQshQBCG/Sri/bsgjEaUFboQDUO/9FJqhFMncdk /home/taschan/.ssh/id_rsa (RSA)
The docs say
the extension will automatically forward your local SSH agent if one is running
But if I do ssh-add -l in the devcontainer, it responds with Could not open a connection to your authentication agent.; and of course starting one (with eval "$(ssh-agent -s)") only starts one that doesn't know of my private key.
What am I missing?
I had basically the same issue. Running Windows 11 with WSL2 and my VSCode Devcontainer wouldn't show any ssh keys (running ssh-add -l inside the container showed an empty list) despite having Git configured on my host machine with working ssh keys.
For me, there were 3 separate instances of ssh-agent on my machine:
WSL2
Git Bash
Windows host 🠆 This is the one VSCode is forwarding to the devcontainer
My existing ssh keys were set up inside Git Bash (as per Github's instructions) so running ssh-add -l only ever showed my ssh keys from inside a Git Bash terminal, nowhere else.
However, as explained in the previous answer, digging through the Devcontainer startup logs shows that VSCode is forwarding only the host machine's ssh-agent, it doesn't look at the WSL2 or Git Bash ones.
Solution: I suggest following the below Microsoft docs page. You need to enable an "Optional Feature" in Windows, then run a few commands in PowerShell (as admin) to activate the ssh-agent service. With this set up, the ssh-agent/ssh-add commands will work from a regular CMD terminal too.
You can use these with the usual keygen commands etc to generate and add new keys on the host (I just ssh-add'ed the same keys generated by Git Bash originally). The added keys should immediately be detected by ssh-add -l inside the container.
https://learn.microsoft.com/en-us/windows-server/administration/openssh/openssh_keymanagement
I tried many things but did not work. Finally after devcontainer is created , I note down the container name and copy the id_rsa and id_rsa.pub key inside container using docker cp command.
syntax:
docker cp <sourcefile> container_id:/dir
Copy both private and public key:
docker cp /root/.ssh/id_ed25519 eloquent_ritchie:/root/.ssh/
docker cp /root/.ssh/id_ed25519.pub eloquent_ritchie:/root/.ssh/
change the permission of private key so that you can do git operations
docker exec eloquent_ritchie chmod 600 /root/.ssh/id_ed25519
eloquent_ritchie is sample container name. Your container name will differ. Use your container name
Then I was able to do Git operations inside devcontainer.
If you rebuild your container again you need to copy the file to devcontainer again.
I also had quite a lot of trouble to get this to work. The following steps might help troubleshooting:
Check that ssh-agent is running on your host and the key is added
Run ssh-agent -l on Windows and expect to see the name of your key
Check that VSCode forwards the socket
Search ssh-agent in the startup log. I had the message
ssh-agent: SSH_AUTH_SOCK in container (/tmp/vscode-ssh-auth-a56c4b60c939c778f2998dee2a6bbe12285db2ad.sock) forwarded to local host (\\.\pipe\openssh-ssh-agent).
So it seems that VSCode is directly forwarding the Windows SSH agent here (and not an SSH agent running in your WSL).
I have a weird problem and don't really know where it's coming from. I have machine A, B and C. I want to connect my VSCode to machine C with the remote-ssh extension for vscode, here's my config:
# Jump box
Host jump-box
HostName machineB
User myuser
# Target machine
Host target-box
HostName machineC
User myuser
ProxyCommand ssh -q -W %h:%p jump-box
The machine C is a weird server used by a lot of people, when I try to connect, the connection to machine B is fine but then, the extension tries to ssh -D -T some5XXXXport machineC bash from B.
The last command passes fine, and I have tested it manually, however the bash at the end makes it run the root bash or something, because I lose my home directory and get an admin one.
So in consequence, the extension fails because it tries to mkdir /some/admin/home/.vscode-server/bin/somecommithash: Permission Denied. My ~doesn't belong to me anymore when the ssh command is bash.
Any ideas how to overwrite or even hack the command?
Do you know why when you ssh address bash it breaks everything?
I also don't think the B->C ssh connection is picking any ~/.bash_profile, ~/.bashrc nor ~/.profile from machine C, perhaps because ~ points to another home.
Solved it. Added all what I need directly in the extension.js.
We are trying to build CRAN reposiotry locally in windows using cwrsync. we are using rsync command to download all the packages etc. Since the machine where we are running is behind proxy we are also using RSYNC_PROXY variable. when we execute rsync command we are getting "rsync : did not see server greeting". Can you please help us.
rsync -av --chmod u+rwx -e "ssh -i d:\r-download" "/cygdrive/d/r-download"
rsync -rtlzv --delete cran.r-project.org::CRAN "/cygdrive/d/download"
Not sure whether its because of -e ssh.
Regards...Srini
Yes,
with daemon you cannot use '-e' ('--rsh') option.
From man page:
CONNECTING TO AN RSYNC DAEMON
It is also possible to use rsync without a remote shell as the
transport. In this case you will directly connect to a remote rsync
daemon, typically using TCP port 873. (This obviously requires the
daemon to be running on the remote system, so refer to the STARTING AN
RSYNC DAEMON TO ACCEPT CONNECTIONS section below for information on
that.)
Using rsync in this way is the same as using it with a remote shell
except that:
o you either use a double colon :: instead of a single colon to
separate the hostname from the path, or you use an rsync:// URL.
o the first word of the "path" is actually a module name.
o the remote daemon may print a message of the day when you
connect.
o if you specify no path name on the remote daemon then the list
of accessible paths on the daemon will be shown.
o if you specify no local destination then a listing of the
specified files on the remote daemon is provided.
o you must not specify the --rsh (-e) option.
Check last line.
Regards.
I was trying to setup an SSH connection with Github following this tutorial:
Testing your SSH connection
I came across the following command:
$ ssh -T git#github.com
# Attempts to ssh to github
Curious, I looked at the ssh manual. It said the following:
-T Disable pseudo-tty allocation.
What is tty allocation? What does tty stand for? Why are we disabling it?I earnestly tried to look it up but I was unable to find even a definition.
As explained in "gitolite: PTY allocation request failed on channel 0", it is important to do ssh test connection with -T, because some server could abort the transaction entirely if a text-terminal (tty) is requested.
-T avoids requesting said terminal, since GitHub has no intention of giving you an interactive secure shell, where you could type command.
GitHub only wants to reply to your ssh request, in order to ascertain that the ssh command does work (you have the right public/private keys, and the public one has been registered to your GitHub account)
PuTTy would be an example of a terminal emulator, serial console and network file transfer application. It supports several network protocols, including SCP, SSH, Telnet and rlogin.
The name "PuTTY" has no definitive meaning, though "tty" is the name for a terminal in the Unix tradition, usually held to be short for Teletype.
Other use-cases for -T (beside testing)
Transferring binary files
Execute commands on a remote server
SSH tunneling: ssh -fnT -L port:server:port user#server (-f for background: you don't want to execute command, don't need a TTY and just want to establish a tunnel)
I'm stuck in a bit of annoying situation.
There's a chain of machines between my desktop and the production servers. Something like this:
desktop -> firewall 1 -> firewall 2 -> prod_box 1
-> prod_box 2
-> ...
I'm looking for a way to automate deployment to the prod boxes via ssh.
I'm aware there are a number of solutions in general, but my restrictions are:
No changes permitted to firewall 2
No config changes permitted to prod boxes (only content)
firewall 1 has a local user account for me
firewall 2 and prod are accessed as root
port 22 is the only open port between each link
So, in general the command sequence I do to deploy is:
scp archive.tar user#firewall1:archive.tar
ssh user#firewall1
scp archive.tar root#firewall2:/tmp/archive.tar
ssh root#firewall2
scp /tmp/archive.tar root#prod1:/tmp/archive.tar
ssh root#prod1
cd /var/www/
tar xvf /tmp/archive.tar
Its a bit more complex than that in reality, but that's a basic summary of the tasks to do.
I've put my ssh key in firewall1:/home/user/.ssh/authorized_keys, so that's no problem.
However, I can't do this for firewall2 or prod boxes.
It'd be great if I could run this (commands above) from a shell script locally, type my password in 4 times and be done with it. Sadly I cannot figure out how to do that.
I need some way to chain ssh commands. I've spent all afternoon trying to use python to do this and eventually given up because the ssh libraries don't seem to support password-entry-style login.
What can I do here?
There must be some kind of library I can use to:
login via ssh using either a key file OR a dynamically entered password
remote remote shell commands through the chain of ssh tunnels
I'm not really sure what to tag this question, so I've just left it as ssh, deployment for now.
NB. It'd be great to use ssh tunnels and a deployment tool to push these changes out, but I'd still have to manually login to each box to setup the tunnel, and that wont work anyway, because of the port blocking.
I am working on Net::OpenSSH::Gateway, an extension for my other Perl module Net::OpenSSH that does just that.
For instance:
use Net::OpenSSH;
use Net::OpenSSH::Gateway;
my $gateway = Net::OpenSSH::Gateway->find_gateway(
proxies => ['ssh://user#firewall1',
'ssh://password:root#firewall2'],
backend => 'perl');
for my $host (#prod_hosts) {
my $ssh = Net::OpenSSH->new($host, gateway => $gateway);
if ($ssh->error) {
warn "unable to connect to $host\n";
next;
}
$ssh->scp_put($file_path, $destination)
or warn "scp for $host failed\n";
}
It requires Perl available in both firewalls, but no write permissions or installing any additional software there.
Unfortunately this isn't possible to do as one shell script. I did try, but ssh's password negotiation requires an interactive terminal, which you don't get with chained ssh commands. You could do it with passwordless keys, but since that's highly insecure and you can't do it anyway, nevermind.
The basic idea is that each server sends a bash script to the next one, which is then activated and sends the next one (and so on) until it reaches the last one, which does the distribution.
However, since this requires an interactive terminal at each stage, you're going to need to follow the payload down the chain manually executing each script as you go, somewhat as you do now but with less typing.
Obviously, you will need to customise them a bit, but try these scripts:
script1.sh
#!/bin/bash
user=doug
firewall1=firewall_1
#Minimise password entries across the board.
tar cf payload1.tar script3.sh archive.tar
tar cf payload2.tar script2.sh payload1.tar
scp payload2.tar ${user}#${firewall1}:payload2.tar
ssh ${user}#${firewall1} "tar xf payload2.tar;chmod +x script2.sh"
echo "Now connect to ${firewall1} and run ./script.sh"
script2.sh
#!/bin/bash
user=root
firewall2=firewall_2
# Minimise password entries
scp payload1.tar ${user}#${firewall2}:/tmp/payload1.tar
ssh ${user}#${firewall2} "cd /tmp;tar xf payload1.tar;chmod +x script3.sh"
echo "Now connect to ${firewall2} and run /tmp/script3.sh"
script3.sh
#!/bin/bash
user=root
hosts="prod1 prod2 prod3 prod4"
for host in $hosts
do
echo scp archive.tar ${user}#${host}:/tmp/archive.tar
echo ssh ${user}#${host} "cd /var/www; tar xvf /tmp/archive.tar"
done
It does require 3 password entries per firewall which is a bit annoying, but such is life.
This do you any good?