How can I automate deployment through multiple ssh firewalls (using PW auth)? - deployment

Question

I'm stuck in a bit of annoying situation.
There's a chain of machines between my desktop and the production servers. Something like this:
desktop -> firewall 1 -> firewall 2 -> prod_box 1
-> prod_box 2
-> ...
I'm looking for a way to automate deployment to the prod boxes via ssh.
I'm aware there are a number of solutions in general, but my restrictions are:
No changes permitted to firewall 2
No config changes permitted to prod boxes (only content)
firewall 1 has a local user account for me
firewall 2 and prod are accessed as root
port 22 is the only open port between each link
So, in general the command sequence I do to deploy is:
scp archive.tar user#firewall1:archive.tar
ssh user#firewall1
scp archive.tar root#firewall2:/tmp/archive.tar
ssh root#firewall2
scp /tmp/archive.tar root#prod1:/tmp/archive.tar
ssh root#prod1
cd /var/www/
tar xvf /tmp/archive.tar
Its a bit more complex than that in reality, but that's a basic summary of the tasks to do.
I've put my ssh key in firewall1:/home/user/.ssh/authorized_keys, so that's no problem.
However, I can't do this for firewall2 or prod boxes.
It'd be great if I could run this (commands above) from a shell script locally, type my password in 4 times and be done with it. Sadly I cannot figure out how to do that.
I need some way to chain ssh commands. I've spent all afternoon trying to use python to do this and eventually given up because the ssh libraries don't seem to support password-entry-style login.
What can I do here?
There must be some kind of library I can use to:
login via ssh using either a key file OR a dynamically entered password
remote remote shell commands through the chain of ssh tunnels
I'm not really sure what to tag this question, so I've just left it as ssh, deployment for now.
NB. It'd be great to use ssh tunnels and a deployment tool to push these changes out, but I'd still have to manually login to each box to setup the tunnel, and that wont work anyway, because of the port blocking.

Answer 1

I am working on Net::OpenSSH::Gateway, an extension for my other Perl module Net::OpenSSH that does just that.
For instance:
use Net::OpenSSH;
use Net::OpenSSH::Gateway;
my $gateway = Net::OpenSSH::Gateway->find_gateway(
proxies => ['ssh://user#firewall1',
'ssh://password:root#firewall2'],
backend => 'perl');
for my $host (#prod_hosts) {
my $ssh = Net::OpenSSH->new($host, gateway => $gateway);
if ($ssh->error) {
warn "unable to connect to $host\n";
next;
}
$ssh->scp_put($file_path, $destination)
or warn "scp for $host failed\n";
}
It requires Perl available in both firewalls, but no write permissions or installing any additional software there.

Answer 2

Unfortunately this isn't possible to do as one shell script. I did try, but ssh's password negotiation requires an interactive terminal, which you don't get with chained ssh commands. You could do it with passwordless keys, but since that's highly insecure and you can't do it anyway, nevermind.
The basic idea is that each server sends a bash script to the next one, which is then activated and sends the next one (and so on) until it reaches the last one, which does the distribution.
However, since this requires an interactive terminal at each stage, you're going to need to follow the payload down the chain manually executing each script as you go, somewhat as you do now but with less typing.
Obviously, you will need to customise them a bit, but try these scripts:
script1.sh
#!/bin/bash
user=doug
firewall1=firewall_1
#Minimise password entries across the board.
tar cf payload1.tar script3.sh archive.tar
tar cf payload2.tar script2.sh payload1.tar
scp payload2.tar ${user}#${firewall1}:payload2.tar
ssh ${user}#${firewall1} "tar xf payload2.tar;chmod +x script2.sh"
echo "Now connect to ${firewall1} and run ./script.sh"
script2.sh
#!/bin/bash
user=root
firewall2=firewall_2
# Minimise password entries
scp payload1.tar ${user}#${firewall2}:/tmp/payload1.tar
ssh ${user}#${firewall2} "cd /tmp;tar xf payload1.tar;chmod +x script3.sh"
echo "Now connect to ${firewall2} and run /tmp/script3.sh"
script3.sh
#!/bin/bash
user=root
hosts="prod1 prod2 prod3 prod4"
for host in $hosts
do
echo scp archive.tar ${user}#${host}:/tmp/archive.tar
echo ssh ${user}#${host} "cd /var/www; tar xvf /tmp/archive.tar"
done
It does require 3 password entries per firewall which is a bit annoying, but such is life.
This do you any good?