I'm trying to create a custom DDEV Provider, to import the current database and also user generated files from the web server.
I want to use it with TYPO3 Projects, where I develop the EXT locally with DDEV (because its awesome :) ) and I want to update my local database and also the "fileadmin" files with the help of the ddev pull function.
I've read the docs: Introduction to Hosting Provider Integration and I tested the bash commands locally within the DDEV Container (ddev ssh) and I'm able to connect to the remote Webserver and make a database dump and transfer it to the local DDEV container.
So I added the bash commands to the my custom provider .yaml file in the /provider/ folder.
Here is the current file:
environment_variables:
DB_NAME: db_name
DB_USER: password
DB_PASSWORD: password
HOST_IP: 11.11.11.11
SSH_USERNAME: username
SSH_PASSWORD: password
SSH_PORT: 22
db_pull_command:
command: |
# Creates the .download folder if it doesn't exist
mkdir -p /var/www/html/.ddev/.downloads
# execute the mysqldump on the remote webserver via SSH
ssh -p ${SSH_PORT} ${SSH_USERNAME}#${HOST_IP} 'mysqldump -h 127.0.0.1 -u ${DB_USER} -p ${DB_PASSWORD} ${DB_NAME} > /tmp/${DB_NAME}.sql.gz'
# download to sql file to the ddev folder
scp -P ${SSH_PORT} ${SSH_USERNAME}#${HOST_IP}:/tmp/${DB_NAME}.sql.gz /var/www/html/.ddev/.downloads/db.sql.gz.
If I execute the pull with ddev pull my-provider I get the following Error:
Downloading database...
bash: 03: command not found
Pull failed: Failed to exec mkdir -p /var/www/html/.ddev/.downloads
I assumed that the commands are executed like I would within the DDEV Container (with ddev ssh). What am I missing?
My Environment:
TYPO3 v10.4.20
Windows 10 (WSL)
Docker Desktop 3.5.2
DDEV-Local version v1.17.7
architecture amd64
db drud/ddev-dbserver-mariadb-10.3:v1.17.7
dba phpmyadmin:5
ddev-ssh-agent drud/ddev-ssh-agent:v1.17.0
docker 20.10.7
docker-compose 1.29.2
The web server is running on Plesk.
Note: I only tried to implement the db pull command so far.
UPDATE 09.11.21:
So I've gotten this far that I'm able update and also download the files. However I'm only able to do it, if I hardcode the variables. Everytime I'm trying to setup the environment_variables: I get the following error, if I run the ddev pull myProvider:
Downloading database...
bash: 03: command not found
Here is my current .yaml file with the environment_variables:, which currently don't work. I've tested all the commands within ddev ssh
and it works if I call them manually.
environment_variables:
DB_NAME: db_name
DB_USER: db_user
DB_PASSWORD: 'Password$'
HOST_IP: 10.10.10.10
SSH_USERNAME: username
SSH_PORT: 21
auth_command:
command: |
ssh-add -l >/dev/null || ( echo "Please 'ddev auth ssh' before running this command." && exit 1 )
db_pull_command:
command: |
mkdir -p /var/www/html/.ddev/.downloads
ssh -p ${SSH_PORT} ${SSH_USERNAME}#${HOST_IP} "mysqldump -h 127.0.0.1 -u ${DB_USER} -p'${DB_PASSWORD}' ${DB_NAME} > /tmp/${DB_NAME}.sql"
scp -P ${SSH_PORT} ${SSH_USERNAME}#${HOST_IP}:/tmp/${DB_NAME}.sql /var/www/html/.ddev/.downloads/db.sql
gzip -f /var/www/html/.ddev/.downloads/db.sql
files_pull_command:
command: |
scp -P ${SSH_PORT} -r ${SSH_USERNAME}#${HOST_IP}:/path/to/public/fileadmin/user_upload /var/www/html/.ddev/.downloads/files
Do I declare the variables the wrong way? Or what is it that I'm missing?
For anyone who has trouble connecting via ssh without the password promt, you can run the following commands:
ssh-keygen -t rsa
ssh-copy-id -i ~/.ssh/id_rsa.pub -p 22 username#host
Afterward you should be able to connect without a password promt. Try the following: ssh -p 22 username#host
before you try to ddev puul you have to execute ddev auth ssh
Thanks to #rfay for pointing me into the right direction.
The Problem was, that my password containted a special charater (not a $ though) which needed to be escaped.
After escpaing it correctly like so
environment_variables:
DB_PASSWORD: 'Password\&\'
the ddev pull works.
I hope my .yaml file helps someone else that needs to pull from a webserver.
I have an odd issue where as of recently I am unable to run mkdir -p for a different user running as root and get the following error:
[root#ip-192-168-1-146 ~]# sudo -u myuser mkdir -p /some/target/path
sudo: mkdir -p: command not found
When I test as the 'myuser' user the mkdir -p command works fine. I have additionally tried the following without success:
su - myuser -c "mkdir -p /some/target/path"
sudo -u myuser -i mkdir -p /some/target/path
sudo -u myuser -i -c "mkdir -p /some/target/path" <---dont think syntax is right on this but tried anyways.
Context: I am executing a script to setup my AWS EC2 instance that populates all defined directories. This has been working fine until recently. Not including my script here as the above command doesn't work by itself.
Env output for 'mysuser':
$ env
XDG_SESSION_ID=1
HOSTNAME=ip-192-168-1-146.ec2.internal
SHELL=/bin/bash
TERM=xterm-256color
HISTSIZE=1000
USER=myuser
LS_COLORS=rs=0:di=38;5;27:ln=38;5;51:mh=44;38;5;15:pi=40;38;5;11:so=38;5;13:do=38;5;5:bd=48;5;232;38;5;11:cd=48;5;232;38;5;3:or=48;5;232;38;5;9:mi=05;48;5;232;38;5;15:su=48;5;196;38;5;15:sg=48;5;11;38;5;16:ca=48;5;196;38;5;226:tw=48;5;10;38;5;16:ow=48;5;10;38;5;21:st=48;5;21;38;5;15:ex=38;5;34:*.tar=38;5;9:*.tgz=38;5;9:*.arc=38;5;9:*.arj=38;5;9:*.taz=38;5;9:*.lha=38;5;9:*.lz4=38;5;9:*.lzh=38;5;9:*.lzma=38;5;9:*.tlz=38;5;9:*.txz=38;5;9:*.tzo=38;5;9:*.t7z=38;5;9:*.zip=38;5;9:*.z=38;5;9:*.Z=38;5;9:*.dz=38;5;9:*.gz=38;5;9:*.lrz=38;5;9:*.lz=38;5;9:*.lzo=38;5;9:*.xz=38;5;9:*.bz2=38;5;9:*.bz=38;5;9:*.tbz=38;5;9:*.tbz2=38;5;9:*.tz=38;5;9:*.deb=38;5;9:*.rpm=38;5;9:*.jar=38;5;9:*.war=38;5;9:*.ear=38;5;9:*.sar=38;5;9:*.rar=38;5;9:*.alz=38;5;9:*.ace=38;5;9:*.zoo=38;5;9:*.cpio=38;5;9:*.7z=38;5;9:*.rz=38;5;9:*.cab=38;5;9:*.jpg=38;5;13:*.jpeg=38;5;13:*.gif=38;5;13:*.bmp=38;5;13:*.pbm=38;5;13:*.pgm=38;5;13:*.ppm=38;5;13:*.tga=38;5;13:*.xbm=38;5;13:*.xpm=38;5;13:*.tif=38;5;13:*.tiff=38;5;13:*.png=38;5;13:*.svg=38;5;13:*.svgz=38;5;13:*.mng=38;5;13:*.pcx=38;5;13:*.mov=38;5;13:*.mpg=38;5;13:*.mpeg=38;5;13:*.m2v=38;5;13:*.mkv=38;5;13:*.webm=38;5;13:*.ogm=38;5;13:*.mp4=38;5;13:*.m4v=38;5;13:*.mp4v=38;5;13:*.vob=38;5;13:*.qt=38;5;13:*.nuv=38;5;13:*.wmv=38;5;13:*.asf=38;5;13:*.rm=38;5;13:*.rmvb=38;5;13:*.flc=38;5;13:*.avi=38;5;13:*.fli=38;5;13:*.flv=38;5;13:*.gl=38;5;13:*.dl=38;5;13:*.xcf=38;5;13:*.xwd=38;5;13:*.yuv=38;5;13:*.cgm=38;5;13:*.emf=38;5;13:*.axv=38;5;13:*.anx=38;5;13:*.ogv=38;5;13:*.ogx=38;5;13:*.aac=38;5;45:*.au=38;5;45:*.flac=38;5;45:*.mid=38;5;45:*.midi=38;5;45:*.mka=38;5;45:*.mp3=38;5;45:*.mpc=38;5;45:*.ogg=38;5;45:*.ra=38;5;45:*.wav=38;5;45:*.axa=38;5;45:*.oga=38;5;45:*.spx=38;5;45:*.xspf=38;5;45:
MAIL=/var/spool/mail/myuser
PATH=/sbin:/bin:/opt/home/myuser/.local/bin:/opt/home/myuser/bin
PWD=/opt/home/myuser
LANG=en_US.UTF-8
HISTCONTROL=ignoredups
SHLVL=1
HOME=/opt/home/myuser
LOGNAME=myuser
LESSOPEN=||/usr/bin/lesspipe.sh %s
_=/bin/env
Env output for 'root':
$ env
XDG_SESSION_ID=1
HOSTNAME=ip-192-168-1-146.ec2.internal
SHELL=/bin/bash
TERM=xterm-256color
HISTSIZE=1000
USER=root
LS_COLORS=rs=0:di=38;5;27:ln=38;5;51:mh=44;38;5;15:pi=40;38;5;11:so=38;5;13:do=38;5;5:bd=48;5;232;38;5;11:cd=48;5;232;38;5;3:or=48;5;232;38;5;9:mi=05;48;5;232;38;5;15:su=48;5;196;38;5;15:sg=48;5;11;38;5;16:ca=48;5;196;38;5;226:tw=48;5;10;38;5;16:ow=48;5;10;38;5;21:st=48;5;21;38;5;15:ex=38;5;34:*.tar=38;5;9:*.tgz=38;5;9:*.arc=38;5;9:*.arj=38;5;9:*.taz=38;5;9:*.lha=38;5;9:*.lz4=38;5;9:*.lzh=38;5;9:*.lzma=38;5;9:*.tlz=38;5;9:*.txz=38;5;9:*.tzo=38;5;9:*.t7z=38;5;9:*.zip=38;5;9:*.z=38;5;9:*.Z=38;5;9:*.dz=38;5;9:*.gz=38;5;9:*.lrz=38;5;9:*.lz=38;5;9:*.lzo=38;5;9:*.xz=38;5;9:*.bz2=38;5;9:*.bz=38;5;9:*.tbz=38;5;9:*.tbz2=38;5;9:*.tz=38;5;9:*.deb=38;5;9:*.rpm=38;5;9:*.jar=38;5;9:*.war=38;5;9:*.ear=38;5;9:*.sar=38;5;9:*.rar=38;5;9:*.alz=38;5;9:*.ace=38;5;9:*.zoo=38;5;9:*.cpio=38;5;9:*.7z=38;5;9:*.rz=38;5;9:*.cab=38;5;9:*.jpg=38;5;13:*.jpeg=38;5;13:*.gif=38;5;13:*.bmp=38;5;13:*.pbm=38;5;13:*.pgm=38;5;13:*.ppm=38;5;13:*.tga=38;5;13:*.xbm=38;5;13:*.xpm=38;5;13:*.tif=38;5;13:*.tiff=38;5;13:*.png=38;5;13:*.svg=38;5;13:*.svgz=38;5;13:*.mng=38;5;13:*.pcx=38;5;13:*.mov=38;5;13:*.mpg=38;5;13:*.mpeg=38;5;13:*.m2v=38;5;13:*.mkv=38;5;13:*.webm=38;5;13:*.ogm=38;5;13:*.mp4=38;5;13:*.m4v=38;5;13:*.mp4v=38;5;13:*.vob=38;5;13:*.qt=38;5;13:*.nuv=38;5;13:*.wmv=38;5;13:*.asf=38;5;13:*.rm=38;5;13:*.rmvb=38;5;13:*.flc=38;5;13:*.avi=38;5;13:*.fli=38;5;13:*.flv=38;5;13:*.gl=38;5;13:*.dl=38;5;13:*.xcf=38;5;13:*.xwd=38;5;13:*.yuv=38;5;13:*.cgm=38;5;13:*.emf=38;5;13:*.axv=38;5;13:*.anx=38;5;13:*.ogv=38;5;13:*.ogx=38;5;13:*.aac=38;5;45:*.au=38;5;45:*.flac=38;5;45:*.mid=38;5;45:*.midi=38;5;45:*.mka=38;5;45:*.mp3=38;5;45:*.mpc=38;5;45:*.ogg=38;5;45:*.ra=38;5;45:*.wav=38;5;45:*.axa=38;5;45:*.oga=38;5;45:*.spx=38;5;45:*.xspf=38;5;45:
MAIL=/var/spool/mail/root
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin
PWD=/root
LANG=en_US.UTF-8
HISTCONTROL=ignoredups
SHLVL=1
HOME=/root
LOGNAME=root
LESSOPEN=||/usr/bin/lesspipe.sh %s
_=/bin/env
mkdir is on the system:
[root#ip-192-168-1-146 ~]# ls /bin/mkdir
/bin/mkdir
[root#ip-192-168-1-146 ~]# which mkdir
/bin/mkdir
[myuser#ip-192-168-1-146 ~]$ which mkdir
/bin/mkdir
I am at a loss, if anyone has any suggestions I'ld be greatful. Again this has been working for few months now.
Thanks!
so...no idea how it happened but somehow I pasted a special character M-BM- in place of a proper space char.
The M-BM- characters are an ASCII representation of byte sequence 0xc2 0xa0, which is the UTF8 encoding of unicode character A0 - a non-breaking space character. This character can be inserted in both LibreOffice and Microsoft Word documents using the key sequence Ctrl+Shift+SPACE.
¯\(ツ)/¯
I have the need to start a java rest server with concourse that lives on an Ubuntu 18.04 machine. The version of concourse my company uses is 5.5.11. The server code is written in Java, so a simple java -jar <uber.jar> suffices from the command line (see below). In production, I will not have this simple luxury, hence my question.
I have an scp command working that copies the .jar from concourse to the target Ubuntu machine:
scp -i /tmp/key.p8 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null ./${NEW_DIR}/${ARTIFACT_NAME}.${ARTIFACT_FILE_TYPE} ${SRV_ACCOUNT_USER}#${JAVA_VM_HOST}:/var/www
Note that my private key is passed with -i and I can confirm that is working.
I followed this other SO Q&A that seemed to be promising: Getting ssh to execute a command in the background on target machine
, but after trying a few permutations of the suggested solution and other answers, I still don't have my rest service kicked off.
I've tried a few permutations of this line in my concourse script:
ssh -f -i /tmp/pvt_key1.p8 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null ${SRV_ACCOUNT_USER}#${JAVA_VM_HOST} "bash -c 'nohup java -jar /var/www/${ARTIFACT_NAME}.${ARTIFACT_FILE_TYPE} -c \"/opt/testcerts/clientkeystore\" -w \"password\" > /dev/null 2>&1 &'"
I've tried with and without the -f and -t switches in ssh, with and without the file stream redirection, with and without nohup and the Linux background ('&') command and various ways to escape the quotes.
At the bash prompt, this line successfully starts my server. The two switches are needed to point to the certificate and provide the password:
java -jar rest-service.jar -c "/opt/certificates/clientkeystore" -w "password"
I really think this is possible to do in Concourse, but I'm stuck at this point.
After a lot of trial an error, it seems I needed to do this:
ssh -f -i /tmp/pvt_key1.p8 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null ${SRV_ACCOUNT_USER}#${JAVA_VM_HOST} "bash -c 'sudo java -jar /var/www/${ARTIFACT_NAME}.${ARTIFACT_FILE_TYPE} -c \"/path/to/my/certificate\" -w \"password\" > /var/www/log.txt 2>&1 &'"
The key was I was missing the 'sudo' portion of the command. Using nohup as opposed to putting in a Linux bash background indicator ('&') seems to give me an error in the pipeline. This works for me, but others are welcome to post responses with better answers or methods that might be a better practice.
Hi i am trying to stop workflow through PMCMD but unsuccessfully.
i am doing it through powershell
&"$INFS_ROOT\pmcmd.exe" stopworkflow -usd DS -u $IFPC_USER -p $IFPC_PASS -sv ISUD -d DomainIF -f ("$Folder") ("$wf");
But everytime i got error:
ERROR: Option value cannot start with one leading '-'. Usage: pmcmd
stopworkflow
<<-service|-sv> service <-domain|-d> domain [<-timeout|-t> timeout]>
[<<-user|-u> username|<-uservar|-uv> userEnvVar>]
[<<-password|-p> password|<-passwordvar|-pv> passwordEnvVar>]
[<<-usersecuritydomain|-usd> usersecuritydomain|<-usersecuritydomainvar|-usdv> userSecuritydomainEnvVar>]
[<-folder|-f> folder] [<-runinsname|-rin> runInsName]
[-wfrunid workflowRunId] [-wait|-nowait] workflow
For example getworkflowstatus works clear
&"$INFS_ROOT\pmcmd.exe" getworkflowdetails -usd DS -u $IFPCUser -p $IFPCPass -sv ISUD -d DomainIF -f ("$Folder") ("$wf")
Can anyone help me with stopping workflow through pmcmd? Thanks
I'm writing a C Shell program that will be doing su or sudo or ssh. They all want their passwords in console input (the TTY) rather than stdin or the command line.
Does anybody know a solution?
Setting up password-less sudo is not an option.
expect could be an option, but it's not present on my stripped-down system.
For sudo there is a -S option for accepting the password from standard input. Here is the man entry:
-S The -S (stdin) option causes sudo to read the password from
the standard input instead of the terminal device.
This will allow you to run a command like:
echo myPassword | sudo -S ls /tmp
As for ssh, I have made many attempts to automate/script it's usage with no success. There doesn't seem to be any build-in way to pass the password into the command without prompting. As others have mentioned, the "expect" utility seems like it is aimed at addressing this dilemma but ultimately, setting up the correct private-key authorization is the correct way to go when attempting to automate this.
I wrote some Applescript which prompts for a password via a dialog box and then builds a custom bash command, like this:
echo <password> | sudo -S <command>
I'm not sure if this helps.
It'd be nice if sudo accepted a pre-encrypted password, so I could encrypt it within my script and not worry about echoing clear text passwords around. However this works for me and my situation.
For ssh you can use sshpass: sshpass -p yourpassphrase ssh user#host.
You just need to download sshpass first :)
$ apt-get install sshpass
$ sshpass -p 'password' ssh username#server
For sudo you can do this too:
sudo -S <<< "password" command
I've got:
ssh user#host bash -c "echo mypass | sudo -S mycommand"
Works for me.
The usual solution to this problem is setuiding a helper app that performs the task requiring superuser access:
http://en.wikipedia.org/wiki/Setuid
Sudo is not meant to be used offline.
Later edit: SSH can be used with private-public key authentication. If the private key does not have a passphrase, ssh can be used without prompting for a password.
Maybe you can use an expect command?:
expect -c 'spawn ssh root#your-domain.com;expect password;send "your-password\n";interact
That command gives the password automatically.
This can be done by setting up public/private keys on the target hosts you will be connecting to.
The first step would be to generate an ssh key for the user running the script on the local host, by executing:
ssh-keygen
Enter file in which to save the key (/home/myuser/.ssh/id_rsa): <Hit enter for default>
Overwrite (y/n)? y
Then enter a blank password. After that, copy your ssh key onto the target host which you will be connecting to.
ssh-copy-id <remote_user>#<other_host>
remote_user#other_host's password: <Enter remote user's password here>
After registering the ssh keys, you would be able to perform a silent ssh remote_user#other_host from you local host.
When there's no better choice (as suggested by others), then man socat can help:
(sleep 5; echo PASSWORD; sleep 5; echo ls; sleep 1) |
socat - EXEC:'ssh -l user server',pty,setsid,ctty
EXEC’utes an ssh session to server. Uses a pty for communication
between socat and ssh, makes it ssh’s controlling tty (ctty),
and makes this pty the owner of a new process group (setsid), so
ssh accepts the password from socat.
All of the pty,setsid,ctty complexity is necessary and, while you might not need to sleep as long, you will need to sleep. The echo=0 option is worth a look too, as is passing the remote command on ssh's command line.
Take a look at expect linux utility.
It allows you to send output to stdio based on simple pattern matching on stdin.
ssh -t -t me#myserver.io << EOF
echo SOMEPASSWORD | sudo -S do something
sudo do something else
exit
EOF
Set SSH up for Public Key Authentication, with no pasphrase on the Key. Loads of guides on the net. You won't need a password to login then. You can then limit connections for a key based on client hostname. Provides reasonable security and is great for automated logins.
echo <password> | su -c <command> <user>
This is working.
a better sshpass alternative is: passh
https://github.com/clarkwang/passh
Login to a remote server
$ passh -p password ssh user#host
Run a command on remote server
$ passh -p password ssh user#host date
other methods to pass the password
-p The password (Default: `password')
-p env: Read password from env var
-p file: Read password from file
here I explained why it is better than sshpass, and other solutions.
You can also pass various parameters as follows:
echo password | echo y | sudo -S pacman -Syu
(Although that's a bad idea, it's just an example)
I had the same problem. dialog script to create directory on remote pc.
dialog with ssh is easy. I use sshpass (previously installed).
dialog --inputbox "Enter IP" 8 78 2> /tmp/ip
IP=$(cat /tmp/ip)
dialog --inputbox "Please enter username" 8 78 2> /tmp/user
US=$(cat /tmp/user)
dialog --passwordbox "enter password for \"$US\" 8 78 2> /tmp/pass
PASSWORD = $(cat /tmp/pass)
sshpass -p "$PASSWORD" ssh $US#$IP mkdir -p /home/$US/TARGET-FOLDER
rm /tmp/ip
rm /tmp/user
rm /tmp/pass
greetings from germany
titus
Building on #Jahid's answer, this worked for me on macOS 10.13:
ssh <remote_username>#<remote_server> sudo -S <<< <remote_password> cat /etc/sudoers
I once had a use case where I needed to run Sudo and ssh in the same command without stdin specifying all the variables needed.
This is the command I used
echo sudopassword | sudo -S -u username sshpass -p extsshpassword ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no username#ipaddress " CMD on external machine"
Breaking that command into pieces!
This will allow you to run commands through your machine using Superuser:
echo password | sudo -S -u username
This will allow you to pass ssh password and execute commands on external machines:
sshpass -p sshpassword ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no username#ipaddress " CMD on external machine"
make sure you install the sudo and openssh packages on your machine.
One way would be to use read -s option .. this way the password characters are not echoed back to the screen. I wrote a small script for some use cases and you can see it in my blog:
http://www.datauniv.com/blogs/2013/02/21/a-quick-little-expect-script/
USE:
echo password | sudo command
Example:
echo password | sudo apt-get update; whoami
Hope It Helps..
You can provide password as parameter to expect script.
su -c "Command" < "Password"
Hope it is helpful.