I'm trying to create a custom DDEV Provider, to import the current database and also user generated files from the web server.
I want to use it with TYPO3 Projects, where I develop the EXT locally with DDEV (because its awesome :) ) and I want to update my local database and also the "fileadmin" files with the help of the ddev pull function.
I've read the docs: Introduction to Hosting Provider Integration and I tested the bash commands locally within the DDEV Container (ddev ssh) and I'm able to connect to the remote Webserver and make a database dump and transfer it to the local DDEV container.
So I added the bash commands to the my custom provider .yaml file in the /provider/ folder.
Here is the current file:
environment_variables:
DB_NAME: db_name
DB_USER: password
DB_PASSWORD: password
HOST_IP: 11.11.11.11
SSH_USERNAME: username
SSH_PASSWORD: password
SSH_PORT: 22
db_pull_command:
command: |
# Creates the .download folder if it doesn't exist
mkdir -p /var/www/html/.ddev/.downloads
# execute the mysqldump on the remote webserver via SSH
ssh -p ${SSH_PORT} ${SSH_USERNAME}#${HOST_IP} 'mysqldump -h 127.0.0.1 -u ${DB_USER} -p ${DB_PASSWORD} ${DB_NAME} > /tmp/${DB_NAME}.sql.gz'
# download to sql file to the ddev folder
scp -P ${SSH_PORT} ${SSH_USERNAME}#${HOST_IP}:/tmp/${DB_NAME}.sql.gz /var/www/html/.ddev/.downloads/db.sql.gz.
If I execute the pull with ddev pull my-provider I get the following Error:
Downloading database...
bash: 03: command not found
Pull failed: Failed to exec mkdir -p /var/www/html/.ddev/.downloads
I assumed that the commands are executed like I would within the DDEV Container (with ddev ssh). What am I missing?
My Environment:
TYPO3 v10.4.20
Windows 10 (WSL)
Docker Desktop 3.5.2
DDEV-Local version v1.17.7
architecture amd64
db drud/ddev-dbserver-mariadb-10.3:v1.17.7
dba phpmyadmin:5
ddev-ssh-agent drud/ddev-ssh-agent:v1.17.0
docker 20.10.7
docker-compose 1.29.2
The web server is running on Plesk.
Note: I only tried to implement the db pull command so far.
UPDATE 09.11.21:
So I've gotten this far that I'm able update and also download the files. However I'm only able to do it, if I hardcode the variables. Everytime I'm trying to setup the environment_variables: I get the following error, if I run the ddev pull myProvider:
Downloading database...
bash: 03: command not found
Here is my current .yaml file with the environment_variables:, which currently don't work. I've tested all the commands within ddev ssh
and it works if I call them manually.
environment_variables:
DB_NAME: db_name
DB_USER: db_user
DB_PASSWORD: 'Password$'
HOST_IP: 10.10.10.10
SSH_USERNAME: username
SSH_PORT: 21
auth_command:
command: |
ssh-add -l >/dev/null || ( echo "Please 'ddev auth ssh' before running this command." && exit 1 )
db_pull_command:
command: |
mkdir -p /var/www/html/.ddev/.downloads
ssh -p ${SSH_PORT} ${SSH_USERNAME}#${HOST_IP} "mysqldump -h 127.0.0.1 -u ${DB_USER} -p'${DB_PASSWORD}' ${DB_NAME} > /tmp/${DB_NAME}.sql"
scp -P ${SSH_PORT} ${SSH_USERNAME}#${HOST_IP}:/tmp/${DB_NAME}.sql /var/www/html/.ddev/.downloads/db.sql
gzip -f /var/www/html/.ddev/.downloads/db.sql
files_pull_command:
command: |
scp -P ${SSH_PORT} -r ${SSH_USERNAME}#${HOST_IP}:/path/to/public/fileadmin/user_upload /var/www/html/.ddev/.downloads/files
Do I declare the variables the wrong way? Or what is it that I'm missing?
For anyone who has trouble connecting via ssh without the password promt, you can run the following commands:
ssh-keygen -t rsa
ssh-copy-id -i ~/.ssh/id_rsa.pub -p 22 username#host
Afterward you should be able to connect without a password promt. Try the following: ssh -p 22 username#host
before you try to ddev puul you have to execute ddev auth ssh
Thanks to #rfay for pointing me into the right direction.
The Problem was, that my password containted a special charater (not a $ though) which needed to be escaped.
After escpaing it correctly like so
environment_variables:
DB_PASSWORD: 'Password\&\'
the ddev pull works.
I hope my .yaml file helps someone else that needs to pull from a webserver.
I have the need to start a java rest server with concourse that lives on an Ubuntu 18.04 machine. The version of concourse my company uses is 5.5.11. The server code is written in Java, so a simple java -jar <uber.jar> suffices from the command line (see below). In production, I will not have this simple luxury, hence my question.
I have an scp command working that copies the .jar from concourse to the target Ubuntu machine:
scp -i /tmp/key.p8 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null ./${NEW_DIR}/${ARTIFACT_NAME}.${ARTIFACT_FILE_TYPE} ${SRV_ACCOUNT_USER}#${JAVA_VM_HOST}:/var/www
Note that my private key is passed with -i and I can confirm that is working.
I followed this other SO Q&A that seemed to be promising: Getting ssh to execute a command in the background on target machine
, but after trying a few permutations of the suggested solution and other answers, I still don't have my rest service kicked off.
I've tried a few permutations of this line in my concourse script:
ssh -f -i /tmp/pvt_key1.p8 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null ${SRV_ACCOUNT_USER}#${JAVA_VM_HOST} "bash -c 'nohup java -jar /var/www/${ARTIFACT_NAME}.${ARTIFACT_FILE_TYPE} -c \"/opt/testcerts/clientkeystore\" -w \"password\" > /dev/null 2>&1 &'"
I've tried with and without the -f and -t switches in ssh, with and without the file stream redirection, with and without nohup and the Linux background ('&') command and various ways to escape the quotes.
At the bash prompt, this line successfully starts my server. The two switches are needed to point to the certificate and provide the password:
java -jar rest-service.jar -c "/opt/certificates/clientkeystore" -w "password"
I really think this is possible to do in Concourse, but I'm stuck at this point.
After a lot of trial an error, it seems I needed to do this:
ssh -f -i /tmp/pvt_key1.p8 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null ${SRV_ACCOUNT_USER}#${JAVA_VM_HOST} "bash -c 'sudo java -jar /var/www/${ARTIFACT_NAME}.${ARTIFACT_FILE_TYPE} -c \"/path/to/my/certificate\" -w \"password\" > /var/www/log.txt 2>&1 &'"
The key was I was missing the 'sudo' portion of the command. Using nohup as opposed to putting in a Linux bash background indicator ('&') seems to give me an error in the pipeline. This works for me, but others are welcome to post responses with better answers or methods that might be a better practice.
I'm writing a C Shell program that will be doing su or sudo or ssh. They all want their passwords in console input (the TTY) rather than stdin or the command line.
Does anybody know a solution?
Setting up password-less sudo is not an option.
expect could be an option, but it's not present on my stripped-down system.
For sudo there is a -S option for accepting the password from standard input. Here is the man entry:
-S The -S (stdin) option causes sudo to read the password from
the standard input instead of the terminal device.
This will allow you to run a command like:
echo myPassword | sudo -S ls /tmp
As for ssh, I have made many attempts to automate/script it's usage with no success. There doesn't seem to be any build-in way to pass the password into the command without prompting. As others have mentioned, the "expect" utility seems like it is aimed at addressing this dilemma but ultimately, setting up the correct private-key authorization is the correct way to go when attempting to automate this.
I wrote some Applescript which prompts for a password via a dialog box and then builds a custom bash command, like this:
echo <password> | sudo -S <command>
I'm not sure if this helps.
It'd be nice if sudo accepted a pre-encrypted password, so I could encrypt it within my script and not worry about echoing clear text passwords around. However this works for me and my situation.
For ssh you can use sshpass: sshpass -p yourpassphrase ssh user#host.
You just need to download sshpass first :)
$ apt-get install sshpass
$ sshpass -p 'password' ssh username#server
For sudo you can do this too:
sudo -S <<< "password" command
I've got:
ssh user#host bash -c "echo mypass | sudo -S mycommand"
Works for me.
The usual solution to this problem is setuiding a helper app that performs the task requiring superuser access:
http://en.wikipedia.org/wiki/Setuid
Sudo is not meant to be used offline.
Later edit: SSH can be used with private-public key authentication. If the private key does not have a passphrase, ssh can be used without prompting for a password.
Maybe you can use an expect command?:
expect -c 'spawn ssh root#your-domain.com;expect password;send "your-password\n";interact
That command gives the password automatically.
This can be done by setting up public/private keys on the target hosts you will be connecting to.
The first step would be to generate an ssh key for the user running the script on the local host, by executing:
ssh-keygen
Enter file in which to save the key (/home/myuser/.ssh/id_rsa): <Hit enter for default>
Overwrite (y/n)? y
Then enter a blank password. After that, copy your ssh key onto the target host which you will be connecting to.
ssh-copy-id <remote_user>#<other_host>
remote_user#other_host's password: <Enter remote user's password here>
After registering the ssh keys, you would be able to perform a silent ssh remote_user#other_host from you local host.
When there's no better choice (as suggested by others), then man socat can help:
(sleep 5; echo PASSWORD; sleep 5; echo ls; sleep 1) |
socat - EXEC:'ssh -l user server',pty,setsid,ctty
EXEC’utes an ssh session to server. Uses a pty for communication
between socat and ssh, makes it ssh’s controlling tty (ctty),
and makes this pty the owner of a new process group (setsid), so
ssh accepts the password from socat.
All of the pty,setsid,ctty complexity is necessary and, while you might not need to sleep as long, you will need to sleep. The echo=0 option is worth a look too, as is passing the remote command on ssh's command line.
Take a look at expect linux utility.
It allows you to send output to stdio based on simple pattern matching on stdin.
ssh -t -t me#myserver.io << EOF
echo SOMEPASSWORD | sudo -S do something
sudo do something else
exit
EOF
Set SSH up for Public Key Authentication, with no pasphrase on the Key. Loads of guides on the net. You won't need a password to login then. You can then limit connections for a key based on client hostname. Provides reasonable security and is great for automated logins.
echo <password> | su -c <command> <user>
This is working.
a better sshpass alternative is: passh
https://github.com/clarkwang/passh
Login to a remote server
$ passh -p password ssh user#host
Run a command on remote server
$ passh -p password ssh user#host date
other methods to pass the password
-p The password (Default: `password')
-p env: Read password from env var
-p file: Read password from file
here I explained why it is better than sshpass, and other solutions.
You can also pass various parameters as follows:
echo password | echo y | sudo -S pacman -Syu
(Although that's a bad idea, it's just an example)
I had the same problem. dialog script to create directory on remote pc.
dialog with ssh is easy. I use sshpass (previously installed).
dialog --inputbox "Enter IP" 8 78 2> /tmp/ip
IP=$(cat /tmp/ip)
dialog --inputbox "Please enter username" 8 78 2> /tmp/user
US=$(cat /tmp/user)
dialog --passwordbox "enter password for \"$US\" 8 78 2> /tmp/pass
PASSWORD = $(cat /tmp/pass)
sshpass -p "$PASSWORD" ssh $US#$IP mkdir -p /home/$US/TARGET-FOLDER
rm /tmp/ip
rm /tmp/user
rm /tmp/pass
greetings from germany
titus
Building on #Jahid's answer, this worked for me on macOS 10.13:
ssh <remote_username>#<remote_server> sudo -S <<< <remote_password> cat /etc/sudoers
I once had a use case where I needed to run Sudo and ssh in the same command without stdin specifying all the variables needed.
This is the command I used
echo sudopassword | sudo -S -u username sshpass -p extsshpassword ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no username#ipaddress " CMD on external machine"
Breaking that command into pieces!
This will allow you to run commands through your machine using Superuser:
echo password | sudo -S -u username
This will allow you to pass ssh password and execute commands on external machines:
sshpass -p sshpassword ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no username#ipaddress " CMD on external machine"
make sure you install the sudo and openssh packages on your machine.
One way would be to use read -s option .. this way the password characters are not echoed back to the screen. I wrote a small script for some use cases and you can see it in my blog:
http://www.datauniv.com/blogs/2013/02/21/a-quick-little-expect-script/
USE:
echo password | sudo command
Example:
echo password | sudo apt-get update; whoami
Hope It Helps..
You can provide password as parameter to expect script.
su -c "Command" < "Password"
Hope it is helpful.
In writing my deployment script, I want to set a git checkout url that I want to be secret. I want to create a Travis job to test out my playbook. The easiest approach that I can think of now is by setting my global_vars to look for an env variable say DEPLOYMENT_GIT_URL. I then encrypt this env variable in travis and pass it to docker exec when I am building the docker image to test against my playbook.
Question:
Can I pass my encrypted Travis variable to the instance via docker exec ? Something like sudo docker exec ... export DEPLOYMENT_GIT_URL=$TRAVIS_ENV ansible-playbook -i ....
While this seems the simplest way to do it, appreciate comments on this method.
Thanks
You can pass variables directly to Ansible. If you want nested or complex variables, use a JSON string.
docker exec <container> ansible-playbook -e GITURL="$GITURL" whatever.yml
As gogstad mentions, Ansible has the facility to manage secrets via ansible-vault. Unless this URL is information that Travis needs to be the source of, it might be easier storing it directly in Ansible. Otherwise openssl can manage the secret:
secret=$(echo -n "yourdata" | openssl enc -e -aes-256-cbc -a -k 'passpasspass')
echo $secret | openssl enc -d -aes-256-cbc -a -k 'passpasspass'
If you really want to pass the data with an environment variable you would need to do that at container creation with docker run and -e
docker run -e GITURL="what" busybox sh -c 'echo $GITURL'
I've really been struggling over the past few days trying to setup some docker containers and shell scripts to create an environment for my application to run in.
The tall and short is that I have a web server which requires a database to operate. My aim is to have end users unzip the content onto their docker machine, run a build script (which just builds the relevant docker images), then run a OneTime.sh script (which creates the volumes and databases necessary), during this script, they are prompted for what user name and password they would like for the super user of the database.
The problem I'm having is getting those values to the docker image. Here is my script:
# Create the volumes for the data backend database.
docker volume create --name psql-data-etc
docker volume create --name psql-data-log
docker volume create --name psql-data-lib
# Create data store database
echo -e "\n${TITLE}[Data Store Database]${NC}"
docker run -v psql-data-etc:/etc/postgresql -v psql-data-log:/var/log/postgresql -v psql-data-lib:/var/lib/postgresql -p 9001:5432 -P --name psql-data-onetime postgres-setup
# Close containers
docker stop psql-data-onetime
docker rm psql-data-onetime
docker stop psql-transactions-onetime
docker rm psql-transactions-onetime
And here is the docker file:
FROM ubuntu
#Required environment variables: USERNAME, PASSWORD, DBNAME
# Add the PostgreSQL PGP key to verify their Debian packages.
# It should be the same key as https://www.postgresql.org/media/keys/ACCC4CF8.asc
RUN apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys B97B0AFCAA1A47F044F244A07FCC7D46ACCC4CF8
# Add PostgreSQL's repository. It contains the most recent stable release
# of PostgreSQL, ``9.3``.
RUN echo "deb http://apt.postgresql.org/pub/repos/apt/ precise-pgdg main" > /etc/apt/sources.list.d/pgdg.list
# Install ``python-software-properties``, ``software-properties-common`` and PostgreSQL 9.3
# There are some warnings (in red) that show up during the build. You can hide
# them by prefixing each apt-get statement with DEBIAN_FRONTEND=noninteractive
RUN apt-get update && apt-get install -y python-software-properties software-properties-common postgresql-9.3 postgresql-client-9.3 postgresql-contrib-9.3
# Note: The official Debian and Ubuntu images automatically ``apt-get clean``
# after each ``apt-get``
# Run the rest of the commands as the ``postgres`` user created by the ``postgres-9.3`` package when it was ``apt-get installed``
USER postgres
# Complete configuration
USER root
RUN echo "host all all 0.0.0.0/0 md5" >> /etc/postgresql/9.3/main/pg_hba.conf
RUN echo "listen_addresses='*'" >> /etc/postgresql/9.3/main/postgresql.conf
# Expose the PostgreSQL port
EXPOSE 5432
# Add VOLUMEs to allow backup of config, logs and databases
RUN mkdir -p /var/run/postgresql && chown -R postgres /var/run/postgresql
VOLUME ["/etc/postgresql", "/var/log/postgresql", "/var/lib/postgresql"]
# Run setup script
ADD Setup.sh /
CMD ["sh", "Setup.sh"]
The script 'Setup.sh' is the following:
echo -n " User name: "
read user
echo -n " Password: "
read password
echo -n " Database Name: "
read dbname
/etc/init.d/postgresql start
/usr/lib/postgresql/9.3/bin/psql --command "CREATE USER $user WITH SUPERUSER PASSWORD '$password';"
/usr/lib/postgresql/9.3/bin/createdb -O $user $dbname
exit
Why doesn't this work? (I don't get prompted to enter the text, and it throws an error that the parameters are bad). What is the proper way to do something like this? It feels like it's probably a pretty common problem to solve, but I cannot for the life of me find any non convoluted examples of this behaviour.
The main purpose of this is to make life easier for the end user, so if I could just prompt them for the user name, password, and dbname, (plus calling the correct scripts), that would be ideal.
EDIT:
After running the log file looks like this:
User name:
Password:
Database Name:
Usage: /etc/init.d/postgresql {start|stop|restart|reload|force-reload|status} [version ..]
EDIT 2:
After updating to CMD ["sh", "-x", "Setup.sh"]
I get:
echo -n User name:
+read user
:bad variable nameuser
echo -n Password:
+read password
:bad variable namepassword
echo -n Database Name:
+read dbname
:bad variable dbname