I'm trying to create a custom DDEV Provider, to import the current database and also user generated files from the web server.
I want to use it with TYPO3 Projects, where I develop the EXT locally with DDEV (because its awesome :) ) and I want to update my local database and also the "fileadmin" files with the help of the ddev pull function.
I've read the docs: Introduction to Hosting Provider Integration and I tested the bash commands locally within the DDEV Container (ddev ssh) and I'm able to connect to the remote Webserver and make a database dump and transfer it to the local DDEV container.
So I added the bash commands to the my custom provider .yaml file in the /provider/ folder.
Here is the current file:
environment_variables:
DB_NAME: db_name
DB_USER: password
DB_PASSWORD: password
HOST_IP: 11.11.11.11
SSH_USERNAME: username
SSH_PASSWORD: password
SSH_PORT: 22
db_pull_command:
command: |
# Creates the .download folder if it doesn't exist
mkdir -p /var/www/html/.ddev/.downloads
# execute the mysqldump on the remote webserver via SSH
ssh -p ${SSH_PORT} ${SSH_USERNAME}#${HOST_IP} 'mysqldump -h 127.0.0.1 -u ${DB_USER} -p ${DB_PASSWORD} ${DB_NAME} > /tmp/${DB_NAME}.sql.gz'
# download to sql file to the ddev folder
scp -P ${SSH_PORT} ${SSH_USERNAME}#${HOST_IP}:/tmp/${DB_NAME}.sql.gz /var/www/html/.ddev/.downloads/db.sql.gz.
If I execute the pull with ddev pull my-provider I get the following Error:
Downloading database...
bash: 03: command not found
Pull failed: Failed to exec mkdir -p /var/www/html/.ddev/.downloads
I assumed that the commands are executed like I would within the DDEV Container (with ddev ssh). What am I missing?
My Environment:
TYPO3 v10.4.20
Windows 10 (WSL)
Docker Desktop 3.5.2
DDEV-Local version v1.17.7
architecture amd64
db drud/ddev-dbserver-mariadb-10.3:v1.17.7
dba phpmyadmin:5
ddev-ssh-agent drud/ddev-ssh-agent:v1.17.0
docker 20.10.7
docker-compose 1.29.2
The web server is running on Plesk.
Note: I only tried to implement the db pull command so far.
UPDATE 09.11.21:
So I've gotten this far that I'm able update and also download the files. However I'm only able to do it, if I hardcode the variables. Everytime I'm trying to setup the environment_variables: I get the following error, if I run the ddev pull myProvider:
Downloading database...
bash: 03: command not found
Here is my current .yaml file with the environment_variables:, which currently don't work. I've tested all the commands within ddev ssh
and it works if I call them manually.
environment_variables:
DB_NAME: db_name
DB_USER: db_user
DB_PASSWORD: 'Password$'
HOST_IP: 10.10.10.10
SSH_USERNAME: username
SSH_PORT: 21
auth_command:
command: |
ssh-add -l >/dev/null || ( echo "Please 'ddev auth ssh' before running this command." && exit 1 )
db_pull_command:
command: |
mkdir -p /var/www/html/.ddev/.downloads
ssh -p ${SSH_PORT} ${SSH_USERNAME}#${HOST_IP} "mysqldump -h 127.0.0.1 -u ${DB_USER} -p'${DB_PASSWORD}' ${DB_NAME} > /tmp/${DB_NAME}.sql"
scp -P ${SSH_PORT} ${SSH_USERNAME}#${HOST_IP}:/tmp/${DB_NAME}.sql /var/www/html/.ddev/.downloads/db.sql
gzip -f /var/www/html/.ddev/.downloads/db.sql
files_pull_command:
command: |
scp -P ${SSH_PORT} -r ${SSH_USERNAME}#${HOST_IP}:/path/to/public/fileadmin/user_upload /var/www/html/.ddev/.downloads/files
Do I declare the variables the wrong way? Or what is it that I'm missing?
For anyone who has trouble connecting via ssh without the password promt, you can run the following commands:
ssh-keygen -t rsa
ssh-copy-id -i ~/.ssh/id_rsa.pub -p 22 username#host
Afterward you should be able to connect without a password promt. Try the following: ssh -p 22 username#host
before you try to ddev puul you have to execute ddev auth ssh
Thanks to #rfay for pointing me into the right direction.
The Problem was, that my password containted a special charater (not a $ though) which needed to be escaped.
After escpaing it correctly like so
environment_variables:
DB_PASSWORD: 'Password\&\'
the ddev pull works.
I hope my .yaml file helps someone else that needs to pull from a webserver.
I added a new user as follows
inherit extrausers
EXTRA_USERS_PARAMS = "useradd -P p#ssW0rd user1;"
I am trying to find how to add users to sudoers list. Is there a class like extrausers
Update-1:
In class classes/extrausers.bbclass I see usermod supported. Will the following work?
inherit extrausers
EXTRA_USERS_PARAMS = "useradd -P p#ssW0rd user1;\
usermod -aG sudo user1"
Update-2:
I tried adding IMAGE_INSTALL_append += " sudo " and
inherit extrausers
EXTRA_USERS_PARAMS = "useradd -P foobar -G sudo user1;"
But that does not help me in achieving the effect of adding user1 to sudoers list. I see following error when I do sudo -v
Sorry, user user1 may not run sudo on <machine-name>.
Update-3:
I found that the sudoers file has the sudo group commented as follows:
# %sudo ALL=(ALL) ALL
Hence the reason even adding user1 to group sudo didn't help
Rather than adding user1 to group sudo I adopted approach of adding a drop-in file under /etc/sudoers.d/0001_user1 using recipes-extended/sudo/sudo_1.8.14p3.bbappend
do_install_append () {
echo "user1 ALL=(ALL) ALL" > ${D}${sysconfdir}/sudoers.d/001_first
}
Now I need help in understanding which of following is a better approach in terms of security?
uncomment sudo line in /etc/sudoers and adding user1 to /etc/sudoers
adding user1 in /etc/sudoers.d/001_first
So there are two approaches to add an user with sudo capability
Add user to sudo group and enable sudo group in /etc/sudoers
Create a file under ${D}${sysconfdir}/sudoers.d/ and add the sudo rule for user there.
Now which approach is suitable for your distro is well answered in /etc/sudoers vs /etc/sudoers.d/ file for enabling sudo for a user
I'm trying to find Active Directory users who are:
memberOf::Q049RG9tw6RuZW4tQWRtaW5zLENOPVVzZXJzLERDPXh4eCxEQz1pbnRlcm4=
(this base64 stands for CN=Domänen-Admins,CN=Users,DC=xxx,DC=intern)
Searching for it directly (via API or ldapsearch) yields no result (since it's a unicode DN):
ldapsearch -h ... -D Administrator#xxx.intern -x -w '...' -b dc=xxx,dc=intern '(memberof=CN=Domänen-Benutzer,CN=Users,DC=xxx,DC=intern)'
Following filters didn't work either:
(memberof=Q049RG9tw6RuZW4tQWRtaW5zLENOPVVzZXJzLERDPXh4eCxEQz1pbnRlcm4=)
(memberof=:Q049RG9tw6RuZW4tQWRtaW5zLENOPVVzZXJzLERDPXh4eCxEQz1pbnRlcm4=)
(memberof=::Q049RG9tw6RuZW4tQWRtaW5zLENOPVVzZXJzLERDPXh4eCxEQz1pbnRlcm4=)
I can't find any documentation except for RFC specifying base64 encoding in LDIF files.
UPDATE the above ldapsearch commands are for convenience only, it doesn't work with LDAP API either - using:
ldap.search_s('dc=xxx,dc=intern', ldap.SCOPE_SUBTREE, filter, ['cn'])
with filters:
filter='(memberof=CN=Domänen-Benutzer,CN=Users,DC=xxx,DC=intern)'.encode('utf-8') # raw UTF
filter='(memberof=CN=Domänen-Benutzer,CN=Users,DC=xxx,DC=intern)'.encode('cp1252') # raw 1252
filter=b'(memberof=CN=Dom\\e4nen-Benutzer,CN=Users,DC=xxx,DC=intern)' # hex
filter=b'(memberof=CN=Dom\\xe4nen-Benutzer,CN=Users,DC=xxx,DC=intern)' # python repr
I've also confirmed with Wireshark that the filter is indeed transmitted in UTF8
The attribute type cn has a Directory String syntax according to the standard. Directory Strings are encoded using UTF-8. Saying that searching via API yields no results incorrect. You are just using the incorrect encoding. The ldapsearch tool (assuming you are using OpenLDAP) may not support searching with accented characters.
The ldapsearch utility shipped with the UnboundID Data Store handles this pretty well. Here is the LDIF that I used for testing:
dn:: Q049RG9tw4PCpG5lbi1BZG1pbnMsZGM9ZXhhbXBsZSxkYz1jb20=
objectclass: organizationalPerson
sn: person
dn: cn=mygroup,dc=example,dc=com
objectclass: groupofnames
member:: Q049RG9tw4PCpG5lbi1BZG1pbnMsZGM9ZXhhbXBsZSxkYz1jb20=
Here is my command line test:
$ ldapsearch -b "dc=example,dc=com" "member=CN=Domänen-Admins,dc=example,dc=com"
dn: cn=mygroup,dc=example,dc=com
objectClass: top
objectClass: groupofnames
cn: mygroup
member:: Q049RG9tw4PCpG5lbi1BZG1pbnMsZGM9ZXhhbXBsZSxkYz1jb20=
Also, you may want to read the Directory String syntax from RFC 4517.
UPDATE
I managed to make this working with Active Directory (Windows Server 2012 R2, DataCenter edition) and using the ldapsearch utility shipped with the UnboundID Data Store. This is what I see:
$ ldapsearch --trustAll -Z -h <hostname> -p 636 -D "cn=administrator,cn=users,dc=dom-ad2,dc=local" -w <password> -b "cn=test,dc=dom-ad2,dc=local" "member=CN=Domänen-Benutzer,CN=test,DC=dom-ad2,DC=local"
dn: CN=mygroup,CN=test,DC=dom-ad2,DC=local
objectClass: top
objectClass: group
cn: mygroup
member:: Q049RG9tw4PCpG5lbi1CZW51dHplcixDTj10ZXN0LERDPWRvbS1hZDIsREM9bG9jYWw=
member: CN=Administrator,CN=Users,DC=dom-ad2,DC=local
distinguishedName: CN=mygroup,CN=test,DC=dom-ad2,DC=local
instanceType: 4
whenCreated: 20160514104531.0Z
You can also use the LDAPSearch example class from the UnboundID LDAP SDK (jar download link) to accomplish this. Here is the equivalent command-line I used with the LDAP SDK:
$ java -cp unboundid-ldapsdk-3.1.1.jar com.unboundid.ldap.sdk.examples.LDAPSearch --trustAll -Z -h <host> -p 636 -D "cn=administrator,cn=users,dc=dom-ad2,dc=local" -w <password> -b "cn=test,dc=dom-ad2,dc=local" "member=CN=Domänen-Benutzer,CN=test,DC=dom-ad2,DC=local"
It turned out that it's only impossible to fetch members for built-in AD groups (i.e. (isCriticalSystemObject=TRUE)). memberOf queries for user-created groups work just fine, independently from encoding used. No hex encoding was necessary.
Why are you trying to use the base64 encoded value? You must base64 decode the value before using it in a LDAP filter. It either needs to be the string representation name of the value or a hex escaped version of the string value when used in a LDAP filter.
Edit: After translating the group names a bit from your question it became more apparent to me what your issue actually is. The default "Domain Users" group is actually a primary group for a user. It will not show up in the memberOf list (hence the empty results on a search). To check if a user is a member of "Domain Users" you must inspect the primaryGroupId value for a user. In 99% of all cases this will always be the default "Domain Users" group.
When I tried to dump database using command line, it gave me following error.
Command:
D:\TOOLS\MySQL\MySQL Server 5.6\bin>mysqldump -u root -p webchat > tut_backup.sql
Error:
mysqldump: unknown option '--no-beep'
Add host-name in your command, like:
D:\TOOLS\MySQL\MySQL Server 5.6\bin>mysqldump -h localhost -u root -p webchat > tut_backup.sql
Replace localhost with your host-name if it is different.
I consider webchat is your database name and you will have to type password explicitly after firing this command.
And of course you have to remove --no-beep (as suggested by Mark Setchell ) from mysql configuration file.
Hope it helps, thanks.
During the installation of my app, I want to create a PostgreSQL-Database and some tables and functions.
For that purpose I use PSQL.EXE that ships with PostgreSQL. I have 2 scripts. The first one creates the database and a corresponding user that has rights to execute scripts on that database. I want to execute the second script as this just created user. Unfortunately I can't find a way to pass the password for that user as a command line argument. Omitting the password leads to a stop of execution and a prompt for the user to enter the password, which I would like to avoid - since this is executed during installtion of my app.
Is there any way to pass the password as argument or is there any other command line tool I could use?
To explain the environment a bit further. I use WiX 3.5 setup as a "MSI-Builder".
You can either use a pgpass file as dbenhur answerd, or you can set the environment variable PGPASSWORD before calling psql:
SET PGPASSWORD=my_very_secret_password
psql somedb someuser
All supported environment variables are documented in the manual: http://www.postgresql.org/docs/current/static/libpq-envars.html
You can't supply password via cmdline arg (and don't want to as that's poor security practice).
You can provide a .pgpass file to support automatic script authentication. Here's the docs.
Better still, if you have access to create the db role then you already have all the access you need without having to carefully log in with a password. Have the second script operate under the same user as the first but include the following line to switch user:
set role my_new_user;
Where my_new_user is the name of the role you want to run it as.
If you only divided the scripts because of the different logins then with this they can go in the same file and just switch role mid way through.
Note:
On the off chance that you are not creating the DB and new role as a super user this may be a little more complex. If this is the case you will need to create the new role with:
create role my_new_role ... ADMIN my_role;
Where my_new_role is the role you're creating and my_role is your current user. Then when you're finished simply:
revoke my_new_role from my_role;
For completion, you can also use URI (doc link)
List dbs
psql "postgresql://username:password#localhost/postgres" -l
I also crafted this command to have only names (please tell me if you know a better way):
psql "postgresql://username:password#localhost/postgres" -l | awk -F '|' '{print $1}'| sed -e '/^\s*$/ d' -e '1,3d'|sed '$d'|awk '{print $1}'
You can also use unix socket to connect:
# ss -x -a |grep postgres|awk '{print $5}'
/var/run/postgresql/.s.PGSQL.5432
Note that the parent directory of the socket is used:
# sudo -u postgres psql -d "postgresql:///postgres?host=/var/run/postgresql/" -l
You can only do this if you have this line in your pg_hba.conf:
local all postgres ident
"ident" uses unix user for authent
dump a db
Here I added a different port number
pg_dump -Fc "postgresql://username:password#localhost:9001/${db}" > "backup_${db}.pgdump"
With dumpall you need a super user or role (with CREATE ROLE ... SUPERUSER). And it must have access to all DB. By default postgres can.
but in my case I couldn't use pg_dumpall with postgres because his password was removed by devs.
So I used:
sudo -u postgres pg_dumpall -d "postgresql:///?host=/var/run/postgresql/" > all.dump
tested version
# cat /opt/postgresql/PG_VERSION
9.6
hth