I am trying to move all the *.csv files to another folder on server but every time i get access failed error , I am able to get all the files to local server using mget but mv fails everytime , i can see the file on the server and got full permissions on the files, sh script is not working with wild characters. struck here with the simple command .
Download to local directory
localDir="/home/toor/UCDownloads/"
[ ! -d $localDir ] && mkdir -p $localDir
#sftp in the file directory to be downloaded
remoteDir="/share/CACHEDEV1_DATA/Lanein1/Unicard/"
#The file to be downloaded is fileName
lftp -u ${sftp_user},${password} sftp://${host}:${port}<<EOF
PS4='$LINENO: '
set xfer:log true
set xfer:log-file "$logfileUCARC"
set xfer:clobber true
set xfer:auto-rename true
debug 9
cd ${remoteDir}
lcd ${localDir}
#mget *.CSV
ls -l
mv "/share/CACHEDEV1_DATA/Lanein1/Unicard/"*.csv "/share/CACHEDEV1_DATA/Lanein1/Unicard/Archives/"
#rm /share/CACHEDEV1_DATA/Lanein1/Unicard/!(*.pdf)
bye
EOF
This is not a shell or Bash problem. It is a LFTP problem.
From the manual of LFTP:
mv file1 file2
Rename file1 to file2. No wildcard expansion is performed.
LFTP just does not support what you asking for. It will treat *.csv as a part of the file name.
See here for an alternative.
I use terraform to create an EC2 instance, and I use user_data to place a file in /var/lib/cloud/scripts/per-once. This is not executed - my question now is: is cloud-init run before user_data?
===EDIT===
A longer reply to Dude0001's very helpful answer:
I have tried the following, now - this is my user_data:
#!/bin/bash
cat >/var/lib/cloud/scripts/per-once/install_mysql <<!
#cloud-config
package_update: true
packages:
- mysql-server
!
cat >>/root/.bashrc <<!
set -o vi
unalias -a
alias ll='ls -lp'
!
cat >>/home/admin/.bashrc <<!
set -o vi
unalias -a
alias ll='ls -lp'
!
cat /root/.vimrc <<!
set t_ti= t_te=
set compatible
set expandtab ts=2 sw=2 ai
!
cat >/home/admin/.vimrc <<!
set t_ti= t_te=
set compatible
set expandtab ts=2 sw=2 ai
!
This creates all the files, as expected (I'm really old-fashioned and don't like most of vim's new features). I tried to reboot after the instance was created: no mysqld. I changed the permissions, chmod 755 /var/lib/cloud/scripts/per-once/install_mysql, and rebooted: no result either (the reason I changed permissions is that it appears from the python code that cloud-init looks for executables only).
===EDIT===
Some explanations to my user_data above:
This construction may mystify some, since it isn't too common:
cat >/some/path/to/a/file <<!
...
!
cat is a command that simply read from the standard input and writes to the standard output without change - it is often used with redirection < and >. In the construction above, I direct any output to a file /some/path/to/a/file. The other part, involving <<! and ! is known as a here document, something that has its origin in the JCL language used on mainframes, I suspect, but it is really useful. What is means is read the following lines until the end-marker (here: !, but it could be any string). So, all in all, it says create a file with the following content: ....
The first file, /var/lib/cloud/scripts/per-once/install_mysql, contains:
#cloud-config
package_update: true
packages:
- mysql-server
My hope is that this should tell cloud-init to update the package repository and install mysql-server - this doesn't happen.
The next 4 files are just some setup in the root and admin users' environments; basically, I create a .vimrc and add a few lines to .bashrc to ensure that certain things are set up to my liking.
The files are all created, but the one with #cloud-config doesn't seem to get touched at all. I have done a few experiments yesterday, by placing this file in different directories under /var/lib/cloud/scripts/, but it looks a lot as if these files aren't in place for when cloud-init reads the directories. Reading through cloud-init's source code, it looks as if it runs through 10 stages - user_data is fetched in stage 5, and it should be read in stage 7. I can also see that it seems to require the execute permission bit to be set; however this is what is in the log after a reboot:
2019-10-02 08:06:52,884 - handlers.py[DEBUG]: start: modules-final/config-scripts-per-boot: running config-scripts-per-boot with frequency always
2019-10-02 08:06:52,884 - helpers.py[DEBUG]: Running config-scripts-per-boot using lock (<cloudinit.helpers.DummyLock object at 0x7f677362acc0>)
2019-10-02 08:06:52,885 - util.py[DEBUG]: Running command ['/var/lib/cloud/scripts/per-boot/install_mysql'] with allowed return codes [0] (shell=False, capture=False)
2019-10-02 08:06:52,887 - util.py[WARNING]: Failed running /var/lib/cloud/scripts/per-boot/install_mysql [-]
2019-10-02 08:06:52,887 - util.py[DEBUG]: Failed running /var/lib/cloud/scripts/per-boot/install_mysql [-]
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 1992, in subp
env=env, shell=shell)
File "/usr/lib/python3.7/subprocess.py", line 775, in __init__
restore_signals, start_new_session)
File "/usr/lib/python3.7/subprocess.py", line 1522, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
OSError: [Errno 8] Exec format error: b'/var/lib/cloud/scripts/per-boot/install_mysql'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 835, in runparts
subp(prefix + [exe_path], capture=False)
File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 2000, in subp
stderr="-" if decode else b"-")
cloudinit.util.ProcessExecutionError: Exec format error. Missing #! in script?
Command: ['/var/lib/cloud/scripts/per-boot/install_mysql']
Exit code: -
Reason: [Errno 8] Exec format error: b'/var/lib/cloud/scripts/per-boot/install_mysql'
Stdout: -
Stderr: -
2019-10-02 08:06:52,897 - cc_scripts_per_boot.py[WARNING]: Failed to run module scripts-per-boot (per-boot in /var/lib/cloud/scripts/per-boot)
2019-10-02 08:06:52,898 - handlers.py[DEBUG]: finish: modules-final/config-scripts-per-boot: FAIL: running config-scripts-per-boot with frequency always
2019-10-02 08:06:52,898 - util.py[WARNING]: Running module scripts-per-boot (<module 'cloudinit.config.cc_scripts_per_boot' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_scripts_per_boot.py'>) failed
2019-10-02 08:06:52,898 - util.py[DEBUG]: Running module scripts-per-boot (<module 'cloudinit.config.cc_scripts_per_boot' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_scripts_per_boot.py'>) failed
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/cloudinit/stages.py", line 800, in _run_modules
freq=freq)
File "/usr/lib/python3/dist-packages/cloudinit/cloud.py", line 54, in run
return self._runners.run(name, functor, args, freq, clear_on_fail)
File "/usr/lib/python3/dist-packages/cloudinit/helpers.py", line 187, in run
results = functor(*args)
File "/usr/lib/python3/dist-packages/cloudinit/config/cc_scripts_per_boot.py", line 41, in handle
util.runparts(runparts_path)
File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 842, in runparts
% (len(failed), len(attempted)))
RuntimeError: Runparts: 1 failures in 1 attempted commands
So, it definitely doesn't like the format of the file - it wants to see a #!... or perhaps a binary executable.
I will try out Dude0001's suggestions in more detail now.
===EDIT===
In the end, what does work is using the multipart/mixed format, as suggested by Dude0001:
Content-Type: multipart/mixed; boundary="//"
MIME-Version: 1.0
--//
Content-Type: text/cloud-config; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename="cloud-config.txt"
#cloud-config
package_update: yes
package_upgrade: all
packages:
- mariadb-server
- apt-file
--//
Content-Type: text/x-shellscript; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename="userdata.txt"
#!/bin/bash
cat >>/root/.bashrc <<!
set -o vi
unalias -a
alias ll='ls -lp'
!
cat >>/home/admin/.bashrc <<!
set -o vi
unalias -a
alias ll='ls -lp'
!
cat /root/.vimrc <<!
set t_ti= t_te=
set compatible
set expandtab ts=2 sw=2 ai
!
cat >/home/admin/.vimrc <<!
set t_ti= t_te=
set compatible
set expandtab ts=2 sw=2 ai
!
--//
Just specifying #cloud-config doesn't seem to work, but this way does. For me, at least. In the present moment.
Short answer:
A user_data value set to a shell script will cause the given shell script to be ran during in the final stage of cloud-init (and I believe after the cloud-init directives in the one-time folder you reference).
If you want to use a custom cloud-init directive and a shell script both in EC2 user_data property you need to use the multipart/mixed mime format https://aws.amazon.com/premiumsupport/knowledge-center/execute-user-data-ec2/
Long answer:
The user_data can hold raw data to be read through the EC2 meta data, a script or a cloud-init directive. Additionally, you can set it up as a multipart/mixed mime type and provide each of these.
If user_data is raw data, it can be fetched with a curl command inside the EC2 instance. It is up to the calling command to interpret the data, it can be whatever the user chooses.
[ec2-user ~]$ curl http://169.254.169.254/latest/user-data
If user_data is a script (e.g. #!/bin/bash in the first line), it is ran as a step in cloud-init in the final stage of cloud-init https://cloudinit.readthedocs.io/en/latest/topics/boot.html#final.
If user_data is a cloud-init directive (e.g. #cloud-config in the first line), it is ran as the specified cloud-init directive.
From https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/user-data.html#user-data-cloud-init
"To pass cloud-init directives to an instance with user_data... enter your cloud-init directive text in the user_data text."
Like so
#cloud-config
repo_update: true
repo_upgrade: all
packages:
- httpd
- mariadb-server
runcmd:
- [ sh, -c, "amazon-linux-extras install -y lamp-mariadb10.2-php7.2 php7.2" ]
- systemctl start httpd
- sudo systemctl enable httpd
- [ sh, -c, "usermod -a -G apache ec2-user" ]
- [ sh, -c, "chown -R ec2-user:apache /var/www" ]
- chmod 2775 /var/www
- [ find, /var/www, -type, d, -exec, chmod, 2775, {}, \; ]
- [ find, /var/www, -type, f, -exec, chmod, 0664, {}, \; ]
- [ sh, -c, 'echo "<?php phpinfo(); ?>" > /var/www/html/phpinfo.php' ]
The multipart/mixed mime format is described here https://aws.amazon.com/premiumsupport/knowledge-center/execute-user-data-ec2/, with the example
Content-Type: multipart/mixed; boundary="//"
MIME-Version: 1.0
--//
Content-Type: text/cloud-config; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename="cloud-config.txt"
#cloud-config
cloud_final_modules:
- [scripts-user, always]
--//
Content-Type: text/x-shellscript; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename="userdata.txt"
#!/bin/bash
/bin/echo "Hello World" >> /tmp/testfile.txt
--//
I have a postfix server and procmail installed and working.
The problem is when I try to output the content of an e-mail to a file.
I have the following script:
/var/log/user1/fooscript.sh
#!/bin/bash
echo "Trying to get e-mail" > success.txt
echo $1 >> success.txt
/var/log/user1/.procmailrc
VERBOSE=off
PMDIR=$HOME/.procmail
LOGFILE=$PMDIR/procmail.log
INCLUDERC=$PMDIR/rc.filters
/var/log/user1/.procmail/rc.filters
:0
* ^From:(.*\<)?(test#gmail\.com)\>
| /var/log/user1/fooscript.sh
After sending an e-mail, /var/log/user1/.procmail/rc.filters
contains:
From test#gmail.com Thu Jul 18 05:08:13 2013
Folder: /var/log/user1/fooscript.sh 513
but the success file only shows:
Trying to get e-mail
(empty line)
I've chmod 777 all files and directories, so don't think its a permissions issue.
Any help would be greatly appreciated.
Your script gets the message via standard input (STDIN). Try:
#!/bin/bash
echo "Trying to get e-mail" > success.txt
# append data read from STDIN to success.txt file
cat >> success.txt
BTW for more complicated scripts use custom lock to avoid running two scripts in parallel:
:0 w :fooscript.lock
* ^From:(.*\<)?(test#gmail\.com)\>
| /var/log/user1/fooscript.sh
I have a bash code to backup my iOS files and send them to my website FTP in the directory: (http://mywebsite.com/sms) but when I run this code, it isn't .zip'ing the files and leaves the file 'zippyy.db' in the root of my website, not in the /sms folder.
I will be running this script from a few devices so when I execute the code, if there is already a file in the FTP called zippyy.zip, it will change it to zippyy1.zip, zippyy2.zip etc..
I would be really grateful for somebody to re-write the script for me. Thank you in advance! Here's my code:
#!/bin/bash
ROOTFOLDER="/var/root"
ZIPNAME="zipfolder"
ZIPFOLDER=$ROOTFOLDER/$ZIPNAME
LIBFOLDER="/var/mobile/Library"
ZIPFILE="zippyy.zip"
mkdir -p $ZIPFOLDER
cp $LIBFOLDER/SMS/sms.db $ZIPFOLDER/
cp $LIBFOLDER/Notes/notes.sqlite $ZIPFOLDER/
cp $LIBFOLDER/Safari/Bookmarks.db $ZIPFOLDER/
cp $LIBFOLDER/Safari/History.plist $ZIPFOLDER/
cd $ROOTFOLDER
zip -r $ZIPFILE $ZIPNAME
HOST=HOSTNAME
USER=USERNAME
PASS=PASSWORD
ftp -inv $HOST << EOF
user $USER $PASS
cd sms
dir . remote_dir.txt
bye
EOF
FILECOUNT=$(grep zippyy remote_dir.txt | wc -l)
NEXTDB="zippyy${FILECOUNT}.db"
mv $ZIPFILE $NEXTDB
ftp -inv $HOST << EOF
user $USER $PASS
put $NEXTDB
bye
EOF
You mean your archive is corrupt once it's been ftp'd?
Its likely your sending the file in default mode on your machine, which must be ASCII mode.
But first, on you local copy of zip file, issue the test option
zip -t $ZIPFILE
If that succeeds, then change you ftp here-doc to
ftp -inv $HOST << EOF
user $USER $PASS
binary
put $NEXTDB
bye
EOF
Note the addition of the ftp command binary, which means send file without translations for ASCII.
It's highly recommended to issue the following command
man ftp
And read through it at least once. Granted there are sections of even a good ftp man page that I have failed to find useful! ;-) . Also be aware that there are many ftp clients, with only a semblance of adherence to a common set of options, parameters and sub-commands. Don't assume that once you get it working at home, that it will work at the office, or at your friends place!
IHTH
Does anybody know if there is an rsync option, so that directories that are being traversed do not show on stdout.
I'm syncing music libraries, and the massive amount of directories make it very hard to see which file changes are actually happening.
I'v already tried -v and -i, but both also show directories.
If you're using --delete in your rsync command, the problem with calling grep -E -v '/$' is that it will omit the information lines like:
deleting folder1/
deleting folder2/
deleting folder3/folder4/
If you're making a backup and the remote folder has been completely wiped out for X reason, it will also wipe out your local folder because you don't see the deleting lines.
To omit the already existing folder but keep the deleting lines at the same time, you can use this expression :
rsync -av --delete remote_folder local_folder | grep -E '^deleting|[^/]$'
I'd be tempted to filter using by piping through grep -E -v '/$' which uses an end of line anchor to remove lines which finish with a slash (a directory).
Here's the demo terminal session where I checked it...
cefn#cefn-natty-dell:~$ mkdir rsynctest
cefn#cefn-natty-dell:~$ cd rsynctest/
cefn#cefn-natty-dell:~/rsynctest$ mkdir 1
cefn#cefn-natty-dell:~/rsynctest$ mkdir 2
cefn#cefn-natty-dell:~/rsynctest$ mkdir -p 1/first 1/second
cefn#cefn-natty-dell:~/rsynctest$ touch 1/first/file1
cefn#cefn-natty-dell:~/rsynctest$ touch 1/first/file2
cefn#cefn-natty-dell:~/rsynctest$ touch 1/second/file3
cefn#cefn-natty-dell:~/rsynctest$ touch 1/second/file4
cefn#cefn-natty-dell:~/rsynctest$ rsync -r -v 1/ 2
sending incremental file list
first/
first/file1
first/file2
second/
second/file3
second/file4
sent 294 bytes received 96 bytes 780.00 bytes/sec
total size is 0 speedup is 0.00
cefn#cefn-natty-dell:~/rsynctest$ rsync -r -v 1/ 2 | grep -E -v '/$'
sending incremental file list
first/file1
first/file2
second/file3
second/file4
sent 294 bytes received 96 bytes 780.00 bytes/sec
total size is 0 speedup is 0.00