Modifying/Combining Text Files with Scripts - powershell

I am running a program on my machine that is triggered by running a .bat file. Right now, I am manually modifying the .bat to point to specific files/folders before running the script, which only takes a few minutes since I am running this on 3 to 4 files at a time. In the very near future, I am going to need to run this script on groups of files ranging from 200 to 500. Manually trying to edit the .bat file each time would be a nightmare.
The finished .bat would look like:
cd\[rootfolder]
mkdir Output
cd\folderpath\to\program
this.is.the.program.exe -i "[rootfolder]\[filename1].pdf" -r "[rootfolder]" -o "[rootfolder]\Output" -u username -p password
this.is.the.program.exe -i "[rootfolder]\[filename2].pdf" -r "[rootfolder]" -o "[rootfolder]\Output" -u username -p password
this.is.the.program.exe -i "[rootfolder]\[filename3].pdf" -r "[rootfolder]" -o "[rootfolder]\Output" -u username -p password
this.is.the.program.exe -i "[rootfolder]\[filename4].pdf" -r "[rootfolder]" -o "[rootfolder]\Output" -u username -p password
What I would like to do is create another script, .ps or .bat, that will take a list of filenames from a .txt file (dir /b output) and add the information from above in the correct place.
cd\[rootfolder]
mkdir Output
cd\folderpath\to\program
would only occur in the head of the script; This can be done in another way, but if it is included, that is fine...
this.is.the.program.exe -i "[rootfolder]\
would be added before each filename in the .txt file; I can manage this so far with a search/replace operation...
.pdf" -r "[rootfolder]" -o "[rootfolder]\Output" -u username -p password
would be added behind the filename. This is where I am having the issue!
The only similarity to the files will be the beginning of the filename; such as "Text_", which is why I can do a search/replace operation. The ending of the files are completely random and could be alpha, numeric or symbols, and could be of any character length.
I guess my question would be:
Is there a way to insert text into a .txt file by line position or something similar? Behind the last character on each line?

try this:
cd /d "rootfolder"
md output
for %%a in (*.pdf) do "folderpath\to\program\this.is.the.program.exe" -i "%%~a" -r "rootfolder" -o "rootfolder\Output" -u username -p password
or this:
cd /d "rootfolder"
md output
cd /d "folderpath\to\program"
for %%a in ("rootfolder\*.pdf") do "this.is.the.program.exe" -i "%%~a" -r "%%~dpa" -o "%%~dpaOutput" -u username -p password

In powershell you'd probably do something like this:
$rootfolder = 'C:\path\to\rootfolder'
$outputfolder = Join-Path $rootfolder 'Output'
$programfolder = 'C:\program\folder'
$filelist = 'C:\path\to\files.txt'
$user = 'username'
$pass = 'password'
if ( -not (Test-Path -LiteralPath $outputfolder) ) {
New-Item -ItemType directory -Path $outputfolder
}
Set-Location $programfolder
Get-Content $filelist | % {
.\this.is.the.program.exe -i (Join-Path $rootfolder $_) -r $rootfolder `
-o $outputfolder -u $user -p $pass
}

cd\[rootfolder]
mkdir Output
cd\folderpath\to\program
for /f "delims=" %%a in (
' the dir /b command that generates the .txt file '
) do (
this.is.the.program.exe -i "[rootfolder]\%%a.pdf" -r "[rootfolder]" -o "[rootfolder]\Output" -u username -p password
)
should execute this requirement.

Related

How to sort the output of Find to a psql copy command to load data in order?

I wish to load data to a PostgreSQL DB from a bunch of files that are in multiple folders. I have to load them in order (i.e. files in folders 2020 must be loaded before folders 2021 and so on). This is what I have currently:
find ~/data/inserts/ -type f -exec psql -h db1.cluster-xxxxx.us-east-1.rds.amazonaws.com -p 5432 dbname -U admin1 -c "\COPY public.db1(col1,col2) FROM '{}' DELIMITER ',' CSV HEADER;" \;
This loads the data in the files, but the files aren't sorted. By Googling, I know you can pipe into sort like so:
find ~/data/inserts/ -type f -print | sort -z | xargs -r0 echo
but I am not sure how to apply it to my case. I am not sure how to use xargs -r0 even after reading the docs.
You need -print0 instead of -print as find argument:
#!/usr/bin/env bash
# Pipe the sorted null delimited output of find to while loop
find ./ -type f -print0 | sort -z |
while IFS= read -r -d '' input_file || [ -n "$input_file" ]; do
# Now execute the pgsql command to copy from STDIN rather than named file
psql \
-h db1.cluster-xxxxx.us-east-1.rds.amazonaws.com -p 5432 -U admin1 dbname \
-c "COPY public.db1(col1,col2) FROM STDIN DELIMITER ',' CSV HEADER;" \
<"$input_file" # This provide the input file as STDIN
done

How to trim/remove leading/left white spaces from a text file using Windows Batch?

I want to remove/trim the leading/left white space which are as a result of me hiding headers and footers from my PostgreSQL query using Windows batch. I am not sure whether these are white spaces or tabs.
My SQL query:
psql -d databasename -p portname -U username -t -f filename -o "C:\text.txt"
I am not aware of any other way to do this since my SQL is a multi line query and I am not sure if we can do this using -c.
Previous the result was something like this:
After removing the header:
So as you can see there is a white space here and I want to remove it.
Can someone please help me with this?
Have a look at the -t and -A psql parameters:
-t removes headers and footers from the results
-A switches off aligned mode (which is most likely where your whitespace is coming from - alignment into columns).
So the command should look something like the following:
psql -d databasename -p portname -U username -t -A -f filename -o "C:\text.txt"
So, basically, you shouldn't need to modify the resulting file - you can modify your psql command to get results in a format you want.
Here is an hybrid script (batch\vbscript) to trim a string left and right :
#echo off
Set "VAR= abc#abc.com "
echo My Variable before the Trim Function VAR="%VAR%"
Call :Trim "%VAR%"
echo(
echo My Variable after the Trim Function VAR="%VAR%"
pause>nul & exit
::*************************************************************************
:Trim <String>
(
echo Wscript.echo Trim("%~1"^)
)>"%tmp%\%~n0.vbs"
for /f "delims=" %%a in ('Cscript /nologo "%tmp%\%~n0.vbs"') do (
set "VAR=%%a"
)
exit /b
::**************************************************************************

Postgres backup script not emailing

I am using autopostgresqlbackup to backup my PostgreSQL database on Debian 8.3. Everything works except the zipped backup does not get emailed to me. When I run the script with "sudo autopostgresqlbackup" I get:
Can't stat <redacted name>#gmail.com: No such file or directory
<redacted name>#gmail.com: unable to attach file.
From my research this may be caused by mutt requiring "the -a option must be placed at the end of command line options"
The relevant part of the script itself seems to be:
ATTSIZE=`du -c $BACKUPFILES | grep "[[:digit:][:space:]]total$" |sed s/\s*total//`
if [ $MAXATTSIZE -ge $ATTSIZE ]
then
if which biabam >/dev/null 2>&1
then
BACKUPFILES=$(echo $BACKUPFILES | sed -r -e 's#\s+#,#g')
biabam -s "PostgreSQL Backup Log and SQL Files for $HOST - $DATE" $BACKUPFILES $MAILADDR < $LOGFILE
elif which heirloom-mailx >/dev/null 2>&1
then
BACKUPFILES=$(echo $BACKUPFILES | sed -e 's# # -a #g')
heirloom-mailx -s "PostgreSQL Backup Log and SQL Files for $HOST - $DATE" $BACKUPFILES $MAILADDR < $LOGFILE
elif which mutt >/dev/null 2>&1
then
BACKUPFILES=$(echo $BACKUPFILES | sed -e 's# # -a #g')
mutt -s "PostgreSQL Backup Log and SQL Files for $HOST - $DATE" $BACKUPFILES $MAILADDR < $LOGFILE
else
cat "$LOGFILE" | mail -s "WARNING! - Enable to send PostgreSQL Backup dumps, no suitable mail client found on $HOST - $DATE" $MAILADDR
fi
else
cat "$LOGFILE" | mail -s "WARNING! - PostgreSQL Backup exceeds set maximum attachment size on $HOST - $DATE" $MAILADDR
I have tried to reverse the order of "-a #g" in the above code, but then I get errors. Everything works fine if have the script backup Postgres and just send a log, but not if I have it email the backup file. Any ideas why the email attachment is not working?
SOLVED: Changed the order of "$BACKUPFILES $MAILADDR < $LOGFILE" to "$MAILADDR $BACKUPFILES < $LOGFILE" for mutt, and this stopped mutt from trying to attached the email address as an attachment, and thus failing.
Like so:
mutt -s "PostgreSQL Backup Log and SQL Files for $HOST - $DATE" $MAILADDR $BACKUPFILES < $LOGFILE

How to compare the content of a tarball with a folder

How can I compare a tar file (already compressed) of the original folder with the original folder?
First I created archive file using
tar -kzcvf directory_name.zip directory_name
Then I tried to compare using
tar -diff -vf directory_name.zip directory_name
But it didn't work.
--compare (-d) is more handy for that.
tar --compare --file=archive-file.tar
works if archive-file.tar is in the directory it was created. To compare archive-file.tar against a remote target (eg if you have moved archive-file.tar to /some/where/) use the -C parameter:
tar --compare --file=archive-file.tar -C /some/where/
If you want to see tar working, use -v without -v only errors (missing files/folders) are reported.
Tipp: This works with compressed tar.bz/ tar.gz archives, too.
It should be --diff
Try this (without the last directory_name):
tar --diff -vf directory_name.zip
The problem is that the --diff command only looks for differences on the existing files among the tar file and the folder. So, if a new file is added to the folder, the diff command does not report this.
The method of pix is way slow for large compressed tar files, because it extracts each file individually. I use the tar --diff method loking for files with different modification time and extract and diff only these. The files are extracted into a folder base.orig where base is either the top level folder of the tar file or teh given comparison folder. This results in diffs including the date of the original file.
Here is the script:
#!/bin/bash
set -o nounset
# Print usage
if [ "$#" -lt 1 ] ; then
echo 'Diff a tar (or compressed tar) file with a folder'
echo 'difftar-folder.sh <tarfile> [<folder>] [strip]'
echo default for folder is .
echo default for strip is 0.
echo 'strip must be 0 or 1.'
exit 1
fi
# Parse parameters
tarfile=$1
if [ "$#" -ge 2 ] ; then
folder=$2
else
folder=.
fi
if [ "$#" -ge 3 ] ; then
strip=$3
else
strip=0
fi
# Get path prefix if --strip is used
if [ "$strip" -gt 0 ] ; then
prefix=`tar -t -f $tarfile | head -1`
else
prefix=
fi
# Original folder
if [ "$strip" -gt 0 ] ; then
orig=${prefix%/}.orig
elif [ "$folder" = "." ] ; then
orig=${tarfile##*/}
orig=./${orig%%.tar*}.orig
elif [ "$folder" = "" ] ; then
orig=${tarfile##*/}
orig=${orig%%.tar*}.orig
else
orig=$folder.orig
fi
echo $orig
mkdir -p "$orig"
# Make sure tar uses english output (for Mod time differs)
export LC_ALL=C
# Search all files with a deviating modification time using tar --diff
tar --diff -a -f "$tarfile" --strip $strip --directory "$folder" | grep "Mod time differs" | while read -r file ; do
# Substitute ': Mod time differs' with nothing
file=${file/: Mod time differs/}
# Check if file exists
if [ -f "$folder/$file" ] ; then
# Extract original file
tar -x -a -f "$tarfile" --strip $strip --directory "$orig" "$prefix$file"
# Compute diff
diff -u "$orig/$file" "$folder/$file"
fi
done
To ignore differences in some or all of the metadata (user, time, permissions), you can pipe the result to awk:
tar --compare --file=archive-file.tar -C /some/where/ | awk '!/Mode/ && !/Uid/ && !/Gid/ && !/time/'
That should output only the true differences between the tar and the directory /some/where/
I recently needed a better compare than what "tar --diff" produced so I made this short script:
#!/bin/bash
tar tf "$1" | while read ; do
if [ "${REPLY%/}" = "$REPLY" ] ; then
tar xOf "$1" "$REPLY" | diff -u - "$REPLY"
fi
done
The easy way is to write:
tar df file This compares the file with the current working directory, and tell us about if any of the files has been removed.
tar df file -C path/folder This compares the file with the folder.

Use grep / sed for filename search & replace?

I have a bunch of image files that were incorrectly named 'something#x2.png' and they need to be 'something#2x.png'. They're spread across multiple directories like so:
/images
something#x2.png
/icons
icon#x2.png
/backgrounds
background#x2.png
How can I use grep + sed to find/replace as needed?
Ruby(1.9+)
$ ruby -e 'Dir["**/*#x2.png"].each{|x| File.rename( x, x.sub(/#x2/,"#2x") ) }'
Look at qmv and rename
find -iname '*.png' -print0 | xargs -0 qmv -d
will launch your default editor and allow you to interactively edit the names
rename s/#x2/#2x/ *.png
Slashes look linuxy/unixoid to me. Do you have find and rename?
find -name "*#x2*" -execdir rename 's/#x2/#2x/' {} +
rename is worth installing, comes in some perl-package.
With bash 2.x/3.x
#!/bin/bash
while IFS= read -r -d $'\0' file; do
echo mv "$file" "${file/#x2/#2x}"
done < <(find images/ -type f -name "something*#x2*.png" -print0)
With bash 4.x
#!/bin/bash
shopt -s globstar
for file in images/**; do
[[ "$file" == something*#x2*.png ]] && echo mv "$file" "${file/#x2/#2x}"
done
Note:
In each case I left in an echo so you can do a dry-run, remove the echo if the output is sufficient