i have a file.txt with contents
2021-12-03;12.20.31;13;00000.00;00000.00;NO LINK
2021-12-03;12.33.31;15;00199.94;00000.00;Status OK
2021-12-03;12.35.33; 2;01962.33;00015.48;;Status OK
2021-12-03;13.05.31;13;00000.00;00000.00;NO LINK
so what command to output like below
2021-12-03;12:20:31;13;00000.00;00000.00;NO LINK
2021-12-03;12:33:31;15;00199.94;00000.00;Status OK
2021-12-03;12:35:33; 2;01962.33;00015.48;Status OK
2021-12-03;13:05:31;13;00000.00;00000.00;NO LINK
note.
cut -b 12-19 file.txt (is time)
Thanks for your help.
Rido
I assumed that the lines you want to modify are contained in a file (which I called filea.txt). The script should solve your problem.
Contents of the file 'filea.txt':
$> cat filea.txt
2021-12-03;12.20.31;13;00000.00;00000.00;NO LINK
2021-12-03;12.33.31;15;00199.94;00000.00;Status OK
2021-12-03;12.35.33; 2;01962.33;00015.48;;Status OK
2021-12-03;13.05.31;13;00000.00;00000.00;NO LINK
Script File:
$> cat refrm
#!/usr/bin/bash
in_file="filea.txt"
while read -r line || [ -n "$line" ];
do
line=$(echo "${line}" | sed -E 's/;{2,}/;/g')
IFS=$'\n'
line=$(echo ${line} | sed 's/;/\n/g')
arr=($(IFS='\n' ; echo "${line}"))
for ((n=0; n < ${#arr[*]}; n++))
do
if [[ ${arr[n]} =~ ^[0-9]{2}\.[0-9]{2}\.[0-9]{2} ]];
then
arr[n]=`echo ${arr[n]} | sed 's/\./:/g'`
fi
done
nline=$(IFS=";" ; echo "${arr[*]}")
echo "${nline}"
done < ${in_file}
Output:
$> refrm
2021-12-03;12:20:31;13;00000.00;00000.00;NO LINK
2021-12-03;12:33:31;15;00199.94;00000.00;Status OK
2021-12-03;12:35:33; 2;01962.33;00015.48;Status OK
2021-12-03;13:05:31;13;00000.00;00000.00;NO LINK
Related
Often I need to analyze large logs in console.
I use the following command to colorize important keywords:
echo "string1\nerror\nsuccess\nstring2\nfail" | perl -p -e 's/(success)/\e[1;32;10m$&\e[0m/g;' -e 's/(error|fail)/\e[0;31;10m$&\e[0m/g'
It will colorize "success" with green, and error messages with red and keeps others lines unchanged (as they contain some useful info).
But in some cases I need to colorize values after some marker, but not marker itself, i.e. in these lines
Marker1: value1
Marker2: value2
need to highlight only value1 and value2 by known markers.
I'm looking for a way to modify my current oneliner to add this function
Also I tried the following solution, which I like less
#!/bin/bash
default=$(tput op)
red=$(tput setaf 1 || tput AF 1)
green=$(tput setaf 2 || tput AF 2)
sed -u -r "s/(Marker1: )(.+)$/\1${red}\2${default}/
s/(Marker2: )(.+)$/\1${green}\2${default}/" "${#}"`
But it has some problem with buffering, so it's ok for some constant file, but log which is continuosly running is not displayed at all
UPDATE:
Found a solution with help of some perl guru.
echo -e "string1\nerror\nsuccess\nstring2\nfail\nMaker1: value1\nMaker2: value2" | \
perl -p \
-e 's/(success)/\e[32m$&\e[0m/g;' \
-e 's/(error|fail)/\e[31m$&\e[0m/g;' \
-e 's/(Maker1:) (.*)/$1 \e[36m$2\e[0m/m;' \
-e 's/(Maker2:) (.*)/$1 \e[01;34m$2\e[0m/m;'
echo -e "string1\nerror\nsuccess\nstring2\nfail\nMaker1: value1\nMaker2: value2" | \
perl -p \
-e 's/(success)/\e[32m$&\e[0m/g;' \
-e 's/(error|fail)/\e[31m$&\e[0m/g;' \
-e 's/(Maker1:) (.*)/$1 \e[36m$2\e[0m/m;' \
-e 's/(Maker2:) (.*)/$1 \e[01;34m$2\e[0m/m;'
#!/bin/bash
default=$(tput op)
red=$(tput setaf 1 || tput AF 1)
green=$(tput setaf 2 || tput AF 2)
#default='e[0m'
#red='e[0;31;10m'
#green='e[1;32;10m'
# automaticaly use passed argument file if any or stdin if not
sed -u -r \
"/success/ s//${green}&${default}/
/error|fail/ s//${red}&${default}/
/^Marker1:/ {s//\1${red}/;s/$/${default}/;}
/^Marker2:/ {s//\1${green}/;s/$/${default}/;}" \
$( [ ${##} -gt 0 ] && echo ${#} )
For a one line:
remove other line thans sed one
replace newline in sed by ;
use directly the terminal code in place of variable
remove the last line if you pipe or use specific file instead
Could someone help out on this
I want to print all line between the search pattern (START & END) to different files (new_file_name can be any incremental name provided)
But the search pattern repeats in file hence each time it finds the pattern it should dump the line b/w them into different files
The file is something like this
START --- ./body1/b1
##########################
123body1
abcbody1
##########################
END --- ./body1/b1
START --- ./body2/b2
##########################
123body2
defbody2
##########################
END --- ./body2/b2
perl solution,
perl -MFile::Basename -MFile::Path -ne '
($a) = /^START.+?(\S+)$/;
$b = /^END/;
$a..$b or next;
if ($a){ mkpath(dirname $a); open STDOUT,">",$a; }
$a||$b or print;
' file
Here is my awk solution:
# print_between_patterns.awk
/^START/ { filename = $NF ; next } # On START, use the last field as file name
/^END/ { next } # On END, skip
{ print > filename } # For the rest of the lines, print to file
Assume your data file is called data.txt, the following will do what you want:
awk -f print_between_patterns.awk data.txt
Discussion
After the script ran, you will have ./body1, ./body2, and so on.
If you don't want to skip the BEGIN and END parts, remove the next commands.
Update
If you want to control the output filename in a sequential way:
/^START/ { filename = sprintf("out%04d.txt", ++count) ; next }
/^END/ { next }
{ print > filename }
To get automatically generated incremental file names:
awk '
/^END/ { inBlock=0 }
inBlock { print > outfile }
/^START/ { inBlock=1; outfile = "outfile" ++count }
' file
To use the file names from your input:
awk '
/^END/ { inBlock=0 }
inBlock { print > outfile }
/^START/ {
inBlock=1
outdir = outfile = $NF
sub(/\/[^\/]+$/,"",outdir)
system("mkdir -p \"" outdir "\"")
}
' file
The problem #JamesBond was having below was that I wasn't escaping the "/" within the character list in the sub() so I've updated my answer above to do that now. There's absolutely no reason why that should need to be escaped but apparently both nawk and /usr/xpg4/bin/awk require it:
$ cat file
the
quick/brown
dog
$ gawk '/[/]/' file
quick/brown
$ nawk '/[/]/' file
nawk: nonterminated character class [
source line number 1
context is
>>> /[/ <<< ]/
$ /usr/xpg4/bin/awk '/[/]/' file
/usr/xpg4/bin/awk: /[/: [ ] imbalance or syntax error Context is:
>>> /[/ <<<
and gawk doesn't care either way:
$ gawk --lint --posix '/[/]/' file
quick/brown
$ gawk --lint '/[/]/' file
quick/brown
$ gawk --lint --posix '/[\/]/' file
quick/brown
$ gawk --lint '/[\/]/' file
quick/brown
They all work just fine if I escape the backslash without putting it in a character list:
$ /usr/xpg4/bin/awk '/\//' file
quick/brown
$ nawk '/\//' file
quick/brown
$ gawk '/\//' file
quick/brown
So I guess that's something worth remembering for portability in future!
Using awk:
awk 'sub(/^START/, ""){out=sprintf("out%d", c++); p=1}
sub(/^END/, ""){print > out; p=0} p{print > out}' file
This will find and store each match between START and END into separate files named out1, out2 etc.
This is one way to do it in Bash.
#!/bin/bash
[ -n "$BASH_VERSION" ] || {
echo "You need Bash to run this script."
exit 1
}
shopt -s extglob || {
echo "Unable to enable extglob shell option."
exit 1
}
IFS=$' \t\n' ## Use default.
while read KEY DASH FILENAME; do
if [[ $KEY == START && $DASH == --- && -n $FILENAME ]]; then
CURRENT_FILENAME=$FILENAME
DIRNAME=${FILENAME%%+([^/])}
if [[ -n $DIRNAME ]]; then
mkdir -p "$DIRNAME" || {
echo "Unable to create directory $DIRNAME."
exit 1
}
fi
exec 4>"$CURRENT_FILENAME" || {
echo "Unable to open $CURRENT_FILENAME for output."
exit 1
}
for (( ;; )); do
IFS= read -r LINE || {
echo "End of file reached finding END block of $CURRENT_FILENAME."
exec 4>&-
exit 1
}
read -r KEY DASH FILENAME <<< "$LINE"
if [[ $KEY == END && $DASH == --- && $FILENAME == "$CURRENT_FILENAME" ]]; then
break
else
echo "$LINE" >&4
fi
done
exec 4>&-
fi
done
Make sure you save the script in UNIX file format then run it as bash script.sh < file.
I guess you need to see this.
perl -lne 'print if((/START/../END/) and ($_!~/START/ and $_!~/END/))' your_file
Tested below:
> cat temp
START --- ./body1
##########################
123body1
abcbody1
##########################
END --- ./body1
START --- ./body2
##########################
123body2
defbody2
##########################
END --- ./body2
> perl -lne 'print if((/START/../END/) and ($_!~/START/ and $_!~/END/))' temp
##########################
123body1
abcbody1
##########################
##########################
123body2
defbody2
##########################
>
This might work for you:
csplit -z file '/^START/' '{*}'
Files will be named xx00 xx01 xx..
I'm trying to extract data/urls (in this case - someurl) from a file that contains them within some tag ie.
xyz>someurl>xyz
I don't mind using either awk or sed.
I think the best, easiest, way is with cut:
$ echo "xyz>someurl>xyz" | cut -d'>' -f2
someurl
With awk can be done like:
$ echo "xyz>someurl>xyz" | awk 'BEGIN { FS = ">" } ; { print $2 }'
someurl
And with sed is a little bit more tricky:
$ echo "xyz>someurl>xyz" | sed 's/\(.*\)>\(.*\)>\(.*\)/\2/g'
someurl
we get blocks of something1<something2<something3 and print the 2nd one.
grep was born to extract things:
kent$ echo "xyz>someurl>xyz"|grep -Po '>\K[^>]*(?=>)'
someurl
you could kill a fly with a bomb of course:
kent$ echo "xyz>someurl>xyz"|awk -F\> '$0=$2'
someurl
If your grep supports P option then you can use lookahead and lookbehind regular expression to identify the url.
$ echo "xyz>someurl>xyz" | grep -oP '(?<=xyz>).*(?=>xyz)'
someurl
This is just a sample to get you started not the final answer.
I need to run a bash script via cron to update a file.
The file is a .DAT (similar to csv) and contains pipe separated values.
I need to insert a header row at the top.
Here's what I have so far:
#!/bin/bash
# Grab the file, make a backup and insert the new line
sed -i.bak 1i"red|blue|green|orange|yellow" thefilename.dat
Exit
But how can I save the file as a different file name so that it always takes fileA, edits it and then saves it as fileB
do you really rename the old one to xxx.bak or can you just save a new copy?
either way, just use redirection.
sed 1i"red|blue|green|orange|yellow" thefilename.dat > newfile.dat
or if you want the .bak as well
sed 1i"red|blue|green|orange|yellow" thefilename.dat > newfile.dat \
&& mv thefilename.dat thefilename.dat.bak`
which would create your new file and then, only if the sed completed sucessfully, rename the orig file.
In case anyone finds it useful, here is what I ended up doing....
Grab the original file, convert it to the desired file type, whilst inserting a new header row and making a log of this.
#!/bin/bash -l
####################################
#
# This script inserts a header row in the file $DAT and resaves the file in a different format
#
####################################
#CONFIG
LOGFILE="$HOME/bash-convert/log-$( date '+%b-%d-%y' ).log"
HOME="/home/rootname"
# grab original file
WKDIR="$HOME/public_html/folder1"
# new location to save
NEWDIR="$HOME/public_html/folder2"
# original file to target
DAT="$WKDIR/original.dat"
# file name and type to convert to
NEW="$NEWDIR/original-converted.csv"
####################################
# insert a new header row
HDR="header-row-1|header-row-2|header-row-2 \r"
# and update the log file
{
echo "---------------------------------------------------------" >> $LOGFILE 2>&1
echo "Timestamp: $(date "+%d-%m-%Y: %T") : Starting work" >> $LOGFILE 2>&1
touch "$LOGFILE" || { echo "Can't create logfile -- Exiting." && exit 1 ;} >>"$LOGFILE"
# check if file is writable
sudo chmod 755 -R "$NEW"
echo "Creating file \"$NEW\", and setting permissions."
touch "$NEW" || {
echo "Can't create file \"$NEW\" -- Operation failed - exiting" && exit 1 ;}
} >>"$LOGFILE" 2>&1
{
echo "Prepending line \"$HDR\" to file $NEW."
{ echo "$HDR" ; cat "$DAT" ;} > "$NEW"
{
if [ "$?" -ne "0" ]; then
echo "Something went wrong with the file conversion."
exit 1
else echo "File conversion successful. Operation complete."
fi
}
} >>"$LOGFILE" 2>&1
exit 0
I found more clear an consistent the syntax with 'i' between the two single quotes for the pattern to 'insert'.
You can simply add a header, and save it in a different file with:
sed '1i header' file > file2
In your case:
sed '1i red|blue|green|orange|yellow' file > file2
If you wanted to save it on the same file, you'd use -i option:
sed -i '1i red|blue|green|orange|yellow' file
I'm trying to update a file in Bourne Shell. The user inputs a name and then is prompted to change the persons name, age and courses. Here's part of the code I've written:
echo "please enter the name: \c"
read updateInput
updateNumber=$(grep -cwi "$updateInput" database.txt)
updateRecord=$(grep -i "$updateInput" database.txt)
test=$(! grep -i "$updateInput" database.txt)
if [ "$updateNumber" -gt "1" ]
then
echo "ambiguous input"
elif [ "$updateRecord" = "" ]
then
echo "record not found"
else
lineNumber=$(grep -ni $updateInput database.txt | cut -f1 -d:)
grep -i $updateInput database.txt > tmp
read first last age course1 course2 < tmp
echo "record found, enter new value now:"
echo "name ($first $last): \c"
read uFirst uLast
if [ "$uFirst" != "" ]
then
sed "$lineNumber s/$first/$uFirst/" database.txt
fi
if [ "$uLast" != "" ]
then
sed "$lineNumber s/$last/$uLast/g" database.txt
fi
When run, sed outputs the correct output with the right things changed, but it doesn't actually update the database file at all. I've tried googling all sorts of things, but nothing is working. If someone could point me in the right direction, that would be awesome. Thanks so much :)
If this is GNU sed, you can use the -i option to edit the file in place:
sed -i "$lineNumber s/$first/$uFirst/" database.txt
Otherwise, you will need to capture the sed output into a temporary file, then copy it over the original file.