Hello i have some csv files like that:
"N.º","Fecha Tiempo, GMT-03:00","Temp, °C (LGR S/N: 10466185, SEN S/N: 10466185, LBL: Temperatura)","Acoplador separado (LGR S/N: 10466185)","Acoplador adjunto (LGR S/N: 10466185)","Host conectado (LGR S/N: 10466185)","Parado (LGR S/N: 10466185)","Final de archivo (LGR S/N: 10466185)"
1,03/03/14 01:00:00 PM,25.477,Registrado,,,,
2,03/03/14 02:00:00 PM,24.508,,,,,
3,03/03/14 03:00:00 PM,26.891,,,,,
4,03/03/14 04:00:00 PM,25.525,,,,,
5,03/03/14 05:00:00 PM,27.358,,,,,
Then i wanna convert the second field of data-hour in two fields: date, hour
I'm ok with split date and hour, but when i try to convert hours in am-pm to hours in 24hrs i failed.
Using for all files this command:
awk -F"," '{print $2}' *.csv|awk '{print $1","$2" "$3}'
I'm arriving to that command, in particular:
echo "11:04:44 PM" | awk -F, -v hora=$1 '{system("date --date=$hora +%T");print $hora}'
00:00:00
11:04:44 PM
The problem is the variable inside system(date... beacuse it returns 0 or empty.
Then the question is about how to do thath.
And finnally how to insert tath changes inside the file.
Thanks, very thanks!
On my machine (Mac OS), the command you need is
echo "11:22:33 AM" | awk '{split($1,a,":"); if($2=="PM") {a[1]=a[1]+12;} print a[1] ":" a[2] ":" a[3]}'
This does the splitting of the time manually (rather than relying on date which is a bit platform dependent) and adds 12 to the time if it's PM.
So the whole thing becomes:
awk -F"," '{print $2}' *.csv | awk '{split($1,a,":"); if($2=="PM") {a[1]=a[1]+12;} print a[1] ":" a[2] ":" a[3]}'
Although you really want to skip the first line in the file, so
awk -F"," 'if(NR>1){print $2}' *.csv | awk '{split($1,a,":"); if($2=="PM") {a[1]=a[1]+12;} print a[1] ":" a[2] ":" a[3]}'
Thanks!
Now, after many hours i can convert the time using 'date', then the code is:
echo "11:04:44 PM" | awk -F, -v hora=$1 '{system("date --date=\""$hora"\" +%T");print $hora}'
With thaat you can compare the time in 24hrs and AM/PM
The details was the '\"' before and after the '$hora' variable
;)
Then, to converte the complete DAte-Hour from the csv file you have to put:
awk -F"," '{if (FNR>=3) print $2}' *.csv | awk '{print $1","$2" "$3}'| awk -F, '{system("printf "$1", & date --date=\""$2"\" +%T")}'
Now, i have to design a new file qith the id and values columns....
Related
I am looking for how to convert all dates in a csv file row into this format ? example I want to convert 23/1/17 to 23/01/2017
I use unix
thank you
my file is like this :
23/1/17
17/08/18
1/1/2
5/6/03
18/05/2019
and I want this :
23/01/2017
17/08/2018
01/01/2002
05/06/2003
18/05/2019
I used date_samples.csv as my test data:
23/1/17,17/08/18,1/1/02,5/6/03,18/05/2019
cat date_samples.csv | tr "," "\n" | awk 'BEGIN{FS=OFS="/"}{print $2,$1,$3}' | \
while read CMD; do
date -d $CMD +%d/%m/%Y >> temp
done; cat temp | tr "\n" "," > converted_dates.csv ; rm temp; truncate -s-1 converted_dates.csv
Output:
23/01/2017,17/08/2018,01/01/2002,05/06/2003,18/05/2019
This portion of the code converts your "," to new lines and makes your input DD/MM/YY to MM/DD/YY, since the date command does not accept date inputs of DD/MM/YY. It then loops through re-arranged dates and convert them to DD/MM/YYYY format and temporarily stores them in temp.
cat date_samples.csv | tr "," "\n" | awk 'BEGIN{FS=OFS="/"}{print $2,$1,$3}' | \
while read CMD; do
date -d $CMD +%d/%m/%Y >> temp
done;
This line cat temp | tr "\n" "," > converted_dates.csv ; rm temp; truncate -s-1 converted_dates.csv converts the new line back to "," and puts the output to converted_dates.csv and deletes temp.
Using awk:
awk -F, '{ for (i=1;i<=NF;i++) { split($i,map,"/");if (length(map[3])==1) { map[3]="0"map[3] } "date -d \""map[2]"/"map[1]"/"map[3]"\" \"+%d/%m/%y\"" | getline dayte;close("date -d \""map[2]"/"map[1]"/"map[3]"\" \"+%d/%m/%y\"");$i=dayte }OFS="," }1' file
Explanation:
awk -F, '{
for (i=1;i<=NF;i++) {
split($i,map,"/"); # Loop through each comma separated field and split into the array map using "/" as the field seperator
if (length(map[3])==1) {
map[3]="0"map[3] # If the year is just one digit, pad out with prefix 0
}
"date -d \""map[2]"/"map[1]"/"map[3]"\" \"+%d/%m/%y\"" | getline dayte; # Run date command on day month and year and read result into variable dayte
close("date -d \""map[2]"/"map[1]"/"map[3]"\" \"+%d/%m/%y\""); # Close the date execution pipe
$i=dayte # Replace the field for the dayte variable
}
OFS="," # Set the output field seperator
}1' file
Stuck trying to figure out how to single quotes withing single quotes within double quotes. Here's what I'm trying to do....
From perl, I want to run a system command that...
- does an ssh into a remote machine
- executes 'uptime' and then plucks the last field out of that (avg load last 15 min).
\#\!/usr/bin/env perl
my $cmd = "ssh othermachine 'uptime | awk '{print $NF}'' > local_file.dat";
system($cmd);
Of course this won't run ...
% ./try.pl
Missing }.
%
Missing "}" ??? Looks like it's interpreting the $NF} as a var? I tried escaping the {} chars with no luck. I tried escaping the $, no luck. I tried a space before the }, no luck but different msg (Undefined variable).
c-shell BTW and thanks in advance !
You want the following to be ssh's second argument:
uptime | awk '{print $NF}'
To do that, you simply placed single quotes around it. But that doesn't work because it contains single quotes.
You want to build a string that contains $NF, but you did it as follows:
"...$NF..."
That will place the value of (non-existent) Perl variable $NF in the string.
Do it step by step.
Static:
Remote command:
uptime | awk '{print $NF}'
Local command:
ssh othermachine 'uptime | awk '\''{print $NF}'\''' >local_file.dat
String literal:
my $local_cmd = q{ssh othermachine 'uptime | awk '\''{print $NF}'\''' >local_file.dat}
Dynamic:
use String::ShellQuote qw( shell_quote );
my $remote_cmd = q{uptime | awk '{print $NF}'};
my $local_cmd = shell_quote('ssh', 'othermachine', $remote_cmd) . ' >local_file.dat';
Use Net::OpenSSH and let it do the quoting for you:
use Net::OpenSSH;
my $ssh = Net::OpenSSH->new($othermachine,
remote_shell => 'tcsh');
$ssh->system({stdout_file => 'local_file.dat'},
'uptime', \\'|', 'awk', '{print $NF}')
or die "ssh command failed: " . $ssh->error;
I have a file with several rows and with each row containing the following data-
name 20150801|1 20150802|4 20150803|6 20150804|7 20150805|7 20150806|8 20150807|11532 20150808|12399 2015089|12619 20150810|12773 20150811|14182 20150812|27856 20150813|81789 20150814|41168 20150815|28982 20150816|24500 20150817|22534 20150818|3 20150819|4 20150820|47773 20150821|33168 20150822|53541 20150823|46371 20150824|34664 20150825|32249 20150826|29181 20150827|38550 20150828|28843 20150829|3 20150830|23543 20150831|6
name2 20150801|1 20150802|4 20150803|6 20150804|7 20150805|7 20150806|8 20150807|11532 20150808|12399 2015089|12619 20150810|12773 20150811|14182 20150812|27856 20150813|81789 20150814|41168 20150815|28982 20150816|24500 20150817|22534 20150818|3 20150819|4 20150820|47773 20150821|33168 20150822|53541 20150823|46371 20150824|34664 20150825|32249 20150826|29181 20150827|38550 20150828|28843 20150829|3 20150830|23543 20150831|6
The pipe separated value indicates the value for each of the dates in the month.
Each row has the same format with same number of columns.
The first column name indicates a unique name for the row e.g. 20150818 is yyyyddmm
Given a specific date, how do I extract the name of the row that has the largest value on that day?
I think you mean this:
awk -v date=20150823 '{for(f=2;f<=NF;f++){split($f,a,"|");if(a[1]==date&&a[2]>max){max=a[2];name=$1}}}END{print name,max}' YourFile
So, you pass the date you are looking for in as a variable called date. You then iterate through all fields on the line, and split the date and value of each into an array using | as separator - a[1] has the date, a[2] has the value. If the date matches and the value is greater than any previously seen maximum, save this as the new maximum and save the first field from this line for printing at the end.
You couldn't have taken 5 seconds to give your sample input different values? Anyway, this may work when run against input that actually has different values for the dates:
$ cat tst.awk
BEGIN { FS="[|[:space:]]+" }
FNR==1 {
for (i=2;i<=NF;i+=2) {
if ( $i==tgt ) {
f = i+1
}
}
max = $f
}
$f >= max { max=$f; name=$1 }
END { print name }
$ awk -v tgt=20150801 -f tst.awk file
name2
As a quick&dirty solution, we can perform this in following Unix commands:
yourdatafile=<yourdatafile>
yourdate=<yourdate>
cat $yourdatafile | sed 's/|/_/g' | awk -F "${yourdate}_" '{print $1" "$2}' | sed 's/[0-9]*_[0-9]*//g' | awk '{print $1" "$2}' |sort -k 2n | tail -n 1
With following sample data:
$ cat $yourdatafile
Alice 20150801|44 20150802|21 20150803|7 20150804|76 20150805|71
Bob 20150801|31 20150802|5 20150803|21 20150804|133 20150805|71
and yourdate=20150803 we get:
$ cat $yourdatafile | sed 's/|/_/g' | awk -F "${yourdate}_" '{print $1" "$2}' | sed 's/[0-9]*_[0-9]*//g' | awk '{print $1" "$2}' |sort -k 2n | tail -n 1
Bob 21
and for yourdate=20150802 we get:
$ cat $yourdatafile | sed 's/|/_/g' | awk -F "${yourdate}_" '{print $2" "$1}' | sed 's/[0-9]*_[0-9]*//g' | awk '{print $2" "$1}' | sort -k 2n | tail -n 1
Alice 21
The drawback is that only one line is printed the highest value of a day was achieved by more than one name as can be seen with:
$ yourdate=20150805; cat $yourdatafile | sed 's/|/_/g' | awk -F "${yourdate}_" '{print $2" "$1}' | sed 's/[0-9]*_[0-9]*//g' | awk '{print $2" "$1}' | sort -k 2n | tail -n 1
Bob 71
I hope that helps anyway.
I've a tab delimited log file that has date time in format '2011-07-20 11:34:52' in the first two columns:
An example line from the log file is:
2011-07-20 11:34:15 LHR3 1488 111.111.111.111 GET djq2eo454b45f.cloudfront.net /1010.gif 200 - Mozilla/5.0%20(Windows%20NT%206.1;%20rv:5.0)%20Gecko/20100101%20Firefox/5.0 T=F&Event=SD&MID=67&AID=dc37bcff-70ec-419a-ad43-b92d6092c9a2&VID=8&ACID=36&ENV=demo-2&E=&P=Carousel&C=3&V=3
I'm trying to convert the date time to epoch using just awk:
cat logfile.log | grep 1010.gif | \
awk '{ print $1" "$2" UTC|"$5"|"$10"|"$11"|"$12 }' | \
awk 'BEGIN {FS="|"};{system ("date -d \""$1"\" +%s" ) | getline myvar}'
So this gets me some way, in that it gets me epoch less three 000's on the end - however i'm just getting the output of the system command - where as i really want to substitute $1 with the epoch time.
I'm aiming for the following output:
<epoch time>|$5|$10|$11|$12
I've tried just using:
cat logfile.log | grep 1010.gif | awk '{ print d };' "d=$(date +%s -d"$1")"
But this just gives me blank rows.
Any thoughts.
Thanks
This assumes gawk -- can't do any timezone translation though, strictly local time.
... | gawk '
BEGIN {OFS = "|"}
{
split($1, d, "-")
split($2, t, ":")
epoch = mktime(d[1] " " d[2] " " d[3] " " t[1] " " t[2] " " t[3])
print epoch, $5, $10, $11, $12
}
'
I want to change the second column to upper case and I want to do it in shell script only. (no one liners!)
#!/bin/sh
# read file line by line
file="/pdump/country.000000.txt"
while read line
do
mycol=`echo $line | awk -F"," '{print $2}'`
mycol_new=`echo $mycol | tr "[:lower:]" [:upper:]`
echo $line | awk -F"," '{print $1 "," $mycol_new "," $3 "," $4 "," $5 "," $6 "," $7 "," $8}'
done < $file
I am not able to replace the $2 with $mycol_new.
Any suggestion?
awk cannot see $mycol_new because it is a shell variable. Here is one way of passing a shell variable into awk using the -v flag:
echo $line | awk -v var="$mycol_new" -F"," '{print $1 "," var "," $3 "," $4 "," $5 "," $6 "," $7 "," $8}'
Here is an alternative method which lets the shell expand $mycol_new:
echo $line | awk -F"," '{print $1 ",'"$mycol_new"'," $3 "," $4 "," $5 "," $6 "," $7 "," $8}'
why no one liners? Doing homework?
$ cat file
one two three four
five six seven eight
$ awk '{$2=toupper($2)}1' file
one TWO three four
five SIX seven eight
If you want to do this all in the shell, then you don't need awk:
IFS=,
while read line; do
set -- $line
a="$1"
b="${2^^}" # assumes bash, use "tr" otherwise
shift 2
set -- "$a" "$b" "$#"
echo "$*"
done < "$file" > "$file.new"