unix yyyymmddhhmmss format conversion to specific date format - date

There is a bash script running that outputs folder names appended with time logs_debug_20190213043348. I need to be able to extract the date into a readable format yyyy.mm.dd.hh.mm.ss and also may be convert to GMT timezone. I'm using the below method to extract.
echo "${folder##*_}" | awk '{ print substr($0,1,4)"."substr($0,5,2)"."substr($0,7,2)"."substr($0,9,6)}'
Is there a better way to print the output without writing complex shell scripts?

The internal string conversion functions are too limited, so we use sed and tr when needed.
## The "readable" format yyyy.mm.dd.hh.mm.ss isn’t understood by date.
## yyyy-mm-dd hh:mm:ss is. So we first produce the latter.
# Note how to extract the last 14 characters of ${folder} and that, since
# we know (or should have checked somewhere else) that they are all digits,
# we match them with a simple dot instead of the more precise but less
# readable [0-9] or [[:digit:]]
# -E selects regexp dialect where grouping is done with simple () with no
# backslashes.
d="$(sed -Ee's/(....)(..)(..)(..)(..)(..)/\1-\2-\3 \4:\5:\6/'<<<"${folder:(-14)}")"
# Print the UTF date (for Linux and other systems with GNU date)
date -u -d "$d"
# Convert to your preferred "readable" format
# echo "${d//[: -]/.}" would have the same effect, avoiding tr
tr ': -' '.'<<<"$d"
For systems with BSD date (notably MacOS), use
date -juf'%Y-%m-%d %H:%M:%S' "$d"
instead of the date command given above. Of course, in this case the simplest way would be:
# Convert to readable
d="$(sed -Ee's/(....)(..)(..)(..)(..)(..)/\1.\2.\3.\4.\5.\6/'<<<"${folder:(-14)}")"
# Convert to UTF
date -juf'%Y.%m.%d.%H.%M.%S' "$d"

Here's a pipeline that does what you want. It certainly isn't simple looking, but taking each component it can be understood:
echo "20190213043348" | \
sed -e 's/\([[:digit:]]\{4\}\)\([[:digit:]]\{2\}\)\([[:digit:]]\{2\}\)\([[:digit:]]\{2\}\)\([[:digit:]]\{2\}\)\([[:digit:]]\{2\}\)/\1-\2-\3 \4:\5:\6/' | \
xargs -d '\n' date -d | \
xargs -d '\n' date -u -d
The first line is simply printing the date string so sed can format it (so that it can easily be modified to fit the way you are passing in the string).
The second line with sed is converting the string from the format you give, to something like this, which can be parsed by date: 2019-02-13 04:33:48
Then, we pass the date to date using xargs, and it formats it with the timezone of the device running the script (CST in my case): Wed Feb 13 04:33:48 CST 2019
The final line converts the date string given by the first invocation of date to UTC time rather than being stuck in the local time: Wed Feb 13 10:33:48 UTC 2019
If you want it in a different format, you can modify the final invocation of date using the +FORMAT argument.

Related

How to regex today or previous days date using awk and $date?

Column 13 of my data contains date in YYMMDD format. I'm trying to regex using $date for today and previous days. Neither of the following code would work. Could someone give me some insights?
TODAY
awk -F, ($13~/$(date '+%Y%m%d')/) {n++} END {print n+0}' file.csv)
3 DAYS AGO
awk -F, ($13~/$(date -d "$date -3 days" '+%Y%m%d')/) {n++} END {print n+0}' file.csv
Your Awk attempts have rather severe quoting problems. You will generally want to single-quote your Awk script, and pass in any parameters as variables with -v.
awk -F, -v when="$(date -d "-3 days" '+%Y%m%d')" '$13~when {n++} END {print n+0}' file.csv
Perhaps notice also that $date is not defined anywhere. The notation $(cmd ...) is a command substitution which runs cmd ... and replaces the expression with its output.
Probably also notice that date -d is a GNU extension and is not portable, though it will work on Linus and other platforms where you have the GNU utilities installed.
More fundamentally, depending on what's in $13, you might want to implement a simple date parsing for that format, so that you can specify a range of acceptable values, rather than search for matches on static text.
This quoting is correct for Bourne-style Unix shells. If you are on Windows, the quoting rules are quite different, and quite likely often impossible to apply in useful ways.
If you are using GNU AWK then you might use its' Time Functions to check if it does work do
awk 'END{print strftime("%y%m%d")}' emptyfile.txt
which should output current day in YYMMDD format. If it does then you might get what you want following way:
awk 'BEGIN{today=strftime("%y%m%d");threedago=strftime("%y%m%d",systime()-259200)}END{print today, threedago}' emptyfile.txt
output (as of today)
210809 210806
Explanation: strftime first argument is time format %y is year 00...99, %m is month 01...12, %d is day 01...31. Second argument is optional, and it is seconds since start of epoch. If skipped current time is used, systime() return number of seconds since start of epoch, 259200 is 72 hours as seconds.
Example of usage as regexp, let say that I have file.txt as follows
210807 120
210808 150
210809 100
and want to retrieve content of 2nd column for today, then I can do
awk 'BEGIN{today=strftime("%y%m%d")}$1~today{print $2}' file.txt
getting output (as of today)
100
(tested in gawk 4.2.1)

Convert timestamp (unix 13 digits) to datetime format of a complet column of a csv file using awk or sed

I have a csv file multiple columns. The first columns has timestamps like,
1529500027127
1529500027227
1529500027327
1529500027428
1529500027528
1529500027628
1529500027728
I know you can do something like that for a specific timestamp:
date -d #1529500027528
But how can I select all values of the columns and do that? I tried the next command:
date -d "$(awk -F , -v OFS=, '$1/=1000')" file.csv
I am trying to understand how date command works with other commands.
Since sample of expected output is not given so could only test it with given 1st column values, written and tested in GNU awk. You could use strftime function of awk, also since OP hs mentioned Input_file is a csv file so mentioning FS and OFS as , here.
awk 'BEGIN{FS=OFS=","} {$1=strftime("%Y/%m/%d %H:%M:%S",$1/1000)}1' Input_file
From man awk for strftime:
strftime([format [, timestamp[, utc-flag]]]) Format timestamp
according to the specification in format. If utc-flag is present and
is non-zero or non-null, the result is in UTC, otherwise the result is
in local time. The timestamp should be of the same form as returned
by systime(). If timestamp is missing, the current time of day is
used. If format is missing, a default format equivalent to
the output of date(1) is used. The default format is available in
PROCINFO["strftime"]. See the specification for the strftime()
function in ISO C for the format conversions that are guaranteed to be
available.
If you want to use an external date -d#.... command, you could do this:
awk -F, -v 'OFS=,' '{"date -d#"$1 | getline timestamp ; $1=timestamp; print}' filename
Obviously finding a builtin function to do the same job (in this case, the strftime function as suggested by another answer) is a more efficient solution in terms of execution time, but the above gives an example of how to call out to external programs that you may be already familiar with.

How do I replace a substring by the output of a shell command with sed, awk or such?

I'd like to use sed or any command line tool to replace parts of lines by the output of shell commands. For example:
Replace linux epochs by human-readable timestamps, by calling date
Replace hexa dumps of a specific protocol packets by their decoded counterparts, by calling an in-house decoder
sed seems best fitted because it allows to match patterns and reformat other things too, like moving bits of matches around, but is not mandatory.
Here is a simplified example:
echo "timestamp = 1234567890" | sed "s/timestamp = \(.*\)/timestamp = $(date -u --d #\1 "+%Y-%m-%d %T")/g"
Of course, the $(...) thing does not work. As far as I understand, that's for environment variables.
So what would the proper syntax be? Is sed recommended in this case ? I've spent several hours searching... Is sed even capable of this ? Are there other tools better suited?
Edit
I need...
Pattern matching. The log is full of other things, so I need to be able to pinpoint the strings I want to replace based on context (text before and after, on the same line). This excludes column-position-based matching like awk '{$3...
In-place replacement, so that the reste of the line, "Timestamp = " or whatever, remains unchanged. This exclused sed's 'e' command.
To run an external command in sed you need to use e. See an example:
$ echo "timestamp = 1234567890" | sed "s#timestamp = \(.*\)#date -u --d #\1 "\+\%Y"#e"
2009
With the full format:
$ sed "s#timestamp = \(.*\)#echo timestamp; date -u --d #\1 '\+\%Y-\%m-\%d \%T'#e" <<< "timestamp = 1234567890"
timestamp
2009-02-13 23:31:30
This catches the timestamp and converts it into +%Y format.
From man sed:
e
This command allows one to pipe input from a shell command into
pattern space. If a substitution was made, the command that is found
in pattern space is executed and pattern space is replaced with its
output. A trailing newline is suppressed; results are undefined if the
command to be executed contains a nul character. This is a GNU sed
extension.
However, you see it is a bit "ugly". Depending on what you want to do, you'd better use a regular while loop to fetch the values and then use date normally. For example, if the file is like:
timestamp = 1234567890
Then you can say:
while IFS="=" read -r a b
do
echo "$b"
done < file
this will make you have $b as the timestamp and then you can perform a date ....
As commented, use a language with built-in time functions. For example:
$ echo "timestamp = 1234567890" | gawk '{$3 = strftime("%F %T", $3)} 1'
timestamp = 2009-02-13 18:31:30
$ echo "timestamp = 1234567890" | perl -MTime::Piece -pe 's/(\d+)/ localtime($1)->strftime("%F %T") /e'
timestamp = 2009-02-13 18:31:30

Converting date format with sed

I have a question how to convert date from i.e. 'Jun 14 2012 5:00PM' to '2012-06-14' using sed? So ommiting '5:00PM' as well.
I'm trying something like s/(...)(\d\d)(\d\d\d\d)/$2.$1.$3/ but can't get it right.
Thanks!
You really do not need sed to do it - you need date:
$ date --date='Jun 14 2012 5:00PM' '+%Y-%m-%d'
2012-06-14
In this command, I am asking date to convert the date 'Jun 14 2012 5:00PM' to the format composed by the four-digits year (%Y) followed by an hyphen and the two-digits month (%m) followed by another hyphen and the two-digit day (%d).
Now, some notes about your sed command:
Sed uses an specific, somewhat limited regular expression syntax. One of the differences is that the parenthesis groups should be preceded by backslashes for effectively grouping. If there is no backslash, the parenthesis is considered a literal parenthesis char. This is exactly the contrary of the regex in other languages, I know, but it is how it work. So, instead of
s/(...)(\d\d)(\d\d\d\d)/$2.$1.$3/
you should have
s/\(...\)\(\d\d\)\(\d\d\d\d\)/$2.$1.$3/
Also, there is no \d wildcard in sed. Instead, you should use [[:digit:]]:
s/\(...\)\([[:digit:]][[:digit:]]\)\([[:digit:]][[:digit:]][[:digit:]][[:digit:]]\)/$2.$1.$3/
or, better yet:
s/\(...\)\([[:digit:]]\{2\}\)\([[:digit:]]\{4\}\)/$2.$1.$3/
Last but not least, the reference to the matched groups is not marked by $, but instead by a backslash:
s/\(...\)\([[:digit:]]\{2\}\)\([[:digit:]]\{4\}\)/\2.\1.\3/
(Oh, uh, ok, you could use extended regexes with GNU sed, too, but what would be the fun?)
Anyway, this will not match your date either - I just mentioned the syntax errors, there are some semantic errors too (for example, your date is separated by spaces that are not present in the regex). Nonetheless, this is no problem because date can do it better.
This might work for you:
echo "Jun 14 2012 5:00PM" |
sed 's/$/Jan01Feb02Mar03Apr04May05Jun06Jul07Aug08Sep09Oct10Nov11Dec12/;s/\(...\) \(..\) \(....\).*\1\(..\).*/\3-\2-\4/'
2012-14-06
Add a lookup and match using back references.
You can try something like this -
echo "Jun 21 2011 5:00PM" |
awk '{"date \"+%Y/%m/%d\" -d \""$1" "$2" "$3" \"" | getline var; print var}'
Test:
$ echo "Jun 21 2011 5:00PM" |
awk '{"gdate \"+%Y/%m/%d\" -d \""$1" "$2" "$3" \"" | getline var; print var}'
2011/06/21
$

UNIX programming

Hi i want to convert UNIX date to normal date (YYYY-MM-DD)
22222,0,0,0,14387
33333,0,0,0,14170
44444,0,0,0,14244
55555,0,0,0,14190
66666,0,0,0,14528
77777,0,0,0,14200
88888,0,0,0,0
99999,0,0,0,0
here 5th column represents UNIX date
i want to convert into
22222,0,0,0,2009-05-23
and similarly remaining rows
can any body help me
A simple awk script ought to do it
awk -F, 'BEGIN{OFS=","} { print $1,$2,$3,$4,strftime("%Y-%m-%d",$5) }' myFile.txt
Cheers.
EDIT:
You're not using unix timestamps, but I checked your data, and it appears you're using days since the epoch, so here goes:
awk -F, 'BEGIN{OFS=","} { print $1,$2,$3,$4,strftime("%Y-%m-%d",$5*86400) }' myFile.txt