I have such events:
something;<id>abc123<timeStamp>2021-12-10T23:10:12.044Z<timeStamp>2021-12-10T23:08:55.278Z>
I want to extract the Id abc123 and the two timeStamps.
index = something
|rex field=_raw "id>(?<Id>[0-9a-z-]+)"
|rex "timeStamp>(?<timeStamp>[T0-9-\.:Z]+)"
| table _time Id timeStamp
This works with the query above. But what I struggle now is to convert the timeStamp-string to date format to get at the end the min(timeStamp) extracted in order to compute the difference between the event's _time and the min(timeStamp) by the id field. I am struggling because of the special format of the timestamp with T and Z included in it.
There's nothing special about those timestamps - they're in standard form. Use the strptime function to convert them.
index = something
|rex field=_raw "id>(?<Id>[^\<]+)"
|rex "timeStamp>(?<timeStamp>[^\<]+)"
| eval ts = strptime(timeStamp, "%Y-%m-%dT%H:%M:%S.%3N%Z")
| eval diff = ts - _time
| table _time Id timeStamp diff
Check out strftime.org, and the related strptime function used with eval
Something on the order of this (pulled the microseconds out of your rex, since Unix epoch time has no concept of subsecond intervals):
| rex field=_raw "timeStamp\>(?<timeStamp>[^\.]+)\.\d+Z"
| eval unixepoch=strptime(timeStamp,"%Y-%m-%dT%H:%M:%S")
Related
I have a dataset containing a date field in the format of MM/dd/yyyy, my goal is to create a table with the same date format but with timestamp format (at the end, there should be an option to execute date functions on that, if it is int or string date function will not work.)
Things I tried:
my column name: as_of_date
1) cast(unix_timestamp(as_of_date, "MM/dd/yyyy") as timestamp)
i/p -> 01/03/2006, o/p ->2006-01-03 00:00:00
problem - I do not want extra zeros in the output. substr is not working on the date function
2) If i keep the value as string, date functions does't work.
day('01/03/2006')
input: '01/03/2006' , output:null (but expected 3)
Can you please help me a date format that already existing or help me to create a new date format for my logic.
Try with this once
use unix_timestamp function to match your input date format then use from_unixtime function to change the output format then cast to date type.
hive> select date(from_unixtime(unix_timestamp("03/06/2018", "MM/dd/yyyy"),"yyyy-MM-dd"));
+-------------+--+
| _c0 |
+-------------+--+
| 2018-03-06 |
+-------------+--+
In Impala:
hive> select from_unixtime(unix_timestamp("03/06/2018", "MM/dd/yyyy"),"yyyy-MM-dd");
+-------------+--+
| _c0 |
+-------------+--+
| 2018-03-06 |
+-------------+--+
I have some very strange looking 18 character alphanumeric datetimes in a SQL database, they seem to be using Hexadecimal?
I can find out what the dates are through the application which uses them, but I was looking for a way to convert them via a query. Do you know how I would convert these with TSQL?
000B3E4Bh01F2D947h - 29/05/2018 09:04:52
000B3E0Dh03A16C1Eh - 23/05/2018 10:22:26
000B3E4Eh0248C3D8h - 01/06/2018 10:38:43
000B3E4Eh0249B449h - 01/06/2018 10:39:44
I assume the date and time are separated as below, but I don't know to convert the individual parts if anyone can help with this? Thanks!!
000B3E4Eh (date) - 0249B449h (time)
(The dates are in dd/mm/yyyy format)
Your hex values are separated as you have assumed (with the h used as a delimiter) and represent integer values to add to a baseline date and time value.
Using your 000B3E54h0221CBFEh - 07/06/2018 09:56:09 value this translates as:
Date portion: 000B3E54
Integer Value: 736852
Time portion: 0221CBFE
Integer Value: 35769342
These integer values are then added as days to the date 0001/01/00 (which SQL Server can't handle, hence the +/-1 below) and milliseconds to 00:00:00 respectively, which you can see working in this script:
select convert(int, 0x000B3E54) as DateIntValue
,dateadd(day,convert(int, 0x000B3E54)-1,cast('00010101' as datetime2)) as DateValue
,convert(int, 0x0221CBFE) as TimeIntValue
,cast(dateadd(millisecond,convert(int, 0x0221CBFE),cast('19000101' as datetime2)) as time) as TimeValue
,cast(datediff(day,cast('00010101' as datetime2),'20180607')+1 as binary(4)) as DateHexValue
,cast(datediff(millisecond,cast('20180607' as date),cast('2018-06-07 09:56:09.342' as datetime2)) as binary(4)) as TimeHexValue
Which outputs:
+--------------+-----------------------------+--------------+------------------+--------------+--------------+
| DateIntValue | DateValue | TimeIntValue | TimeValue | DateHexValue | TimeHexValue |
+--------------+-----------------------------+--------------+------------------+--------------+--------------+
| 736852 | 2018-06-07 00:00:00.0000000 | 35769342 | 09:56:09.3420000 | 0x000B3E54 | 0x0221CBFE |
+--------------+-----------------------------+--------------+------------------+--------------+--------------+
Note the judicious use of datetime2 values to ensure the right amount of milliseconds are output/returned, as SQL Server datetime is only accurate to the nearest 3 milliseconds.
I'm trying to convert value for DIM_DT_ID to MMddYY. I'm successful in doinf that. However, query fails because ultimately I'm comparing a character value to date here. Is there a way by which I can get value for DIM_DT_ID in MMddyy format and its data type still remains DATE ?
Here DIM_DT_ID
SELECT DIM_DT_ID
DIM_DT_ID >= FORMATDATE('MMddyy',ADDDAY(TO_date('yyyy-MM-dd','2016-12-21'), -25)); from abc;
Regards,
Ajay
In Denodo, to convert a string to a date field, use "to_date()" (which returns a date).
Then, don't convert back to a string, leave that field as a date (so don't use "Formatdate()", which returns a string).
So:
SELECT *
FROM MyTable
WHERE now() >= to_date('yyyy-MM-dd',myStringFieldThatLooksLikeADate)
In my example, "now()" is a date, and so is the output of the to_date function... so you can do a comparison.
If you try to convert the date back to a string using formatdate, it won't work:
#This doesn't work:
SELECT *
FROM MyTable
WHERE now() >= formatdate('MMddyy',to_date('yyyy-MM-dd',myStringFieldThatLooksLikeADate))
It doesn't work because we are comparing a date ("now()") to a string.
My variable DateBecame looks like this:
19550101
It is of long type and the format is %10.0g. So YMD format.
I also have another variable DateLeft:
19961001
This is also of long type and format %10.0g.
How can I calculate the duration between the two dates?
The following works for me:
clear
input long(date1 date2)
19550101 19961001
end
generate diff = daily(string(date1, "%10.0g"), "YMD") - ///
daily(string(date2, "%10.0g"), "YMD")
list
+---------------------------------+
| date1 date2 diff |
|---------------------------------|
1. | 19550101 19961001 -15249 |
+---------------------------------+
Note that variables of long type are generally not ideal for this kind of calculations.
Type help datetime from Stata's command prompt for more information on working with dates.
Say I have a column that contains unix timestamps - an int representing the number of seconds since the epoch. They look like this: 1347085827. How do I format this as a human-readable date string in my SELECT query?
Postgresql has a handy built-in function for this: to_timestamp(). Just wrap that function around the column you want:
Select a, b, to_timestamp(date_int) FROM t_tablename