I have a dataset containing a date field in the format of MM/dd/yyyy, my goal is to create a table with the same date format but with timestamp format (at the end, there should be an option to execute date functions on that, if it is int or string date function will not work.)
Things I tried:
my column name: as_of_date
1) cast(unix_timestamp(as_of_date, "MM/dd/yyyy") as timestamp)
i/p -> 01/03/2006, o/p ->2006-01-03 00:00:00
problem - I do not want extra zeros in the output. substr is not working on the date function
2) If i keep the value as string, date functions does't work.
day('01/03/2006')
input: '01/03/2006' , output:null (but expected 3)
Can you please help me a date format that already existing or help me to create a new date format for my logic.
Try with this once
use unix_timestamp function to match your input date format then use from_unixtime function to change the output format then cast to date type.
hive> select date(from_unixtime(unix_timestamp("03/06/2018", "MM/dd/yyyy"),"yyyy-MM-dd"));
+-------------+--+
| _c0 |
+-------------+--+
| 2018-03-06 |
+-------------+--+
In Impala:
hive> select from_unixtime(unix_timestamp("03/06/2018", "MM/dd/yyyy"),"yyyy-MM-dd");
+-------------+--+
| _c0 |
+-------------+--+
| 2018-03-06 |
+-------------+--+
Related
I have such events:
something;<id>abc123<timeStamp>2021-12-10T23:10:12.044Z<timeStamp>2021-12-10T23:08:55.278Z>
I want to extract the Id abc123 and the two timeStamps.
index = something
|rex field=_raw "id>(?<Id>[0-9a-z-]+)"
|rex "timeStamp>(?<timeStamp>[T0-9-\.:Z]+)"
| table _time Id timeStamp
This works with the query above. But what I struggle now is to convert the timeStamp-string to date format to get at the end the min(timeStamp) extracted in order to compute the difference between the event's _time and the min(timeStamp) by the id field. I am struggling because of the special format of the timestamp with T and Z included in it.
There's nothing special about those timestamps - they're in standard form. Use the strptime function to convert them.
index = something
|rex field=_raw "id>(?<Id>[^\<]+)"
|rex "timeStamp>(?<timeStamp>[^\<]+)"
| eval ts = strptime(timeStamp, "%Y-%m-%dT%H:%M:%S.%3N%Z")
| eval diff = ts - _time
| table _time Id timeStamp diff
Check out strftime.org, and the related strptime function used with eval
Something on the order of this (pulled the microseconds out of your rex, since Unix epoch time has no concept of subsecond intervals):
| rex field=_raw "timeStamp\>(?<timeStamp>[^\.]+)\.\d+Z"
| eval unixepoch=strptime(timeStamp,"%Y-%m-%dT%H:%M:%S")
I have a column datetime object declare as decimal (38,0) not timestamp or date and the data input is 'yyyMMdd'. How do I select data with that column convert as date format as 'yyyy-MM-dd' in spark sql (scala) within a day or two days old?
I have tried:
select count(*) from table_name where to_date('column_name','yyyy-MM-dd') = date_sub(current_date(),1));
this gives me 0 count when a data have quiet more than 500000 records
I tried:
select count(*) from table_name where from_unixtime(cast(load_dt_id as string), 'yyyy-MM-dd') = date_sub(current_date(), 1));
I got data in year 1970-01-31 which those year data are not in the table, even when I select that column where it's like '1970%', I got "OK" with bulk sign that accelerate query with Delta. The data select in order of that column started with 20140320
The format argument for to_date is the format of the input, not the desired output. Assuming you have yyyymmdd:
Seq(("20200208")).toDF("RawDate").select(col("RawDate"),to_date(col("RawDate"),"yyyyMMdd").as("formatted_date")).show()
+--------+--------------+
| RawDate|formatted_date|
+--------+--------------+
|20200208| 2020-02-08|
+--------+--------------+
Expanding this to filter by the derived date column:
val raw = Seq(("20200208"),("20200209"),("20200210")).toDF("RawDate")
raw: org.apache.spark.sql.DataFrame = [RawDate: string]
raw.select(col("RawDate"),to_date(col("RawDate"),"yyyyMMdd").as("formatted_date")).filter($"formatted_date".geq(date_add(current_date,-1))).show
+--------+--------------+
| RawDate|formatted_date|
+--------+--------------+
|20200209| 2020-02-09|
|20200210| 2020-02-10|
+--------+--------------+
I have some very strange looking 18 character alphanumeric datetimes in a SQL database, they seem to be using Hexadecimal?
I can find out what the dates are through the application which uses them, but I was looking for a way to convert them via a query. Do you know how I would convert these with TSQL?
000B3E4Bh01F2D947h - 29/05/2018 09:04:52
000B3E0Dh03A16C1Eh - 23/05/2018 10:22:26
000B3E4Eh0248C3D8h - 01/06/2018 10:38:43
000B3E4Eh0249B449h - 01/06/2018 10:39:44
I assume the date and time are separated as below, but I don't know to convert the individual parts if anyone can help with this? Thanks!!
000B3E4Eh (date) - 0249B449h (time)
(The dates are in dd/mm/yyyy format)
Your hex values are separated as you have assumed (with the h used as a delimiter) and represent integer values to add to a baseline date and time value.
Using your 000B3E54h0221CBFEh - 07/06/2018 09:56:09 value this translates as:
Date portion: 000B3E54
Integer Value: 736852
Time portion: 0221CBFE
Integer Value: 35769342
These integer values are then added as days to the date 0001/01/00 (which SQL Server can't handle, hence the +/-1 below) and milliseconds to 00:00:00 respectively, which you can see working in this script:
select convert(int, 0x000B3E54) as DateIntValue
,dateadd(day,convert(int, 0x000B3E54)-1,cast('00010101' as datetime2)) as DateValue
,convert(int, 0x0221CBFE) as TimeIntValue
,cast(dateadd(millisecond,convert(int, 0x0221CBFE),cast('19000101' as datetime2)) as time) as TimeValue
,cast(datediff(day,cast('00010101' as datetime2),'20180607')+1 as binary(4)) as DateHexValue
,cast(datediff(millisecond,cast('20180607' as date),cast('2018-06-07 09:56:09.342' as datetime2)) as binary(4)) as TimeHexValue
Which outputs:
+--------------+-----------------------------+--------------+------------------+--------------+--------------+
| DateIntValue | DateValue | TimeIntValue | TimeValue | DateHexValue | TimeHexValue |
+--------------+-----------------------------+--------------+------------------+--------------+--------------+
| 736852 | 2018-06-07 00:00:00.0000000 | 35769342 | 09:56:09.3420000 | 0x000B3E54 | 0x0221CBFE |
+--------------+-----------------------------+--------------+------------------+--------------+--------------+
Note the judicious use of datetime2 values to ensure the right amount of milliseconds are output/returned, as SQL Server datetime is only accurate to the nearest 3 milliseconds.
I have to insert dates in MySQL with the format dd-mm-yy.
I know that if I change the date format to yy-mm-dd it will work, but my client wants it in the format dd-mm-yy, and the default format of a MySQL date is 0000-00-00.
As Honeyboy said in the comments, you can use the SELECT_TO_DATE function with the format string "%d-%m-%y". You can test it like this:
CREATE DATABASE test;
USE TEST;
CREATE TABLE test (date1 date);
INSERT INTO test (date1) VALUES (STR_TO_DATE("03-04-15","%d-%m-%y"));
then with
SELECT * FROM test;
you obtain:
+------------+
| date1 |
+------------+
| 2015-04-03 |
+------------+
If you use PHP, you can then make a prepared statement such as:
$stmt = $conn->prepare("INSERT INTO test (date1) VALUES (STR_TO_DATE(?,'%d-%m-%y'));");
$stmt->bind_param("s", $client_date);
$stmt->execute();
Same principle with other languages.
The table schema is like this:
Column | Type | Modifiers
--------+--------------------------+--------------------------------------------------
time | timestamp with time zone | default now()
The time format is like this:
time
------------------------
2016-07-11 18:58:28+00
2016-07-11 18:58:37+00
2016-07-12 00:59:31+00
How to group by date with time truncated?
I would like to see the result as:
date
------------------------
2016-07-11
2016-07-11
2016-07-12
If you want compare or group by dates instead of timestamps you can cast to DATE:
SELECT time::DATE, ... FROM ... GROUP BY time::DATE;
or simpler
SELECT time::DATE, ... FROM ... GROUP BY 1;
Take a look at the current Postgresql documentation for datetime functions: https://www.postgresql.org/docs/current/static/functions-datetime.html
You can use date_trunc, extract, to_char or the simplest way is to cast to date (as I would do):
SELECT time::date; // 2016-07-11 18:58:28+00 -> 2016-07-11
Cheers!
use date_trunc:
select date_trunc('day', time)