I am trying to covert a timestamp value in my data to 'yyyy-mm-dd HH:mm:ss' using UNIXTIMEFORMAT formula but it is giving a wrong result as you can see in the screenshot here.
Unixtime 1592574691 translates to ==> 2020-06-19 17:51:31 but dataprep is converting to 1970-22-19 10:22:54
Your click_time is in seconds but UNIXTIMEFORMAT needs miliseconds. Try to multiply your column by 1000
UNIXTIMEFORMAT($col * 1000, 'yyyy-mm-dd HH:mm:ss')
Related
I have a Spark data frame with the column timestamp. I need to create event_hour in unix_timestamp format out of this column. The current issue is that the timestamp is in unix_timestamp format with a granularity of milliseconds while I need the granularity of hours.
Current values for timestamp:
1653192037
1653192026
1653192025
1653192024
1653192023
1653192022
Expected values:
1653192000
1653195600
1653199200
1653202800
How can I achieve that using Spark functions?
I've already tried to convert it to timestamp and then format it but I got null as the result:
inputDf
.withColumn("event_hour", unix_timestamp(date_format($"timestamp".cast(TimestampType), "MM-dd-yyyy HH")))
A (not very explicit but) efficient way would be to use modulus operation with 3600 (as 3600 seconds = 1 hour):
timestamp_hour = timestamp_second - (timestamp_second % 3600)
This assumes you are manipulating data as numeric.
You can use DateUtils API,
import org.apache.commons.lang3.time.DateUtils;
Long epochTimestamp_hour = DateUtils.truncate(Timestamp_column, Calendar.HOUR)).getTime();
create new column of type timestamp
use that column to truncate timestamp to epochTimestamp_hour
I am using Hive 1.2.1000.
I am actually dealing with conversion to unix timestamp. I'm trying to convert a date with format:
dd/mm/yyyy hh:mm:ss
to unix_timestamp, hence:
unix_timestamp(date,"dd-mm-yyyy hh:mm:ss")
have been used.
More precisely, I have run the following code:
select '09/06/2012 04:02:32', regexp_replace('09/06/2012 04:02:32',"/","-")
which seems to work, in fact the result is: 1326081752
I have also noticed that I have N distinct dates and the distinct on unix_timestamp of these dates is M with M < N.
Doing some manipulations on the data i have seen that there are different dates with the same unix_timestamp.
Hence i've tried to go deeper and i have found a lot of dates, for example
09/06/2012 04:02:32
and
09/12/2012 04:02:32
Now, if I try to run the following code:
select
'09/06/2012 04:02:32',
regexp_replace('09/06/2012 04:02:32',"/","-"),
unix_timestamp(regexp_replace('09/06/2012 04:02:32',"/","-"),"dd-mm-yyyy hh:mm:ss"),
unix_timestamp('09-06-2012 04:02:32',"dd-mm-yyyy hh:mm:ss")
UNION ALL
select '09/12/2012 04:02:32',
regexp_replace('09/12/2012 04:02:32',"/","-"),
unix_timestamp(regexp_replace('09/12/2012 04:02:32',"/","-"),"dd-mm-yyyy hh:mm:ss"),
unix_timestamp('09-12-2012 04:02:32',"dd-mm-yyyy hh:mm:ss")
That's the output:
09/06/2012 04:02:32 09-06-2012 04:02:32 1326081752 1326081752 1326081752
09/12/2012 04:02:32 09-12-2012 04:02:32 1326081752 1326081752 1326081752
Which is clearly the same.
This result can be extended for all the dates where everything is identical except one between dd and mm.
Could you explain me why?
Thanks in advance,
Manuel
Ps. I have also tried with dates with other format for example:
select '2012-06-09 04:02:32', unix_timestamp(regexp_replace('2012-06-09 04:02:32',"/","-"),"yyyy-mm-dd hh:mm:ss")
UNION ALL
select '2012-12-09 04:02:32', unix_timestamp(regexp_replace('2012-12-09 04:02:32',"/","-"),"yyyy-mm-dd hh:mm:ss")
But result is the same.
The problem was that mm stands for minutes.
dd-MM-yyyy hh:mm:ss
instead of
dd-mm-yyyy hh:mm:ss
was the solution to the problem.
I am trying to convert a long utc value into "yyyy-MM-dd HH:mm:ss" formatted pattern. I am expecting my data to be converted on 24 hours range scale and in GMT. My code passes all the test cases, I push the data into database using the jar that is newly built with this code -
dbRecord("order_dt_utc") = if (orderTs.isDefined) Some(new DateTime(orderTs.get, DateTimeZone.UTC).toString("yyyy-MM-dd HH:mm:ss")) else None
and now, when I query my database, I find that the data is still converting on 12 hours range. The query -
SELECT order_id, order_dt, order_dt_utc, order_ts_utc, from_unixtime(order_ts_utc/1000) FROM order_items where order_dt >= '2018-08-01' AND order_dt <= '2018-08-02' ORDER BY order_dt_utc LIMIT 1000;
And you can see the the values are not matching in the columns from_unixtime(order_ts_utc/1000) and order_dt_utc -
I am not able to figure the reason for this behaviour.
To convert Time Zone use the function first:
CONVERT_TZ (dateobj, oldtz, newtz)
After that use the date_format function:
date_format(from_unixtime(order_ts_utc), '%Y-%m-%d %H:%i:%s');
to format your time to 00-23 format.
I am converting from SQL server to PostgreSQL. I have a function that take a float and converts it to a time. For example if the float number is 12.5 it converts to 12:30. Is there anyway to do this in postgres?
Thanks
Something like this?
select to_char(to_timestamp((12.5) * 60), 'MI:SS');
DEMO
If seconds is the only thing you need, you can write something like this:
select date_part('second', to_timestamp((12.37) * 60))::int;
or
select to_char(to_timestamp((12.37) * 60), 'SS');
In my personal case, this worked:
to_char(to_timestamp(table.time_stamp/1000), 'DD/MM/YYYY HH24:MI:SS')
You can even change the date format (for instance, 'mm/dd/yyyy' or 'dd-mm-yyyy) and the space between the date and the time (another example: 'dd/mm/yyyy | HH24:MI:SS')
SELECT 6.75*'1 HOUR'::INTERVAL
I've processed my data by datenum(DATE(xy), 'dd-mmm-yyyy HH:MM:SS') and when I tried to convert it back to the date formate by datestr(DATE(xy), 'dd-mmm-yyyy HH:MM:SS') I've got different date. The original date was 8.4.2013 4:12:31 and the converted is 04-Oct-2013 04:12:31
Any idea, where is the error?
Thanks Tomas
What is the format of the input-date? Is it 08-04-2013 04:12:31? If it is given as 8.4.2013 4:12:31 your second input to datenum is incorrect. You specify the format: 'dd-mmm-yyyy HH:MM:SS'), while your input seems to be 8.4.2013 4:12:31.
Try:
datenum(DATE(xy), 'dd.mm.yyyy HH:MM:SS')