I'm very new to sql/hive. At first, I loaded a txt file into hive using:
drop table if exists Tran_data;
create table Tran_data(tran_time string,
resort string, settled double)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' LINES TERMINATED BY '\n';
Load data local inpath 'C:\Users\me\Documents\transaction_data.txt' into table Tran_Data;
The variable tran_time in the txt file is like this:10-APR-2014 15:01. After loading this Tran_data table, I tried to convert tran_time to a "standard" format so that I can join this table to another table using tran_time as the join key. The date format desired is 'yyyymmdd'. I searched online resources, and found this: unix_timestamp(substr(tran_time,1,11),'dd-MMM-yyyy')
So essentially, I'm doing this: unix_timestamp('10-APR-2014','dd-MMM-yyyy'). However, the output is "NULL".
So my question is: how to convert the date format to a "standard" format, and then further convert it to 'yyyymmdd' format?
from_unixtime(unix_timestamp('20150101' ,'yyyyMMdd'), 'yyyy-MM-dd')
My current Hive Version: Hive 0.12.0-cdh5.1.5
I converted datetime in first column to date in second column using the below hive date functions. Hope this helps!
select inp_dt, from_unixtime(unix_timestamp(substr(inp_dt,0,11),'dd-MMM-yyyy')) as todateformat from table;
inp_dt todateformat
12-Mar-2015 07:24:55 2015-03-12 00:00:00
unix_timestamp function will convert given string date format to unix timestamp in seconds , but not like this format dd-mm-yyyy.
You need to write your own custom udf to convert a given string date to the format that you need as present Hive do not have any predefined functions. We have to_date function to convert a timestamp to date , remaining all unix_timestamp functions won't help your problem.
select from_unixtime(unix_timestamp('01032018' ,'MMddyyyy'), 'yyyyMMdd');
input format: mmddyyyy
01032018
output after query: yyyymmdd
20180103
To help someone in the future:
The following function should work as it worked in my case
to_date(from_unixtime(UNIX_TIMESTAMP('10-APR-2014','dd-MMM-yyyy'))
unix_timestamp('2014-05-01','dd-mmm-yyyy') will work, your input string should be in this format for hive yyyy-mm-dd or yyyy-mm-dd hh:mm:ss
Where as you are trying with '01-MAY-2014' hive won't understand it as a date string
Related
I have a parquet file with a start_date and end_date columns
Formatted like this
01-Jan-2021
I've tried every combination conversion toDate, toString, toInterger functions but I still get Nulls returned when viewing the data (see image).
I would like to have see the result in two ways YYYYMMDD as integer column and YYYY-MM-DD as Date columns.
eg 01012021 and 01-01-2021
I'm sure the default format has caused this issue.
Thanks
First, for the Date formatter, you need to first tell ADF what each part of your string represents. Use dd-MMM-yyy for your format. Then, use a string formatter to manipulate the output as such: toString(toDate('01-Jan-2021', 'dd-MMM-yyyy'), 'yyyy-MM-dd')
For the integer representation: toInteger(toString(toDate('01-Jan-2021', 'dd-MMM-yyyy'), 'yyyyMMdd'))
Ah, you say *"I would like to have see the result in two ways YYYYMMDD as integer column and YYYY-MM-DD as Date columns. eg 01012021 and 01-01-2021"* Do you want in YYYYMMDD or dd-mm-yyy cause your example is in the later format.
Anyways, please see below expression you could use:
My source:
Use derived column:
Edit expression:
start_date_toInteger : toString(toDate(substring(start_date,1,11), 'dd-MMM-yyyy'), 'yyyymmdd')
start_date_toDate: toString(toDate(substring(start_date,1,11), 'dd-MMM-yyyy'), 'yyyy-mm-dd')
Final results:
In presto SQL, the date is saved as a string like '2020-06-10'. I want to convert into a date format (yyyy-mm-dd)
This is how I did it:
select date_format(date_parse('2020-06-10', '%Y-%m-%d'),'%Y-%m-%d')
First I convert string to a timestamp format, then convert the new timestamp to date_format.
My question is that is there a function such that I convert only once?
For example
date_parse(string, format, expect_out_put_time_format)
You can use date function, which is a shortcut for CAST(x AS date).
presto> SELECT date('2020-06-10');
_col0
------------
2020-06-10
My job Talend is about mapping between a csv file and a postregresql table.
I need to insert a column date which can be with normal format yyyyMMdd or(0/99999999) in the csv file. So if the date is equal to 0 or 99999999 it's will be mapping as a null variable in the database, else the data must be loaded as a date type timestamp yyyy-mm-dd HH:mm:ss.
In the csv file I declared the date as an int, so I must parse int to a datetime in the tmap and loaded the 0/99999999 as a null variable.
Any help please.
if I understand the problem correctly, its solution is as follows:
// correspondent expression to convert string with special "0/99999999" values is:
(row1.dateAsString.equals("0")||row1.dateAsString.equals("99999999"))?null:routines.TalendDate.parseDate("yyyyMMdd", row1.dateAsString)
I have these character fields actv_date and actv_time in a hive table
ap_actv_date
171205 (YYMMDD)
ap_actv_time
1954359 (HHMMSSF)
I want the output to be a time stamp field of format YYYY-MM-DD HH:MM:SS.f
Use this.
from_unixtime(unix_timestamp(concat(ap_actv_date,ap_actv_time), 'YYMMDDHHMMSSF'))
I am reading a csv file with date fields of formatted mm/dd/yyyy. I expected the same kind of format from a Postgres table after the import, but I see yyyy-mm-dd hh:mm:ss.
The date fields in my table are defined as timestamp without time zone data type.
How do I maintain the same format of data? I am using PostgreSQL 9.3.
Postgresql only stores the value, it doesn't store formatting (which would waste space).
You can use the to_char function in your query if you like to get the output formatted in a special way. Details are in the manual.