I have hour column with nvarchar format ... like example column is
14:45:56
20:03:25
etc
we know in ADF have toDate and ToTimeStamp to create expression in derived column for example:
I have try
toTimestamp(toString((currentUTC()+ hours(7))),'HH:mm:ss')
but the result is null when i'm trying to review. many big thanks to post the answer ...
You must specify the time format both inside toString() and toTimestamp():
toTimestamp(toString(currentUTC()+ hours(7),'HH:mm:ss'),'HH:mm:ss')
Related
Trying to convert string value(2022-07-24T07:04:27.5765591Z) into datetime/timestamp to insert into SQL table in datetime format without losing any value till milliseconds. String which I am providing is actually a datetime and my source is ADLS CSV. I tried below options in data flow.
Using Projection-> Changed the datatype format for specific column into timestamp and format type-yyyy-MM-dd'T'HH:mm:ss.SSS'Z' however getting NULL in output.
Derived column-> Tried below expressions but getting NULL value in output
toTimestamp(DataLakeModified_DateTime,'%Y-%m-%dT%H:%M:%s%z')
toTimestamp(DataLakeModified_DateTime,'yyyy-MM-ddTHH:mm:ss:fffffffK')
toTimestamp(DataLakeModified_DateTime,'yyyy-MM-dd HH:mm:ss.SSS')
I want the same value in output-
2022-07-24T07:04:27.5765591Z (coming as string) to 2022-07-24T07:04:27.5765591Z (in datetime format which will be accepted by SQL database)
I have tried to repro the issue and it is also giving me the same error, i.e., null values for yyyy-MM-dd'T'HH:mm:ss.SSS'Z' timestamp format. The issue is with the string format you are providing in source. The ADF isn’t taking the given string as timestamp and hence giving NULL in return.
But if you tried with some different format, like keeping only 3 digits before Z in last format, it will convert it into timestamp and will not return NULL.
This is what I have tried. I have kept one timestamp as per your given data and other with some modification. Refer below image.
This will return NULL for the first time and datetime for second time.
But the format you are looking for is still missing. With the existing source format, the yyyy-MM-dd'T'HH:mm:ss would work fine. This format also works fine in SQL tables. I have tried and it’s working fine.
Try to use to String instead of timestamp and use this to create your Desired timestamp
toString(DataLakeModified_DateTime, 'yyyy-MM-dd HH:mm:ss:SS')
I have a parquet file with a start_date and end_date columns
Formatted like this
01-Jan-2021
I've tried every combination conversion toDate, toString, toInterger functions but I still get Nulls returned when viewing the data (see image).
I would like to have see the result in two ways YYYYMMDD as integer column and YYYY-MM-DD as Date columns.
eg 01012021 and 01-01-2021
I'm sure the default format has caused this issue.
Thanks
First, for the Date formatter, you need to first tell ADF what each part of your string represents. Use dd-MMM-yyy for your format. Then, use a string formatter to manipulate the output as such: toString(toDate('01-Jan-2021', 'dd-MMM-yyyy'), 'yyyy-MM-dd')
For the integer representation: toInteger(toString(toDate('01-Jan-2021', 'dd-MMM-yyyy'), 'yyyyMMdd'))
Ah, you say *"I would like to have see the result in two ways YYYYMMDD as integer column and YYYY-MM-DD as Date columns. eg 01012021 and 01-01-2021"* Do you want in YYYYMMDD or dd-mm-yyy cause your example is in the later format.
Anyways, please see below expression you could use:
My source:
Use derived column:
Edit expression:
start_date_toInteger : toString(toDate(substring(start_date,1,11), 'dd-MMM-yyyy'), 'yyyymmdd')
start_date_toDate: toString(toDate(substring(start_date,1,11), 'dd-MMM-yyyy'), 'yyyy-mm-dd')
Final results:
I have table ABC in which I have column Z of datatype Date. The format of the data is YYYYMMDD. Now I am looking to convert the above format to YYYY-MON-DD format. Can someone help?
You can use to_char
TO_CHAR(Z,'YYYY-MON-DD')
Depending on what the purpose of the reformatting is, you can either explicitly cast it to a VARCHAR/CHAR and define the format, or you can change your display format to however you'd like to see all dates:
ALTER SESSION SET DATE_OUTPUT_FORMAT = 'YYYY-MON-DD';
It's important to understand that if the data is in a DATE field, then it is stored as a date, and the format of the date is dependent on your viewing preferences, not how it is stored.
Since the value of the date field is stored as a number, you have to convert it to date.
ALTER SESSION SET DATE_OUTPUT_FORMAT = 'YYYY-MON-DD';
select to_date(to_char( z ), 'YYYYMMDD');
(adding this answer to summarize and resolve the question - since the clues and answers are scattered through comments)
The question stated that column Z is of type DATE, but it really seems to be a NUMBER.
Then before parsing a number like 20201017 to a date, first you need to transform it to a STRING.
Once the original number is parsed to a date, it can be represented as a new string formatted as desired.
WITH data AS (
SELECT 20201017 AS z
)
SELECT TO_CHAR(TO_DATE(TO_CHAR(z), 'YYYYMMDD'), 'YYYY-MON-DD')
FROM data;
# 2020-Oct-17
one column in my csv file is a date that is read as a string and it follows this pattern : 2018-09-19 10:27:28.409Z. I am struggling to convert the column from string to date.
The conversion options in spotfire didn't allow me to change the column type. however, I found the solution, at the moment of importing the data set (file) you need to specify the type (date time) and magically spotfire manages the conversion.
I'm very new to sql/hive. At first, I loaded a txt file into hive using:
drop table if exists Tran_data;
create table Tran_data(tran_time string,
resort string, settled double)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' LINES TERMINATED BY '\n';
Load data local inpath 'C:\Users\me\Documents\transaction_data.txt' into table Tran_Data;
The variable tran_time in the txt file is like this:10-APR-2014 15:01. After loading this Tran_data table, I tried to convert tran_time to a "standard" format so that I can join this table to another table using tran_time as the join key. The date format desired is 'yyyymmdd'. I searched online resources, and found this: unix_timestamp(substr(tran_time,1,11),'dd-MMM-yyyy')
So essentially, I'm doing this: unix_timestamp('10-APR-2014','dd-MMM-yyyy'). However, the output is "NULL".
So my question is: how to convert the date format to a "standard" format, and then further convert it to 'yyyymmdd' format?
from_unixtime(unix_timestamp('20150101' ,'yyyyMMdd'), 'yyyy-MM-dd')
My current Hive Version: Hive 0.12.0-cdh5.1.5
I converted datetime in first column to date in second column using the below hive date functions. Hope this helps!
select inp_dt, from_unixtime(unix_timestamp(substr(inp_dt,0,11),'dd-MMM-yyyy')) as todateformat from table;
inp_dt todateformat
12-Mar-2015 07:24:55 2015-03-12 00:00:00
unix_timestamp function will convert given string date format to unix timestamp in seconds , but not like this format dd-mm-yyyy.
You need to write your own custom udf to convert a given string date to the format that you need as present Hive do not have any predefined functions. We have to_date function to convert a timestamp to date , remaining all unix_timestamp functions won't help your problem.
select from_unixtime(unix_timestamp('01032018' ,'MMddyyyy'), 'yyyyMMdd');
input format: mmddyyyy
01032018
output after query: yyyymmdd
20180103
To help someone in the future:
The following function should work as it worked in my case
to_date(from_unixtime(UNIX_TIMESTAMP('10-APR-2014','dd-MMM-yyyy'))
unix_timestamp('2014-05-01','dd-mmm-yyyy') will work, your input string should be in this format for hive yyyy-mm-dd or yyyy-mm-dd hh:mm:ss
Where as you are trying with '01-MAY-2014' hive won't understand it as a date string