I have a column which contians date as string but in many formats like - dd/MM/yy, dd/MMM/yyy .. etc etc. And I am using the following code to convert all strings to one specific date format (yyyy-MM-dd) in hive :
select
from_unixtime(unix_timestamp('31/02/2021','dd/MM/yyyy'),'yyyy-MM-dd')
but this gives me 2021-03-03 in HIVE.
Is there any other way to identify such invalid dates and give null.
Assume, you recognized format correctly and it is exactly 'dd/MM/yyyy' and date is invalid one '31/02/2021'.
unix_timestamp function in such case will move date to the next month and there is no way to change it's behavior. But you can check if the date double-converted from original string to timestamp and back to original format is the same. In case it is not the same, then the date is invalid one.
case
-- check double-converted date is the same as original string
when from_unixtime(unix_timestamp(date_col,'dd/MM/yyyy'),'dd/MM/yyyy') = date_col
--convert to yyyy-MM-dd if the date is valid
then from_unixtime(unix_timestamp('31/02/2021','dd/MM/yyyy'),'yyyy-MM-dd')
else null -- null if invalid date
end as date_converted
Related
I have a parquet file with a start_date and end_date columns
Formatted like this
01-Jan-2021
I've tried every combination conversion toDate, toString, toInterger functions but I still get Nulls returned when viewing the data (see image).
I would like to have see the result in two ways YYYYMMDD as integer column and YYYY-MM-DD as Date columns.
eg 01012021 and 01-01-2021
I'm sure the default format has caused this issue.
Thanks
First, for the Date formatter, you need to first tell ADF what each part of your string represents. Use dd-MMM-yyy for your format. Then, use a string formatter to manipulate the output as such: toString(toDate('01-Jan-2021', 'dd-MMM-yyyy'), 'yyyy-MM-dd')
For the integer representation: toInteger(toString(toDate('01-Jan-2021', 'dd-MMM-yyyy'), 'yyyyMMdd'))
Ah, you say *"I would like to have see the result in two ways YYYYMMDD as integer column and YYYY-MM-DD as Date columns. eg 01012021 and 01-01-2021"* Do you want in YYYYMMDD or dd-mm-yyy cause your example is in the later format.
Anyways, please see below expression you could use:
My source:
Use derived column:
Edit expression:
start_date_toInteger : toString(toDate(substring(start_date,1,11), 'dd-MMM-yyyy'), 'yyyymmdd')
start_date_toDate: toString(toDate(substring(start_date,1,11), 'dd-MMM-yyyy'), 'yyyy-mm-dd')
Final results:
I am trying to change the format of a timestamp field to a date field in a certain format with AWS Redshift.
I googled a lot and the common "best practice" that I found was to cast the timestamp to a date and then use to_char to bring it into the right format. In the end I want to use the date field in a Qlik Sense dashboard where it is input for a date picker extension that apparently requires the format DD/MM/YYYY to work.
Current date format in the DB:
9/2/2019 6:38:00 AM (which I would describe as M/D/YYYY H:MM:SS ZZ)
Desired output:
DD/MM/YYYY, resulting in the value 02/09/2019
Current status:
to_char(cast(timestamp_field as date), 'DD/MM/YYYY') --> result: 02/09/2019
However, the date picker extension in Qlik Sense still does not work and I guess that's because the output is a string and not a date. Casting the string to a date returns in an error.
cast(to_char(cast(timestamp_field as date), 'DD/MM/YYYY') as date) as date_picker_date
Connector reply error: SQL##f - SqlState: 57014, ErrorCode: 30,
ErrorMsg: [Amazon][Amazon Redshift] (30) Error occurred while trying
to execute a query: [SQLState 57014] ERROR: Error converting text to
date
I am new to Redshift and would have expected to be able to cast to date with a format string as parameter, but apparently that's not a thing. Can someone enlighten me on how to solve this?
You can try with the timestamp at the beginning of the script of the QlikSense.
It will be at the start of the script. in the Main section as shown below:
SET TimestampFormat='M/D/YYYY h:mm:ss[.fff] TT';
here SET is being used to define the time stamp in that application.
You can try changing here to your requirements.
Else 2 options 1 use a stamp which you have already used or create your own extension from the lib
Stamp use can be like this:
Datestamp_Table:
Load *,
Date(Date#(StringDate,'M/D/YY'),'DD/MMM/YY') as Date;
LOAD * INLINE [
StringDate
8/7/97
8/6/97];
Output will be like this:
Stringdate Date
8/7/97 07/Aug/97
8/6/97 06/Aug/97
I am trying to extract a month from a date in SAS, but so far all my new month variables are coming up as missing.
I have attempted to use some combinations of the month() function in SAS, but so far it just comes up as missing. The dates are formatted as follows: 01/31/2017 (MMDDYY10.)
I have tried
month = month(end_date)
Month =catx('/',put(month(end_date),z2
I would like the Month to show up as a number (01) or a 3 letter code (JAN), currently it is just missing (.)
Thanks in advance!
For month() to return a missing value the end_date variable must be numeric and missing. If end_date were a character variable the log would show invalid numeric data.
Use the monname3. format to convert a date value to a $3. character value mon
monthname = put (end_date, monname3.);
Other alternatives are:
keep the date value unchanged and change the format, or
map the date value to the first of the month value and also format that
For example:
end_date_copy = end_date;
format end_date_copy monname3.;
end_date_month = intnx('month', end_date, 0);
format end_date_month monname3.;
What you ultimately do depends on how the mon is to be used downstream in reporting or aggregating.
I have a string associated with date in ‘Teradata’ tables
Var1=09OCT2017-EMRT
I need to extract the date from the above string in ‘mm/dd/yyyy’ format
I tried the following
Cast(cast(substr(var1,1,9) as char(20)) as date format ‘mm/dd/yyyy’) as date
I am getting error as ‘invalid date supplied for var1’
I would appreciate your help
You need to apply a format matching the input string:
To_Date(Substr(var1,1,9), 'ddmonyyyy')
returns a DATE.
If you want to cast it back to a string:
To_Char(To_Date(Substr(var1,1,9), 'ddmonyyyy'), 'mm/dd/yyyy')
In a data flow from one table to another, I would like to cast a date.
The date leaves the source table as a string in this format: "2009-01-05 00:00:00:000 + 01:00".
I tried to convert this to a date using a tConvertType, but that is not allowed apparently.
My second option is to cast this string to a date using a formula in a tMap component.
At the moment I tried these formulas:
- TalendDate.formatDate("yyyy-MM-dd",row3.rafw_dz_begi);
- TalendDate.formatDate("yyyy-MM-dd HH:mm:ss",row3.rafw_dz_begi);
- return TalendDate.formatDate("yyyy-MM-dd HH:mm:ss",row3.rafw_dz_begi);
None of these worked. When inserting the result into the target-table (MySQL, InnoDB) a receive the error message that the date is not valid. The format of the target field is a MySQL Date field.
How can I cast the date to the desired format?
Talend offers you a nice way of handling date formats.
You can easily change the date format in the Schema editor tab at the tMap window.
It works for both tMap input and output flows.
I've added a picture for a better illustration.
to cast this string to a date using a formula.... error message that
the date is not valid. The format of the target field is a MySQL Date
field.
What I understand from your question is, you want to insert a date into MySQL Date field.
But the method that you are using, returns the 'String' type.
TalendDate.formatDate(String pattern, Date date); //formats a date into Date/Time string
So in that case, if your field is of 'Date' type.
TalendDate.parseDate("yyyy-MM-dd", TalendDate.formatDate("yyyy-MM-dd",row3.rafw_dz_begi));