Using derived column i am adding 3 columns -> 2 columns for date and 1 for timestamp. for the date columns i am passing a string as parameter. for eg: 21-11-2021 and timstamp i am using currenttimestamp fucntion.
i wrote expressions in derived columns to convert them as date and timestamp datatype and also in a format that target table needs which is dd-MM-yyyy and dd-MM-yyyy HH:mm:ss repectively
For date->
expression used: toDate($initialdate, 'dd-MM-yyyy')
data preview output: 2021-01-21 --(not in the format i want)
After pipline Debug Run, value in target DB(Azure sql database) column:
2021-01-21T00:00:00 -- in table it shows like this I dont understand why
For Timstamp conversion:
Expression used:
toTimestamp(toString(currentTimestamp(), 'dd-MM-yyyy HH:mm:ss', 'Europe/Amsterdam'), 'dd-MM-yyyy HH:mm:ss')
Data preview output: 2021-11-17 19:37:04 -- not in the format i want
After pipline Debug Run, value in target DB(Azure sql database) column:
2021-11-17T19:37:04:932 -in table it shows like this I dont understand why
question 1: I am NOT getting values in the format the target requires ???and it should be only in DATE And Datetime2 dataype respectively so no string conversions
question 2: after debug run i dont know why after insert the table values look different from Data preview???
Kinldy let me know if i have written any wrong expressions??
--apologies i am not able post pictures---
toDate() converts input date string to date with default format as yyyy-[M]M-[d]d. Accepted formats are :[ yyyy, yyyy-[M]M, yyyy-[M]M-[d]d, yyyy-[M]M-[d]dT* ].
Same goes with toTimestamp(), the default pattern is yyyy-[M]M-[d]d hh:mm:ss[.f...] when it is used.
In Azure SQL Database as well the default date and datetime2 formats are in YYYY-MM-DD and YYYY-MM-DD HH:mm:ss as shown below.
But if your column datatypes are in string (varchar) format, then you can change the output format of date and DateTime in azure data flow mappings.
When loaded to Azure SQL database, it is shown as below:
Note: This format results when datatype is varchar
If the datatype is the date in the Azure SQL database, you can convert them to the required format using date conversions as
select id, col1, date1, convert(varchar(10),date1,105) as 'dd-MM-YYYY' from test1
Azure SQL Database always follows the UTC time zone. Using “AT TIME ZONE” convert it another non-UTC time zone.
select getdate() as a, getdate() AT TIME ZONE 'UTC' AT TIME ZONE 'Central Standard Time' as b
You can also refer to sys.time_zone_info view to check current UTC offset information.
select * from sys.time_zone_info
Related
I have a column which contians date as string but in many formats like - dd/MM/yy, dd/MMM/yyy .. etc etc. And I am using the following code to convert all strings to one specific date format (yyyy-MM-dd) in hive :
select
from_unixtime(unix_timestamp('31/02/2021','dd/MM/yyyy'),'yyyy-MM-dd')
but this gives me 2021-03-03 in HIVE.
Is there any other way to identify such invalid dates and give null.
Assume, you recognized format correctly and it is exactly 'dd/MM/yyyy' and date is invalid one '31/02/2021'.
unix_timestamp function in such case will move date to the next month and there is no way to change it's behavior. But you can check if the date double-converted from original string to timestamp and back to original format is the same. In case it is not the same, then the date is invalid one.
case
-- check double-converted date is the same as original string
when from_unixtime(unix_timestamp(date_col,'dd/MM/yyyy'),'dd/MM/yyyy') = date_col
--convert to yyyy-MM-dd if the date is valid
then from_unixtime(unix_timestamp('31/02/2021','dd/MM/yyyy'),'yyyy-MM-dd')
else null -- null if invalid date
end as date_converted
I am trying to change the format of a timestamp field to a date field in a certain format with AWS Redshift.
I googled a lot and the common "best practice" that I found was to cast the timestamp to a date and then use to_char to bring it into the right format. In the end I want to use the date field in a Qlik Sense dashboard where it is input for a date picker extension that apparently requires the format DD/MM/YYYY to work.
Current date format in the DB:
9/2/2019 6:38:00 AM (which I would describe as M/D/YYYY H:MM:SS ZZ)
Desired output:
DD/MM/YYYY, resulting in the value 02/09/2019
Current status:
to_char(cast(timestamp_field as date), 'DD/MM/YYYY') --> result: 02/09/2019
However, the date picker extension in Qlik Sense still does not work and I guess that's because the output is a string and not a date. Casting the string to a date returns in an error.
cast(to_char(cast(timestamp_field as date), 'DD/MM/YYYY') as date) as date_picker_date
Connector reply error: SQL##f - SqlState: 57014, ErrorCode: 30,
ErrorMsg: [Amazon][Amazon Redshift] (30) Error occurred while trying
to execute a query: [SQLState 57014] ERROR: Error converting text to
date
I am new to Redshift and would have expected to be able to cast to date with a format string as parameter, but apparently that's not a thing. Can someone enlighten me on how to solve this?
You can try with the timestamp at the beginning of the script of the QlikSense.
It will be at the start of the script. in the Main section as shown below:
SET TimestampFormat='M/D/YYYY h:mm:ss[.fff] TT';
here SET is being used to define the time stamp in that application.
You can try changing here to your requirements.
Else 2 options 1 use a stamp which you have already used or create your own extension from the lib
Stamp use can be like this:
Datestamp_Table:
Load *,
Date(Date#(StringDate,'M/D/YY'),'DD/MMM/YY') as Date;
LOAD * INLINE [
StringDate
8/7/97
8/6/97];
Output will be like this:
Stringdate Date
8/7/97 07/Aug/97
8/6/97 06/Aug/97
There are two date columns in Teradata table with following definition.
Date1 DATE FORMAT 'DD-MM-YYYY'
Date2 DATE FORMAT 'DD-MM-YYYY'
As you can see, the format of the date is same for both the columns.
The values I see in Teradata SQL Assistant are different for Date1 & Date2 ...
For Date1 it is shows as MM/DD/YYYY
For Date2 it is shown as DD/MM/YYYY
Also, due to this, EXTRACT(MONTH FROM Date...) is not working same for both the columns.
Please note: Date2 column is updated by me with values like '04-28-2016' i.e. 'MM-DD-YYYY'. And this is to acheive the Date1 format as shown in SQL Assistant.
Can you please let me know where I made a mistake ?
Any advise would be helpful.
Thanks,
The FORMAT is used for casting from/to a string, but SQL Assistant uses the format specified under Tools -> Options -> Data Format -> Display dates in this format. And a DATE is stored in an internal format, thus EXTRACT is independent of the it.
Btw, the only recommended way to write a date is Standard SQL's DATE '2016-04-28'
My query needs to format my timestamp without time zonecolumns into a format which in Java I can obtain like this:
SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss.SSSZZZZZ")
For example:
2015-12-21T16:29:07.000-06:00 or 2016-04-18T10:10:09.000+0000
Now, I know the format wants a timezone, but it's OK to assume timezone = UTC.
What I tried:
select to_char(updated_at, 'yyyy-MM-dd''T''HH:mm:ss.SSSZZZZZZ') from teams
2016-04-07'T'05:04:47.47SZZZZZZ
select to_char(updated_at::TIMESTAMPTZ, 'yyyy-MM-dd''T''HH:mm:ss.SSSZZZZZZ') from teams
2016-04-07'T'05:04:47.47SZZZZZZ
Also tried with postgres' 'tz' and 'TZ' it's just not capable to output a numeric timezone!
select to_char(updated_at::TIMESTAMPTZ, 'yyyy-MM-dd''T''HH:mm:ss.SS TZ') from teams
2016-04-07'T'05:04:47.47 BST
I just can't get the numeric representation of the timezone, is this intentional? :(
I have create one field in sql server database as nvarchar datatype and store some date like 'd/MM/yyyy' and 'dd/MM/yyyy' format previously. Now i want to get all data in 'dd/MM/yyyy' format using query it is possible?
You can cast the field to datetime in the query:
select cast(YourField as datetime)
from YourTable
where isdate(YourField) = 1
The where isdate(YourField) = 1 part is necessary to filter out rows where the value is no valid date (it's a nvarchar field, so there could be things like abc in some rows!)
But you should really change the field to datetime in the long term, as already suggested by Christopher in his comment.
Casting like described above is always error-prone because of the many different data formats in different countries.
For example, I live in Germany where the official date format is dd.mm.yyyy.
So today (December 9th) is 9.12.2011, and running select cast('9.12.2011' as datetime) on my machine returns the correct datetime value.
Another common format is mm/dd/yyyy, so December 9th would be 12/9/2011.
Now imagine I have a nvarchar field with a date in this format on my German machine:
select cast('12/9/2011' as datetime) will return September 12th (instead of December 9th)!
Issues like this can easily be avoided by using the proper type for the column, in this case datetime.