Date format of the machine is yyyy/mm/dd HH:mm:ss
(yyyy-MM-dd-HH.mm.ssssss) 1988-12-25-17.12.30.000000 this is my time format input, Can this time format be used to query logs from historic_log_info table? Irrespective of the date format set in the machine.
example query - SELECT * FROM TABLE(HISTORY_LOG_INFO( START_TIME => '2021-02-22-09.35.16.508075'))WHERE MESSAGE_ID IS NOT NULL
Based on the description for end-time, that formating looks correct:
start-time
A timestamp expression that indicates the starting timestamp to use when returning history log information.
If this parameter is omitted, the default of CURRENT DATE - 1 DAY is used.
end-time
A timestamp expression that indicates the ending timestamp to use when returning history log information.
If this parameter is omitted, the default of '9999-12-30-00.00.00.000000' is used.
https://www.ibm.com/support/knowledgecenter/ssw_ibm_i_74/rzajq/rzajqudfhistoryloginfo.htm
Related
Using derived column i am adding 3 columns -> 2 columns for date and 1 for timestamp. for the date columns i am passing a string as parameter. for eg: 21-11-2021 and timstamp i am using currenttimestamp fucntion.
i wrote expressions in derived columns to convert them as date and timestamp datatype and also in a format that target table needs which is dd-MM-yyyy and dd-MM-yyyy HH:mm:ss repectively
For date->
expression used: toDate($initialdate, 'dd-MM-yyyy')
data preview output: 2021-01-21 --(not in the format i want)
After pipline Debug Run, value in target DB(Azure sql database) column:
2021-01-21T00:00:00 -- in table it shows like this I dont understand why
For Timstamp conversion:
Expression used:
toTimestamp(toString(currentTimestamp(), 'dd-MM-yyyy HH:mm:ss', 'Europe/Amsterdam'), 'dd-MM-yyyy HH:mm:ss')
Data preview output: 2021-11-17 19:37:04 -- not in the format i want
After pipline Debug Run, value in target DB(Azure sql database) column:
2021-11-17T19:37:04:932 -in table it shows like this I dont understand why
question 1: I am NOT getting values in the format the target requires ???and it should be only in DATE And Datetime2 dataype respectively so no string conversions
question 2: after debug run i dont know why after insert the table values look different from Data preview???
Kinldy let me know if i have written any wrong expressions??
--apologies i am not able post pictures---
toDate() converts input date string to date with default format as yyyy-[M]M-[d]d. Accepted formats are :[ yyyy, yyyy-[M]M, yyyy-[M]M-[d]d, yyyy-[M]M-[d]dT* ].
Same goes with toTimestamp(), the default pattern is yyyy-[M]M-[d]d hh:mm:ss[.f...] when it is used.
In Azure SQL Database as well the default date and datetime2 formats are in YYYY-MM-DD and YYYY-MM-DD HH:mm:ss as shown below.
But if your column datatypes are in string (varchar) format, then you can change the output format of date and DateTime in azure data flow mappings.
When loaded to Azure SQL database, it is shown as below:
Note: This format results when datatype is varchar
If the datatype is the date in the Azure SQL database, you can convert them to the required format using date conversions as
select id, col1, date1, convert(varchar(10),date1,105) as 'dd-MM-YYYY' from test1
Azure SQL Database always follows the UTC time zone. Using “AT TIME ZONE” convert it another non-UTC time zone.
select getdate() as a, getdate() AT TIME ZONE 'UTC' AT TIME ZONE 'Central Standard Time' as b
You can also refer to sys.time_zone_info view to check current UTC offset information.
select * from sys.time_zone_info
I have a column which contians date as string but in many formats like - dd/MM/yy, dd/MMM/yyy .. etc etc. And I am using the following code to convert all strings to one specific date format (yyyy-MM-dd) in hive :
select
from_unixtime(unix_timestamp('31/02/2021','dd/MM/yyyy'),'yyyy-MM-dd')
but this gives me 2021-03-03 in HIVE.
Is there any other way to identify such invalid dates and give null.
Assume, you recognized format correctly and it is exactly 'dd/MM/yyyy' and date is invalid one '31/02/2021'.
unix_timestamp function in such case will move date to the next month and there is no way to change it's behavior. But you can check if the date double-converted from original string to timestamp and back to original format is the same. In case it is not the same, then the date is invalid one.
case
-- check double-converted date is the same as original string
when from_unixtime(unix_timestamp(date_col,'dd/MM/yyyy'),'dd/MM/yyyy') = date_col
--convert to yyyy-MM-dd if the date is valid
then from_unixtime(unix_timestamp('31/02/2021','dd/MM/yyyy'),'yyyy-MM-dd')
else null -- null if invalid date
end as date_converted
i make a query about bigint to timestamp and value is '1494257400'
i will use a presto query
but presto is not collect result about from_unixtime() function.
hive version.
select from_unixtime(1494257400) result : '2017-05-09 00:30:00'
presto version.
Blockquote
select from_unixtime(1494257400) result : '2017-05-08 08:30:00'
hive gave a collect result, but presto is not collect result. how i can solve about it?
The presto from_unixtime returns you a date at UTC when the one from Hive returns you a date on your local time zone.
According to https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF, from_unixtime:
Converts the number of seconds from unix epoch (1970-01-01 00:00:00
UTC) to a string representing the timestamp of that moment in the
current system time zone in the format of "1970-01-01 00:00:00".
The output of Hive is not that good because ISO formatted strings should show GMT data if they have any which are not GMT+00.
With Hive, you can use to_utc_timestamp({any primitive type} ts, string timezone) to convert your timestamp to the proper timezones. Take a look at the manual whose link is provided above.
My query needs to format my timestamp without time zonecolumns into a format which in Java I can obtain like this:
SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss.SSSZZZZZ")
For example:
2015-12-21T16:29:07.000-06:00 or 2016-04-18T10:10:09.000+0000
Now, I know the format wants a timezone, but it's OK to assume timezone = UTC.
What I tried:
select to_char(updated_at, 'yyyy-MM-dd''T''HH:mm:ss.SSSZZZZZZ') from teams
2016-04-07'T'05:04:47.47SZZZZZZ
select to_char(updated_at::TIMESTAMPTZ, 'yyyy-MM-dd''T''HH:mm:ss.SSSZZZZZZ') from teams
2016-04-07'T'05:04:47.47SZZZZZZ
Also tried with postgres' 'tz' and 'TZ' it's just not capable to output a numeric timezone!
select to_char(updated_at::TIMESTAMPTZ, 'yyyy-MM-dd''T''HH:mm:ss.SS TZ') from teams
2016-04-07'T'05:04:47.47 BST
I just can't get the numeric representation of the timezone, is this intentional? :(
So I have one big file (13 million rows) and date formatted as:
2009-04-08T01:57:47Z. Now I would like to split it into 2 columns now,
one with just date as dd-MM-yyyy and other with time only hh:MM.
How do I do it?
You can simply use tMap and parseDate/formatDate to do what you want. It is neither necessary nor recommended to implement your own date parsing logic with regexes.
First of all, parse the timestamp using the format yyyy-MM-dd'T'HH:mm:ss'Z'. Then you can use the parsed Date to output the formatted date and time information you want:
dd-MM-yyyy for the date
HH:mm for the time (Note: you mixed up the case in your question, MM stands for the month)
If you put that logic into a tMap:
you will get the following:
Input:
timestamp 2009-04-08T01:57:47Z
Output:
date 08-04-2009
time 01:57
NOTE
Note that when you parse the timestamp with the mentioned format string (yyyy-MM-dd'T'HH:mm:ss'Z'), the time zone information is not parsed (having 'Z' as a literal). Since many applications do not properly set the time zone information anyway but always use 'Z' instead, so this can be safely ignored in most cases.
If you need proper time zone handling and by any chance are able to use Java 7, you may use yyyy-MM-dd'T'HH:mm:ssXXX instead to parse your timestamp.
I'm guessing Talend is falling over on the T and Z part of your date time stamp but this is easily resolved.
As your date time stamp is in a regular pattern we can easily extract the date and time from it with a tExtractRegexFields component.
You'll want to use "^([0-9]{4}-[0-9]{2}-[0-9]{2})T([0-9]{2}:[0-9]{2}):[0-9]{2}Z" as your regex which will capture the date in yyyy-MM-dd format and the time as mm:HH (you'll want to replace the date time field with a date field and a time field in the schema).
Then to format your date to your required format you'll want to use a tMap and use TalendDate.formatDate("dd-MM-yyyy",TalendDate.parseDate("yyyy-MM-dd",row7.date)) to return a string in the dd-MM-yyyy format.