MongoDB - datetime type for mongosqld - mongodb

I have a collection in MongoDB that have a Date field :
date: 2021-02-17T18:40:01.000+00:00
I want to expose this collection to a BI tool thanks to MongoDB BI Connector, mongosqld. I used mongodrdl to create the data model.
mongodrdl converts MongoDB's Date type to MySQL's timestamp type. When i read the date column in a BI application, the time is only zeros :
17/02/2021 00:00:00
This is catastrophic because i need the time. I tried to edit the drdl generated by mongodrdl and put the SqlType to datetime. But when i restart mongosqld i get the following error :
unable to create column "date" from drdl: unsupported SQL type: "datetime" on column "date"
How can i preserve the time for this date field to be properly exposed to BI tools ?


date: 2021-02-17T18:40:01.000+00:00
This is a stringified (iso8601) representation of a timestamp. If this is how your timestamps are stored, they are of the wrong type (string, not timestamp) and hence produce zeroed SQL timestamps.
To fix, store the timestamps as timestamps (BSON date type).

Related

Is there a way we can get date alone from a datetime field stored as string in MongoDB. We are using Spring MongoDB driver

Let's say I have an date attribute stored in the format dd/MM/YYYY HH:mm:ss i.e. 22/12/2021 10:15:23. Now I just want 22/12/2021 as a date value so that I can do date comparisons. I want this operation to happen at database itself so that I can do date comparisons as part of the query/aggregation itself. Please note I am using Spring MongoDB driver.

Getting date alone from a datetime string is not possible in MongoDB since it always returns an ISO date with time. As a workaround we can convert the time HH:mm:ss to 00:00:00 either by applying String operations on the string attribute or by using Arithmetic operations on the date. We will have to add a temporary field in the query which will hold the converted value using add field operations. And then we can compare them with a given date.

Epoch date converion fails with apache-nifi

I have QueryDatabaseTableRecord to get data from Oracle table.
In my oracle table, I have following data
id,name,bday
1,sachith,17-SEP-1990
2,nalaka,16-MAR-2020
When I run and get data, its changed into bigint.
1,sachith,653523824000
2,nalaka,1584311083000
In Record Writer : CSV
: Date format : yyyy-mm-dd
But yet its not working. Do I have to use intermediate UpdateRecord processor and update date fields as described here?
Edit :
After doing some research, I could add UpdateRecord processor with
/my_date_colum : ${filed.value:format("yyyy-MM-dd HH:mm:ss.SSS")}
But this fails with negative epoch values.
Error : Could not implicitly convert input to Date -104697000000
How can I handle this?

This statement should work:
${field.value:toDate():format('yyyy-MMM-dd')}

mongoDB is adding 4 hours and 5 hours to date field while inserting date format(YYYY-MM-DD) through datastage ? is this the normal behavior?

case#1 -- when inserted the date format (2015-08-01 ) , in mongodb it shows as 2015-08-01T04:00:00.000Z, where i expected it to be 2015-08-01T00:00:00.000Z
case#2 -- when inserted the date format (2016-01-01 ) , in mongodb it shows as 2016-01-01T05:00:00.000Z
where i expected it to be 2016-01-01T00:00:00.000Z
source data type used data stage : Date
target data type specified in mongoDB : Date
Question here is why 04:00:00 timestamp is coming even though source has only YYYY-MM-DD(with out time stamp) ?
can some one help to clarify this ? is this usual behavior of mongodb ? is there any thing i can do to have only timestamp as T00.00.00.000Z at target.
requirement is to not convert the date into string format ..

presto from_unixtime function is right?

i make a query about bigint to timestamp and value is '1494257400'
i will use a presto query
but presto is not collect result about from_unixtime() function.
hive version.
select from_unixtime(1494257400) result : '2017-05-09 00:30:00'
presto version.
Blockquote
select from_unixtime(1494257400) result : '2017-05-08 08:30:00'
hive gave a collect result, but presto is not collect result. how i can solve about it?

The presto from_unixtime returns you a date at UTC when the one from Hive returns you a date on your local time zone.
According to https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF, from_unixtime:
Converts the number of seconds from unix epoch (1970-01-01 00:00:00
UTC) to a string representing the timestamp of that moment in the
current system time zone in the format of "1970-01-01 00:00:00".
The output of Hive is not that good because ISO formatted strings should show GMT data if they have any which are not GMT+00.
With Hive, you can use to_utc_timestamp({any primitive type} ts, string timezone) to convert your timestamp to the proper timezones. Take a look at the manual whose link is provided above.

Using AS400 date in SSRS Report

Running a SELECT against the AS400 using the IBMDA400 OleDb provider appears to return dates as string values, SSRS just laughs at you when you try and apply a date format to the field. I've tried a simple CAST in the SELECT to no avail.
How can I get an actual DBTYPE_DBDATE struct back from the iSeries OleDb provider?
I should mention that the dates in question are all being returned by a UDF with a type of DATE. IBM appears to map DATE type into a DBTYPE_STR OleDb type.

The field(s) in the table(s) are probably not defined as a date type. You will need to convert them using the DATE function as part of the query.
You can use the DSPFFD command, the Navigator, or query the SYSIBM.SQLCOLUMNS table to view the field definitions.
UPDATE
After further testing with the IBMDA400 provider I found the Convert Date Time To Char property hidden away in the OLE DB Technical Reference installed as part of the Programmer's Toolkit with Access. The default value is TRUE. Set Convert Date Time To Char=FALSE in the connection string or properties to disable this 'feature'.
Here's a quick VBA test:
Set cn = CreateObject("ADODB.Connection")
cn.Open "Provider=IBMDA400;Data Source=...;User ID=...;Password=...;Convert Date Time To Char=FALSE"
Set rs = cn.Execute("SELECT DATE(NOW()) FROM SYSIBM.SYSDUMMY1")
MsgBox "Returned ADO type: " & rs.Fields(0).Type
See MSDN: DataTypeEnum for the possible ADO data types. adDBDate is 133.
IBM i Access for Windows OLE DB Technical Reference
IBM i OLE DB provider functions > Special Properties
Convert Date Time To Char
Specifies conversion of DB2 for IBM i Date, Time, and Timestamp data types to corresponding PC data types and vice versa.
Settings and Return Values
Sets or returns one of the following string values. The default value is "TRUE".
"TRUE"
DB2 for IBM i Date, Time, and Timestamp data types are treated as character strings. When reading IBM i data, these values are converted to character strings. When writing data to the system, character strings are expected as input for these values. The supported character string format for the Date data type is ISO format: yyyy-mm-dd. The supported character string format for the Time data type is the earlier version of the ISO format: hh.mm.ss. The supported character string format for the Timestamp data type is: yyyy-mm-dd-hh.mm.ss.nnnnnn.
"FALSE"
DB2 for IBM i Date, Time, and Timestamp data types will be converted to PC Date, Time, and Timestamp data types. Care should be taken when using this value in an environment that only supports the Variant Date data type (such as Visual Basic). You may encounter unexpected errors due to truncation or overflow caused by the limitations of the Variant Date data type.
Following are additional considerations when Convert Date Time To Char is FALSE.
The variant Date data type, which is actually a timestamp, does not support micro-seconds - the precision of the DB2 for IBM i timestamp. The OLE DB provider will truncate the fractional timestamp without reporting an error. For example, 1990-03-02-08.30.00.100517 will become 1990-03-02-08.30.00.000000. All updated or inserted timestamp values will have 0 micro-seconds.
Leap second overflow error. The OLE DB timestamp allows up to two leap seconds (a value of 60 or 61). DB2 for IBM i supports a maximum value of 59. An overflow error is returned if leap seconds are set.
The variant Date data type does not support the data limits of an ISO date or timestamp. The value "0001-01-01", used as a default date in many databases, including DB2 for IBM i, will cause an overflow.
DB2 for IBM i supports a time value of 24:00:00 for certain older formats of the TIME data type. The OLE DB provider will convert values of 24:00:00 to 00:00:00 without any error message or warning.
Typically for VB variant, a date value of 1899-12-30 (which is a 0 date) is used to imply a Time only variant date. A time of Midnight (00:00:00) is used to imply a Date only variant date.
Remarks
This custom property is available on the ADO connection object. The property is read/write when the connection is closed and read-only when the connection is open.
Delphi example
<connection>.Provider := 'IBMDA400';
<connection>.Properties('Convert Date Time To Char') := "TRUE";
OR
<connection>.Open('Provider=IBMDA400;Data Source=SystemA;Convert Date Time To Char =TRUE', 'Userid', 'Password');
PowerBuilder example
<connection>.Provider = "IBMDA400"
SetProperty(<connection>), "Convert Date Time To Char", "TRUE")
OR
<connection>.Open("Provider=IBMDA400;Data Source=SystemA;Convert Date Time To Char=TRUE", "Userid", "Password")
Visual Basic example
<connection>.Provider = "IBMDA400"
<connection>.Properties("Convert Date Time To Char") = "TRUE"
AND/OR
<connection>.Open "Provider=IBMDA400;Data Source=SystemA;Convert Date Time To Char=TRUE", "Userid", "Password")

It appears the correct answer is, you can't. No way, no how, does the IBMDA400 provider map any type into a DBTYPE_DBDATE.
What you can do is use the DateValue() SSRS function to convert the returned DBTYPE_STR value to a date/time serial. From there the format functions will work on it.
I didn't have a problem, here in North America, with the DateValue() function directly interpreting the returned DBTYPE_STR value, however, this could be an issue in other locales due to date format differences.