I have Excel data and trying to insert the data into MongoDB using Talend Big Data for Open Studio. This is my job,
tFileInputExcel --> tMap --> tMongoDBOutput
In excel sheet, i have a date value column in this format 7/13/2017(MM/dd/yyyy) as string type and I am trying to insert this column value as ISO format ISODate("2017-07-13T00:00:00.000Z") in MongoDB.
This is my Job:
tFileInputExcel:
tMap:
tMongoDBOutput:
When execute this job, I'm getting the below error.
Error:
When i change the parse format like this TalendDate.parseDate("MM/dd/yyyy",row1.ClosingDate) , I'm getting SimpleDateFormat error. Simple Date Format Error
How to resolve this issue?
you can do simply if you mongodb column schema is date:
TalendDate.parseDate("MM/dd/yyyy",row3.newColumn)
That will automatically convert the date in the date model that your mongoDB column have.
You can change in your schema in Talend the date Model like "yyyy-MM-dd'T'HH:mm:ss.SSS'Z'".
This is a very common mistake doing in reading data without understanding the underlying data types.
I have blogged about this especially for Talend: https://www.tobiasmaasland.de/2017/07/20/using-date-in-talend-etl-jobs/
But let me explain a bit.
Sometimes Excel tries to convert data in the cell even if one might think the cell type is set to String. Insted, it is set to Date. As such, no conversion is needed and the type needs to be Date in the input component.
If it is a String and an error occurs, the the structure of the String is either not the same everywhere or some cells are empty (null). So you might be lucky with
TalendDate.parseDate("MM/dd/yyyy", (row1.ClosingDate == null), "01/01/1970", row1.ClosingDate)
I just assumed you might want to use a placeholder date insted of having null.
This heavily depends on the actual data type in the cells, if every cell has the same data type and if all the data is formatted correctly.
To sum up one of the facts in my blog post: Don't use String for dates. Use Date for dates in Excel. It makes everything easier.
Related
Trying to convert string value(2022-07-24T07:04:27.5765591Z) into datetime/timestamp to insert into SQL table in datetime format without losing any value till milliseconds. String which I am providing is actually a datetime and my source is ADLS CSV. I tried below options in data flow.
Using Projection-> Changed the datatype format for specific column into timestamp and format type-yyyy-MM-dd'T'HH:mm:ss.SSS'Z' however getting NULL in output.
Derived column-> Tried below expressions but getting NULL value in output
toTimestamp(DataLakeModified_DateTime,'%Y-%m-%dT%H:%M:%s%z')
toTimestamp(DataLakeModified_DateTime,'yyyy-MM-ddTHH:mm:ss:fffffffK')
toTimestamp(DataLakeModified_DateTime,'yyyy-MM-dd HH:mm:ss.SSS')
I want the same value in output-
2022-07-24T07:04:27.5765591Z (coming as string) to 2022-07-24T07:04:27.5765591Z (in datetime format which will be accepted by SQL database)
I have tried to repro the issue and it is also giving me the same error, i.e., null values for yyyy-MM-dd'T'HH:mm:ss.SSS'Z' timestamp format. The issue is with the string format you are providing in source. The ADF isn’t taking the given string as timestamp and hence giving NULL in return.
But if you tried with some different format, like keeping only 3 digits before Z in last format, it will convert it into timestamp and will not return NULL.
This is what I have tried. I have kept one timestamp as per your given data and other with some modification. Refer below image.
This will return NULL for the first time and datetime for second time.
But the format you are looking for is still missing. With the existing source format, the yyyy-MM-dd'T'HH:mm:ss would work fine. This format also works fine in SQL tables. I have tried and it’s working fine.
Try to use to String instead of timestamp and use this to create your Desired timestamp
toString(DataLakeModified_DateTime, 'yyyy-MM-dd HH:mm:ss:SS')
Can I accept "yyyy-mm-dd" instead of "yyyymmdd" when I put data into a date type column in DataStudio?
From the following Q&A and the official documentation, it looks like "yyyymmdd" only.
Google data studio table shows date column as null
https://support.google.com/datastudio/answer/6401549?hl=en#zippy=%2Cin-this-article
However, since the date type of Snowflake, the source of the linkage, only allows "yyyy-mm-dd", the
I'd like to know how to do this efficiently, because I'm changing the date type to a character type, linking it to DataStudio, and then changing it back to a date type in DataStudio.
https://docs.snowflake.com/en/sql-reference/data-types-datetime.html#date
I would appreciate it if you could tell me.
Not exactly sure how things are mapped from Snowflake to DataStudio, but Snowflake can produce yyyymmdd dates.
SELECT TO_VARCHAR('2021-08-19'::DATE, 'YYYYMMDD');
If DataStudio requires that the data type from the source be a DATE then perhaps you could alter the date_output_format of your connection session or user to be 'YYYYMMDD'.
https://docs.snowflake.com/en/sql-reference/parameters.html#label-date-output-format
Import your data as it is into Data Studio. In the schema screen, ensure your source date field ("yyyy-mm-dd") is formatted as text. Then add a calculated field with the formula:
TODATE(source_date_field,"%Y-%m-%d","%Y%m%d")
TODATE reference
Formate the new calculated field as date.
Data imported from sharepoint to qliksense is in the format "2020-08-30T18:30:00Z". And I want to filter out the columns based on the month(2020-Aug). I have converted the date format "2020-08-30T18:30:00Z" to "31/08/2020" using the expression:
Date(ConvertToLocalTime([REPORTED_x0020_DATE],'New Delhi'),'DD/MM/YYYY')
In filterpane, given expression is :
[REPORTED_x0020_DATE.autoCalendar.YearMonth]
The problem is that after conversion the dates are just previous to the dates in sharepoint. So that, result getting after filtering is not right.
The mismatch of dates are shown in the image attached. Because of this mismatch not getting actual results on filtering.
The problem is that although you see just dates due to the format string, the underlying data still have timestamps returned from ConvertToLocalTime.
Try to truncate the timestamp by using floor:
Date(Floor(ConvertToLocalTime([REPORTED_x0020_DATE],'New Delhi')),'DD/MM/YYYY')
one column in my csv file is a date that is read as a string and it follows this pattern : 2018-09-19 10:27:28.409Z. I am struggling to convert the column from string to date.
The conversion options in spotfire didn't allow me to change the column type. however, I found the solution, at the moment of importing the data set (file) you need to specify the type (date time) and magically spotfire manages the conversion.
In a data flow from one table to another, I would like to cast a date.
The date leaves the source table as a string in this format: "2009-01-05 00:00:00:000 + 01:00".
I tried to convert this to a date using a tConvertType, but that is not allowed apparently.
My second option is to cast this string to a date using a formula in a tMap component.
At the moment I tried these formulas:
- TalendDate.formatDate("yyyy-MM-dd",row3.rafw_dz_begi);
- TalendDate.formatDate("yyyy-MM-dd HH:mm:ss",row3.rafw_dz_begi);
- return TalendDate.formatDate("yyyy-MM-dd HH:mm:ss",row3.rafw_dz_begi);
None of these worked. When inserting the result into the target-table (MySQL, InnoDB) a receive the error message that the date is not valid. The format of the target field is a MySQL Date field.
How can I cast the date to the desired format?
Talend offers you a nice way of handling date formats.
You can easily change the date format in the Schema editor tab at the tMap window.
It works for both tMap input and output flows.
I've added a picture for a better illustration.
to cast this string to a date using a formula.... error message that
the date is not valid. The format of the target field is a MySQL Date
field.
What I understand from your question is, you want to insert a date into MySQL Date field.
But the method that you are using, returns the 'String' type.
TalendDate.formatDate(String pattern, Date date); //formats a date into Date/Time string
So in that case, if your field is of 'Date' type.
TalendDate.parseDate("yyyy-MM-dd", TalendDate.formatDate("yyyy-MM-dd",row3.rafw_dz_begi));