I have a source data(.csv) with a Date column which a format
"dd/mm/yyyy" and when I try to output this date column into Excel writer, it gives me an error and also Excel writer step doesn't has the same format built-in in the content tab-->fields-->Type of this step. I even tried using select values step and specified the metadata of date format as "dd/mm/yyyy".
Transformation:
Drop-down date format list from Microsoft Excel Writer step:
Microsoft Excel Writer does support Date type with dd/MM/yyyy format which is properly saved in a cell within a created file. I suggest you don't choose the option from drop-down, as it may not be full, but rather type it by hand.
Here's an example I've run which proves it:
As an additional note it's good to remember, that if you ever face such issues and find that even though you can ensure formatting of a column in previous steps in the step where it can't be enforced you could go for String type with # format - this would treat the input column value as-is and thus not altering it in any way that it came from the previous step.
Related
I'm using the copy data utility in Azure Data Factory to copy data from a REST source to a CSV file. When I preview that source data in ADF the date format is the correct ISO format however when it is written to a csv file or a database table the format changes to something that looks a bit like a unix timestamp e.g. '/Date(340502400000)/'.
Source:
Preview data from the source in ADF
Destination:
Actual data written to the csv file
I've been trying to figure out how to change this to write the date to the file in the ISO format but I'm getting nowhere. Any assistance will be greatly appreciated
I tested the same string date with your source and all works well in my side.
You could manually set the sink data type, like String or Datatime
Csv output(Datetime):
After doing these, if the issue still exist, the best way is that ask Azure support for the deep helps. Only them can tell you what happened in your pipeline.
HTH.
Thanks for the answer but still no joy, all I get is a type conversion error.
Error
Operation on target Copy_Lake_CSV failed: Failure happened on 'Sink' side. ErrorCode=UserErrorInvalidDataValue,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Column 'lastModifiedDateTime' contains an invalid value '/Date(1615206773000+0000)/'. Cannot convert '/Date(1615206773000+0000)/' to type 'DateTime'.,Source=Microsoft.DataTransfer.Common,''Type=System.FormatException,Message=String was not recognized as a valid DateTime.,Source=mscorlib,'
UPDATE:
I logged a call with Microsoft who initially didn't know how to resolve this issue either. Eventually it was escalated all the way up to the ADF product team (developers) who came back with a formula to use in a Data Flow.
What you have to do to convert the date format is to first use a Data Copy Task to dump the data into a "raw" dataset, then using a Data Flow Task you can transform the date values using a Derived Column. In the Derived column use the below formula to convert the JSON date format to an ISO date format, then write it to the destination dataset.
toTimestamp("1970-01-01 00:00:00") + seconds(toInteger((ltrim(rtrim([SourceColumn], "+0000)/"),"/Date("))))
I have Excel data and trying to insert the data into MongoDB using Talend Big Data for Open Studio. This is my job,
tFileInputExcel --> tMap --> tMongoDBOutput
In excel sheet, i have a date value column in this format 7/13/2017(MM/dd/yyyy) as string type and I am trying to insert this column value as ISO format ISODate("2017-07-13T00:00:00.000Z") in MongoDB.
This is my Job:
tFileInputExcel:
tMap:
tMongoDBOutput:
When execute this job, I'm getting the below error.
Error:
When i change the parse format like this TalendDate.parseDate("MM/dd/yyyy",row1.ClosingDate) , I'm getting SimpleDateFormat error. Simple Date Format Error
How to resolve this issue?
you can do simply if you mongodb column schema is date:
TalendDate.parseDate("MM/dd/yyyy",row3.newColumn)
That will automatically convert the date in the date model that your mongoDB column have.
You can change in your schema in Talend the date Model like "yyyy-MM-dd'T'HH:mm:ss.SSS'Z'".
This is a very common mistake doing in reading data without understanding the underlying data types.
I have blogged about this especially for Talend: https://www.tobiasmaasland.de/2017/07/20/using-date-in-talend-etl-jobs/
But let me explain a bit.
Sometimes Excel tries to convert data in the cell even if one might think the cell type is set to String. Insted, it is set to Date. As such, no conversion is needed and the type needs to be Date in the input component.
If it is a String and an error occurs, the the structure of the String is either not the same everywhere or some cells are empty (null). So you might be lucky with
TalendDate.parseDate("MM/dd/yyyy", (row1.ClosingDate == null), "01/01/1970", row1.ClosingDate)
I just assumed you might want to use a placeholder date insted of having null.
This heavily depends on the actual data type in the cells, if every cell has the same data type and if all the data is formatted correctly.
To sum up one of the facts in my blog post: Don't use String for dates. Use Date for dates in Excel. It makes everything easier.
I am pulling in date information via MySQL and the date is in the string format of "MM/dd/YY hh:mm:ss"
Using the built in conversion to date isnt working; so I tried using DATEPARSE and also using DATE and manually parsing it. I have had no luck and I have no idea why I keep getting "null" as the answer or it is putting the month as year, day as month, and year as day.
See screenshots below for the different formulas I have used:
DATEPARSE
Original date format
DATE with manual parsing
This worked for me using your example original data
DATEPARSE('mm/dd/yy hh:mm:ss',[Time] )
I used tableau's documentation of custom date formats to find the right formula for the format.
If this doesn't solve the problem then it is most likely due to a Locale issue. The Dateparse function relies on the locale specified by your computer settings to interpret and then display the strings you want to convert and can affect whether a certain format can be specified. This means that if a certain format is not recognised it will return a null. This will often occur is the windows region and language format is changed to another language other than the original language the workbook was created it.
So check your locale and if required execute the following workaround (documented by tableau here):
Open Windows control panel > region and language > format
Click format dropdown and change to original language of workbook creation
click ok or apply and reopen tableau work book
I have been tasked with converting a string which can come in different formats like such(mm/dd/yyyy, m/dd/yyyy, or mm/d/yyyy). However, it needs to be converted into a Timestamp with the following format (yyyy-mm-dd-00.00.000000). I have tried multiple conversion techniques within the Transformer stage; however, I have been unsuccessful. Basically, I pull the data from a file and stage it into a file in the same format as the table. I then insert into the database using the second file.
The main issue I was running into was the fact that the format could be mm/dd/yyyy or m/d/yyyy. The solution to this was to add a ",s" to my StringToTimestamp function.
StringtoTimestamp(Input,,"%(m,s)/%(d,s)/%yyyy %hh:%nn:%ss").
I have a decimal date field (TDDATR) that is in the YYYYMMDD format.
I would like to create a field that is TDDATR + 30 days but I am unable to.
Using 'Define Results Field' I have tried a few things;
Simply doing this;
TDDATR + 30 DAYS
But it returned this error: Labeled duration not used correctly.
I tried using the DIGITS and SUBSTR commands to create a field in the DDMMYYYY format and then +30 days but got the same error.
Same as above but in the DD/MM/YYYY format - same error.
Using DATE(TDDATR) but all I see is +'s in the field.
Using DATE( ) on the fields created in step 2 and 3 - still get +'s
I've ran out of ideas - any help would be greatly appreciated.
Query/400 lacks a lot of the features that an SQL based interface has.
I'd urge you to consider switching to Query Manager (STRQM) which is a fully SQL based product. You can even convert Query/400 queries to Query Manager queries with the RTVQMQRY command by having the ALWQRYDFN parm set to *YES.
The other option that IBM is pushing is Web Query. Again, fully SQL based and you can convert Query/400 queries into it.
Having said that, the problem is that FLD + 30 DAYS only works when FLD is a DATE data type. Query/400 includes a DATE() function to convert non-date types into date. But it's very limited in that it only works with character fields formatted according to your job defaults. Assuming you're in the US, it'd only work with a character value of '07/01/15'.
You could do a lot of manipulation in Query/400 and end up with a result field that meets DATE()'s requirements. But a better solution would be to create an SQL view over your table and have your numeric date converted into a date data type in the view.
You can find code examples that show how to convert a numeric YYYYMMDD to a actual date data type in the view. However, I'd recommend create a user defined function (UDF) that will do the conversion for you. That will make it much easier to use in the view and to reuse in other places.
If you'd like, there's an open source package called iDate, that includes all the code required for convert to/from date data types.
Download that, install/compile it and your SQL view becomes
select ... idate(TDDATR,'*CCYMD') as TD_DATE
from myfile
The use of days is as follow
Field Expression
CURDATE_30 days(current(date)) + 30
The solution to your problem is: given the field A dec(8,0)
Field Expression
YYYYMMDD_ date(substr(digits(a),5,2)||'/'||
substr(digits(a),7,2)||'/'||
substr(digits(a),3,2))
NEXT_MONTH DAYS(YYYYMMDD_) + 30
Remember to check the date format in your job description. In the example the format is MDY or MM/DD/YY.
More info here
Based on the information here, I created the below 2 fields;
TDDIGI DIGITS(TDDATR)
TDDAT1 SUBSTR(TDDIGI,7,2)||'/'||
SUBSTR(TDDIGI,5,2)||'/'||
SUBSTR(TDDIGI,3,2)
From here I was able to create a date field;
TDDAT2 DATE(TDDAT1)
Which allowed me to perform the necessary calculations.
The format of TDDAT1 is based on your job description which can be found by;
WRKJOB
Option 2
Page down
Date format..: X
Mine was *DMY, so TDDAT1 was formatted based on this.