I have some data which I need to pivot in Talend. This is a sample:
brandname,metric,value
A,xyz,2
B,xyz,2
A,abc,3
C,def,1
C,ghi,6
A,ghi,1
Now I need this data to be pivoted on the metric column like this:
brandname,abc,def,ghi,xyz
A,3,null,1,2
B,null,null,null,2
C,null,1,6,null
Currently I am using tPivotToColumnsDelimited to pivot the data to a file and reading back from that file. However having to store data on an external file and reading back is messy and unnecessary overhead.
Is there a way to do this with Talend without writing to an external file? I tried to use tDenormalize but as far as I understand, it will return the rows as 1 column which is not what I need. I also looked for some 3rd party component in TalendExchange but couldn't find anything useful.
Thank you for your help.
Assuming that your metrics are fixed, you can use their names as columns of the output. The solution to do the pivot has two parts: first, a tMap that transposes the value of each input-row in into the corresponding column in the output-row out and second, a tAggregate that groups the map's output-rows according to the brandname.
For the tMap you'd have to fill the columns conditionally like this, example for output colum named "abc":
out.abc = "abc".equals(in.metric)?in.value:null
In the tAggregate you'd have to group by out.brandname and aggregate each column as sum ignoring nulls.
I have a tExtractRegexField which extracts a date from a string of text coming from a ExcelFileInput and will output the dates to tLogRow but I can't connect the same output as a lookup column to a tMap with a 2nd ExcelFileInput as its main input.
If I connect the ExtractRegexField to tMap first I can't then connect the 2nd ExcelFileInput and vis versa
I'm using Talend 6.3.1 and for testing I am able to connect 2 x ExcelFileInput to a tMap so I dont think its a problem with my system setup.
I have also tried tJoin instead of tMap but I encounter the same issue (can't connect both inputs together but can connect "A" or "B" first)
Overview of Process
Problem Area
The tExcelFileInput uses globalMap to get the path to the excel file from the preceding tFlowToIterate
Based on discussions on the talend forum the issue may have been down to a desire by taland DI to avoid circular references
An alternative solution is to extract the regexfield from the header row and store them in a global variable using a tJavaRow and globalMap.put("MyVal", row.Data); and then OnComponentOk read the remaining data from the body rows and in the tMap recall the global variable MyVal and include it as needed in your tMap Output
I have an XML file (tFileInputXML) as the start point of my job, from that XML, i'd like to "combine" all its rows in a java List/Array/Whatever and get that List as output.
Is there a component in Talend that offers such mechanism ?
NB : I've already tried the TJavaFlex component but it still output many rows.
Thank you in advance.
You need to read the file, map its fields using tXMLMap and then process them in a java component :
I am inserting records from CSV to salesforce using talend tool. I want failed records in a separate CSV. Please provide me some solution?
Thanks!!!
Deactivate tSalesforceOutput / Advanced Settings / Extended Output (might result in slower performance)
Add another row with right click, Row / Rejects
Use this row in a csv component
Try this :
Create another tFileoutputdelimited
Go in tmap after create a new output in tmap
Go in the option in new output and put "catch output reject" to true.
Join the new output to the tfileoutputdelimited
The format is
tJDBCInput
main
tAggregateRow
main
tJavaRow
main
tLogRow
as shown in the image:
Under tAggregateRow basic setting I have this:
What should I write in tJava to get the value of rowcount?
If you want to get the row number of the data read by tjdbcinput, Talend provide it natively with no need to make aggregation, the row number is stored in the global map and you can get it using this line of code ((Integer)globalMap.get("tJDBCInput_1_NB_LINE"))
You can use it in a tJava component and wite it in your console using
System.out.println(((Integer)globalMap.get("tJDBCInput_1_NB_LINE")));