Talend: Split a data file into two flows/streams (header_info, data_rows) - talend

Question

Please, I don't need no solution, just a few hints on how-tos. Anyhow, here is the problem I am tackeling with:
I have a file (bloomberg answer file) which is built as follows:
we have a header part (I am only interested in the
START-OF-FIELDS[...]END-OF-FIELDS; varying amount of fields!)
then there is the data part: START-OF-DATA[...]END-OF-DATA. Where each row: unique_id|some_val|some_val|EXCH_CODE|ID_BB_GLOBAL|NAME|SECURITY_TYP|TICKER\n
Shortened example file:
START-OF-FILE
RUNDATE=20150921
PROGRAMFLAG=oneshot
DATEFORMAT=yyyymmdd_sep
FIRMNAME=dl111111
FILETYPE=pc
REPLYFILENAME=r150921020044_20426_01_00
SECMASTER=yes
DERIVED=yes
CREDITRISK=yes
USERNUMBER=1111111
WS=0
SN=111111
CLOSINGVALUES=yes
SECID=BB_GLOBAL
PROGRAMNAME=getdata
START-OF-FIELDS
EXCH_CODE
ID_BB_GLOBAL
NAME
SECURITY_TYP
TICKER
END-OF-FIELDS
TIMESTARTED=Mon Sep 21 01:01:18 BST 2015
START-OF-DATA
BBG004C5BLW2|0|5|LABUAN INTL FIN|BBG004C5BLW2|1MDB GLOBAL INVESTMENTS|EURO-DOLLAR|OGIMK|
BBG000MGZ064|0|5|HK|BBG000MGZ064|361 DEGREES INTERNATIONAL|Common Stock|1361|
BBG000QVRHX9|0|5|AV|BBG000QVRHX9|3BG EMCORE CONVRT GLB-A|Open-End Fund|EMBDGCA|
BBG000BP52R2|0|5|US|BBG000BP52R2|3M CO|Common Stock|MMM|
BBG0068TPTD9|0|5|TRACE|BBG0068TPTD9|51JOB INC|US DOMESTIC|JOBS|
BBG0069D1BR3|0|5|NOT LISTED|BBG0069D1BR3|51JOB INC|EURO-DOLLAR|JOBS|
BBG000BJD1D4|0|5|US|BBG000BJD1D4|51JOB INC-ADR|ADR|JOBS|
BBG008CTTWK1|0|5|FRANKFURT|BBG008CTTWK1|AABAR INVESTMENTS PJSC|EURO MTN|AABAR|
BBG008D4J9S9|0|5|FRANKFURT|BBG008D4J9S9|AABAR INVESTMENTS PJSC|EURO MTN|AABAR|
BBG008B2BXH2|0|5|SIX|BBG008B2BXH2|AARGAUISCHE KANTONALBANK|DOMESTIC|KBAARG|
BBG0016WJL30|0|5|LX|BBG0016WJL30|AB-AMERICAN INCOME PT-ATEURH|Open-End Fund|ABAATEH|
BBG006F3D598|0|5|BH|BBG006F3D598|ABBEY CAPITAL DAILY FUTURE-B|Fund of Funds|ABBDFUB|
END-OF-DATA
TIMEFINISHED=Mon Sep 21 01:03:22 BST 2015
END-OF-FILE
And now my questions
How can I split this file into 2 flows (field_names; data_rows)?
My problem was:
The regex component only works on row level...
The tFileInputMSDelimited does bring me nowhere...
I don't want to start parsing the file by hand (tJava)... or do I have to?
Thanks for any hints in advance,
Marco

Answer 1

No need to java code, check out this very simple job:
Generally, the header have fixed row count, so we need only to play with rows numbers:
tFileInputDelimited1: header 11 and limit 5
tFileInputDelimited2: header 20 and footer 3
and it works fine, if you have a dynamic rows positions, try to find these positions, save them in variables then use this job based on variables. You can also refeer to my answer here.

Answer 2

I'd use tJavaFlex and some Java code. If you look at the actual code its not that hard to understand how it works even if you don't really know java.
Begin:
boolean header = false;
boolean data = false;
String headerData = "";
String line;
Main:
line = input_row.line;
if(line.equalsIgnoreCase("START-OF-FIELDS") ) { header = true; }
if(line.equalsIgnoreCase("END-OF-FIELDS") ) { header = false; }
if(line.equalsIgnoreCase("START-OF-DATA") ) { data = true; }
if(line.equalsIgnoreCase("END-OF-DATA") ) { data = false; }
if(header && !line.equalsIgnoreCase("START-OF-FIELDS")) {
headerData += line + "|";
}
if (data) {
if(line.equalsIgnoreCase("START-OF-DATA")) {
output_row.line = headerData.substring(0,headerData.length()-1); //remove the trailing delimiter.
} else {
output_row.line = line;
}
} else {
continue; //lets go to the next line.
}
End:
//if you want to handle the header separately:
globalMap.put("headerData",headerData);
Hope this helps.