I have a question in Talend :
I need to create a file with name like "File_" + TalendDate.getDate("CCYY-MM-DD hh:mm:ss") + ".txt" and populate it with the result of a SQL query and add a "\t" separator on each column of each row.
After that, I need to connect to a FTP (Through tFTPConnection component), and put this file on a folder (Through tFTPPut component)
The main problem i encounter is that i don't know which composent i should use when i'll create the text file ? Should I use a tFileOutputPositional ? tFileOutputDelimited ? Another component ?
Moreover, i have another issue : When i'm connecting to the FTP, no worries but when i'm on the tFTPPut component, i have this issue :
java.net.SocketTimeoutException: Accept timed out
Any idea ?
Thanks
First you need to execute your SQL query.
To generate the file you should use tFileOutputDelimited on the row data and change the field separator to tab "\t".
Set the filename directly in the tFileOutputDelimited component. Keep in mind that the path only contains forward slashes, e.g.:
"C:/my-folder/File_" + TalendDate.getDate("CCYY-MM-DD hh:mm:ss") + ".txt"
Depending on your configuration it might help to set the ftp connection to passive mode (see more here https://community.boomi.com/docs/DOC-1643)
Related
I have an input CSV file and if one or more rows failed during validation check, then I want to move the input CSV file from InProcess folder to Failed folder and none of the rows should be processed.
I've created a job but the above scenario doesn't seem to work. Can you please help?
tFileInputDelimited --> tSchemaValidationCheck --> tFileCopy
Try to output the reject data into the same file and then run the "tFileCopy_1" in the same branch.
My solution is not very elegant , but it should work.
You want to know how many lines have been rejected : if it is 0 , you can continue with the main task. If it is more than 0, you'll have to stop and copy your input file.
Unfortunately, tSchemaCompliance does not have a global variable like NB_LINES_REJECTED , as you can have with DBOutput components for example.
Here, the reject flow goes to a tLogRow : in my IF links , I can then test how many lines appear in this tLogRow ( ((Integer)globalMap.get("tLogRow_1_NB_LINE"))>0) : if it is 0 , you can stop and copy the input file.
Otherwise, you'll have to read your input file again.
Here, the tJava component is only used to output IF links.
I have a requirement to move files from SFTP to a local directory and it is simple.
I have used
tftpconnection-->tfileexists-->(if)tfileinput-->tmap--tfileoutput to move files and then i have other subjobs as tfileexists-->(if)tfileinput-->tmap--tfileoutput.
I have ten subjobs in the same job moving different files from different directories of SFTP.
I have to also capture all the file related details i.e size of file, number of rows and time of processing, source and destination.
Now i guess i can achieve this if i use
tfileproperties--->iterate--->tfilerowcount--->onComponentOk--->tfixedFlowInput
-- tfileoutput
However I just want one subjob that can capture all the details of the files.But in the above flow i have to hard code subjob file details i want to capture in the tfixedFlowInput which is not what i am looking for. Is there any way i can make this happen in a single subjob by dynamically changing file details in tFixedFlowInput.
Any guidance will be great-full.
Thanks.
You have two choice:
You can use the component tFileProperties (or tFTPFilePropertie).
You can use intern variable of the component tFileinput ( Windows -> Show view -> General -> Structure -> Your Component )
PS : If you just want to only move file without do any modifications you can use tfilecopy maybe.
UPDATE: in the workbench/J log file I am seeing this error:
ERROR Variable names may only contain characters (a-z, A-Z), numbers and underscores
I'm sure this is what is causing my process to fail, but I have no idea why because my variables are named appropriately. I've tried renaming them a few times just in case and the same thing happens.
ORIGINAL POST:
I am working on an automated process to dump the contents of a Postgres query to a text file and FTP it to someone. The process I have been using successfully is a windows batch script that runs SQL Workbench to run the query and write the entire contents of the table to a text file and FTP it.
Now I want to be able to use WBVarDef to load a variable from a text file and use it in my query. For reference, the variable is the unique id of the last record that was FTPed. This is the code i have:
WBVarDef -variable=id -contentFile=id.txt;
WBVardef today=#"select to_char(current_date,'mmddyyyy')";
WBExport -type=text
-file='c:/CLP/FTP/$[today]circ_trans.txt'
-delimiter='|'
-quoteAlways=true
-lineEnding=crlf
-encoding=utf8;
SELECT
*
FROM
transactions
WHERE
transactions.id > $[id]
ORDER BY
transactions.id;
The only thing new here is the reference to the text file that contains the id on the first line. This completely breaks the process but as far as I can tell, I am using this according to the SQL Workbench documentation.
Any help would be greatly appreciated.
I have figured this one out. I was running an older version of workbench that did not support this functionality. Now that I upgraded to build 119 this is working. I'm having other issues but that's a different story....
I tried my luck on the Talend forum and no luck there, so I will try here as well.
I have a job that is reading a large table and then writing the data to .csv files in increments of 25000 rows. What I have noticed is that all .csv files created after the first .csv file have the data loaded all in one row versus the first .csv file that has the data loaded in 25000 rows (as I want it).
Is there a setting that needs to get set on the tFileOutputDelimited component that will allow for the rows in all subsequent .csv files to get loaded as they are in the first (and 'good') .csv file? I am thinking it may be due to what is being used for the 'Escape char' value on the 'Advance settings' tab but am not sure.
On the tFileOutputDelimited component's 'Basic settings' tab, the CSV Row Separator value is CRLF("\r\n") and the field separator is ",". On the component's 'Advanced settings' tab, the Escape char value is """ and the Text enclosure value also is """.
Also, this is being run in a Windows 7 environment.
Unfortunately the documentation I found for the tFileOutputDelimited component's 'Advance settings' tab is lacking in regards to the CSV options.
Below is an example of what is being encountered. As listed below, the first file looks great but all files that follow do not break on the line break and end up placing all of the data on one row versus individual rows.
File #1
header row
row 1
row 2
row 3
...
row 25000
File #2...
header rowrow1row2...row25000
File #3...
header rowrow1row2...row25000
If you need more details, let me know and I'll send them right off. Thank you in advance.
Figured it out. As mentioned in my initial post, the CSV Row Separator had been set to the CRLF("\r\n") option. I changed this to the LF("\n") and that addressed the problem. I had looked atthe generated java code and noticed that it was not treating the CRLF("\r\n") as one of the default options - only \n and \r were. This pointed me in the direction of trying the \n option.
Please help me to achieve the below problem:
I am having one "view" in the Oracle database,I want the output of that view and store that output in the .txt file on some other folder in UNIX box.
The output which is generated from view is a report and I want to save that report in .txt format in one folder on UNIX box.Oracle is present on the UNIX box.
I thought you might be able to use data pump, but maybe the easiest way is to just run this into the standard oracle sql command line app like:
set long 10000
set termout off
set trimspool off
set feedback off
set heading off
spool test.txt
select a ||','||b||','||c from myview;
spool off;
If you put this in a file called extractSql.sql, then you could run:
${ORACLE_HOME}/bin/sqlplus -L ${USER}/${PASS}#${DB_SERVER} #extractSql.sql