What is the best way to iterate over files and feed them into tMongoDBBulkLoad? It sees that you cannot feed into this component from a tFileList componet (Iterate) - which would make the most sense.
I want to import 80 files, rather than create one massive file which is too large to open in notepad if I have issues during the import.
Thanks
---Update----
I know how to do this with other components, my issue is I cannot feed an Iterate component into the tMongoBulkLoad
The simplified job will be like this :
tFileList ---------iterate--------tMongoDBBulkLoad
and in the tMongoDBBulkLoad settings you set the Data file to :
((String)globalMap.get("tFileList_1_CURRENT_FILEPATH"))
Here, the tFileList will iterate over files, in each iteration, the tMongoDBBulkLoad will be triggered to load the current file, which is indicated by the global variable.
--- Reply for the Update ---
To connect an iterate trigger to the component, you can add a dummy tjava with no code, it will be like this :
tFileList -----(iterate)-----tJava-------(onComponentOk)-------tMongoDBBulkLoad
Related
I want to fetch file details like path of file, column names, and other things .In my job there are null values while using tfileproperties as shown in image added. look
if somebody can do something.
To Use tFileProperties you have to pass a valid file Path
What i suggest is to pass the file Path in a context variable as such
Then , pass the context variable in my case file_Input in the tFileProperties as such :
And if your files are stored in the same Folder you can use a tFileList component see the link below :
https://help.talend.com/r/en-US/7.3/tfilelist/tfilelist-standard-properties
I'm new to talend, I have java code which need to get data from files. I want to use them in talend job. Now am facing problem how to use this java code in talend, I created routine but facing problem in creating jar files, and also how should I use this routine in my job.
I don't know what you want to do but normally you would use the build-in Talend components for reading a file.
Depending on your File you are going to read you can use:
tFileInputRaw - for reading a file line by line
tFileInputDelimited - for reading CSV files (getting a set of columns)
tFileInputExcel - for XLS/XLSX files (getting a set of columns)
If you want to use your code anyway you have to make your routine available to your job. To do that, close your job, click right on the job and choose "setup routine dependencies". You should be able to add a routine by click the green "+"-button.
After that you are able to use your functions in a tJava or tJavaRow component with routines.ClassName.functionName().
What is the best way to apply logic to objects during an iteration of tFileList.
The issue is that if I use a tFileList to get a list of files, i am not able to use tJavaRow or jMap to create the filename that i want the file to be renamed. Basically, if i have zip files with years(2010,2011,2012 etc) and each zip file contains files with the same name (f1.csv, f2.csv, f3.csv), i want to iterate through the compressed files, uncompress them and rename the files with
f1_2010.csv, f2_2010.csv, f3_2010.csv..f1_2012.csv etc.
Thanks!
Iterate links are providing a way to execute components based on events or facts while main links are transfering data between components.
With something looking like that you should be able to resolve your problem :
tFileList_1 --iterate--> tFileUnarchive_1
|
onComponentOK
|
tFileList_2 -- iterate --> tFileCopy_1
|
onComponentOK
|
tFileArchive_1
Use ((String)globalMap.get("tFileList_1_CURRENT_FILEPATH")) in your tFileUnarchive to get the ZIP path.
In tFileCopy use ((String)globalMap.get("tFileList_2_CURRENT_FILEPATH")) to get the path of file and config it to be a rename.
For your name modification you can add tJava on "onComponentOK" links. By using globalMap.put("year",((String)globalMap.get("tFileList_1_CURRENT_FILEPATH")).substring(x,x)) or more complicated code. And use these variables in your others components parameters.
I have a txt file that is holding a string inside, I want to be able to use this string in one of my scripts, so I'm wondering if there is a way to set the content of the file as one of the build properties or parameters which I'll be able to use in my scripts it should be the same as using one of the build environment properties.
For example : ${JOB_NAME} which is holding the the job name, so in the same way I want to access the content of the file which is holding some value inside.
Is it possible?
You can upload a file from your computer to the workspace through the File parameter of the job.
You can use Extended Choice plugin parameter, to read value(s) from a file and display them in a dropdown/radio-button/checkbox for the user to select, dynamically, every time the build is triggered.
You can use EnvInject plugin to read value(s) from a file and inject them into the build as environment variables, so that they can be used by the rest of the build steps/scripts.
Your question is very unclear on what your are trying to do. Pick one of the 3 methods above based on what you need, or clarify your question.
I have a large amount of data in an XML file, and I'd like to append data to this file without rewriting it every time. I already know how to write the entire file out, but I'm struggling with how to append data to this file. Do you have a suggestion for how to do this?
XML isn't a good format for this - if you append to a previously-complete document, it's no longer a complete document.
One option (depending on the APIs available to you) is not to write the root tag or document declaration, but to fake them when you read the file. So you'd have:
Fake document declaration
Fake root open tag
Real data from the file
Fake root close tag
Then you can just append elements to the end of your file at will. It will depend on what you're trying to do with this file though - and whether you can fake a stream input which effectively "tops and tails" the real data in the file.