I am trying to download from SOAP API. The supplier has set a limit to download 1000 records per select. I would like to set up "for loop" container to iterate all records in one file.
The Url expect two variables for data i.e Select and Skip. For this I have added two variables #select and #skip and create a web service task and set up the variable with #Select and #Skip.
this webservice task runs ok, but i am stuck on working out a logic to loop through and download all rows in one file.
for the first run the #skip should be 0 for the consecutive run #skip should be previous value of #skip+1000.
Can someone please help to achieve this?
Thanks
Related
I've a question, in my main agent I've some variables which I want to export to a excel row based on a parameter value (in this case AGVs) after the simulation duration. Now I want to vary this parameter by using the parameter variation experiment, but somehow it does not work and only updates the excel sheet for the last run. So in the figure below you see the exportdata function which is the function which puts the data in the excel sheet columns.
Now I want to vary a parameter in this example I choose the number of AGVs, so I created an parameters variation experiment and set this number of AGVs in a specified range see figure below:
Now I want for every parameter the ExportData function to run, and put it in new rows for every AGV value:
AGV=13 ExcelRow=1 ExportData()
AGV=14 ExcelRow=2 ExportData()
....
So I tried to do this by putting the variable NrOfExperiment in the ParametersVariation agent and let it do +1 after every run see the next figure:
But somehow he only export the data for the last run instead of all the runs? How can I fix this problem? I already tried to put it in after iteration by java actions but this would not work.
Thanks!
Easiest option is to just write all model results into a dbase table at the end, with an extra iteration and replication column (fill with getExperiment().getCurrentIteration/Replication()).
Write to that at the end of each model (on destroy of Main).
At the end of the experiment, simply setup the dbase to export to an Excel file.
I'm working with DataStage 11.3 with parallel jobs.
My input is a date and I use the "DateOffsetByComponents" function in a -transformer- stage to obtain 4 dates with different rules, these 4 results ends in different -sequential file- stages.
The next step of my transformation is to make a query in Sybase but, the conditional clause "Where" uses the 4 dates mentioned before as parameters to get the proper information from the DB.
Does anybody have an idea of how can I read the date in each sequential file and put these as a parameter in the next step?
I read a similar question of it in which, he suggested use the -execute command- stage in a Sequence job but I'm new using DataStage and it isn't clear for me on how can I achieve this, although, I can see that, in this type of job (Sequence) you can select parameters that are contained in others -job activities- stages.
Thanks a lot in advance
Yes, an Execute Command activity to read the date from each file is the way to go. Figure out an operating system command to extract the information you need, perhaps some combination of grep and sed, and use this as the command. The command output is available in an activity variable called $CommandOutput.
For example, if your file contains the line "Date: 2021-11-17" then a suitable command might be grep Date: filename | cut -f2
You might like to add a tr command to the pipeline to remove the trailing newline character.
You'll need one Execute Command activity per file. Access the required date from the $CommandOutput activity variable for each.
I have a task to copy the newly added files everyday from SAP Server and store it in ADLS. There are 2 types files on the server (Recurring and Upfront) appended with date. I need to store the files in separate folder and everyday add the latest file from the SAP Server to the ADLS.
File name format:
R_07292021.orc
Recurring_08312021.orc
U_07292021.orc
Upfront_08312021.orc
Below are the steps I have taken so far
Get Metadata Activity to get the list of files from the server
Use filter activity to separate the files based on the names, so filtering with the initial letter
I tried using the Foreach activity and If Condition, but it doesn't seem to be working.
I am stuck at this point trying to figure out how to proceed. Any help would be very much appreciated.
If you are trying to get the latest modified date of a file from a folder, you can refer to the below process.
I tested it with one type of file which starts with “U”.
Create 2 variables, one to store maxdate and the other to store the latestfilename. Assign any initial value (which is the past date) in maxdate variable.
Using Get Metadata activity getting the list of files that starts with “U” by hardcoding the filename parameter value as “U”.
Output of Get Metadata1:
Pass the Get Metadata activity output child items to the ForEach activity to loop through all the files from the list.
Inside ForEach--> Use another Get Metadata activity to get the metadata (last modified date & filename) of the current file in the loop.
Output of Get Metadata2:
Connect Get Metadata to If Condition and use greater function and ticks function to evaluate the If condition expression.
Ticks function returns Integer value of specified timestamp and using greater function compare the 2 values.
#greater(ticks(activity('Get_lastmodified_date_and_name').output.lastModified),ticks(formatDateTime(variables('maxdate'))))
When the expression evaluates to true add (2 set variable) activities to store the last modified date and respective file name into the variables created initially.
Maxdate:
LatestFileName:
Note: These variable's values will be overridden if the next iteration file in the loop contains a timestamp greater than the previous (or first) loop file.
In the next activities, Use the variable filename (latestfilename) to assign the value for the filename parameter in the source.
Is there a way for me to know if the stream of rows has ended? That is, if the job is on the last row?
What im trying to do is for every 10 rows do something, my problem are the last rows, for example in 115 rows, the last 5 wont happen but i need them to.
There is no built-in functionality in Talend which tells you if you're on the last row. You can work around this using one of the following:
Get the row count beforehand. For instance, if you have a file, you
can use tFileRowCount to count the number of rows, then when you
process your file, you use a variable for your current row
number, and so you can tell if you've reached the last row. If your
data come from a database, you could either issue a query that
returns the total number of rows beforehand, or modify your main
query to return the total number of rows in an additional column and
use that (using ranking functions).
Do some processing after the subjob has ended: There may be situations
where you need a special processing for the last row, you can achieve
this by getting the last row processed by the previous subjob (which
you have already saved, for instance, by putting a tSetGlobalVar
after your target, when your subjob is done, your variable contains the last written value).
Edit
For your use case, what you could do is first store the result of the API call in memory using tHashOutput, then read it with a tHashInput in order to process it, and you'll know then how many rows you have retrieved using tHashOutput's global variable tHashOuput_X_NB_LINE.
Having some problems in a job, and I suspect it is due to a lack of understanding of tJavaFlex. I am generating 10 rows in this test job, and am generating loop inside a tJavaFlex:
So there are 10 rows coming in, and a loop in the Start and End section. I was expecting that for each row coming in, it would generate 10 identical rows coming out. And that I would see iterations 0,1,2,3....9 for each row.
What I got was this. This looks to me like the entire job is running 10 times, and so I have 100 random values coming through the flow from the tRowGenerator.
If I move the for loop into the Main Code section, I get close to the behaviour I was expecting. I am expecting each row when it comes in to be repeated 10 times, and for 1 row coming in to produce 10 output rows. What I get is this.
But even then my tLogRow is only generating one row for each 10 iterations it seems (look at the tLogRow output after iteration 9 above why not 10 items?). I had thought I would be getting 10 rows for each single row coming in and I would see this in the tLogRow.
What I need to do is take a value from a field coming in, do some reg exp parsing and split into an array, and then for each item in the array create lines in the output flow. i.e. 1 row coming in can be turned into x number of rows coming out using a string.split() method.
Can someone explain the behaviour above, and also advise on the best approach to get one value coming in, do some java manipulation and then generate multiple rows coming out?
Any advice appreciated.
Yes you don't use it correctly.
The initial part is for initiate variable. (executed one time before the first tow)
In the principal you put your loop (executed one time at each row)
In the final you store in global variable for example.(executed one time after the last row)
The principal code will be executed at each row in a tjavaflex. So don't put a for loop inside you can do like the example in the screen.
You tjavaflex comportement is normal. you have ten row so each row the for loop wil be executed 10 time (i<10)
You can use it like :
You dont need to create your own loop.
By putting the for loop in the Start code, your main code will be triggered by the loop and by incoming rows, and it will be executed n*r times.
The behaviour of subjob that contains a tJavaFlex, reveils that component before tJavaFlex is included into its starting code, and the after component is included in the ending code, but that may depend to many conditions like data propagation and trigger type.
start code :
System.out.print("tJavaFlex is starting...");
int i = 0;
Main code :
i++;
System.out.print("tJavaFlex inside Main Code...iteration:"+i);
row8.ITEM_NAME = row7.ITEM_NAME;
row8.ITEM_COUNT = row7.ITEM_COUNT;
End code :
System.out.print("tJavaFlex is ending...");
System.out.print(row7.ITEM_NAME);
Instead of main flow in row5, try using iterate flow to connect tJavaFlex