In Talent open studio. i have to add different source file to one output table..How can i fetch the last id in that output table and generate the very next id and continue insertion with different sources
Add a subjob prior to your current subjob.
in a tDBINPut component select your lastID through a query (select maxID , select top 1 , etc) .
Put the result in a variable (global variable or context variable), in a tJavaRow for example. (context.lastID=input_row.id)
Use this variable in a tMap to generate the next ID, through Numeric.sequence function.
In the output mapping of your tmap, you should add something like Numeric.sequence(context.lastID,1,1)
I think there are plenty of solutions to get the lastID and generate a sequence from there. you can also check advanced output parameters on your tDBOutput.
Related
Is it possible to get values from a Excel file and use them to filter rows in Talend?
For example:
I have an Excel file with account numbers list.
In Talend, I have a query in a tDBInput linked to a tMap, that is linked to a tFilter. This filter is based on Excel columns values with the accounts list.
Or if I can use the account list in the where clause in tDBInput. The problem is that the account list can change anytime.
Thanks a lot.
There are multiple ways to proceed :
Link your tDBInput to tMap as the main flow, and your excelInput as the lookup. Make the join between the two flow in tMap as an inner join : this way you can filter main data with data coming from excel. The inconvenient is that you will read all data from DBInput and filter after that, which is less efficient.
You can also make it through two subjobs :
First subjob : tFileInputExcel -> tJavaRow . Push your list of accounts to a String in a context variable to construct your "where" query. (you can also use a tAggregateRow with 'list' mode to build a list )
Second subjob : use this constructed context variable as filter in your DBInput query. This way you'll read only valuable data in this component.
I have one already existing Talend Open Studio tMySQLInput component with some sql code inside it, in order to retrieve some joined columns linked to a tMySQLOuput component (pointing to an already existing MySQL table) with few records.
QUESTION:
Will the "tMySQLInput" component overwrite the already existing table data that the tMySQLOutput component relates to? I mean is there an option to check in the tMySQLInput our output in order to say, overwrite each time this job is executed ?
Thank you all.
Yes, there is an option where in tMySQLOutput where you can specify what action you want to do to your table. Follow following steps:
Go to component tab of tMySQLOutput, it will open the basic settings of this component.
If you will look closer you will find Action on table. This is the action which you can perform on the table which is pointed by tMySQLOutput. It has options as Default, Drop and Create Table etc.
Then you have Action on data. These are the options which you can perform on the data like Insert, Update etc.
In your case I suppose you can choose Action on Table as Default and Action on Data as Insert. Default action would not do anything on the table and Insert option would insert the records at the end of table. But in case of Insert if you will have duplicate rows then job would stop the moment it will find any duplicate row.
I want to select certain columns from 2 tables by coding the appropriate SQL code in the "query" zone of my tMySQLInput component. Then I've been told to use a "tMap" component in order to map each column from my tMySQLInput component to my tMySQLOutput component.
Is this the correct way to to do what I need to do : Take information (SELECT ...) from 2 tables and then insert them into another 3 rd table?
Inorder to insert data from 2 tables (source) into another table(target), it is necessary to use tmap component.
1.tMysqlinput(with query) and check the edit schema section for appropriate values you wish to send to the output/target
2.tMap component map all the necessary columns from source to target
3.tMysqlOutput check the edit schema section for appropriate columns that you wish to see in your target.
I can get the results of a Stored Proc from Visual Studio by following these steps:
0) Select View > Server Explorer
1) Locate the Data Connection that contains the SP of interest.
2) Right-click the SP and select Execute; if any args are required, it will allow you to populate those in a dialog and then proceed with the execution
Provided that you have provided valid args (and the SP returns data), the results will display in a grid in a T-SQL Results tab.
Now, though, what if you want to query that "dataset" that has been returned, such as to sum the values of a particular column - can that be done right there from within Visual Studio.Server Explorer.T-SQL? If so, how?
UPDATE
I believe Khazratbek, but I can only get this far in a New Query instantiated by right-clicking the Tables folder beneath the Data Connection:
SELECT SUM(QtyShipped), SUM(QtyOrdered) FROM CPSData.
...the options available as a drop down following that final "." do not contain the results of the Stored Proc (SQLQuery1.sql).
Is it really possible?
Server Explorer -. Right click on Tables -> New query. Write your query (you may select from your SP execution results) and just click on triangle sign. Sorry, if I misunderstand your exact question.
Let's think that your stored procedures gives you logins and passwords of users:
select login, password from users;
You are taking one million result, for example. And you don't need to change your stored procedure, but you want to make some job with your results (selecting in some order, changing the order, select by some condition, so on).
Let's continue. Your stored procedure gives you a two column: it is column login and password. So we may consider the results of your SP execution as some table. To work with your result do next steps:
Server Explorer -> Your_database_connection -> Right click on Tables -> New query
Then write the following code there:
Declare #temporary_table_variable table(login varchar, password varchar); --or without datatypes
insert into #temporary_table_variable(login, password) exec MyStoredProc;
SELECT * from #temporary_table_variable where id > 1000; --it is just a simple example
Hope it helps
In my Talend job I have some xml files. Each of these files contains a field that I will use in the WHERE clause of an UPDATE statement.
I will not describe the whole job and how I've done it, but only the small excerpt: I've modeled this by using a tFileInputXML and a tMysqlRow component.
In the tMysqlRow component I've built an UPDATE query like the following (simplified) one:
"UPDATE `my_table`
SET `my_table`.`oneField` = '" + row1.ONEFIELD + "'
WHERE `my_table`.`id` = '" + row1.ID
This works fine for me. But I don't understand how to model the exceptionnel case if there will be no updated rows, because the ID wasn't be found in the table. I know that there is a trigger "run if", but I don't how to use it exactly.
Can anybody help?
I think you should use a preceding component (tMap) that will filter the content of your XML files by defining a join between the ID column of your table and the ID column in your XML.
Because your main flow (row1) comes from the XML files, you have to reject the rows where the IDs won't be found in the database. That's (one of) the role of tMap component.
You can also have a look on the tJoin component but I haven't used it till now.