I have a csv file where insert queries are present.
I want to create a job to execute query against the DB.
How would I do this?
Use tFileInputDelimited component to read the csv file based on your file configuration like fields & row delimiter. Connect this component to tFlowToIterate and Connect tFlowToIterate to db component (tOracleRow, tMySQLRow likewise based on your database) with iterate link.
In tFileInputDelimited, define the schema like : Query
tFlowToIterate will iterate each Row(insert query) and convert it in key value pair and than this will pass to DB component to execute.
In DB component, ((String)globalMap.get("row3.Query"))
Hope this help
Related
I want to access the table names from tFileInputDelimited, so how to write SQL squery in tDBInput so that I can able to access the data of that table.please just see the image you can understand, there is SQL query which i have written.
I tried various ways but it's not working.
try using this query
"select * from "+((String)globalMap.get("row2.Table_name"))+""
I supposed that your getting the right result from tFileInputDelimited to check that you have to link
tFileInputDelimited -> tLogRow
I have to read 10 files from a folder in blob container with different schema(most of the schema among the table macthes) and merge them into a single SQL table
file 1: lets say there are 25 such columns
file 2: Some of the column in file2 matches with columns in file1
file 3:
output: a sql table
How to setup a pipeline in azure data factory to merge these columns into a single SQL table.
my approach:
get Metadata Activity---> for each childitems--- copy activity
for the mapping--- i constructed a json that containes all the source/sink columns from these files
You can create a JSON file which contains your each source file name and Tabular Translator. Then use Lookup activity to get this file's content(Don't check first row only). Loop this array in For Each activity and pass source file name in your dataset. Finally, create a copy data activity and use Tabular Translator as your mapping.
I am trying to read data using a simple select query and create a csv file with the resultset data.
As of now,I have the select query present in application.properties file and I am able to generate the csv file.
Now, I want to move the query to a static table and fetch it as an initialization step before the batch job starts(Something like a before job).
Could you please let me know what would be the best strategy to do so.i.e. reading from a database before the actual batch job of fetching the data and creating a CSV file starts.
I am able to read the data and write it to a CSV file
application.properties
extract.sql.query=SELECT * FROM schema.table_name
I want it moved to database and fetched before actual job starts
1) I created a job with one step(Read and then write).
2) Implemented JobExecutionListener. In the beforeJob method, used JdbcTemplate to fetch the relevant details(A query in my case) from the DB.
3) Using jobExecution.getExecutionContext() , I set the query in the execution context.
4) Used a step scoped reader to retrieve the value using late binding. #Value("#{jobExecutionContext['Query']}") String myQuery.
5) The key to success here is to pass a placeholder value of null so that the compilation is successful.
I need to apply ETL process on many csv files having different schema, to load them at the end into a single star schema.
How can I do that using talend ETL tool ?
Firstly import the csv file into talend. Create a job tom move csv data to database. In output component (if oracle tOracleOutput) you have an option to create table if it does not exists. Like wise you can import all the csv files and create schema in the database.
Thank you
I am importing an excel file in Talend.
I want to select all the distinct values in column "A" and then dump that data into the database. Is it possible to do that with Talend?
If not, what are the alternatives available. Any help is appreciated
Yes you can do that easily with Talend Open Studio.
Create a new job like this one:
You can replace the tOracleOutput component by the component corresponding to your database.
Then parameterize the tAggregateRow component like this :
Distinct values of ColumnA will be transfered to distinctColumnA in the output schema.
You can also get the number of occurences by adding a count of columnB in the operations table.
Using tUniqRow in Talend Open Studio 6.3 works very well and you get to keep all your columns.