I am very new to talend open studio and I am using Talend 7.3.
We have customers accross multiple zone and we have seperate table per zone.
We have multiple zone specific files named based on zone. I want to write a generic job which will process files to correct table(each zone has a separate table).
Can someone help me with this?
Thank you.
You have use multiple tFileinput and multiple targets in the same code. Your design should be something as below
tFileinput1 If you need any ---- Table1
tFileList ----- tFileinput2 transformations ---- Table2
tfileinput3 you can apply it ---- Table3
Provide the proper file pattern names in each tFileinput components and it will process into required target alone.
Related
Currently, I have multiple jobs to load data from source to target (Oracle Connector -> Transformer Stage -> Oracle Connector). I want to get those job's information to a statistic table to track the progress every day.
My thought is after the job has done, it will automatically insert 1 row for each job to my statistic table. For example, after Job_1 (load to Target_1 table) and Job_2 (load to Target_2 table) finished, each job will insert 1 row to my statistic table and it will look like below:
TABLE_NAME DATE_1 DATE_2 TIME_STAMP TOTAL_RECORD
---------- ------ ------- ------------------- ------------
Target_1 041120 2020309 2020-11-04 11:09:00 500
Target_2 041120 2020309 2020-11-04 11:10:00 1000
Is it possible to do with a routine or something else?
Such load stats are really useful and there are several way to achieve this.
The main question is how to get the total record information amd how those are defined.
The is a number of rows getting read from the source and a number written into the target - only if you do not have any filtering in your Transformer (and no rejects) those shoule be the same.
You can get this from
the link information via DSGetLinkInfo
by running a count on your target table
or - my recommendation - by using the DSODB
Check out the Section Monitoring job and job runs by using the Operations Console in the Knowledge Center
or if you want to create your own table Extracting Monitor data from the operations database
I am using Talend free version
I have below requirement:
My source is MS ACCESS; table SRC_CUST.
SRC_CUST
CUST_ID CUST_NAME
101 ABC
102 LMN
My target is .csv file TGT_CUST
Requirement: I am using tAccessInput component for MS Access table and I want to load that table into .csv file. My columns are varying day to day on daily basis.
Day 1: SRC_CUST has 2 columns CUST_ID and CUST_NAME so I need to load as it is into .csv file
Day 2: SRC_CUST has 3 columns CUST_ID, CUST_NAME, CUST_ADD so on day 2 I need to load these 3 columns without changing any code, means, I need to achieve column change dynamically .
Note: I am using Talend free version so I neither use any dynamic component nor dynamic data type. I cannot even add columns in "Edit schema" under basic settings of tAccessInput component because my columns are varying.
Please help me for same.
Thanks,
Vaishali Shinde
I am currently trying to create a rota within filemaker 16 and I can't figure out how to create records that share a date.
I want to be able to have people assigned to jobs and jobs assigned to dates but currently when I create jobs with the same date it creates a new record instead of assigning it to the one already existing.
I have 3 tables currently jobs, date and people. I have a 4th layout with a portal where I wanted to view records related to jobs that are set for a certain day.
Any help would be much appreciated.
Many thanks.
I am not 100% convinced you need a Dates table. Do you have anything specific to record about a date, other than its existence?
However, you certainly need a join table of Assignments, with fields for:
PersonID
JobID
Date
(this is assuming your rota is daily, otherwise you will need to indicate the shift or hours too).
I think your structure should change on this.
So instead have:
Parent:
ProjectId
Date
PersonId
JobId
Date
Then make the project Id your 'Primary Key' so your parent record.
Then you are just assigning the person, job & date as the child.
That way you can always add to the previous record without relying on date field.
You could then filter via dates etc.
I have data in the following form on HDFS:-
/basedir/yyyymmdd/fixedname/files
Where the yyyymmdd is the date folder and files are the list of files added in the directory. I need a table in hive to pick up data from yyyymmdd/fixedname directory. This should also work when i added a new date. e.g. i add something on 5th March 2013 so all files added on that day would go to 20130305/fixedname folder. On 6th March 2013, all files would go to 20130306/fixedname folder.
How do i alter a hive table to pickup data from the changing date but fixed folder within it?
Do you have a partitioned table? Let's say that you already have a partitioned table by the column date and you want to add new data. In this case, you will have to add the data to the new directory and tell to hive table (specifically to the metastore) that it has a new partition using ALTER TABLE ADD PARTITION COMMAND.
Let's say that you do have not created any table yet. In this case you will have to create a partitioned table and then insert the data into this table from queries. The magic comes up when you set these two flags:
set hive.exec.dynamic.partition=yes
set hive.exec.dynamic.partition.mode = nonstrict;
These flags allow dynamic partitions (For more details read here).
Remember that you will have directories like:
/date=YYYYMMDD/fixedname/files
So you have to tell to Hive to pick up all the data into subdirectories in a recursive way. You should set the following flag (here there is a better explanation)
SET mapred.input.dir.recursive=true;
Finally you will able to make queries by date and get all the data in the subdirectories from the date you specified in the query (/date=YYYYMMDD/...).
Hope this helps you.
We are using a 3rd party HR Tool (NuView) and are creating custom reports in Crystal. The source data is a csv file. In order for NuView to process the Crystal Report it REQUIRES that the:
csv file name
datasource name in Crystal
the report name
be the same.
For example, if it was a report on ADDRESSES - Address.csv, Address#csv.fieldname, Address.rpt
Here's our dilemma - we are creating some complex reports and want to REUSE them. So, we want to take an existing CR report and copy it and then * CHANGE THE DATASOURCE * name - and we can not figure out how to to this?!?
We can change the source file (e.g., Address.csv) and the report name - but for some reason can not figure out how (or if it is possible) to change the Address#csv (which I'm calling the datasource name.
Just trying to save our dev staff some time from having to re-create very similar reports from scratch. We are using MS ACCESS/Jet to read the csv file (per NuView)
Any help is greatly appreciated.
From the Database Expert you can simply right click on it in the right hand pane selected tables and choose rename. Is that what you're looking for?