Delta load with Talend - talend

Question

I am new in using Talend.
I want to use delta load in my ETL.
I am extracting from Mysql datasource and loading into Postgresql database.
Mysql datasource has created_at and updated_at timestamps which I would like to use to extract new or update data.
I have already implemented this in Sql Server with SSIS before.
I am not sure how to implement with Talend.
Has anybody implemented delta load with timestamps using Talend?
Thanks in advance.

Answer 1

As you have the date filed to identify the delta, it wil be easy.
Have two files one to have the current date(when the job flow starts). Another file last run date with low date 19000101. In the first load run the job and read date from last run date file, have this as a where clause to check source data timestamp col>run date and run the job. Next move the current date file value to next run date file. In the incremental run again the same process. So you will get the delta records.

Answer 2

I did for one of the project.
First we will create one log table, columns like Job_id, Job_name, start_time, end_time, status.
whenever you job run once, you need to updated this table at the end of the job.
for next run, first we have to check when job last done successfully last time from that take last job start time and put into one variable.
In the below give condition to table input like
created_at > variable of last start time
see below image
or
updated_at > variable of last start time
check below image for job flow

Answer 3

A couple options exist -- this answer is NOT specific to timestamps, but can still be helpful.
You can use the built-in CDC directly for certain databases (outside of Talend), like SQL Server and Oracle. May not be relevant to your situation.
You can use the built-in CDC for certain databases (inside of Talend). Requires the subscription version. Includes MySQL, Oracle, DB2, PostgreSQL, Sybase, MS SQL Server, Informix, Ingres, and Teradata. https://help.talend.com/reader/4UeRbZs9GU5n8b9nm3hUrQ/8yztvpROOkQauOWwo_0twA
You can go through the manual method via SQL and either Java (create a custom routine) OR the included component, tAddCDCRow. This will involve the use of an MD5 hash value for the key and, or non-key values. * Always use a hash for the non-key fields.
A. Hash for key combination and include remaining fields in comparison.
B. Include in comparison all key fields and use hash for remaining.
C. Use one hash for key combination and another for remaining fields.
If using tAddCDCRow, use one component for the key combination and another for the remaining fields. Of course, if hashing only the key, only one component is necessary.
If using the custom Java function, call once or twice depending on needs.
Hash Java function:
// data = Text for which to generate hash value. Will be a combination of one or
more fields. Recommend padding strings with spaces to accurately combine. Can use |
or similar dividers. Convert integers and other non-text to strings, as necessary.
public static String getMD5(String data)
{
java.security.MessageDigest digest; // Message digest of type, MD5.
byte[] hash; // Byte array containing passed text converted to hash value.
// Create a message digest with the specified algorithm name -- MD5.
digest = java.security.MessageDigest.getInstance("MD5");
// Convert passed text to hash value -- as a byte array.
hash = digest.digest(data.getBytes("UTF-8"));
// Return the hash value converted to a string.
return javax.xml.bind.DatatypeConverter.printHexBinary(hash);
}
Here is a link to another code example: https://community.talend.com/t5/Design-and-Development/sha1-hash-key/td-p/109750.
Comparison: Compare the old and new information using the hash (and possibly other) fields. Use a full outer join on the key fields (or related hash) and the hash for the non-key fields. When a null is found on the new, but not the old, side, an insert is needed. When a null is found on the old, but not the new, side, a deletion is needed (or simply ignore, based on your needs). When no null occurs, perform an update.

Answer 4

Here an example:
Requirements:
Dim_Control (Job_Id, Job_Name, Table_Name, Last_Success, Created_Date)
CREATE TABLE Dim_Control(
job_id BIGINT IDENTITY(1,1) PRIMARY KEY
,job_name NVARCHAR(255)
,table_name NVARCHAR(255)
,last_success DATETIME2(0)
,created_date DATETIME2(0) DEFAULT GETDATE()
)
Contexts (Last_Success, Job_Name, Table_Name, Current_Run)
Steps:
1.Get last successful job name & date :
"Select job_name, table_name, MAX(last_success) as last_success
FROM Dim_Control
WHERE table_name ='Employee'
GROUP BY job_name, table_name;"
2.Write Log:
Using tLogRow component - You can prefer Table(print values in cells of a table)
3.Match Context Varable & Dim_Control Values
System.out.println("*** Job Name = "+input_row.job_name);
System.out.println("*** Table Name = "+input_row.table_name);
System.out.println("*** Last Success = "+input_row.last_success);
System.out.println("*** (Before) context last_success:" +context.last_success);
context.last_success = TalendDate.formatDate("yyyy-MM-dd HH:mm:ss",input_row.last_success);
context.current_run = TalendDate.formatDate("yyyy-MM-dd HH:mm:ss",TalendDate.getCurrentDate());
context.table_name = input_row.table_name;
System.out.println("*** (After) context last_success:" +context.last_success);
System.out.println("*** (After) context current_run:" +context.current_run);
4.Truncate Target Stage Table
5.Insert New records to Target Stage Table:
"SELECT distinct *
FROM dbo.Source_Employee a WITH(NOLOCK)
WHERE FORMAT(ISNULL(a.UpdateDate, a.CreatedDate),'yyyy-MM-dd HH:mm:ss') >= '" + context.last_success +"' OPTION (MAXDOP 32);"
6.Insert New Successful Job Info to Dim_Control
"INSERT INTO Dim_Control (job_name, table_name, last_success)
VALUES ('"+context.job_name+"', '"+context.table_name+"', '"+context.current_run+"' ); "
7. Merge Stage & Main Target Table
"MERGE dbo.Main_Target_Table t1
USING dbo.Stage_Target_Table t2
ON t1.Id = t2.Id
WHEN MATCHED
THEN UPDATE SET Id = t2.Id, Name= t2.Name
WHEN NOT MATCHED BY TARGET
THEN INSERT ( Id, Name ) VALUES ( t2.Id, t2.Name);"
Merge shouldn't include Delete Part & Id should be Primary Key
All contexts type are String
Default Talend Date Format: Thu Jun 28 00:00:00 EET 2021
Also you can watch Rohan's Video
Workflow of ETL