Capturing CDC in Talend without using CDC Components

The ability to capture only the changed source data and to move it from a source to a target system(s) in real-time is known as Change Data Capture (CDC).

Here is the method used to capture the CDC without using CDC components in Talend and by comparing the source and target to capture the inserts, deletes, and updates individually.

Below is an example of how inserts,updates and deletes are captured.

Considering source to be a transactional data source and target to be a target table which captures the changes in the source.

1. Capture Inserts

Capturing Inner Join rejects from join of source and target.

Here the source becomes the main link.

Make data easy with Helical Insight.
Helical Insight is the world’s best open source business intelligence tool.

Grab The 30 Days Free Trail



Source consists of 4 records. And target consists of 3 records, which implies that there is one new record in the source which is an insert.


Source


Target

Tmap settings for capturing the inserts


tMap

Output from the above step



2. Capture Deletes

Deletes are obtained by performing an inner join between target and the source and capturing the rejects.

Here, the target becomes the main link.



logic for deletes


Target

Source


Logic for deletes

Output from the join to capture reject :



3. Capturing updates using inner join between source and target :

For capturing the updates a CRC component is used, which generates a CRC bit based on the columns which are subjected to change. Any change in the columns selected will also change the CRC bit. Hence this CRC bit is also compared to identify the update.

Logic for updates

Source

Target

Settings in CRC component :

Logic to fetch the updates in TMap

Output for capturing deletes

In case if you have any queries please get us at support@helicaltech.com

Rajitha
ETL Developer
Helical IT Solutions Pvt Ltd

Best Open Source Business Intelligence Software Helical Insight Here

A Business Intelligence Framework

Free
Download

Best Open Source Business Intelligence Software Helical Insight is Here

A Business Intelligence Framework

Free
Download