The ability to capture only the changed source data and to move it from a source to a target system(s) in real-time is known as Change Data Capture (CDC).
Here is the method used to capture the CDC without using CDC components in Talend and by comparing the source and target to capture the inserts, deletes, and updates individually.
Below is an example of how inserts,updates and deletes are captured.
Considering source to be a transactional data source and target to be a target table which captures the changes in the source.
- Capture Inserts
- Capture Deletes
- Capturing updates using inner join between source and target :
Capturing Inner Join rejects from join of source and target.
Here the source becomes the main link.
Source consists of 4 records. And target consists of 3 records, which implies that there is one new record in the source which is an insert.
Tmap settings for capturing the inserts
Output from the above step
Deletes are obtained by performing an inner join between target and the source and capturing the rejects.
Here, the target becomes the main link.
logic for deletes
Logic for deletes
Output from the join to capture reject :
For capturing the updates a CRC component is used, which generates a CRC bit based on the columns which are subjected to change. Any change in the columns selected will also change the CRC bit. Hence this CRC bit is also compared to identify the update.
Logic for updates
Settings in CRC component :
Logic to fetch the updates in TMap
Output for capturing deletes
In case if you have any queries please get us at email@example.com
Helical IT Solutions Pvt Ltd
Best Open Source Business Intelligence Software Helical Insight Here
A Business Intelligence Framework
Best Open Source Business Intelligence Software Helical Insight is Here