Data migration is one of the most common database related task. In this blog we will talk about the best practices which should be followed while doing any data migration project
Data Migration ::
It is a process of moving data from one platform to another platform. This is one of the most common task with databases with setting up of new servers, new databases, new applications, load balancing, HA etc. For a typical data migration project, for ease, any ETL tool can be used (can be open source or proprietary).
The different stages of a typical data migration project is highlighted below
Step 1: First step is taking data from the source stage (which could be ERP, flat files, any type of database etc) and putting into the database. ER model is generally not changes. At times, additional metadata information, timestamp information can be added.
Step 2: Data moved from stage 1 to stage 2 (only those columns are moved which are necessary). Filtering is applied, duplicates are removed, erroneous data is removed. All the logs should be maintained of this step.
Step 3: In this step, we should map the data from old ER model to new ER model. All actions should be logged and metadata should be transferred. Extreme caution should be taken while defining data types just to make sure that no data is rejected because of data length.
Step 4 : All data translations and calculations take place here. Data is translated to meet target system rules, implement functional requirements. Metadata and SID are maintained, all the logs and audit information should also be maintained. Data translation information is maintained in data dictionary.
A check should also be put at this stage to make sure that there is no data duplication happening, no data type error is present etc.
Step 5 : Referential integrity check to be performed to make sure that data structures are completely ready to be written to the target system.
Step 6 : This is the final step. In case if the data is to be shown to outside world in the form of any API, that is created here.
Please get in touch with us at nikhilesh@helicaltech.com for any data migration or ETL project.
Nikhilesh
Best Open Source Business Intelligence Software Helical Insight is Here
It’s interesting to learn more about data migration. I didn’t know that it was used each time you moved servers or databases. The place I work at uses computers everyday, so I wonder when the last data migration we did was?