New article
Recently updated
Understanding data conversions
Who is this article for?
Data Administrations migrating information into the system
Begin preparation for data migration during the initial stages of requirement gathering of the project. Failure to do so could impact the project timeline.
Data conversion is the process of moving historical data from a customer’s existing system or application into a new application. The data conversion process identifies what information needs to be migrated into the system from initiate through to close, including in-flight data, which is data added during various work steps.
Careful consideration should be taken to discuss the conversion of data considering the following for better operational efficiency and decision-making:
- Compliance with regulations
- Company policies and archiving needs of historical data, etc.
- Trending capabilities and leveraging the capabilities of modern technology
- Reporting
- In-flight data
- Expectations
- Additional discussions with data experts who understand the subscriber's data structure
Metadata Conversion Process
Metadata refers to data that provides information about an object or record, such as a title, creator, creation date, location, and more. During a data conversion, metadata headings are matched to module definition fields.
| Process | Subscriber | Ideagen |
|---|---|---|
| Collection of information | Provide Raw Metadata Samples (CSV) | Provide Module Definitions |
| Mapping Documentations | Mapping Document (Excel) | The Mapping Document headers are converted to a JSON script that will gather the customer’s information and transfer it into the applications using a REST API. |
| Documentation Validation | Requires communication between both parties to create documentation | Requires communication between both parties to create documentation |
| UAT Script | None | Internal Development |
| UAT – Initial Validation | Testing and approval (Subset) | Internal validation |
| UAT – Small Sample Validation | QA testing and approval | Modifications and reload data, if necessary, until approved |
| PROD - Load | Script testing and approval (Subset) | QA Validation |
| PROD - Load | Script testing and approval (Remaining Data - Newest to Oldest) | QA Validation |
Mapping documentation
- Customer Raw Data: Collection of the Raw Data helps define the scope of the project as well as the previous data structure
- Total number of records: this will identify how long it will take to convert the data
- Number of workflows per type of objects: this defines the complexity of the module
- Number of references: this identifies the complexity of the script
- This may require multiple (CSV) files
- Missing reference data such as HR/Person data needs to be clarified
- Relationship within objects: this identifies the complexity of the script
- Total number of records: this will identify how long it will take to convert the data
- Mapping Documents: Mapping Documents establish a contract that describes how the data will be migrated and are used during the validation process to identify if an issue is due to an error within the script or a mapping error.
- Module Definition rules: Required to ensure that the business logic within the platform will not modify the metadata. The module definition should consider the data migration needs during the initial development of the module.
Verification rule changes
Changes to module rules during the end stage of DVT or UAT testing may require a rewrite of the mapping script and delay project timelines.
- Environments:
- The Data Management team will initially work with sample data during the initial mapping validations
- For the UAT - Small Sample Validation the Data Management team will work with the subscribers to establish a data migration access plan.