Managing source records after migration

Migrating records

Electronic systems may have a shorter life than the records they contain. Technology changes rapidly and systems regularly improve and update. When systems are superseded but the record is still required for business or operational purposes, the  records must be transferred to the new system in a process called migration. 

Migration creates a new version of a record but keeps the old version (source record) and in most cases, you don’t need to keep both versions after migration.

The General authority for source records that have been migrated (GA48) explains when source records can be destroyed after migration. This helps make sure the process is safe and reliable. 

What needs to happen before destroying old records?

Before destroying any old records public offices must:

• Manage the migration process:  Plan and document to ensure the process is organised.
• Meet requirements: The conditions of disposal of General authority for source records that have been migrated (GA48) must be met before any destruction is undertaken.
• Test before and after migration: Check that the new records are accurate, complete, easy to access, and usable.
• Keep source records for an appropriate time: Keep the source records long enough to confirm that the migration was successful.  Determination of the specific retention period must be based on an organisational risk assessment. 

Following these steps ensures the new records are trustworthy and can be used properly.

This advice doesn't apply to records that have been digitised. The disposal of paper records that have been scanned or digitised is covered by the General Retention and Disposal Authority: original records that have been copied (GA45).

Planning for migration

It’s important to plan carefully to ensure the records are authentic, complete, accessible, and usable. The migration must be planned properly for source records to be destroyed after migration.

To understand what is to be migrated, the plan should:

• understand user needs ,system usage and process
• identify the information that users input in existing fields and whether the uses of data fields are consistent
• identify the data to be migrated and any external data sources
• check the quality of data and clean if necessary
• consider destroying records due for destruction which are no longer required for business purposes. This reduces the quantity of records to be migrated.
• identify essential relationships between records and their hardware/software needs
• ensure metadata that supports records’ integrity is preserved, including structural and relational metadata
• exclude metadata that’s no longer needed for business purposes. 

Develop an appropriate migration plan

The migration plan should outline:

• data structure, volume, and quality
• migration strategy, including metadata mapping and format transformations
• testing and validation methods (before and after migration)
• risks, mitigation strategies, and recovery plans
• actions on source records and planning for decommissioning
• roles and responsibilities for migration tasks.

The migration plan is a key document, and any project changes should be reflected in it.

Maintaining authentic, complete, accessible and useable records

Successful migration operations will produce authentic, complete, accessible and useable records.

It is a condition for the destruction of source records that have been migrated that pre and post migration testing proves that authentic, complete, accessible and useable records can and have been migrated.

Migration testing

Pre-migration testing

Pre-migration testing helps ensure the migration plan works as expected. It involves testing the migration process on a small sample of records to check for errors. Relevant personnel should verify the records' completeness and accessibility.

Any issues found must lead to a revised migration strategy, which should then undergo further testing. Document the results of testing, which will inform the final migration plan. Once pre-migration testing is complete, the results and finalised plan should be validated as part of the migration project governance.

Post-migration testing

Post-migration testing checks that:

• all identified records have been migrated
• necessary functionality and essential characteristics are preserved
• users are satisfied with the migrated records.

Testing can be done through sampling, with the sample size proportional to the business risk. Document the results and validate them according to the project governance arrangements.

Retention period for source records

Public offices must keep the source records after migration to ensure the migration was successful. Retaining the source records helps identify any issues that might arise after post-migration testing. The retention period should be calculated after successful post-migration testing and the validation of any issues.

Risk assessment

The retention period for source records should be based on a risk assessment. 

Organisations should conduct this risk assessment in accordance with their internal risk management framework (as required by the Internal Audit and Risk Management Policy for the NSW Public Sector). 

The Risk Management Toolkit developed by NSW Treasury provides advice on risk management processes and includes risk assessment templates.

This assessment considers:

• the business purpose and associated risk of the records
• potential consequences of lost or corrupted records
• the complexity of the migration and capacity of the target system
• the nature of metadata not being migrated.

For high-risk records, it may be appropriate to retain source records longer. However the best way to guarantee that records are authentic, complete, accessible and useable is to ensure that the migration planning, testing and documentation is robust. Retaining source records for long periods of time will not substitute for performing those processes adequately. 

Incidental quality assurance

Public offices should also consider the extent of ‘incidental’ quality assurance when determining an appropriate retention period for source records.

If data is migrated and then immediately and frequently used in business as usual processes, anomalies and unforeseen issues with the migrated data are more likely to be found, and found within a shorter period of time. In these scenarios, and subject to the findings of the risk assessment, it may be appropriate to retain source records for a short period of time.

Understanding user behaviour and requirements is critical to successful migration projects. If the business is not involved in migration projects, key fields or functionality may not be migrated. In these scenarios, it may be appropriate to retain source records for a longer period so that the business has the opportunity to identify any anomalies or issues.

The migration of data between organisations following administrative change is a scenario where incidental quality assurance is likely to be delayed as users in the receiving organisation take time to use and understand the data, and identify any issues requiring rectification.

The migration of data between organisations is a high risk scenario. It is critical that migration processes are supported by a plan with agreed upon timeframes for quality assurance testing and retention of source records, and agreed upon sign-off arrangements before the destruction of source records by the transferring organisation can proceed.

See Managing records in administrative change for further advice. 

Retaining source records

The General Authority for source records that have been migrated (GA48) does not require source records to be destroyed after migration. Some public offices may choose to keep a reference copy for business needs or as a mitigation strategy in high-risk environments. However, if source records are retained, clear rules must be established to determine which version is the "official" record.

Documenting migration

The entire migration process must be documented to ensure the authenticity of records. 

Migration should be undertaken in accordance with best practice project management methodologies. 

Comprehensive records of the project should document: 

• the factors triggering the migration project
• the records being migrated and the business functions and activities they relate to
• the identified essential characteristics of the records
• comparisons between original system functionality and target system functionality
• technical requirements of the original and target systems
• all system configurations, including metadata definitions and mappings
• all decisions, including decisions not to migrate certain metadata components of a record
• risk assessments
• any disposal and data clean-up performed prior to migration, including records of decisions and the processes undertaken
• management approval of migration plans
• any variations to plans
• technical reports on the migration itself, including date and time of the migration and all personnel involved
• details of all testing, pre and post migration, including comprehensive test reports which confirm the target system is functioning successfully and that all expected data and metadata has been migrated
• any necessary variation in records structure, design, metadata, format or content that will result or has resulted from the migration. 

This documentation ensures the integrity of records, especially if needed for legal purposes. 

Retention and disposal

The Standard on Records Management requires NSW public offices to document the disposal of records (requirement 3.7). NSW public offices must document the destruction of source records and keep this documentation in accordance with the general retention and disposal authority for administrative records. This documentation might take the form of:

• metadata at the aggregate or system level which states that the previous version of this record group or system was destroyed on the specified date, by the identified authorised officer, in accordance with the conditions outlined in this authority
• metadata at a more granular level that documents the destruction of the previous version of specific files in accordance with necessary disposal requirements
• a final migration report that outlines the destruction of source records, providing enough detail to identify all records or groups of records that have been destroyed
• standard destruction approval forms used to authorise the destruction of records within the organisation.

NSW public offices must also document the transfer of records (e.g. to another organisation or, for State archives, to Museums of History NSW) and keep this documentation in accordance with the general retention and disposal authority for administrative records. This documentation must identify:

• the records that have been transferred
• where the records were transferred to
• the date of the transfer.