Principles for implementing metadata for records and information
The effective implementation of metadata for records and information should focus on identifying necessary metadata for authoritative records, ensuring it is scalable, well-documented, and linked persistently. Additionally, organisations should recognise that metadata evolves over time and should be treated as a record itself.
Principles for implementing metadata for records and information
Consider metadata in appraisal: When deciding which records to keep, your organisation should also determine what metadata is needed to create, capture, and manage authoritative records, as well as support business needs.
Scalability of metadata: Your organisation should decide how much metadata is necessary for different situations to effectively meet business requirements.
Standardised metadata: Metadata should be clearly described and managed using metadata schemas and encoding methods to ensure consistency and standardisation across records.
Dynamic nature of metadata: Recognise that metadata will continue to grow and change as records are used and managed over time.
Link metadata with records: Ensure that metadata stays linked with the records it describes, even when the records move to a new system or environment.
Manage metadata as a record: Your organisation should document how metadata is applied and managed in its systems, treating the metadata itself as a valuable record.
Metadata is everywhere in digital systems. Records and information management professionals need to identify what metadata is necessary for creating, capturing, and managing authoritative records, as well as what supports organisational recordkeeping and business needs.
Appraisal is the process of assessing business activities and work processes to determine which records and information should be created and kept, and for how long. The results of appraisal can help identify and define metadata requirements, including those for generating or capturing metadata and maintaining it during system changes over time.
Your organisation will likely need to make choices about what metadata to create, capture, and keep at key moments, including:
- system design
- system selection
- system implementation
- system configuration
- start of a service agreement
- system upgrade
- system migration
Example: Appraising metadata requirements when establishing contracts for cloud-based services
The NSW Government Cloud Policy states that when agencies set up contracts for cloud services, they must ensure the data's accuracy, integrity, and reliability for ongoing availability.
This includes provisions for safely returning or transferring data if the cloud provider changes ownership and what will happen to the data when the agreement ends (for example, transferring it to a new provider, returning it to the agency, or permanently deleting it). These provisions should consider what metadata needs to be returned with the content to show how it was managed and what processes were performed on it.
Metadata can describe records and information at different levels, such as:
- Individual records: For example, in a case management system, you might apply titles and unique identifiers to each document.
- Groups of records: In an EDRMS, you might apply retention and disposal rules at the file level to manage records related to the same matter or transaction effectively.
- Systems: You might apply access rules metadata to an entire system, such as a personnel or finance system, to standardise management actions and access control.
You need to determine which levels of metadata will best meet your organisation's various business needs.
Other uses of metadata
Metadata can also describe entities beyond records and information, such as:
- people
- workgroups
- organisations
- business transactions
- business activities
- business functions
- laws
- regulations
- policies
- business rules
This type of metadata provides context for how records and information are created and used, making it easier to interpret, understand, validate, trust, and reuse them. You should consider whether metadata descriptions of these entities would benefit your organisation.
Importance of metadata management
For metadata to be a key organisational resource, it must be clearly specified and well managed over time. Metadata shouldn’t be created randomly.
Using metadata schemas and encoding schemes helps ensure that metadata is meaningful, standardised, and consistent, which improves data quality and facilitates data sharing and reuse.
Metadata schemas:
- specify the metadata fields that will be used in the system
- provide a definition for each of the fields, indicating what can and cannot be applied within them
- identify the data that will be used to populate each field if metadata will be collected automatically from other systems
- identify the encoding schemes or ‘picklists’ that will be used to provide data values in the system if metadata collection cannot be automated.
Encoding schemes
Encoding schemes define the content of specific metadata values.
Encoding schemes:
- set out what information can be entered against a value (one or more of a list of set entries or numbers, or free text, or a combination)
- specify how that information is arranged
- define which symbols (for example, dashes, commas, colons) are used to separate the individual chunks of information in each value.
Use of encoding schemes:
- promotes standardisation, consistency and accuracy
- makes it easier for users to automatically apply metadata values
- makes searching and browsing easier and more efficient
- facilitates metadata reuse in other business areas
- enables automation of records and information management processes, e.g. disposal
- can be critical for enabling interoperability between systems
- contributes to system sustainability.
Good encoding schemes enhance a system's value and usability, so investing time and effort into their development is crucial.
Developing a metadata schema
If your organisation is creating a custom-built business system, you must define a comprehensive metadata schema tailored to your specific needs. For off-the-shelf software, develop a schema that clearly identifies and defines the system's elements for your business purposes.
When using a service-based system, collaborate with the service provider to identify key metadata that aligns with your requirements. It’s vital to understand and document the metadata within a service offering for effective records management during any transition out of that arrangement.
The following table highlights the consequences of standardising metadata creation and use in an EDRMS:
| Scenario | Organisation A implemented an EDRMS with over 500 possible data fields. Organisation A did not recommend standard elements for its staff to use when registering records. Organisation A did not define elements for staff or explain what each element should and should not be used for. | Organisation B implemented an EDRMS with over 500 possible data fields. Organisation B issued procedures to staff specifying the specific elements that should be used to register records. Organisation B developed business rules for the system that identified the specific relationships, access rules and disposal requirements that could be applied in the system. Organisation B employed a range of encoding schemes that identified the specific values that could be used in a range of fields. |
|---|---|---|
| Outcome | There was no standardisation in the data fields used by staff when registering records – for example, some staff had used the ‘Creator’ field while others had used the ‘Author’ field to capture information about a record’s creator; some used ‘Provenance’ while others used ‘Creating organisation’ to capture information about the controlling agency. Multiple options had been used for all commonly used metadata values. | Organisation B had clearly defined documents that outlined all the fields used in the EDRMS and rules that defined how these fields were used. Regular monitoring by records staff had ensured that staff complied with these rules. Data in the EDRMS was consistent and well defined. |
| Consequences | Migrating to the next generation of the EDRMS was impeded as simple metadata mapping and translations were impossible. A lot of data cleansing was necessary and multiple metadata mappings were required. A lot of the metadata could not be migrated because the costs of translating multiple fields and values into specific fields in the target system were considered too excessive. This has ongoing implications for the authenticity and useability of the records which could potentially have legal and cost implications for the organisation. | Migrating to the next generation of the EDRMS was facilitated by the relatively simple one to one mappings which could be achieved for all record types. There was minimal data cleansing and all data was successfully migrated from one system to another. |
Tip: Use or adapt existing metadata schemas where possible
Using or adapting an existing schema presents a range of benefits, including:
- the schema and its usage guidelines have already been developed, saving your organisation time and effort
- an existing schema is likely to have a community of users, which means there will be access to help and advice about how best to use it
- using an established schema facilitates interoperability between systems and in the event of administrative change.
AS/NZS 5478:2015 Recordkeeping metadata property reference set provides a reference set of recordkeeping metadata to support systems interoperability and records sustainability.
Additionally, State Records NSW has developed the Document Form Metadata Scheme which establishes common record types for NSW public sector organisations.
It is important to note that existing schemas may not specifically address all of your organisation’s identified requirements for metadata for managing records and information. Your organisation will need to adapt or add to these schemas to ensure that they include all metadata necessary for effectively managing your records and information.
Tip: Automatically harvest metadata where possible
Automating as much metadata capture as possible reduces the amount of time your staff must spend on manually entering metadata and enables more consistent metadata capture.
Example: Digital images
If an image has been created by a digital camera, it is likely that the camera has also written a certain amount of information about the digital capture into the file header. This may include information about the camera make and model, its settings and the date the photograph was taken. This metadata can be automatically extracted, saving a user from having to enter it manually.
Tip: Reuse metadata where possible
You can also improve organisational efficiency if metadata created for one business purpose is shared with other applications for other business purposes. Metadata reuse in this way saves money and can dramatically improve how business is performed in your organisation. Mapping and standardising your metadata is critical to enabling this type of functionality.
Example: Reusing personnel data
Human resource management systems contain information about the people who work in your organisation, including which business unit they work in and their position title. Your organisation could establish links between its human resource management system and EDRMS or ECM system to enable the EDRMS or ECM system to reuse some of this personnel-related metadata to provide context about who created records and the business context in which they were created.
Tip: Consider business needs when developing metadata schemas
The metadata schemas you develop need to be based on specific knowledge of the business that needs to be performed within the system. Talk to staff and perform system assessments so that you know what is currently done and so that you can gain an awareness of how things can be done better.
For example:
- Is information retrieval an issue? What metadata fields could be included to address this problem?
- Does the business unit need to run standard reports? What data elements are necessary to generate these?
- What system structure and functionality that can be enabled by metadata are required?
- What information access or restrictions need to be enabled across a system?
- What privacy management or data collection and management requirements can be met through metadata design and application?
- What metadata is needed to account for or authenticate transactions or to enable employees to understand where a process is up to? Can the identified metadata tell the story of how the information has been used, processed and acted upon across the organisation?
Metadata schemas do not have to be uniform across your organisation, as business needs may differ between business units. Remember to support individual business requirements wherever possible through the inclusion of specific metadata fields.
Tip: Ensure users understand your metadata schema
Systems are only as good as the people who use them. You could spend significant amounts of time and money developing standardised metadata schema for business systems but if people do not implement these as intended, all the efforts involved in their development will be wasted.
Explain the roles of the different metadata fields in your business systems to users and demonstrate how the particular encoding schemes that provide values for these fields operate. This type of work may be difficult and repetitive but it is a critical component of the effective operation of your systems and also for the long-term sustainability and integrity of the system.
Tip: Manage metadata schema and encoding schemes as records
Your organisation should regularly review metadata schema and encoding schemes to ensure they remain relevant and continue to meet needs. Versions of schema and encoding schemes that have been superseded should be retained and managed as records.
Keep your metadata up to date
Business environments are often subject to regular change. It is important to update your encoding schemes when:
- workgroups are renamed
- business processes change
- business locations change
- business rules change
- employees change roles.
However, existing metadata should not be changed when circumstances change. So if an employee changes roles, any metadata for records and information they created in their previous role should reflect their role at the time they were created.
If the metadata was changed, it would no longer provide an accurate representation of the business as it was when it was transacted.
Similar work processes may require different metadata based on the nature of the business being conducted.
Example: Different types of records and information need different contextual metadata
Systems that manage records and information, like electronic document and records management systems (EDRMS), enterprise content management (ECM) systems, web content management systems, and digital asset management systems, can be set up to capture various contextual metadata.
Your organisation will need to determine the specific contextual metadata it needs for different types of records and information.
For example:
- If your organisation keeps documents in an EDRMS or ECM system, you may need to capture information about who created or received the document, the date it was created, sent, received, and added to the system, who edited, viewed, approved, or deleted the document, and who had access at specific times.
- If your organisation manages images in a digital asset management system, you may need to capture information about who took the picture, what it depicts, when it was taken, what incident or event it relates to, who owns the picture, how it was used in business processes, what file format it is, and its size.
This metadata is crucial for understanding and using records and information over time.
Further guidance on implementing metadata in an EDRMS can be found in FAQs about EDRMS.
Your organisation can implement metadata at different levels; records of greater value or with higher risks will likely need more detailed metadata.
Your organisation should regularly review its identified metadata requirements as part of the ongoing appraisal process.