Report on the State of Government Recordkeeping 2021/22

By the numbers

• 37: public offices’ information management practices audited

• 226: public offices and local authorities invited to participate in the annual survey

• 94%: response rate to the annual survey

Compliance: a snapshot

• 15: active cases at 1 July 2021 (carried over from 2020/21)

• 33: incidents that required assessment against the Public Records Act 2005 (the PRA) (in addition to those from 2020/21)

• 20: assessments closed with satisfactory or required remediation issues raised with the organisation concerned

• 7: assessments on hold requiring ongoing remediation or support following resolution of the initial issue

• 6: assessments continuing at the time of this report

Highlights and Trends

• The PRA Audit Programme (the Audit programme), part of our Monitoring Framework, shows organisations are still developing in the area of disposal. This has been an ongoing trend over recent years. Capacity and capability needs to improve in Te Rua Mahara and in regulated parties. While most public sector organisations have moved towards a modern digital information model, few have established robust digital disposal systems and Te Rua Mahara is not yet taking digital transfers systematically or at scale. This is resulting in terabytes of information across the sector being maintained regardless of its perceived value to the organisation. Disposal through transfer of physical records is also stalled while new Te Rua Mahara storage is being developed. Internal capability and expertise in information management continues to be an issue for organisations, with both the Survey of Public Sector Information Management 2021/22 and Audit Programme uncovering deficits across the public sector.

• Information management software solutions and upgrades are also an issue facing the sector. More organisations are moving to the Microsoft 365 Enterprise suite. When configured well it is a robust information management system, but the ‘evergreen’ and constantly evolving nature of these products require active oversight and vigilance to ensure that the system configurations remain relevant and effective for the user organisation. The Microsoft 365 suite should be adopted with careful configuration to ensure compliance with the Information and Records Management Standard (16/S1) (the Standard) and the PRA. Te Rua Mahara provides guidance to organisations on how to use the software and meet their obligations, with best configuration, correct license agreements and integration with electronic document and records management systems or enterprise content management systems.

Collections Search

Te Rua Mahara is acutely aware there have been performance issues with its Collections Search system since it went live in February 2022, replacing Archway, which needed replacing due to age and a lack of ongoing technical support.

The intent of Collections Search is to make it easier for our customers to access collections, and we accept that this has not always been the case. We remain committed to working with our provider to improve Collections Search to ensure a fully functioning and positive, on-going customer experience as the system is rectified.

For as long as there have been official governments and magistrates, there have been documents that required conserving and archiving. The Romans established its first public archives in 509 B.C., however it wasn’t until 1794 that the first legislation surrounding the organisation and conservation of archives was established (in France). The United Kingdom created a Public Record Office in 1838 formalising its long history of archival practices beginning with the Master of Rolls in 13th Century England.

New Zealand’s formal archival practices are relatively young in comparison. The Archives Act 1957 established the National Archives, within the Department of Internal Affairs, and a regulatory framework for government archives. Ensuring effective current information and records management across the public sector was not a major part of this legislation. Over the next four decades, technology developed, and legislation and recordkeeping practices were refined. The importance of public access to government information to supporting accountable government was also embedded, notably through the Official Information Act 1982. In 2005, the Archives Act 1957 was repealed and replaced with the Public Records Act 2005, the Act that governs the work of Te Rua Mahara.

The 2005 legislation established the role for the Chief Archivist as the regulator of information created by the public sector. Te Rua Mahara seeks to ensure that information is created and managed well, so that it supports transparency and accountability, the shift to digital government, and the rights and entitlements of New Zealand citizens.

Te Rua Mahara remains the guardian of New Zealand’s public archives. We oversee the government recordkeeping framework, and support and monitor the public sector to ensure each organisation complies with its obligations under the PRA.

Our name in Te Reo Māori, Te Rua Mahara o te Kāwanatanga, means “the repository of the thoughts and memories of government”, and this is exemplified in the range of materials we store and protect that include the Treaty of Waitangi, government documents, maps, paintings, photographs and film.

What the Act does

The PRA provides a regulatory framework for information management across the public sector. Under the PRA, government and public sector organisations are required to create and maintain full and accurate records of their activities. This supports government business and helps to ensure accountability and transparency.

The PRA also establishes the statutory role and duties of the Chief Archivist, which include:

• leading information management practices across the public sector

• setting standards for public sector information management

• authorising the disposal of records when they are no longer required for business purposes

• advising and supporting organisations so they can comply with the requirements of the PRA.

Who the Act applies to

The PRA applies to around 3000 public entities, including school boards of trustees (which make up around 2500 of these), to local authorities, public service agencies, tertiary education entities, Ministers of the Crown and State-owned enterprises. The services delivered and functions performed by these entities varies widely, as does the range and complexity of the information generated.

The importance of our role as regulator

We are the government regulator, under the PRA, of information created by the public sector. With the shift to digital technologies, our role is even more important as we assist public sector organisations to understand best practice methods of digital records creation, accessibility, preservation and archiving.

The Chief Archivist issues standards for public and local authority records under section 27 of the PRA. The Information and Records Management Standard is mandatory across the public sector.

Our Monitoring Framework helps us to understand information management practice across the public sector. It includes an Annual Survey of public sector information management, the Information Management Maturity Assessment, and our Audit Programme (providing a point-in-time view of information practice within an organisation). This framework helps us to focus on the right areas of assistance, guidance and intervention.

Introduction

The audit programme of Te Rua Mahara is a cornerstone of our Monitoring Framework, complementing the annual survey of public sector information management. Audit provides a point-in-time snapshot of the state of recordkeeping of individual public offices. The audit identifies the maturity levels of an organisation’s information management practices and generates actionable recommendations for improvement.

2021/22 was the second year of our refreshed audit programme with audits of 37 public offices conducted under section 33 of the PRA between July 2021 and June 2022. The COVID-19 pandemic lockdowns forced Te Rua Mahara and our auditors, Deloitte and KPMG, to vary the approach to our audit programme, with audits successfully undertaken by remote engagement in some cases. Most public offices were well set up to provide information electronically and meet with our auditors virtually. The move to digital information management has reduced the number of physical records held within public offices. Since the previous audit programme finished in 2014/15, many organisations now do not have significant amounts of physical records on site – with the exception of District Health Boards.

This year saw the final audits of individual District Health Boards within the PRA audit programme. With the transformation of the health sector, they have now been superseded by Te Whatu Ora - Health New Zealand and Te Aka Whai Ora - Māori Health Authority. Both these new entities are subject to the PRA and will be added to future audit schedules.

Audit scope and methodology

Although the role of Te Rua Mahara as regulator includes 3000 public offices and local authorities (the total skewed by the c. 2500 Schools Boards of Trustees), the scope of the audit programme is narrower. Local authorities are not included in the audit mandate under the PRA. The current programme is not yet sufficiently resourced to extend audits to schools and Ministers of the Crown.

Overall, the audit programme covers more than 200 public offices. The key audit contact within each organisation is the Executive Sponsor, the person in the organisation who has strategic and executive responsibility for overseeing its information and records management. Ahead of an audit, organisations are asked to complete the Information Management Maturity Assessment. This self-assessment gives an understanding of the areas and questions that will be covered in the audit.

PRA audit process

The audits are based on the requirements of the Standard and the PRA, as represented by the Information Management Maturity Assessment.

The audit programme assesses the maturity of information management practice across 20 topics grouped into eight categories: governance, self-monitoring, capability, creation, management, storage, access, and disposal. Using this assessment provides a basis for workable recommendations for improvement to be made and measured. A simplistic compliance “pass or fail” approach would not be helpful, given the complexity of information management in the government context and the widely varying size, functions, and structures of public offices. Measuring maturity across a broad range of important components allows targeting of areas for improvement.

Independent auditors conduct onsite and/or virtual audit activities including interviews, focus groups, and storage assessments of physical and digital records (as appropriate). The auditors meet with the Executive Sponsor, specialist staff, general staff members and contractors.

Maturity trends in 2021/22

Audits assess the maturity level of each organisation’s information management practice in each of the 20 topics. The maturity ratings are Beginning, Progressing, Managing, Maturing, and Optimising. Managing is broadly meeting the minimum requirements expected under the mandatory Standard.

Of the 37 public offices that were audited in the 2021/22 year, 19 audit reports had been released at the time of publication. The other 18 audits had completed the onsite audit component during 2021/22 and were moving through final report development. Within the 19 published audits, just three organisations achieved an Optimising maturity rating across any topics.

From audits finalised so far this year, the Commerce Commission of New Zealand emerged as the standout performer achieving a rating of Managing or higher for 17 out of 20 topics (85%). Most organisations, however, are operating at the Beginning or Progressing levels and, for many, significant improvements are required to achieve a higher rating for their management of information and records.

Maturity by category and topic

For the 2021/22 audits released to date, the most mature topic is once again storage requirements, with 47% of organisations at the Managing level with appropriate storage requirements, 16% of organisations are at Maturing and just one organisation at Optimising. This is closely followed by the information management policy and processes topic from the governance category with 47% of organisations at Managing and 11% at Maturing.

Disposal topics continue to be problematic for most organisations audited, with 74% only at Beginning level in their transfer of information to Te Rua Mahara. 68% at Beginning when implementing disposal and 58% with no current organisation-specific disposal authority. This is to be expected. Systems, processes and regulation to support lawfully correct disposal of digital records at scale are still not yet embedded across the system. Physical records transfer to Archives is also restricted while new storage is developed.

Governance

Within the Governance category, Te Tiriti o Waitangi (Topic 6), outsourcing functions and collaborative arrangements (Topic 5) and information management strategy (Topic 1) were found to be the most challenging for those audited. Most organisations rated as Beginning or Progressing in each topic.

In contrast, the organisations audited have had more success in establishing information management policies and processes (Topic 2) and in integrating information management into business processes (Topic 4) with most achieving a maturity rating of Managing or above.

Self-Monitoring

The 2021/22 audits continued the trend seen last year with most organisations achieving a Beginning or Progressing maturity rating for self-monitoring. As there is at least a five year gap between audits, it is imperative that organisations become more accustomed to self-monitoring their information management. The introduction of the Information Management Maturity Assessment tool should assist with this.

Capability

Internal capability continues to be an issue for many organisations. Information management is an area that requires specialist expertise, yet too few organisations have specialist staff. The 2021/22 audit found most organisations were at Progressing in their information management specialist capacity and capability (Topic 8) and in general staff awareness of their information management roles and responsibilities (Topic 9) at (47% in both topics).

Besides employing specialist staff, smaller less complex organisations can improve in this area by upskilling existing staff or seeking external specialist information management services.

Creation

While 42% of organisations achieved maturity ratings of Managing or Maturing in their creation and capture of information (Topic 10), the related high value / high risk information (Topic 11) appeared to be an area of concern for the organisations audited in the 2021/22 year, continuing the trend from last year. More than half the organisations were rated in the Beginning or Progressing stage of identifying high value and high-risk information. This is a crucial area of analysis for organisations as they improve the way information is managed within their organisations and one that requires senior leadership and executive oversight.

Management

Most organisations were at Progressing for requirements built into technology systems (Topic 12), information maintenance and accessibility (Topic 14) and business continuity and recovery (Topic 15). As with last year’s audits, the integrity of information (Topic 13) was an area where most organisations were rated Managing or Maturing with information management controls in place to maintain the integrity, accessibility and usability of information.

Storage

Appropriate storage arrangements (Topic 16) had the highest maturity level of all topics last year. In the 2021/22 audits, 68% organisations were rated Managing or above in their storage arrangements, with just under a third of organisations Progressing in this area. Good storage is important for information protection and security. Appropriate storage arrangements for both physical and digital information means that information remains accessible and usable for as long as it is required for business and accountability purposes. It also complements the implementation of disposal.

Access

Ongoing access to and use of information allows staff to do their work and the public to hold government accountable. It is expected that public offices will have mechanisms for finding and using information efficiently. Information and data sharing between public offices and external organisations should be documented in specific information sharing agreements that are monitored.

The 2021/22 audits found just over half the organisations were at Progressing maturity for information access, use and sharing (58%). Only six organisations were rated at Managing maturity or above. Not one organisation audited had reached Optimising maturity in this topic.

Disposal

As with 2021/21 audits, disposal featured prominently as an area for improvement across every organisation audited in 2021/22. Government organisations have access to the General Disposal Authorities GDA6 and GDA7 and are also expected to have disposal authorisation for their organisation’s core functions. More than half the organisations audited were at Beginning maturity for a current organisation-specific disposal authority (Topic 20). Just a quarter of organisations achieved a maturity rating of Managing. There are multiple causes of low disposal maturity, including the suspension of physical transfers to Te Rua Mahara while new storage capacity is developed and the ongoing challenges of digital transfers. Despite these barriers, the implementation of other disposal activities, notably authorised destruction, is still beneficial.

In keeping with last year’s results, the maturity ratings for implementation of disposal decisions (Topic 21) and transfer to Archives (Topic 22) were the lowest across all topic areas, reaffirming that disposal using GDAs or the organisation-specific disposal authorities is not happening.

Figure Note: The topic Transfer to Archives New Zealand was not applicable in the audit of the Real Estate Agents Authority.

Maturity by public office

Best Practice: Commerce Commission of New Zealand

The Commerce Commission of New Zealand (the Commission) stood out among those public office audits completed to date in 2021/22 for its high levels of maturity, particularly in governance.

The audit of the Commission involved reviews of selected documentation, interviews with selected staff, including the Executive Sponsor, information management staff, the information technology team, and a sample of other staff members from various areas of the Commission. Due to COVID-19 restrictions at the time of the audit, KMPG conducted the audit remotely rather than in-office.

The Commission implemented its comprehensive information management strategy in June 2021. This was developed by the Executive Sponsor and a dedicated information management governance group, the Information Service and Security Governance (ISSG) Group. The strategy considers the implications of organisation-wide risks, which are documented in the Information Security Operational Risk Register. The ISSG receives monthly reports from the Knowledge and Information team that highlights and discusses information management issues and initiatives and provides support in addressing issues as they arise.

Information management is considered an organisation-wide role and capacity is considered as part of the Commission’s annual budget process. Responsibilities are clearly communicated to all staff and contractors through induction and refresher training and are further discussed as part of staff performance reviews.

The Commission’s audit reinforced the important role of the Executive Sponsor in promoting information management within an organisation. The auditor found that the Executive Sponsor actively promotes the value and importance of information management to the senior leadership team and the wider organisation and creates opportunities for the Knowledge and Information team to communicate directly with the organisation. Staff interviewed noted that the Knowledge and Information team are considered trusted partners throughout the organisation. A crucial point of difference was the level of networking undertaken by the Commission’s Executive Sponsor, meeting regularly with Executive Sponsors from other organisations to share experience and knowledge in information management.

Note: The topic Transfer to Archives New Zealand was not applicable in the audit of the Real Estate Agents Authority.

Follow-up after audits – 2021/22

Improving information management within an organisation is a continual process and there is a huge variance in the maturity levels of New Zealand’s public offices. It has only been two years since Te Rua Mahara reintroduced the PRA audit programme and introduced the Information Management Maturity Assessment. Feedback to date has been positive with public offices reporting the experience motivated them to reflect on and improve their practices internally.

The process

Te Rua Mahara provides follow-up with organisations audited within the year. After an audit, an organisation receives a final audit report and letter from the Chief Archivist that prioritises areas for improvement based on the auditor’s recommendations. A follow up discussion about the audit process is also offered.

Each organisation is then tasked with producing an action plan that sets out a schedule of activities that the organisation will undertake to address these priority recommendations. The organisation has six months to submit the action plan to Te Rua Mahara.

Te Rua Mahara meets with each organisation six months after the action plan is produced and then after another year to assess improvements made based on the recommendations and to assist with implementing changes. These ‘check-ins’ conclude two years after the organisation’s audit.

Public offices audited 2020/21

Of the 31 audited organisations (30 audits as Wellington Institute of Technology and Whitireia Community Polytechnic were audited together) 26 action plans to address the prioritised recommendations have been received.

The Health Research Council, New Zealand Film Commission and Drug Free Sport New Zealand are yet to submit their plans. The Waikato District Health Board is no longer a separate public office and will not be providing a plan.

Public offices audited 2021/22

37 organisations were audited and to date 5 action plans have been received with none overdue.

The action plans are followed up with meetings after six months to discuss activity against the prioritised recommendations. These meetings are progressing to plan with many coming due in the later part of this year. Generally, organisations are engaged and working through their actions as scheduled.

In 2021/22, Te Rua Mahara conducted its fourth survey of information management practices in public offices and local authorities. We surveyed 226 public sector organisations and had a 94% response rate, with only the 12 organisations listed below not responding. Public offices are required to respond by a Direction to Report under section 31 of the PRA. As section 31 does not cover local authorities, these are invited to complete the survey. Survey questions were derived from the monitoring criteria from our Monitoring Framework and questions related to risks, challenges, opportunities and emerging trends in the field.

Data and analysis of survey responses can be found in the separate Findings Report. The report includes four sections:

1. Key indicators

2. Governance, capability and self-monitoring

3. Creation and management

4. Disposal

5. Information management environment

Highlights from this year’s responses include:

• 61% of responding organisations had formal governance groups for information management (1% increase from 2020/21 survey)

• 60% of responding organisations had built information management requirements into their business systems (52% in 2020/21 survey)

• The most common response among responding organisations was having either one or less than one staff member dedicated to IM. Organisations with fewer than 100 FTE commonly have no IM staff.

• Identifying high-value and/or high-risk information has remained generally static in the last three years.

• 39% of responding organisations reported they had identified records with te ao Māori value (compared to 35% in 2020/21).

• 57% of responding organisations reported doing authorised destruction of physical and/or digital information compared to 56% in 2020/21.

Raw data from the survey will be published on data.govt.nz.

List of non-respondents (A-Z)

• Grey District Council

• Kawerau District Council

• Manawatu District Council

• Matamata-Piako District Council

• Napier City Council

• New Zealand Forest Research Institute Limited (SCION)

• Northland Regional Council

• Southland District Council

• Te Wānanga o Raukawa

• Timaru District Council

• Wairoa District Council

• Waitaki District Council

Compliance trends in 2021/22

Te Rua Mahara seeks to ensure New Zealand’s public offices and local authorities comply with the PRA and maintain robust, accountable and transparent recordkeeping. We champion information management best practice and providing guidance and advice on good recordkeeping.

How we find out about potential non-compliance issues

There is a variety of ways that Te Rua Mahara may be notified about a potential breach of the PRA or the Standard. These can include:

• self-reporting from a public office and / or local authority

• indirect - notification from Te Rua Mahara staff members working in other areas

• media reports and proactive monitoring / environmental scanning by Te Rua Mahara

• referrals from external regulators, such as the Office of the Ombudsman Tari o te Kaitiaki Mana Tangata.

We assess each case of potential non-compliance against the Standard, and the requirements of the PRA. Cases of non-compliance offer an opportunity for business improvement. Our response is not punitive.

We continue to work closely with the Office of the Ombudsman Tari o te Kaitiaki Mana Tangata on issues. Under section 28(6) of the Official Information Act 1982 and section 27(6) of the Local Government Official Information and Meetings Act 1987, the Ombudsman may notify the Chief Archivist when an information request has been refused by an organisation for reasons relating to information management. The relevant teams in our organisations work together to ensure coordination of what are often overlapping issues.

Over the 2021/22 year we followed up on 33 new incidents, and continued to assess 15 active cases from the 2020/21 year.

Non-compliance themes

2021/22 were:

• failure to create and maintain records

• access to information

• missing records

• unauthorised disposal.

When assessing the notified incidents, often more than one theme of non-compliance was involved.

Organisational change in the public sector is a constant. The country’s 16 Institutes of Technology and Polytechnics are being merged into the newly formed entity Te Pūkenga by 1 January 2023. The introduction of Te Whatu Ora – Health New Zealand and Te Aka Whai Ora – Māori Health Authority replaces the district health board model. Proposals are underway to create four publicly-owned water services entities under the Three Waters Reform Programme.

Information management during organisational change

From the perspective of Te Rua Mahara, organisational change requires careful consideration of information management, the control of transferred records and the management of disposal authorisations across the impacted public offices and local authorities.

Disposal authorities help public offices determine how to dispose of their information and records. Traditionally there have been different types of disposal authorities including general disposal authorities and organisation-centric or sector-based disposal authorities, which have been used to provide this disposal coverage.

When an organisational change takes place, a new disposal authority usually needs to be established. Each year staff at Te Rua Mahara assist public offices with documenting the transfer of records during functional shifts and establishing new disposal authorities as part of our regulatory function.

In an effort to help manage some of the disposal implications arising from organisational change, Te Rua Mahara began working on functional disposal authorities (FDAs). These represented a new disposal model in which similar functions of organisations are incorporated into a functional disposal authority that could be applied across multiple agencies. This FDA model was first piloted with the DHBs with the establishment of three broad FDAs to provide disposal coverage across their functions:

• FDA1 – Clinical Health Care

• FDA2 – Health Administration

• FDA3 – Populations Health and Wellbeing.

This has proved to have been timely with the disestablishment of the DHBs and the creation of Te Whatu Ora – Health New Zealand and Te Aka Whai Ora – Māori Health Authority taking place on 1 July 2022. This is the biggest health reform New Zealand has experienced in more than 20 years and has some significant implications for the future of information management across the health sector. However, rather than needing to develop a new disposal authority, Te Whatu Ora – Health New Zealand has been able to adopt the existing FDAs because the functions remain the same as they were when delivered by DHBs.

Changing our approach

The restructuring of the health and tertiary training sectors will likely see changes in the way Te Rua Mahara supports and audits these sectors in the coming years. This may influence our approach to other large, complex organisations with a distributed nationwide presence.

For example, Te Pūkenga – New Zealand Institute of Skills and Technology will have multiple sites nationwide and a complex inheritance of systems from previously distinct institutions. This will likely require some tailoring of the audit process.

The establishment of Te Whatu Ora – Health New Zealand and Te Aka Whai Ora – Māori Health Authority means that we will no longer audit district health boards. The breadth and complexity of the new organisations will require a distinct audit approach.

Te Rua Mahara is working closely with the new health entities as they work over the coming year to consolidate their records and evolve their IM systems to support national health outcomes and protect public health records. This work will inform our approach to supporting and auditing Te Whatu Ora and Te Aka Whai Ora in the future.

Te Rua Mahara has continued its critical role in enabling access to a vast amount of records for the Royal Commission of Inquiry into Historical Abuse in State Care and in the Care of Faith-based Institutions Te Kōmihana Karauna mō ngā Tūkino o Mua ki te Hunga i Tiakina e te Kāwanatanga i Tiakina hoki e ngā Whare o te Whakapono (The Royal Commission of Inquiry).

Te Rua Mahara took an early decision to enable access to information by digitising and providing digital copies of the archives requested by the agencies involved in the inquiry. In the 2021/22 year Te Rua Mahara dedicated a Royal Commission team that reviewed more than 2 million database entries likely to be of interest to this inquiry and assessed the classifications and access rights. Through the process, we became aware that some records transferred to Te Rua Mahara prior to the introduction of the Privacy Act may have had open access classifications. Accordingly, more than 700,000 of these records were reclassified to restrict access in line with modern privacy expectations.

Over 4000 records, comprising more than a million individual scanned images, were digitised for the inquiry. We have implemented a standardised approach for efficient document transfer and safeguarding information. Watch a video showing how we find, process, and securely provide information for the inquiry.

Te Rua Mahara is acutely aware that each piece of information relates specifically to a person’s real-life experience. It is not simply a case of ensuring compliant recordkeeping. We are a key part of the records design work underway by the Crown Response Unit – as detailed in the Cabinet paper entitled “Responding to the Royal Commission into Historical Abuse in Care’s redress findings – Immediate projects to improve survivors’ experience of seeking redress”. Established in 2019 within Oranga Tamariki, the Crown Response Unit oversees the Government’s response to the inquiry and coordinates agency responses. We are providing expertise to the Crown Response Unit’s complex design work around accessibility, support, and cultural requirements of diverse survivors.

We have listened closely to the evidence from the hearings facilitated by the Inquiry and reflected on the implications for recordkeeping. As members of the Crown Response Records Working Group, we are exploring how improved access to information can facilitate the creation of personal and collective narratives, truth telling, identify and empowerment. Information creation and management is regarded as a critical part of the care approach rather than simply a compliance exercise. Sometimes these records are the only place where survivors of the care system are able to source information about their backgrounds. The records are core to the survivors’ identity, connections with their families, and connecting with where they came from.

Ongoing disposal moratorium

The moratorium on the disposal of records held by all public offices that may be relevant to the inquiry remains in effect (General Notice). This was put in place in March 2019 under section 20 of the PRA.

The disposal moratorium has been and continues to be necessary to ensure that relevant records are protected and can be made available to the inquiry, reinforcing the Government’s expectations of agency transparency, accountability and co-operation.

Public offices have been given clear direction and guidelines to follow to ensure that no records that are required are inadvertently disposed of by what would otherwise usually be considered prudent business practices. Public offices have been able to seek approval from the inquiry for specific exceptions to the coverage of the moratorium.

The inquiry is set to conclude in 2023, however, it is expected that the moratorium will stay in place beyond this point. This will allow thorough consideration of the inquiry’s recommendation to review all disposal authorities covering public records relevant to care. It is essential that we understand what records are required now and into the future.

Pressure points for disposal

Currently all physical records are being retained in public offices. Te Rua Mahara is acutely aware that the moratorium on the disposal of records is putting pressure on the sector for storage. This is compounded by the fact that Te Rua Mahara is not currently accepting the transfer of records in physical format in Wellington, whilst work is underway to build the new repository space. Digital format transfers are also constrained, and the adequacy of digital storage infrastructure is being examined.

Te Rua Mahara has obligations and statutory responsibilities under the PRA, alongside those of the National Library of New Zealand under its legislation, to ensure the effective stewardship of New Zealand’s record of government, documentary heritage and taonga. At present, across the entire portfolio, these physical records total over 271,000 linear metres (271km) and are continually growing.

While Te Rua Mahara has physical storage facilities in Auckland, Christchurch, and Dunedin, it is essential that we develop a space in the Wellington region to ensure we have capacity to accept the transfer of archives in the future, particularly following the conclusion of the moratorium on the disposal of records. Despite digitisation, there will continue to be a need to hold physical records in perpetuity.

Regulatory capability and system uplift sought

Te Rua Mahara is seeking to improve its regulatory capability and the public records system. The main initiatives focus on the disposal system and on a set of strategy and operating model improvements for Te Rua Mahara as a regulator. Initial focus has been on the Appraisal, Disposal, and Implementation (ADI) redesign project. This seeks to fundamentally transform appraisal and disposal processes and to ensure their successful implementation by public offices. This will require Te Rua Mahara to both transform our service into a fit-for-purpose, digitally focussed service and to transform how we manage the remaining paper information in the coming decades.

The regulatory uplift project seeks a clearer operating model for Te Rua Mahara as a regulator and relevant performance measures, while clarifying our high-level and operational strategies. There will also be work on ensuring we are working effectively with other entities involved in the public records system across government.

Some progress was made on both initiatives during 2021/22, including engagement with key public sector stakeholders in the discovery phase of ADI. However, to ensure that constrained resources are used most effectively, ADI has been paused and work on regulatory uplift components will take priority in 2022/23.

Archives in the cloud: Exploring machine learning to transform digital services for agencies

Te Rua Mahara wanted to see if machine learning tools and hyperscale cloud capabilities could help to analyse information and solve other information and archival challenges that have arisen in the digital era. Te Rua Mahara received funding from the Digital Government Partnership Innovation Fund to carry out a proof of concept (PoC) from February to July 2022. The PoC aimed to test if it was possible to use these tools to:

1. Streamline the appraisal process, specifically whether auto-classification could determine the appropriate disposal authority to apply to information and records.

2. Identify material of importance to communities, specifically whether available tools could identify and surface information of interest to Māori.

We worked with agencies, technology partners (Microsoft and Amazon Web Services) and information management experts for the PoC. The aim was to test if machine learning and cloud computing tools could classify data in line with disposal authorities – the rules for keeping or disposing of information. We found that it was possible to do so with high accuracy.

The journey of a government record: from creation to archive

The diagram below sets out at a high level the journey of a record from creation to its disposal, including transfer to Te Rua Mahara.

Auto-classification solution overview: Amazon Web Services (AWS)

The processing pipeline was built on an existing open-source solution designed and built by AWS. It uses a number of AWS services to enable understanding of document content and to provide the user interface.

Auto-classification solution overview: Microsoft

Microsoft’s solution architecture was built on a series of existing tools and systems within the Microsoft ecosystem. The Microsoft solution imagines agencies using a Microsoft 365 or SharePoint environment to store and create their information. Files would then be copied or transferred into an Azure Storage data lake, before being managed and classified using Microsoft Purview and Azure Text Analytics tools.

PoC Conclusions

The PoC demonstrated the huge potential of machine learning tools to assist with the auto-classification of records and information at scale, and potentially the identification of information that is of interest to Māori.

Alongside this capability, we think there is a broad range of other exciting possibilities for using advanced digital tools to transform the way we work, to support agencies and ensure we can deal with the significant volumes of digital data coming our way.

structured to potentially support findability, interoperability, business efficiency and artificial intelligence. Ontologies (and other vocabularies) currently exist within the New Zealand public sector but until recently no work has been done to explore the potential of creating an all-of-government ontology.

The All-of-government ontology options paper in September 2021 was pivotal in progressing discussions on the topic. There was positive support for the initiative with more than 70 public offices and organisations were involved in the compilation of the report. Since its release, Te Rua Mahara has continued to work as a member of a cross-agency working group led by Statistics New Zealand on data system design which also factors in Treaty of Waitangi obligations.

With numerous public offices moving from legacy information management software and platforms to the M365 suite, it is an opportune time to review systems. The All-of-Government Ontology provides a potential standardised approach to bridging information management between the two systems and provides users with a more user-friendly method of describing terms across government departments.

In 2021/22 a demonstrator model was built using software built specifically for ontologies. Te Rua Mahara continues to work with partners on the next steps for this project.

Another key project has been exploring what a Māori metadata solution to support the needs and practices of Māori people in accessing and using the content held by Te Rua Mahara might look like.

The project’s genesis comes from a 2009-2011 proposal to create a set of Māori metadata elements that was paused while Archway, the database and public access interface used by Te Rua Mahara, was underway. The project posed several open questions, such as the use of controlled vocabularies and the source of values for fields such as Iwi and Hāpu.

In 2020 a concept brief was created setting out a framework (He Anga), objectives (Ngā Uaratanga) and process (Ngā Whakataunga) for the work. Over the past year the team created a Work Plan and reviewed previous metadata work looking at other indigenous models including Te Pūtē Routiriata o Taranaki, the community archive of Taranaki whānui; a Māori rendition of the Dublin Core Metadata Element set implemented by Te Puna Mātauranga o Aotearoa/National Library of New Zealand, and Digital Pasifik used by National Library of New Zealand in collaboration with the National Library of Australia. The team also completed an international literature review on engagement with ‘indigenous communities’ of Australia, Canada, and the United States of America. A research report was released internally in June 2021.The next phase of the project would be to engage with Māori stakeholders to determine and co-design the actual metadata solution. For example, some organisations use a standard reo (e.g. Ngā Taonga Sound & Vision), others work with dialects (e.g. Iwi projects), and a decision would need to be made about which direction to take.

At present, this phase is being included in wider plans along with those of our colleagues National Library and Ngā Taonga Sound and Vision. Te Rua Mahara has a clear approach to progress the initiative at a future stage, subject to resourcing, including the potential expansion of the Collections system to include Māori metadata fields.

Sharing expertise across the Tasman

Te Rua Mahara has long fostered a close and cooperative relationship with its Australian counterparts at the state and national archives of Australia.

We are a member of the Council of Australasian Archives and Records Authorities (CAARA). Over the past several years we have pooled resources and expertise to find better ways to ensure the preservation and accessibility of public records through the Australasian Digital Recordkeeping Initiative (ADRI).

The introduction of the cloud-based office productivity suite Microsoft 365 (M365) as an information management and recordkeeping platform has transformed the public sector both here and across the Tasman, with organisations now able to access more than 100 applications with one licence. A leading question posed by public sector agencies and services providers has been “is M365 compliant?”.

To answer this question, Te Rua Mahara released guidance for the public sector on their obligations under the Standard, and how to meet these obligations using M365. This includes assistance with configuration, ensuring the appropriate level of governance licencing, using third party add-ons for information and records management functionality, and improving organisational knowledge of the administrative applications and tools used for information management within M365. Read more.

In December 2021, ADRI and CAARA took this further with the release of “Functional Requirements for Managing Records in Microsoft 365”. The technical document is aimed at information management staff, information technology staff, and business analysts who are responsible for ensuring their technical systems are fit for their individual organisations. ADRI staff have since presented the document to several public offices around Australia with positive feedback.

In New Zealand M365 has transformed information management in the public sector, reinforced by the All-of-Government Microsoft, Cloud, Software and Services Agreement. Microsoft’s investment in cloud infrastructure on our shores with its Azure cloud platform means New Zealand will soon also have the ability to safely and confidentially store data and information within New Zealand.

This year, working closely with the Public Service Commission and the Office of the Ombudsman, Te Rua Mahara released “How to develop an OIA information search policy”. This guidance was recommended by the report of the 2015 then Chief Ombudsman, Dame Beverley Wakem, “Not a game of hide and seek - Review of government OIA practices”.

The recent “Ready or Not?” follow-up report by the current Chief Ombudsman, Peter Boshier, re-emphasises the complementarity of improving access to information and ensuring the foundation of information management and recordkeeping is in place. The Chief Ombudsman’s recommendations strongly align with advice and guidance and requirements from Te Rua Mahara for the public sector.

Public offices and local authorities are covered by the PRA. There are two instances where an organisation can be defined as a public office by legislation:

• if an organisation is defined as a public office in section 4 of the PRA

• if an organisation is otherwise defined as such in other legislation.

The types of organisations are that defined as public offices under the PRA include:

• all departments as defined in section 5 of the Public Service Act 2020 including departmental agencies and interdepartmental executive boards

• interdepartmental ventures as defined in section 5 of the Public Service Act 2020

• all Offices of Parliament as defined in section 2(1) of the Public Finance Act 1989

• all state enterprises as defined in section 2 of the State-Owned Enterprises Act 1986

• all Crown entities as listed in Schedule 1 and Schedule 2 of the Crown Entities Act 2004 including:

  • Crown agents

  • autonomous Crown entities

  • independent Crown entities

  • Crown entity companies

• All Crown entities as defined in section 7(1) of the Crown Entities Act 2004 including for example:

  • boards of trustees for state and state integrated schools

  • tertiary education institutions including colleges of education, polytechnics, specialist colleges, universities, and wānanga

  • Crown entity subsidiaries (dependent on control factors).

A recent example of PRA coverage being established through other legislation is Te Aka Whai Ora – Māori Health Authority. The Pae Ora (Healthy Futures) Act 2022 establishes this organisation as an independent statutory entity, which is not an entity type covered by section 4 of the PRA. However, the Pae Ora (Healthy Futures) Act 2022 specifically makes it “a public office for the purposes of the Public Records Act 2005”.

Local authorities include all regional councils and territorial authorities, council-controlled organisations, local government organisations, and council-controlled trading organisations.

View a list of definitions for terms used in this report.