Report on the State of Government Recordkeeping 2017/18

E ngā minita, me ngā Kaiwhiriwhiri o te Whare Pāremata - tēnā koutou, tēnā - koutou, tēnā- koutou katoa.

Archives New Zealand Te Rua Mahara o te Kāwanatanga works to ensure effective, trusted government recordkeeping for the benefit of all New Zealanders.

We preserve and protect more than seven million official records, from 19th century treaties to 21st century documents and data.

Our goal is for all New Zealanders to easily access and use this taonga, connecting you to your rights and entitlements and stories – now and for the future.

My vision for Archives is succinctly captured in the title of this report – Kia pono ai Te Rua Mahara o te Kāwanatanga Enabling trusted government information. This, the eighth Chief Archivist’s Report on the State of Government Recordkeeping, draws on our long-term strategy, Archives 2057 Te Rautaki Rua Kāwanatanga 2057, to focus on the collaboration and co-design needed to ensure that the information created by government can be trusted. This report covers some of the steps we’ve taken over the 2017/18 year to deliver on our long-term vision and tells you a bit about what we have planned for 2018/19.

We don’t currently have a good picture of the health of the information management system across the public sector, and our focus for this year is to further develop our tools for establishing this baseline. We are keen to hear from those we regulate and those with a direct interest in our work at every stage, so that we can address the information management challenges we all face.

As well as working with our stakeholders and regulated parties, we work together across government. Our work with the Government Chief Digital Officer and the Government Chief Data Steward, as well the Office of the Ombudsman and other regulators, gives us a good foundation to build an all-of-system governance structure for information management.

In September 2018, the Minister Responsible for State Services, Hon Chris Hipkins, launched the reform of the State Services Act 1988. The reform proposes changes that aim to provide improved leadership, agility and adaptability for New Zealanders. This should see the public service operate better as one joined-up system to tackle the big, complex challenges facing New Zealand.

The Public Records Act 2005 (PRA) is a fundamental part of the successful implementation of an open government strategy, with sound information and records management being pivotal to the accountability of the public sector.

The importance of Archives New Zealand’s regulation of public sector information and records management in ensuring transparency and accountability in government has been formally recognised through a new commitment in the Open Government Partnership’s (OGP) National Action Plan 2018-2020. Over the 2018/19-year, a new monitoring framework and measures will be finalised, external engagement on the framework will take place and a new survey will be piloted. Once implemented, regulated parties and the public will have a transparent view of information management performance across the sector.

Our interactions with the sector this year have shown that while some organisations are actively managing their information, others need substantial guidance in managing digital records for enduring access and preservation.

Information is still being under-managed in shared drives and email inboxes, potentially undermining long-term information management and access.

The digital paradigm is no longer new. While the suite of apps and tools we all use now to conduct our work is convenient, it should be remembered that, where public records are concerned, records regardless of format must be captured and preserved to ensure accountability.

I want to thank my staff at Te Rua Mahara, our partners across government, and the organisations subject to the PRA for their work in upholding the integrity of government information over the past year. I hope you enjoy reading this year’s report.

Ngā mihi ki a koutou katoa.

1.1 How we identify issues

Records are a vital accountability and transparency measure and form the basis for significant decision-making.

When problems with information management frustrate accountability, the Chief Archivist’s regulatory role allows intervention. We find out about these problems through:

• our monitoring and compliance work

• our daily interactions with regulated parties

• complaints from directly affected or concerned members of the public

• information reported in the media or received as part of journalists’ investigations

• complaints from concerned third-party organisations

• referrals from the Office of the Ombudsman.

Our new monitoring framework will help us to systematically identify issues across the system.

1.2 Working with the Ombudsman

Archives New Zealand Te Rua Mahara o Te Kāwanatanga (Te Rua Mahara) and the Office of the Ombudsman Tari o Te Kaitiaki Mana Tangata (the Ombudsman) have formed a closer working relationship in 2017/18. While the number of cases is still small, the Ombudsman is a growing source of referrals about potential breaches of the Public Records Act and we have many interests in common.

The PRA intersects with several other Acts, notably the Official Information Act 1982 (OIA) and the Local Government Official Information and Meetings Act 1987 (LGOIMA). Some complaints made to the Ombudsman include instances where organisations cannot supply information because:

• despite extensive searching, it cannot be located

• it would require significant collation and research to be made available

• it is determined not to be held when it could be reasonably expected that the information should exist.

Under section 28(6) of the OIA and section 27(6) of the LGOIMA, the Ombudsman may notify the Chief Archivist when an information request has been refused by an organisation for these reasons.

The number of OIA and LGOIMA requests across the public sector has climbed steeply in recent years. With this trend, the number of complaints to the Ombudsman and subsequent referrals to the Chief Archivist appears likely to grow.

Example: Records for evidence

A successful example of this notification process comes from a case involving Christchurch City Council and Sport New Zealand. An individual, who had been seeking information about the rebuilding of sport and recreation facilities in Christchurch, complained to the Ombudsman that the organisations could not supply information about discussions at a 2012 meeting between the Mayor of Christchurch and the Minister for Sport and Recreation.

Although there were records to confirm the meeting had taken place, searches for records documenting the discussion proved fruitless and the information request was therefore refused.

Having received the notification from the Ombudsman, Te Rua Mahara requested information from the Christchurch City Council and issued a direction to Sport New Zealand to report under the PRA. Both organisations responded readily and undertook further extensive searches for records. Sport New Zealand was subsequently able to find records that its staff had made of the meeting and released these to the requestor.

These had not been migrated into a new information management system and were effectively inaccessible to Sport New Zealand and external requestors under initial searches.

Christchurch City Council had not made records of the discussion. The Mayor was not accompanied by council staff and, as the meeting was requested by Sport New Zealand, reasonably assumed that they would make a record, especially given no decisions were to be made. The Council also confirmed that it makes extensive efforts to ensure that elected members are aware of their PRA obligations and how to meet these.

In this case, both organisations were able to satisfy Te Rua Mahara that the core PRA obligation to create and maintain full and accurate records had been met. This case highlights, however, the challenges posed in maintaining ready accessibility to records once they have been made.

1.3 Understanding your PRA obligations

As well as picking up complaints notified by the Ombudsman and received from members of public, we have been proactive in instances where media attention has publicised recordkeeping concerns within the public sector.

Example: Public vs private records

In March 2018 a voicemail message was left for the then Chair of the Radio New Zealand Board by the then Minister for Broadcasting, Communications and Digital Media. The message related to evidence given at a Parliamentary select committee and was withheld when the then Chair claimed it was a personal record, having been received on a personal phone. Te Rua Mahara sought to use the provisions of the PRA to ensure that the voicemail was treated correctly as a public record. A range of responses, including prosecution, was considered and cautions were issued to the parties involved, with strong reminders of their obligations under the PRA.

This case raised two issues of concern about public recordkeeping practices. The first was the question of whether the voicemail was a public record; the second was the challenge public offices have in creating and maintaining records across multiple technologies or devices and in a variety of formats. The PRA is clear in its definitions that public records are information in its original form or otherwise, without limitation on format.

As a public sector we must rise to the challenge of preserving records in all their formats, so they can remain accessible for accountability.

It is immaterial that a voicemail was received on a personal phone if it relates to the conduct of the business of a public office. A message left for the Chair of a Crown Entity by the Minister responsible for that entity about the business of the entity is a public record under the PRA, whatever the number or device used.

In this case, the voicemail message should have been managed as a public record. As such, it should have been captured in the organisation’s business systems in its original form or an alternative format, such as a written memo with contextual metadata.

Organisations often do not have processes in place to capture staff communications if they are conducted on personal devices or through personal email accounts. We’re working on making our guidance on these issues clearer.

1.4 Records demonstrate openness and transparency of business practices

Organisations are required to ‘create and maintain full and accurate records in accordance with normal, prudent business practice’. This is to enable the Government to be held accountable. If information is falsified or recorded inaccurately or completely it does not meet this requirement.

Example: Use of pseudonyms in records

An incident covered by the media in 2017 publicised Ministry of Social Development Te Manatū Whakahiato Ora’s (MSD) use of pseudonyms by the Benefit Review Committee when communicating decisions on an applicant’s entitlement to benefits. It was MSD practice to withhold the actual identities of the committee members to protect them from threats and harassment. In this case, MSD maintained a record of the actual names of decision makers which constituted a full and accurate record.

This use of pseudonyms was brought to light when the complainant appealed the Committee’s decisions. The resulting court case continued into 2018/19 and was concluded when the High Court ruled that the use of pseudonyms in this context was unlawful.

As this matter went to court, Te Rua Mahara monitored the issue with MSD rather than taking direct regulatory action. The Courts found the use of pseudonyms to be improper. The maintenance of a full and accurate underlying record means that the PRA’s purpose of enabling government accountability can be sustained. We’ve worked with MSD to ensure their information management processes are resulting in full and accurate records.

1.5 Taking care of the present and the future

Implementing our strategy

Archives 2057 | Te Rautaki Rua Kāwanatanga 2057

Our long-term goals are about taking archives to the people, upholding transparency and building systems together. Several projects proceeded during 2017/18 to turn the strategy into a reality – we scaled up digitisation efforts, increased outreach, and piloted artificial intelligence and machine learning initiatives. Have a look at our machine learning case study on page 18 for an example of one of these pilots.

Influencing the state of government recordkeeping

This year, we developed a work programme focused on raising information management capability across the public sector and ensuring our regulatory approach is fit for purpose in an increasingly digital environment.

The engagement material to test the programme was structured around four themes, representing what information management practitioners in the public sector need from us to take our requirements and put them into practice.

Engagement with stakeholders will continue during the 2018/19 year and beyond, as we begin to design individual activities in the programme. Working with our stakeholders, we identified the top priorities as:

• developing a monitoring framework (including audit)

• transforming disposal

• implementing a new relationship management model

• clearly and explicitly demonstrating information management leadership

• getting ahead of the technology trends affecting IM, so that we can better advise practitioners on the challenges they’re facing.

1.6 Building on lessons learned

Part of our regulatory role is to maintain a comprehensive, publicly available view of information management performance in public sector organisations. When monitoring is done well, its outputs can support public confidence in the quality and stewardship of public sector information, and awareness of areas for improvement. They can also empower public sector organisations to lift their performance.

We don’t currently have an all-of- system view of the state of government recordkeeping, so the priority is developing an annual survey of the sector. This will help us understand in detail the issues we know the sector is facing, and feed into providing more effective advice and guidance.

The importance of our monitoring activities for supporting transparency and accountability in government has been recognised through a new commitment in the Open Government Partnership’s (OGP) National Action Plan 2018-2020.

Over the 2018/19 year, a new monitoring framework and measures will be finalised, external engagement on the framework will occur and a new survey will be piloted.

This is with an aim to having audits recommence in 2019/2020 or 2020/21 depending on the availability of development resources. We will make audit and survey results available, so that the New Zealand public can more readily understand the state of government information management.

2.1 Digital transformation - what this means for information management

The public sector’s management of information has continued to use evermore diverse technology and has evolved in its concepts from traditional records management to data and information management. It’s important to remember that records still need active management in a digital environment, and that poorly specified technology solutions can undermine effective information management.

Digital transformation is not simply about new technologies, but also about making a culture change and providing effective leadership.

Customer needs should be at the centre of delivery of public services. To create a working culture where digital information management is valued, organisations must understand the benefit of good information management to the customer and have the tools and capability to deliver this. In the public sector, we create value for our customers – the New Zealand public – by creating full and accurate records, and properly protecting and making them available.

Information needs to be managed effectively to get the best value from it.

Information enables an organisation to deliver better public services, maintain accuracy, build trust, make evidence-informed decisions, avoid and reduce costs, and comply with legislation. The right records also need to be permanently preserved to ensure inter-generational government accountability of the type sought through, for example, the Royal Commission of Inquiry into Historical Abuse in State Care and in the Care of Faith-based Institutions – Te Kōmihana Karauna mō ngā Tūkino o Mua ki te Hunga i Tiakina e te Kāwanatanga i Tiakina hoki e ngā Whare o to Whakapono.

Case study: District Health Boards' shared Disposal Authority

Currently only half of all public sector organisations have disposal authorisation for all their information. A lack of disposal coverage means that many organisations cannot legally dispose of a significant proportion of the records they hold.

Extensive disposal coverage for the public sector is a prerequisite for more effective information management. The current approach of organisation-specific disposal authorities being issued for 10 years has proven difficult for Archives New Zealand to maintain and for public offices to implement. As outlined under Influencing the state of government recordkeeping above, preliminary work on transforming disposal machinery has started. While all potential models can be considered, one that is being looked at is the incorporation of similar functions of organisations into functional disposal authorities that could be applied across multiple agencies.

Disposal classes would be developed to cover functions and activities that are common across government or sectors and not separately for individual organisations - functional disposal authorities (FDAs) as a working title.

This could assist disposal authority development and maintenance by reducing the current proliferation of classes that are tailored to each organisation as it develops its own disposal authority with Te Rua Mahara but which are essentially the same across agencies. Common descriptions could enhance consistency of disposal implementation. In this potential future, a viable core of FDAs would be developed for the whole system and organisations would sign up for those FDAs that matched their functions.

A programme for developing new FDAs would focus on addressing known gaps in coverage. Further gaps in coverage identified by individual organisations would also feed the programme, but the default approach would be to develop new broad-based FDAs, rather than point solutions to a single records class in an organisation.

The model would allow a ‘building block’ approach by which disposal authorisation for each organisation would be made up of a combination of relevant FDA blocks. The FDAs could be maintained by a light review regime to ensure ongoing fitness for purpose, perhaps every five years.

This approach could:

• reduce cost and time spent on producing organisation-specific disposal authorities

• increase disposal authorisation coverage and free resources for disposal implementation

• allow identification of high value/risk functions.

Since 2017, we’ve been working with a group of District Health Boards (DHBs) and the Ministry of Health to replace the current disposal authority for DHBs (DA262). We have taken the opportunity to test aspects of what could become a future new approach.

The working group agreed it was necessary to make the disposal authority more inclusive and flexible for easier implementation. As a result, three FDAs have been developed, each encompassing a few classes that together capture the records related to that broad function, for example population health and wellbeing. DHBs would sign up to all three FDAs. Other agencies that have functions that fit within the broadly defined classes could sign up in future. We anticipate that results from the working group could allow approval of the FDAs for DHB in the first half of 2019. The lessons from the development and implementation of FDAs by DHBs will inform our work on a wider transformation of the disposal system.

Case study: Testing machine learning to automatically transcribe handwritten archives

Our archival holdings include many documents and registers written in 19th century cursive handwriting. To find a piece of information that may be held within a document or register (for example, a name or an event) people need to visit one of our offices where the item is retrieved from the shelves for them.

That’s the easy part. The hard part is then reading through the document or register to find the information; sometimes hundreds of pages. This can be a time-consuming process. Our indexes can point a researcher towards a likely document or register, but there is no way to search within the document other than reading its entirety; and reading 19th century handwriting can be difficult.

We wanted to explore the feasibility of using machine learning to see if these handwritten archives could be automatically transcribed so that their content can be easily searched by keyword and provide a transcription of our records that is less resource intensive and easier for users.

We tested Transkribus, a cloud-based machine learning platform designed for archives and humanities researchers for the automated recognition, transcription and searching of historical documents.

We started by selecting a large sample of text from a single record then, using the software, manually transcribed enough pages to train a model in recognising the handwriting. With that model we were then able to test automatically transcribing the remaining pages of the record.

This testing initially produced a 10% error rate, which improved with further model training.

Machine learning can help make our records easier to find and use, particularly for 21st century users, without the use of intensive of manual resources. The transcripts do not fit easily with our archival management system. This timely study has meant we can include capability for this in our future digital channels.

Case Study: Digital transfers from the Education Review Office (ERO)

We reported in 2016/17 on the digital transfer with the Civil Aviation Authority Te Mana Rererangi Tūmatanui o Aotearoa (CAA). In 2017/18 we accepted a further significant digital transfer, from the Education Review Office Te Tari Arotake Mātauranga (ERO). This is the largest born-digital transfer to date, comprising 26,733 separate items.

Born-digital records are those that were originally created in digital form, in contrast to records that were created in a physical format and digitised.

The transfer consisted of institution review reports for schools and early childhood centres prepared between 1992-2007. These reports look at how schools and early childhood centres are reaching positive learning outcomes and where improvements need to be made. ERO currently publishes the latest two reports for each school or centre on its website, and the schools themselves often publish them on their websites too. The transfer provides a complete set of reviews for each school and centre for the above period of time. The records are all open access archives.

While this has been a successful transfer story, some of the circumstances supporting that success might not be typical. In this case, ERO possessed a series of records created under a stable statutory mandate and function. ERO has had relatively little change of either function or departmental status that would have affected its records since its creation in 1989. The series of records were also largely digitally-born records, with considerable consistency of format over the years.

Many organisations create and maintain records in both physical and digital form. Even where current information management practice has moved to born-digital approaches there will often be a large amount of information maintained in older digital formats, presenting different challenges to effective transfer and other types of disposal under the PRA. Potential disposal actions are transfer of control to another organisation or to Te Rua Mahara, sale, alteration, destruction or discharge of records.

Many organisations have undergone functional, structural or locational and system changes that mean that defining series of records is not straightforward. These factors must be acknowledged as valid reasons for the slow uptake in digital transfer from public offices.

In future projects we will work together to overcome technical and functional issues and use this as an opportunity for collaboration and sharing knowledge. We are looking at ways of using semantic technologies to increase the accessibility of information that has been created during machinery of government and technology changes. Work has recommenced on a project to create a new archival information management system to replace our current one that will, among other goals, remove current barriers to the full description of born-digital records.

3.1 The Public Records Act 2005 (PRA)

Public sector organisations in New Zealand have always kept records of their activities. However, until the introduction of the PRA, there were very limited legislative requirements for what information and records needed to be created and how they should be managed.

The PRA sets out a regulatory framework for information management across the public sector. Its primary purpose is to enable the accountability and transparency of government decision-making by ensuring that organisations create and maintain appropriate records of their activities. The PRA also establishes the statutory role and duties of the Chief Archivist. These include:

• exercising a leadership role for information management across public offices

• setting standards for public sector information management

• authorising organisations to dispose of records when they are no longer needed for business purposes

• providing advice and support for organisations so they can comply with the requirements of the PRA.

Two types of organisations are covered by the PRA, each with different compliance requirements. These are public offices and local authorities.

A wide range of organisations are public offices, including: government departments; district health boards; Crown entities; state owned enterprises; school boards of trustees; and Government Ministers. Regional councils and territorial authorities are local authorities under the PRA, as are council-controlled organisations.

3.2 Te Rua Mahara is a regulator

The PRA establishes the Chief Archivist as an independent information regulator within government. In delivering this role, Te Rua Mahara has responsibility for supporting, monitoring and directing the sector to facilitate compliance with information management requirements.

We regulate approximately 3000 public offices and local authorities (including around 2500 school boards of trustees). These organisations vary widely in their size, complexity, access to funding, staffing levels, and the number of functions they carry out. These factors all affect the level of information management maturity in organisations, as well as the level of risk associated with not being able to find or access information that has been created.

Public Record

Means a record of class of records, in any form, in whole or in part,created or received by a public office in the conduct of its affairs, this includes estray records.

Disposal

The range of activities, defined in the Public Records Act 2005, that can be applied to public and local authority records that are no longer of active business value. This covers transfer of control; sale; alteration; destruction; or discharging of records.

Public Sector Organisation/Regulated Party

Umbrella term used by Te Rua Mahara to describe all organisation subject to the Public Records Act 2005, including public office and local authorities.

Local Authority

Regional council or territorial authority, including:

1. a council-controlled organisation

2. a council-controlled trading organisation

3. a local government organisation

Metadata

Means data that describes context, content and structure of records and their management through time.

• Archives New Zealand. (2016). Information and records management standard

• Archives New Zealand. (2016). Implementation guide

• Courts of New Zealand. (2018). The Chief Executive of the Ministry of Social Development v L [2018] NZHC 2528

• Digital NZ. (2018). NZ’s digital transformation

• Digital NZ. (2018). Information management

• New Zealand Parliament. (2018). State Sector and Crown Entities Reform Bill 2018 (20-2)(Hon Chris Hipkins)

• Satya Nadella and Jim Euchner (2018). Navigating Digital Transformation. Research- Technology Management (July-August) pp5-7 State Records of SouthAustralia (2018). Digital Records Management

• State Services Commission. (2018). State Sector Act – New Zealand’s Public Service today

• Victoria University. (2018). Free and Frank advice and the Official Information Act