Report on the State of Government Recordkeeping 2016/17
The Chief Archivist’s Report on the State of Government Recordkeeping 2016/17: Managing public sector information and data and why it matters
Presented to the House of Representatives pursuant to section 32 of the Public Records Act 2005.
7 March 2018
ISSN: 2433-091X (Print)
ISSN: 2423-0928 (Online)
Information management can often seem like a cumbersome compliance exercise – something we do because we have to. If we look closer, however, it becomes clear that the effective management of public sector information (including data), provides real value to organisations, individuals, communities and to New Zealand as a whole.
When public sector organisations create and manage information effectively, they can more easily trust it, understand it, protect it, and find it when it is needed. This enables the public to easily access information about the work of the public sector, which is vital for the accountability and transparency of government. It also enables organisations to use, re-use, combine and analyse information to improve the delivery of public services and inform policy development.
This report has a focus on digital information management, with less emphasis on paper. While both are important, we chose to focus on digital because of the rapidly growing volume of information created in this environment each year and specific technical challenges that are not present with the management of paper-based records.
Paper is a sturdy medium that can often last many years without specific preservation care.This is not the case for digital information. While there are many benefits of creating and storing information in a digital environment, we cannot just put it in a virtual box or file room and assume that in 30 years’ time we will have the technology to open it and the contextual information needed to understand when and why it was made. Digital information needs to be actively looked after, especially when technology systems and the structure of organisations change.
Archives New Zealand’s aim for this year’s report is to bolster understanding of our role and what we mean by effective information management. It is intended for the many people who are not information specialists, but nonetheless have key responsibilities for information management across government. This includes Ministers, Chief Executives and Executive Sponsors of public offices and local authorities, and other leaders who have key roles in the government information system. Of course, I hope there will be a wide readership that includes the many information professionals across government and members of the public.
Acting Chief Archivist
7 March 2018
Is a record the same as information?
Under the Public Records Act 2005 (PRA), any information in any form or format is considered a record. This includes paper and digital documents, emails, and data. Other examples given in the PRA include signatures, seals, images, sounds and speech.
In this document, we use the term “information management” instead of “recordkeeping”, as we think this better describes for non-experts the range of activities that organisations need to carry out in order to meet the aims of the PRA.
There’s a lot of talk about data–how does that fit?
Data is a sub-set of information. It refers to a set of discrete, objective facts about events, people or places, without context or interpretation. Data is covered by the PRA and the standards issued by the Chief Archivist apply to it. The Chief Archivist has a particular interest where data forms a record of a transaction or decision making that may be needed to hold government to account.
Part One: Snapshots from the sector
This section looks at three key themes in public sector information management.
Each theme highlights an area of information management practice that needs attention– either because there is room for improvement, or because changes in technology have implications for the way information is managed.
The three themes are:
1.1: Public sector organisations need to actively manage the information they hold.
1.2. Archives New Zealand is accepting the transfer of digital information and record.
1.3. Ensuring the accountability of government when key decisions are made by machines.
The key sources for this years report are a survey of public offices and local authorities carried out in late 2016 and early 2017 and meetings and conversations with stakeholder and regulated parties.
1.1 Public sector organisations need to actively manage the information they hold
Public sector organisations rely on information to function, just as wider society does. Without it, informed decisions could not be made; legislation could not be passed or enforced; citizens would not be able to access services or entitlements; and the public could not hold government to account for its actions and decisions.
To get the most value from information, it needs to be actively managed–from the moment it is created or received, through to when it is no longer needed for business purposes.
To do this most effectively, information management requirements should be designed into organisations’ processes and systems.
If information is actively and effectively managed, it can be found; protected; trusted; tracked; used and reused to deliver services and gain insights; and easily destroyed or archived when it is no longer needed. As a result, organisations can make better decisions and deliver better services; manage information and technology risks; more effectively respond to requests for official information; retain institutional knowledge; and ensure value for money.
Many public sector organisations are large and complex. This challenges coherent and effective information management. Across government, information is stored in many different places, including line-of-business systems; third-party systems in the cloud; enterprise content management systems; mobile devices; shared drives; social media platforms; and a range of physical storage locations. In our 2016/17 survey, one public office indicated that it holds information in over 350 applications. In order to manage information successfully across all these platforms, it must be well understood.
There are two key tools that public sector organisations can use to gain a whole-of organisation view of the information they hold: appraisal and information asset registers.
Appraisal: understanding the information you create and receive and its value
The PRA requires public offices to seek authorisation from the Chief Archivist when they want to destroy information or transfer it to Archives New Zealand. The Chief Archivist gives this authorisation by agreeing to a set of rules called a disposal authority, which defines how long to keep certain types of records and what should happen to them after this time.
In order to develop a disposal authority, organisations must undertake an assessment of what they do, determine the information they need to support their business, and how long this information needs to be kept. This exercise is known as ‘appraisal’. At present, only around 56 per cent of public offices have a disposal authority for all the information and records they create.1 This indicates that many offices have not undertaken a formal appraisal of their information assets, and therefore may not have an overarching view of the information they hold, and the value it has.
This lack of disposal coverage also means that many public offices cannot legally dispose of some of the information and records they hold. Archives New Zealand is seeking to ensure that all public sector organisations have their information covered by a disposal authority. We are investigating options for changing our approach to disposal authorities, which would simplify the process, make it more suited to digital information, and build consistency across organisations.
 This excludes the approximately 2,500 school boards of trustees that are covered by a single disposal authority. It also excludes Ministers of the Crown. At present, only around 56 percent of public offices have a disposal authority for all the information and records they create.
Information asset registers: adding more detail to your understanding
Information asset registers (sometimes called information asset catalogues) are a way of keeping track of the information assets held by an organisation. They build on and complement the information collected during the appraisal process and provide more detail, such as who is responsible for the information, where it is kept, and who is allowed access.
While these registers are not mandatory, they provide a foundation for a comprehensive understanding of the information held by an organisation. Information asset registers can provide a fundamental layer of knowledge about information assets to assist with business continuity, sharing and releasing information, risk assessment, managed disposal and minimising duplication.
There are resources available to assist organisations in compiling a register.
The Department of Internal Affairs’ Government Enterprise Architecture (2017b) team has developed an information asset catalogue template and guidelines which organisations can use as a starting point to create their own register. Archives New Zealand (2017b) has also released additional supporting guidance on identifying and managing information assets.
In our 2016/17 survey, we asked organisations about their use of information asset registers or similar tools. We found that only 18 per cent of respondents use such a tool. However, about 20 percent of respondents do have a register in development, which is positive.
Organisations can use an information asset catalogue template and guidelines as a starting point to create their own register.
1.2 Archives New Zealand is accepting transfers of digital information for long-term preservation
Traditionally, when a public office transferred records of enduring value to Archives New Zealand for long-term preservation, we would receive paper files in cardboard boxes. The move to creating information in a digital environment means that we also need to be able to receive digital content for long-term preservation. We have established a digital repository for central government digital records and developed supporting processes and procedures to ensure their maintenance and accessibility over time.
Public offices need to prepare themselves to transfer digital information of enduring value created from the mid 1990s onwards. Archives New Zealand has the ability to preserve this information for future generations and make it accessible.
As noted in the last annual report, a model was put in place during 2015/16 for accepting transfers into the digital repository. At this stage, only content with certain characteristics is being accepted, typically batches of up to 5000 files with open access so they can be directly accessed by the public online. We are working on being able to scale this process so larger volumes of information can be transferred.
During 2016/17, we continued to seek partners to work with to trial and refine our digital transfer processes. For example, we worked closely with the Civil Aviation Authority throughout 2016/17, which resulted in a full transfer of digital files in early 2017/18. However, fewer organisations than expected have been in touch with us about potential digital transfer. Our 2016/17 survey identified only six public offices that have digital information they plan to transfer in the next two years.
There are a number of reasons for the low level of interest in transferring digital content. Our interactions with the sector indicate that in many cases there is substantial work that has to be done before digital content is ready to be transferred to Archives New Zealand.
This includes identifying information for transfer, which is done through a process called sentencing. This can be especially difficult if information does not have appropriate contextual information (metadata) or it is not stored in the “right place”.
Sentencing should be a manageable activity for small to medium sized public offices that have carried out consistent functions over many years. The task may be more challenging for larger organisations that carry out a variety of functions and have seen considerable organisational change in the past. In complex information environments, systems are not always coherent and sentencing large amounts of information across platforms can be difficult. There are a number of other barriers that may prevent this work being carried out, including technical capability and capacity issues.
We encourage any public office that is considering starting the process of digital transfer to contact Archives New Zealand to start a discussion.
1.3 Ensuring the accountability of government when key decisions are made by machines
In public and private organisations worldwide, data is increasingly being analysed by advanced computer systems to inform decision-making and improve service delivery.
Some public organisations in New Zealand already use algorithms to analyse large quantities of data.
An early example is the Single Transferable Vote (STV) calculator, which has been used since 2004 by local authorities undertaking STV elections. This computer program calculates the outcome of STV elections in line with the vote-counting process set out in Schedule 1A of the Local Electoral Regulations 2001.
This type of computer-based data analysis process is complex, and not easily understood by non-experts. The Chief Archivist has the expectation that when public sector organisations use algorithms to inform important decisions, they ensure the process for coming to these decisions is clearly documented. This is because the PRA requires public organisations to create and maintain full and accurate records of their activities, regardless of the methodology used to come to decisions.
In the case of the STV calculator, the process for coming to decisions is both written into regulation, and explained at a high level online (Department of Internal Affairs, 2017a). In addition, any calculator used for STV local authority elections is required by law to be certified before use (Local Electoral Act 2001).
In most cases there are not specific legislative requirements relating to the use and explanation of algorithms. There is, however, a growing body of research around data ethics and how to ensure the accountability of decisions informed by algorithm.
The Data Futures Partnership (2017) has released A Path to Social Licence, wide ranging guidance for the use of data by both public and private organisations in New Zealand. This guidance includes a set of considerations for when decisions are made using algorithms. This is particularly important as analysis techniques evolve and become more sophisticated. Artificial Intelligence (AI) technologies such as machine learning are already being used widely in the private sector. For example, Google has moved to an “AI first” approach and it is “rethinking all [its] products and applying machine learning and AI to solve user problems” (Pichai, 2017). The use and impact of AI is expected to grow significantly in the near future.
AI processes such as machine learning are fundamentally different from formula based algorithms such as the STV calculator. This is because the statistical processes that lead to decision-making are defined by an algorithm rather a person – machine learning algorithms “learn” and adapt from the data they analyse. Therefore, the process that led to decisions cannot be understood in human-understandable terms (UK Government Office for Science, 2016).
As AI technologies develop and become increasingly complex, we expect that traditional forms of recording decision-making processes will also need to develop.
Public service organisations should ensure that effective information management practices are designed into new systems when they are developed. This includes processes for ensuring the accountability and transparency of decision-making processes.
In May 2017, we published our long-term strategy, Archives 2057 (2017a)2. One of the three key areas of focus is Upholding Transparency. In it, we note that as organisations become increasingly sophisticated in using data for evidence based policy decision-making, it will be important to ensure that those decision making processes can be understood by current and future generations. To this end, Archives New Zealand has identified it will need to build data analysis and ethics capability to ensure that decisions made by organisations based on data are properly recorded and preserved to enable government to be held accountable.
If you use algorithms (formula-based decision tools) to make decisions, for example, to determine whether to grant loans or what interest rate to charge, or as part of a recruitment process:
explain how these algorithms work and which elements (variables) of data are used;
advise clients of other inputs to your decision processes (for example, whether a staff member reviews applications);
offer the applicant the right to contest the decision, including the data that was used in the algorithm.
Extract from A Path to Social Licence: Data Use Guidelines
Part Two: Exercising our regulatory role in 2016/17
We have engaged with executive sponsors, public offices and local authorities, and investigated complaints made. We look to our plan to 2057 and what that means for our government agencies and the public who access the information held by our government agencies.
2.1 Our regulatory role in 2016/17
Regulatory statement and mandatory standard
In July 2016, we released our Regulatory Statement which sets out our approach to regulatory compliance (Archives New Zealand, 2016b). It acknowledges that we have a regulatory stewardship role and outlines how we will work with the sector to achieve the outcomes of the PRA. We will focus our regulatory efforts on removing barriers to full and sustained compliance and assist and support public sector organisations to achieve compliance.
At the same time as releasing the Regulatory Statement, the Chief Archivist issued the Information and Records Management Standard (the Standard) which is mandatory for public offices and local authorities (Archives New Zealand, 2016a). The outcomes-based Standard is centred on a number of key information management principles. It is supported by implementation guidance that illustrates how organisations can reflect these principles in their practices.
Engaging with Executive Sponsors
One new requirement in the Standard is that all public offices and local authorities have an Executive Sponsor, who must be a member of the organisation’s senior leadership team. This provision was added to ensure there is high-level, strategic leadership of information management activities across organisations.
Executive Sponsors have strategic and managerial responsibility for oversight of activities that involve managing information and records.
They are expected to champion the importance of effective information management among their organisation’s leadership.
In 2016/17, the Chief Executives of public offices and local authorities were asked to nominate an Executive Sponsor for their organisation. Archives New Zealand staff have actively worked to ensure that nominations were made, and there is now comprehensive coverage across public offices and local authorities.
Initial correspondence with Executive Sponsors included information about their role and the responsibility of their organisation to comply with the Standard. To further build engagement and awareness, Archives New Zealand started a programme of meetings with Executive Sponsors throughout the country.
During 2016/17, the Chief Archivist and senior managers met with 48 executive sponsors. These meetings were held in Wellington; Auckland; Christchurch; Dunedin; Nelson; Invercargill; and Greymouth. The Executive Sponsors were from the core public service, district health boards, tertiary education providers and local authorities.
These meetings were productive and the executive sponsors, on the whole, were engaged and responsive. We found there were varying levels of awareness about the requirements of the PRA, as well as a number of examples of good practice in organisations. One particular benefit of holding group meetings is they assisted in building networks. While executive sponsors generally belonged to many networks, they had seldom come together before with a focus on information management. We hope that these networks will develop and act as a base for sharing information and best practice.
The Chief Archivist and Archives New Zealand managers will continue the programme of initial engagement with Executive Sponsors during 2017/18.
Survey of public offices and local authorities
As noted in the addendum of the previous annual report, in late 2016, the Chief Archivist conducted a survey of information management practices in public offices and local authorities. The survey asked questions about information governance, storage, creation and disposal in each organisation. The aim of the survey was to collect a whole-of system view of information and records management and gain an understanding of the challenges to government organisations. We received responses from 114 public offices and 35 local authorities4. The findings of this survey have been fed into the work programme for Archives New Zealand and have informed the content of this report.
Key findings from the survey include:
31 per cent of respondents were not able to provide us with an estimate of the total quantity of digital information they held and 40 per cent could not provide a full picture of which systems their digital information were held in. This is a key indicator that many public sector organisations do not have an all-of-organisation view of the information they hold; and
70 per cent of respondents still hold information in shared drives. This is of concern because shared drives only allow for limited automatic capture of metadata. While it is possible to effectively manage information in shared drives, the lack of automated metadata capture makes it harder to control versioning, changes to documents, or who has access.
Suspension of transfer for physical records
In May 2017, Archives New Zealand suspended the transfer of physical archives to our Wellington repository. This was to allow the repair and replacement of shelving damaged in the Kaikoura earthquake and reflects the fact that our Wellington repository is near capacity.
Archives New Zealand finalised a number of planned transfers before the suspension of transfer and has encouraged public offices to focus on the destruction of paper records no longer required, in line with respective disposal authorities.
Budget 2017 allocated $8 million of operating funding and $2.1 million of capital funding to safeguard New Zealand’s heritage collections and record of Government. This work was announced in May 2017, with physical repository requirements as a core component.
Investigation of complaints to the Chief Archivist
The Chief Archivist receives complaints about potential information management issues in public sector organisations. In 2016/17, the Chief Archivist concluded two investigations into complaints made under the PRA. The first, relating to Southern District Health Board records, was summarised as part of the Chief Archivist’s 2015/16 annual report.
The second was a complaint made by a member of the public who was unable to access records from Auckland Transport because they could not be located. The records related to a public consultation carried out by the Auckland Regional Transport Authority (ARTA), before the formation of the Auckland Council.
After requesting information from Auckland Transport (which now carries out ARTA functions) and Auckland Council, Archives New Zealand determined there was an issue with the description of records at ARTA, which meant the requested records could not easily be found. Auckland Council is taking remedial action that will improve its ability to locate protected records and respond to information requests.
The Chief Archivist was satisfied that appropriate action is being taken by the Auckland Council to address the issue of poor description of legacy records. This issue does, however, highlight the importance of ensuring information has appropriate metadata, along with the potential issues that can arise when complex organisational change takes place.
Only one other complaint was received during the year, however, there were insufficient grounds to pursue it under the PRA.
2.2 Planning for the future
Review of our monitoring activities
Part of Archives’ regulatory role is to monitor and gain an understanding of public office and local authority information management capability and performance. By doing so, we can better target our resources to improve compliance.
There are a number of ways we get information to build our understanding. We gather some information as part of our business-as-usual work. Other times, the statutory powers of the Chief Archivist will be used to direct public offices to provide specific information about their records and information management practices.
In 2016/17, we took stock of how we carry out our monitoring role. The aim of this work was to summarise our activities to date; explore internal and external influences on monitoring; and identify areas where we could improve our monitoring and reporting framework.
We identified the need for a refreshed approach to monitoring to ensure we have a more joined-up, comprehensive picture of sector capability and performance. A new framework should ensure we target our efforts; achieve better coverage of the sector; put our tools to best use; and more effectively measure the progress of government information management over time.
Work to establish this monitoring framework is underway and expected to be completed during the 2017/18 year. It will include a plan for how the next audit cycle of public offices will be carried out.
The aim of our monitoring role is to summarise our activities to date; explore internal and external influences on monitoring; and identify areas where we could improve our monitoring and reporting framework.
In May 2017, following a public consultation process, Archives released its long-term strategy, Archives 2057. The strategy will guide our one, four and ten year planning cycles.
Archives 2057 sets out three key focus areas to guide our work in the coming decades.
These areas reflect Archives’ dual roles as regulator of the PRA and guardian of the records and taonga it holds on behalf of New Zealanders.
The three focus areas are:
Taking archives to the people is about getting government information, records and archives “out there” for users, promoting what we do and gearing up for the growth in physical and digital holdings;
Upholding transparency shows our intention to support open government principles, so many of which are enabled by good information management practices; and
Building systems together is about shaping the processes, systems and technology that form the government information system so that effective information management is built into the start.
The strategy recognises that there will be big technological changes in the coming decades, and that information management practices will need to evolve to reflect this.
Over the 2017/18 year, Archives New Zealand will be looking at how it can best target its efforts to build capability and improve compliance with digital information management, both now and into the future.
Appendix One: Understanding the Public Records Act 2005 and the role of Archives New Zealand
The Public Records Act 2005
Public sector organisations in New Zealand have always kept records of their activities. However, until the introduction of the Public Records Act 2005 (the PRA), there were very limited legislative requirements for what information and records needed to be created and how they should be managed.
The PRA sets out a regulatory framework for information management across the public sector.
Its primary purpose is to enable the accountability and transparency of government decision-making by ensuring that organisations create and maintain appropriate records of their activities.
The PRA also establishes the statutory role and duties of the Chief Archivist. These include: exercising a leadership role for information management across public offices; setting standards for public sector information management; authorising organisations to dispose of records when they are no longer needed for business purposes; and providing advice and support for organisations so they can comply with the requirements of the PRA.
Two types of organisations are covered by the PRA, each with different compliance requirements. These are public offices and local authorities.
A wide range of organisations are public offices, including: government departments; district health boards; Crown entities; State owned enterprises; school boards of trustees; and Government Ministers.
Regional councils and territorial authorities are local authorities under the PRA, as are council controlled organisations.
Archives New Zealand is a regulator
The PRA establishes the Chief Archivist as an independent information regulator within government. In delivering this role, Archives New Zealand has responsibility for supporting, monitoring and directing the sector to facilitate compliance with information management requirements.
We regulate around 3000 public offices and local authorities (including approximately 2500 school boards of trustees). These organisations vary widely in their size; complexity; access to funding; staffing levels; and the number of functions they carry out. These factors all affect the level of information management maturity in organisations, as well as the level of risk associated with not being able to find or access information that has been created.
What do public offices and local authorities have to do to comply with the Public Records Act 2005?
The PRA requires public offices and local authorities to create and maintain full and accurate records of their activities in line with good business practice. These records must be kept in an accessible form, so they can be used for reference or business purposes. Specific requirements for how these records are created and maintained can be set by the Chief Archivist through voluntary or mandatory standards. At the time of writing there is one mandatory standard, the Information and Records Management Standard, which was issued in July 2016.
As noted earlier in this report, the Chief Archivist must authorise the disposal of any information by agreeing to a disposal authority, which defines how long to keep certain types of records and what should happen to them after this time. Most records can be destroyed after a certain amount of time. Only 3-4 per cent of information and records created by public offices eventually come to Archives New Zealand, where they become part of the enduring record of central government.
Each organisation should develop their own disposal authority, in partnership with Archives New Zealand, for records that are particular to their business.
There are some records that many organisations create, such as those related to finance and human resources functions. To avoid duplication, there are a number of general disposal authorities for these types of records that can be used by all public offices.
Specific requirements for how these records are created and maintained can be set by the Chief Archivist through voluntary or mandatory standards.
How we fit in the wider government information landscape
For several years, government organisations and many work streams have sought to ensure:
privacy and security of information;
trust in, and accountability of, government; and
that information is used and re-used appropriately to maximise impact.
There is a complex array of leaders, legislation, regulatory frameworks, and strategies in place to ensure that these benefits are realised. These range from frameworks that affect only the core public sector (e.g. the Government ICT Strategy) to legislation that affects all organisations in New Zealand (the Privacy Act 1993).
In order to achieve these three benefits, it is imperative that information and records are well managed.
The Chief Archivist recognises the benefit of working with the other key system leads to achieve these outcomes. We seek to align our work with other related functions, for example the Government Chief Digital Officer (previously the Government Chief Information Officer); the Government Chief Data Steward; the Government Chief Privacy Officer; the Auditor-General; and the Ombudsman.
Archives New Zealand. (2016a). Information and records management standard
Archives New Zealand. (2016b). Regulatory Statement
Archives New Zealand. (2017a). Archives 2057 Strategy
Archives New Zealand. (2017b). Identifying and managing information assets
Data Futures Partnership. (2017). A Path to Social Licence: Guidelines for Trusted Data Use
Department of Internal Affairs. (2017a). More about STV. Retrieved December 4, 2017.
Department of Internal Affairs. (2017b). Information Asset Catalogue Template. Retrieved December 4, 2017.
Government Office for Science. (2016). Artificial intelligence: opportunities and implications for the future of decision making
Local Electoral Act 2001. Retrieved December 4, 2017.
Local Electoral Regulations 2001. Retrieved December 4 2017.
Pichai, S. (2017). Google Keynote. Speech presented to the Google I/O developer conference, California. Retrieved December 4, 2017.
Glossary of key terms used in this report
The range of activities, defined in the PRA, that can be applied to public and local authority records that are no longer of active business value. This covers transfer of control; sale; alteration; destruction; or discharge of records.
Public offices are defined in the PRA and are primarily organisations that make up the legislative, executive, and judicial branches of the Government of New Zealand. This includes State Owned Enterprises, school boards of trustees, District Health Boards and tertiary institutions.
For the purposes of the PRA, this includes regional councils; territorial authorities; council controlled organisations; council controlled trading organisations; and local government organisations. These organisations have different obligations from public offices under the PRA.
Public sector organisation
An umbrella term used by Archives New Zealand to describe all organisations subject to the Public Records Act 2005, including public offices and local authorities.
Metadata is contextual data about data that falls into three categories: descriptive metadata, structural metadata, and administrative metadata.