Creating a national archive for New Zealand
Article for Archives and Records Association of New Zealand
By Greg Goulding
Archives New Zealand
Thank you for this opportunity to share with you something very important to me: that is, my vision for the national archives of the future and how we are concentrating our efforts at Archives New Zealand to become that national archive.
This is a story about change and adjustment to change, about prioritising resources, about working in isolation and working in partnership, and about striving to be continually relevant despite and because of shifts in our operating environment. In many ways it is a story that could apply to any organisation today. In other ways it is uniquely archival. It is a story about Archives New Zealand’s directions and priorities, which are set in the context of these wider issues.
This story is necessarily high level, and much of the important work we do is not visible in the following text. This is deliberate because I want to focus on some of the most significant change points and use examples that best reflect how we are responding to these change points.
The following section provides some context (a crucial starting point for any respectable archivist) about important and rapid social and technological changes of the 21st century that are influencing both what we need to do, and what we can choose to do, to be the most effective Archives we can be.
The first factor that needs to be considered when thinking about social change is people: put simply, just how many of us are there? I can remember quite clearly as a child being stunned to learn that the global population was over three billion (well, when you can barely count to 10 without taking off your shoes and socks, that is quite an impressive number!). That was in the late 1960’s. Today that’s nothing. Today the population is over seven billion and is, of course, rising1. This is a rate of population growth never seen before and set to continue for the foreseeable future.
These numbers themselves don’t mean much, but it is what these many people are doing that is of relevance to our story. According to World Internet Users statistics2, there are currently 2.4 billion people connected to the internet, or about 34 per cent of the population. That number is also rising. It is clear that this technological change has led to social change, as increasing numbers of people are forming online communities that ignore regional and national boundaries. Twelve million people across the globe were members of the World of Warcraft community at its peak in 2010 (although that has declined a little as some players move onto newer platforms). That’s almost three times the population of New Zealand. Someone you know, or you yourself, may belong to this alternate community.
The online community that is arguably the most significant today is Facebook. This community didn’t exist in January 2004. Today there are more than one billion members; making the Facebook nation the world’s third most populous. So globally we have huge numbers connected, and connecting to each other in online, self-defining communities.
That the internet is a significant part of peoples’ lives is if anything, even more compelling in New Zealand. According to New Zealand Internet Statistics3, 86 per cent of New Zealanders have access to online information, an increase from 78 per cent in 2007. Of those who have internet access at home, 91 per cent are on broadband. We have talked much about the digital divide over many years, and it is still there. But clearly it’s shrinking. A good example is the connectivity levels of older New Zealanders. It might be tempting to assume that this cohort would be later adopters than their younger contemporaries, and to an extent this is true. But the difference appears to be becoming less significant. For people more than 60 year olds, 68 per cent are internet users, jumping from 57 per cent in 2007, making theirs the biggest increase of any age group. Today, that number will certainly be higher again.4
Why do I consider this change important enough to talk about at length? Because these numbers make it very clear where most of our customers are; they’re on the net. Soon, all of them will be. The big question for any national archive is: Are we moving fast enough to meet them there?5 I will come back to some practical implications later.
Technological change has also had a significant impact on the “back end” of our business, that is, the creation and maintenance of government information. I have some more numbers that make sobering reading about the scale of the challenge. The volume of information being created worldwide today is simply extraordinary: in 2011 it was 1.8 zetabytes created, transmitted, and stored (if only very temporarily in most cases). That’s 1.8 trillion gigabytes which I’m told is a very large number indeed. And it is expected to double by 20206. The slice of that volume that we have responsibility for is relatively small, but the point is this: the approaches we have taken in the past to manage the volume and growth of information are unlikely to be successful from here on in. The scale is just too big. We will need different approaches.
This massive scale is not limited to digital information. In 2010 Archives New Zealand investigated the quantity of paper records that are held by government agencies, either on site or in storage facilities. We talked with records managers, reviewed what was already in storage, and did some modelling to determine likely growth over time. We were surprised by the results. Over the next 30 years there will be up to 2.5 million linear metres of public records in paper form in storage in New Zealand. If we apply existing selection criteria and processes, over this 30 year period, Archives New Zealand can expect to receive over 100,000 linear metres of paper public archives. That’s more than the total number of records currently in our stacks covering most government activity from 1840 to the 1980s.
At Archives New Zealand we coined the term ‘paper bloat’ to describe this situation. It might be expected that the digital information technology era would have reduced the quantity of paper records but, during the initial stages of this era at least, the opposite was true. Surprisingly, this makes sense. During that time, for most employees, information technology simply provided faster and easier ways of creating paper record and lots of them. Multiple versions of documents created by multiple people, all printed and stored because there were few reliable electronic recordkeeping systems. Projects that would have once been stored on one file became several shelves worth of material. Over time this trend will reverse and the volume of paper dry up, but not before we are left with a legacy that needs to be managed one way or another.
As archivists this leaves us with a big question. If, at the close of the 20th century the same government activities generated a larger volume of records than they did mid-century, is it correct to apply the same appraisal criteria and processes to ascertain whether or not they are archives? Or should we apply a new approach that factors in the bloat problem and filters out the additional information that doesn’t add value? Appraisal is one of the core archival functions, so these are strategically significant questions.
As an information agency this is the environment that Archives New Zealand operates in. It is an environment that is going through constant and intense change. In such an environment, it makes sense to continually re-examine ourselves and what we seek to achieve. So the big question for 2013 and beyond is: What is an Archive today? Or more to the point, what is a national archive? Is it a building? Is it a repository? I believe this is what many of our stakeholders think, because there is so much emotional attachment to the physical manifestation of Archives New Zealand. Archival institutions tend, by their nature, to be large imposing physical spaces, implying by their solid nature the solidity and enduring quality of the archives they contain. This has become a key part of the archival identity. But the national archive is not its buildings. The national archive is always the services that it carries out, the functions that it delivers, and the outcomes that it works to achieve. Along the way, the tools we need to do all of this will develop and change. The imposing physical structures are likely to remain, but they will become decreasingly central to how we do our work. Other tools will assume more importance, and other modes of operating, that are less tied to physical spaces, will become available, and indeed essential to our success.
Over the last year we have been doing some work at Archives New Zealand to articulate what that success looks like. As a result we have developed the following set of statements. At the highest level is our vision:
All New Zealanders have a sustainable connection to trusted government information.
To me this is a bold statement about the impact of the services we provide and what all of our efforts are geared toward achieving. If we unpick the statement some key points emerge. Just to be contrary I will start with the last phrase government information. Public archives are a very important subset of government information, but these are not our only concern. It is now accepted that a national archive must focus energy on the full range of information that the government creates and maintains, and to ensure good recordkeeping outcomes. These outcomes include accessibility and sustainability.
Returning to the beginning of the statement, the first word sets a huge challenge, but one which as a national archive we have to embrace. By saying all New Zealanders, we are committing to pursue approaches that will open up our information to many more people than currently. Placing our information in monumental structures and saying ‘come to us’ won’t deliver this outcome.
The sustainable connection is in one sense a re-formulation of the archival purpose. We preserve to reuse. We ensure that the information that is needed tomorrow or in 100 years is in a usable state and that people who need to can get access to it. That is the sustainable connection. But, sustainability carries other meanings; a sustainable archive is also an archive that is mindful of the resource implications of its work, both financially and environmentally.
And finally the word trusted. Can we say government information is trusted? Citizens may not always trust their government, but if the national archive is successful in its work then citizens can trust that government information is reliable and authentic – that it will be the same tomorrow as it is today – and that it is accessible when it needs to be. This statement isn’t naïve. It does not assert that governments always tell the truth. But it does mean that with good government recordkeeping, any untruth will be captured along with the truth and will eventually out. Within a culture of trusted government information, the incentive to tell the truth will be that much higher.
It is important that our whare is well built and is interconnected. Our vision I liken to the tāhuhu or spine/backbone that runs the length of the inside of the house and is the overarching state we seek to achieve. The pou tokomanawa is representative of our strategic plan that supports the vision and gives strength to our house.
The heke or rafters provide the strong strategic starting points. The heke connect the vision to the poupou who represent the people that implement the strategies.
The four heke in our whare provide the conditions that have to be met to achieve the vision/goal. The heke is where we get our direction, inspiration and knowledge to be used and reused.
- It is easy for New Zealanders to find, use and reuse government information
- We know what the important government information is
- Government information is secured and preserved for as long as it is needed and wherever it is most sensible to do so
- We are the national archive.
If we take these heke seriously, they steer us towards a particular set of choices which I will outline below. Taking each of these statements in turn then, what do they mean?
The first heke talks about government information being easy to find, use and reuse. Is that the case now? I would love to say that it is easy to find information in the public archives, but I suspect there would be a number of users who would disagree with me. We have come a long way since I began my archival career, and those who undertook archival research at that time can testify what a huge difference Archway7 has made. But, there is still information in the archives that you need to be an archivist, or at least think like an archivist, to find. This is a situation that is antithetical to our ambition. Most of our customers will never think like archivists and nor should they, that’s our job. With the new tools becoming available to us, we need to move to a point where the casual user, applying intuitive search requests (as they are used to doing with Google) has a reasonable chance of finding what they are looking for. We clearly need to do more, or do things differently, to get there. I don’t want this to be taken as a criticism of the efforts of our staff or the approaches taken up until now. Working with the tools available, which till not that long ago was pen and paper, we have done some outstanding work. I do think, though, that our new tools must mean new approaches to helping our customers find and use the archives we hold.
The second heke, we know what the important government information is speaks to one of the defining archival functions - appraisal - and the decisions I must make as Chief Archivist when I authorise the disposal of government records. The second pillar is not an arrogant statement; it is a statement that any national archive should feel confident in asserting as it is core to our role. At Archives New Zealand we are currently reviewing our Appraisal Framework to ensure our approach is fit for purpose in the 21st Century. We are beginning with the fundamentals: what is the purpose of appraisal, and what are the reasons we keep records for any period of time? In particular we are closely re-examining the criteria for keeping records permanently as public archives. I will soon be issuing a new Appraisal Policy that answers these questions; this will be followed by a review of the processes used to appraise public records.
The third heke, government information is secured and sustained for as long as it is needed and wherever it is most sensible to do so has two important components. One is that not all government information becomes public archives which need to be sustained forever. Many government records are vitally important to New Zealanders for one, 10, 20 or 50 years but will never become part of the public archives. They are still our concern. The second component is the acknowledgement that our house is not the only place that public archives (and other government information) can be well looked after. There will be instances where it makes more sense for other parties to care for and provide access to public archives. The Public Records Act 2005 (Act) is flexible about this. My view is that, where digital archives are concerned (and possibly other types of records), we may see an increase in the use of these provisions in future.
And finally, heke four, we are the national archive I have used the term national archive frequently here and some may question my grounds for doing this. Are we not called Archives New Zealand? Of course that is correct. But, if we look at the purposes of the Act which includes to “provide for the continuation of the repository of public archives called the national archives…” then we see that as a legal entity the national archives continues. To me, this acknowledges our national leadership role, where we connect New Zealand to the international archival community and provide support to the wider archival community in New Zealand. A national archive is a concept that communicates this role powerfully and succinctly, it echoes the relevance to ‘All New Zealanders’ of the vision. It reminds us of the challenge to be a genuinely national institution, not just serving those who visit our buildings. And importantly, and to paraphrase from the Act, we undertake our work in ways that encourage the spirit of partnership and goodwill envisaged by the Treaty of Waitangi (te Tiriti o Waitangi).9
Given the context in which we operated, how do we propose to reach our goals?
The first and most important single feature for me is: Partnership. Or you can call it collaboration, cooperation, or just working with others. This is a way of operating that extends across all of our activities and is essential to our success. We have been actively collaborating for some time to achieve things we simply could not do working by ourselves. One example of these partnerships is millions of pages of archives have been digitised, stored and are being made available online at marginal cost to taxpayers, thanks to our work with FamilySearch. Partnering with different iwi on research and digitisation projects has added greatly to our knowledge of our Māori language holdings and how we can best provide services to Māori. We are currently embarking on a Strategic Partnership with the National Library of New Zealand which promises to amplify the impact we have by sharing resources and reducing duplication of effort. So in some regards, partnership/collaboration is not a stand-alone strategy. It can and should support all of the other approaches we adopt. Partnerships will always be important to us as they are a fundamental enabler of so much of our work.
Supporting our overall vision and priorities is our new Online Strategy. This is our set of statements and objectives about how we will transform our business to take advantage of the opportunities provided by online channels. This is the work we need to do in order to answer ‘yes’ to my earlier question “are we moving fast enough to meet our customers on the net”? The first step is the strategic objective: online channels become the primary way people choose to engage with us. The word ‘choose’ indicates our desire to make our online channels the preferred way of getting access to our services. Supporting this is the objective: it is easy for people to find, use, and reuse our information online. Without this ease, online channels will not be the way people choose to do business with us. But, we have a lot of work to do. Right now I cannot put my hand on my heart and say that it’s easy for people to find our information online. Our online finding aids are like many archival finding aids; they are very logical and are easily navigable if you are an archivist. That’s significantly less than one per cent of the population. That means that more than 99 per cent of the population will struggle with them. Some of this is due to system design, but I suspect that much of it is the information we provide through those systems, which to a large extent focuses on the agency that produced a record rather than what information the record might contain.
Now it is absolutely true that archivists must ensure that the context of archives are adequately documented. Without the context, we cannot trace records back to the actions and the people that generated them, and their value as evidence is diminished. The challenge though is always to strike the right balance between the level of contextual information, and what we might call ‘discovery’ data. This is information that helps more people find the information they are looking for in the archives. Most people will want to find information that is ‘about’ something rather than ‘from’ an organisation, yet the overwhelming majority of the information we provide them is in the latter category. I believe we need to change this equation. In the online environment, away from the reading room and the assistance of the all-knowing archivist, researchers need to be more self-sufficient. For this to happen we will need to provide them with more of the information that makes sense to them. This means having a good hard look at our descriptive practices, re-prioritising, and making some changes. It means working more with partners to share the load of providing that information; and it means using the power of digital technology and the reach of the online environment to facilitate all of this.
Let me give you an example. One simple way of making archives easier to discover is name indexing. Volunteers have created thousands of index cards for some of our holdings that are of interest to genealogists. On cards, they can only be used in one Archives New Zealand office and by one person at a time. Turned into digital text and placed online, they can be used anywhere by many more people. While it makes sense to do this, the technology for doing this automatically is currently nowhere near good enough, and doing it manually is a resource intensive business. However, we have come up with a way to make this happen by using the power of partnership and online through what is called crowd sourcing. When our Indexer system is operational, we will digitise the index cards, place the raw images online, and invite people to turn the information into digital text. When two people’s rendition of an index card match, it will be locked in, and then be searchable by other researchers. We’re by no means the first to do this internationally, but we will be one of the first institutions in New Zealand to take this approach. A key feature of this method is the institution ‘letting go’ of the control over the information that people use to discover archives, which is quite a cultural shift for all of us. The system will be largely self-validating and the information entirely sourced from third parties. This approach is acknowledgement of current reality: the ratio of archivists (few) to archives (millions) is against us. If we attempt to do all this work ourselves, we will never get there. We have to rely on, and trust and work with others if we are going to fulfil our purpose.
Looking back at our statement about making online the primary way that people choose to use our services, there is a question that should be answered. Why should online be the primary way? Is there something wrong with the physical channels that we still operate in each of our four offices? I don’t believe so. I think that our people do excellent work to support customers in our reading rooms. Indeed, there is much archival research that will always be best handled in this way, particularly specialist academic research that involves working through large volumes of obscure, little used (but none the less valuable) archives. There maybe little benefit in digitising these archives, if others are unlikely to want to see them anytime soon. For this type of research, bringing the researcher, the archivist, and the archives together in one physical location may well continue to be the best and most cost-effective way of working.
Remember that our vision statement refers to All New Zealanders having a connection to our information. Good as our physical service is, we can never do it on a scale to reach all New Zealanders. Online, we can. I will show you how this is already happening. Last year in our reading rooms we received 13,392 visits, produced for our visitors 47,869 archives for inspection, and answered 14,152 research enquiries from those who couldn’t or didn’t visit us in person. These are good numbers and compare well with other archives internationally given our size. And they are near the limit of what we can possibly deliver through physical channels with the level of resource we can ever expect to have for this activity. Let’s compare this with our online activity. Our most popular archives are the World War One military personnel files. Our main contribution to World War One commemorations is to digitise and make these files available online, and we are now over half way through this exercise. In 2011/12, 216,517 pages of personnel files were viewed online. If each file is several pages, it means we are already more or less matching our reading room numbers for archives produced with this one series alone, with only 50 of them available.
I have another example that shows the benefits of working in partnership to increase our online services. New Zealand passenger lists of incoming ships from the 19th and early 20th centuries are another extremely popular record series. Working with FamilySearch, these have now all been digitised and are accessible through the FamilySearch website, with indexing well advanced. This is a significant achievement which has been accomplished at only marginal cost to the taxpayer. Researchers now have free access to this information from anywhere in the world. More numbers for you: in 2011/12, 242,958 pages of these files were viewed; once again, more than matching our efforts in the reading room. We are currently working through probates using a similar arrangement, and will soon begin digitising our Intention to Marry records in the same way. The number of candidates for digitising is large. We are currently prioritising a list and will be looking for potential partners to help us progress this important work.
Up until now I have discussed mainly written records; however, Archives New Zealand is not just about text. We have substantial holdings of artworks, photographs and moving images. These all lend themselves to online channel delivery and to partnership based solutions for making this happen. We could set-up our own online channels for delivering film content to the world, but have chosen not to. Instead we have seen our role as content providers rather than channel developers. There are others who provide the channels for us. Ecast TV is one example; Ecast TV provides a dedicated Archives New Zealand internet television channel where we can stream our extensive collection of government created film and video at no cost to the taxpayer. We also upload the same content to YouTube, which again provides the delivery mechanism at no cost to us, and takes the content right to the audience. This is the key to achieving the widest access possible. Expecting everybody to come to our website to find our archives is both self serving and self defeating. We want as many people as possible to discover and use our archives, and utilising the channels people worldwide already use is the best way to achieve this.
Partnership is also the main theme in our relationship with the National Library. We are now both located within the Department of Internal Affairs which makes cooperation between our organisations that much easier. The National Librarian and I sit at the same management table where everyday we face many of the same challenges. The approach we are taking is to identify and deliver on opportunities for joint work to reduce duplication of effort and improve services, rather than a structural merger of the two institutions. A flagship project in this partnership is the development of a new exhibition space featuring te Tiriti o Waitangi and other iconic public archives in the building currently known as the National Library in Molesworth Street, Wellington. By the time the exhibition opens, the building will have a new name to reflect its enhanced function as a home for Archives New Zealand, the National Library and the Department of Internal Affairs.
For me, this project is about delivering a world class exhibition in one of the best locations in the capital, to a standard that would simply not be possible in the building that currently houses te Tiriti. It is an opportunity to build something that will attract many more people than at present to come to see these documents and think about what they mean as individuals and as a nation. It’s very much about sharing resources and delivering something well beyond what could be achieved by either institution acting independently. Some observers have suggested that we are transferring the Treaty to the National Library. That is not what is happening. As Chief Archivist I remain responsible for the preservation and accessibility of these documents, and I am leading the project to develop the exhibition space. This new space will be a presence of Archives New Zealand in a different building, just as we are in other centres throughout New Zealand.
I have focussed most of my attention so far on the public side of Archives New Zealand. What of our regulatory and advisory role to government? Our purpose is: the creation and maintenance of a reliable record of government. Our overall strategy is leadership and influence; that is, we do not carry out the recordkeeping in government ourselves, we work with records and information managers, chief information officers, corporate managers and chief executives to make it happen within their organisations. We use standards, guidelines, training and the fostering of a community of practice to achieve these ends. These efforts have been largely agency focussed, based on an assumption that recordkeeping happens within organisational domains and as a result of decisions and investments within those domains. This environment is changing. As the government moves more to adopting whole of system approaches, we need to change our focus to ensure these system changes will help us all achieve our goals. We need to be involved at the strategic level as much as, or more than, at the individual agency level. And we are starting to do this. When the Government Chief Information Officer initiated a task force to develop an all of government ICT strategy, Archives New Zealand was invited to participate. Through this we are able to ensure that the information managed this way is accorded as much importance as the technology, and is seen as a strategic asset as opposed to an administrative cost. In this environment, we are much more likely to see good recordkeeping outcomes.
A real expression of this all of government system approach will be the development of information management systems shared across government rather than the current situation of each agency having its own unique system, and often more than one. These systems are unable to talk to each other; they lock information up in individual domains, duplicate costs and prevent knowledge flowing across the systems. As new multi-agency systems are developed, as I believe they will be, we have an opportunity to ensure that good recordkeeping is built into them and implemented. There is potentially much more return on the investment of our limited resources using this approach, than in only focussing one agency at a time.
Our approach to delivering advice to agencies, and to our clients managing archives in the community, will need to change as well. At present we have a suite of recordkeeping advice documents which are accessed via our website. These are all quality documents with good information, but they can quickly become outdated and their production has led us into the role of publishers which is time consuming and expensive. We are reconsidering this approach. Rather than writing and publishing documents, can we work with our recordkeeping community to create a more dynamic, partnership based approach, using social media for example, where the information is shared? This already happens to an extent through email listservs, but an online environment where information and advice can be exchanged has much promise. It mitigates the risk of the authoritative advice becoming stale, and further develops the community.
The final element in this future archive I want to talk about is the set of processes and infrastructure at its centre. Until now it has been largely bricks and mortar based, but we are now on the verge of also being a national archive for the digital world. The Government Digital Archive nearing completion will allow us to preserve and provide access to the digital public archive. Once again, partnership features largely in this endeavour. The digital repository is being built on a software platform that we share with the National Library (Rosetta) and on hardware that is supported by our Internal Affairs colleagues in the Government Technology Services. It will be physically located in the all of government storage centre provided under contract by the private sector company Revera using a state of the art infrastructure that would be massively expensive for Archives New Zealand to build on our own.
As with physical records, computer hardware and software does not an archive make. As part of the programme to create the digital archive, we are re-designing the archival processes of documenting context and creating access channels. Archway, our archive management system, is receiving a major overhaul so that it can manage digital archives that will present us millions of individual objects rather than the aggregated documents within files that we are accustomed to. Our archivists’ work will have to change to handle this new reality.
I believe our archivists are ready for the challenge. Together we aim to work at a strategic level across government and with current and future partners to ensure there will be a reliable record of government for anyone who needs it, for as long as it is needed. In the same way we will unlock the enormous potential value in the public archives by making it not only possible, but easy, for all New Zealanders to find, use, and reuse the information they are looking for. Today, this potential is largely untapped and the use made of this invaluable national resource is only a fraction of what it should be.
The future is our opportunity to change that and in so doing make a real contribution to the New Zealand story. This is a story for all of us to share and to contribute to in our own way. For Archives New Zealand this is the story of our future and one we are striving to make a reality as we come to terms with what the National Archive is now and what it can be in the near future for all New Zealanders.
1 For a slightly scary experience go to the Worldometers world population site at http://www.worldometers.info/world-population/ where you can monitor population growth as it happens. It never stops, 24/7.
3 Compiled by the Auckland University of Technology in 2011 for the World Internet Project, http://www.aut.ac.nz/__data/assets/pdf_file/0005/252077/WorldInternetProjectNZ_2011final.pdf
4 I owe thanks to Richard Foy, a colleague at the Department of Internal Affairs for drawing my attention to some of the preceding information. Richard is the Executive Director Service & System Transformation
5 And the corollary question, if we do meet them on the net will they be glad that we did?
6 Source: EMC2 http://www.emc.com/about/news/press/2011/20110628-01.htm
7 Archway – the Archives New Zealand archival documentation and finding aids system
8 Public Records Act 2005 s 3
9 Public Records Act 2005 s 7