He tauira: te whakawhiti pito-ki-te-pito matihiko tuatahi

Case study — First end-to-end born digital transfer

Please note that the PDF of this page has been removed because the content is the same and our website has a feature to print a webpage. You can also see a preview of this by using the print command (CTRL + P). If you still require a PDF version, please contact us at rkadvice@dia.govt.nz

The Civil Aviation Authority (CAA) partnered with Archives to test mutual capabilities in a live [1] transfer pilot. This transfer was the first born-digital end-to-end transfer from a shared drive completed by us.

[1] In this context, “live” means a current, active information system or environment.

CAA is a Crown entity established under the Civil Aviation Act 1990. It is responsible for the regulation of civil aviation, including the establishment and monitoring of civil aviation safety and security standards in New Zealand, investigation of accidents and incidents, and provision of aviation security services for domestic and international air operations.

CAA’s primary role is to effectively manage risks within the civil aviation system, and ensure continued confidence in the safety and security of the system. CAA uses different methods to manage these risks including education, guidance, and legislative tools like the Civil Aviation Rules. Civil Aviation Rules are developed by CAA under contract to the Ministry of Transport and are the means by which standards are promulgated for the civil aviation sector. The rules must be consistent with International Civil Aviation Organisation (ICAO) standards, and take into account ICAO recommended practices, the level of risk pertaining to a particular activity and the cost of implementing proposed measures.

The rules development programme includes four distinct phases: issue assessment, policy investigation, rules development, and rules finalisation. Rule programme files track the process of developing a new rule from background research, through drafting, consulting with the sector and finalising the rule. This also includes emergency rules, changes and amendments, which are treated as separate project files, and exemptions.

Since 2001, CAA has operated a decentralised and hybrid information and records management system. The digital rule programme files offered for the live transfer pilot were created and stored on CAA’s shared drive and dated from 2003 to 2009. The files are recommended for permanent retention as public archives after 25 years under CAA’s current disposal authority, DA451.

Before the transfer could be initiated, an assessment was made of CAA’s ability to meet Archives’ digital transfer readiness characteristics. Following a delay due to the November 2016 Kaikōura earthquake, an initial trial extract from CAA’s shared drive was ingested into Archives’ test digital repository and analysed for content, technical, metadata and accessibility issues (see Transfer data analysis). Further consultation between Archives and CAA then took place to discuss this analysis, and to determine options and make decisions about the final extract. This resulted in signing off a digital transfer management plan in July 2017 and successfully completing the live transfer in August 2017.

Although a total of 560 individual files were transferred to Archives, only 549 were ingested into the Government Digital Archive as 11 were found to be system files, empty folders and duplicates. The total number of individual file formats was 19. The majority were in MS Word 97-2003 and PDF 1.3, with 12 in MS Excel 97-2000 and other file formats.

The files were accompanied by metadata in the form of a CSV file generated by the TreeSize Pro tool. Initial validation by Archives using original checksums provided by CAA revealed that three files were missing from the transfer. After investigation, it was found that these files had failed to copy to an external USB as the file path was too long (i.e., longer than 256 characters).

The system files, empty folders and duplicates found in the technical analysis were destroyed under Archives’ General Disposal Authority GDA7 after consultation with CAA. A test for sensitive, non-business related or draft material did not reveal any files which needed to be removed from the transfer for these reasons. Four PDF files failed the format validation due to an improperly formatted date (for example, Feb07) and were ingested with an appropriate note assigned to them for future context.

Retention period

CAA’s current disposal authority (DA451) has a retention period of 25 years for the rules development files (class 4.1.1) before transfer to Archives. The files selected for transfer met the required criteria of being open access and sentenced, but were less than 25 years old. Archives agreed to an early transfer option under section 21(2)(b) of the Public Records Act 2005. Future transfers of digital public archives may also need more flexibility in terms of age, retention periods and risk. digital records are more at risk of loss or inaccessibility than their physical counterparts, making early transfer prudent in many cases.

Responsibilities

During the transfer preparation phase, the CAA had to reopen access to the files. Access had been secured to ensure that there was no further editing or additions to network drives (including to the files for transfer) while some shared drive content was migrated to a new enterprise content management system. It is important for public sector organisations to be aware that up to the point of formal transfer to Archives, they are responsible for the information and records and for ensuring their maintenance, safety and security.

It is also important for organisations to be aware of their post-transfer responsibilities. They must destroy any remaining in-house copies of the transferred digital records. They must not download transferred digital public archives from Collections (which provides access to the Government Digital Archive) and save them back into their own systems as this will impact on the authenticity, reliability and integrity of the digital public archives as the ‘authoritative’ version.

Engagement

Each digital transfer is unique and is a learning process both for public sector organisations and for Archives. While an intermittent transfer process can still ultimately succeed, the loss of resources and expertise in both Archives and the transferring organisation during the process is a risk. Staff continuity and capability is particularly important as many transfers require bespoke or tailored solutions. As Archives gains more skills and experience with digital transfers, we will be able to systematise more processes and incorporate them into our guidance and tools.

This successful live transfer has shown how patience, persistence, and taking everything one step at a time can build the working relationship between a public sector organisation and Archives that is essential and beneficial for both parties. It provides a model for use with other organisations.