Case study: First end-to-end born digital transfer
In 2016 we opened the doors to digital transfers with the release of two factsheets outlining the interim operating model and explaining digital transfer readiness characteristics. The Civil Aviation Authority (CAA) partnered with us to test mutual capabilities in a live transfer pilot. This transfer is now complete and is the first born-digital end-to-end transfer from a shared drive completed by us.
2. Transfer overview
CAA is a Crown entity established under the Civil Aviation Act 1990. It is responsible for the regulation of civil aviation, including the establishment and monitoring of civil aviation safety and security standards in New Zealand, investigation of accidents and incidents, and provision of aviation security services for domestic and international air operations.
CAA’s primary role is to effectively manage risks within the civil aviation system, and ensure continued confidence in the safety and security of the system. CAA uses different methods to manage these risks including education, guidance, and legislative tools like the Civil Aviation Rules. Civil Aviation Rules are developed by CAA under contract to the Ministry of Transport and are the means by which standards are promulgated for the civil aviation sector. The rules must be consistent with International Civil Aviation Organisation (ICAO) standards, and take into account ICAO recommended practices, the level of risk pertaining to a particular activity and the cost of implementing proposed measures.
The rules development programme includes four distinct phases: issue assessment, policy investigation, rules development, and rules finalisation. Rule programme files track the process of developing a new rule from background research, through drafting, consulting with the sector and finalising the rule. This also includes emergency rules, changes and amendments, which are treated as separate project files, and exemptions.
Since 2001, CAA has operated a decentralised and hybrid information and records management system. The digital rule programme files offered for the live transfer pilot were created and stored on CAA’s shared drive and dated from 2003 to 2009. The files are recommended for permanent retention as public archives after 25 years under CAA’s current disposal authority, DA451.
3. Transfer process
Before the transfer could be initiated, an assessment was made of CAA’s ability to meet our digital transfer readiness characteristics. Following a delay due to the November 2016 Kaikōura earthquake, an initial trial extract from CAA’s shared drive was ingested into our test digital repository and analysed for content, technical, metadata and accessibility issues (see section 5). Further consultation between us and CAA then took place to discuss this analysis, and to determine options and make decisions about the final extract. This resulted in signing off a digital transfer management plan in July 2017 and successfully completing the live transfer in August 2017.
4. Transfer Data Analysis
Although a total of 560 individual files were transferred to us, only 549 were ingested into the Government Digital Archive as 11 were found to be system files, empty folders and duplicates. The total number of individual file formats was 19. The majority were in MS Word 97-2003 and PDF 1.3, with 12 in MS Excel 97-2000 and other file formats.
The files were accompanied by metadata in the form of a CSV file generated by the TreeSize Pro tool. Initial validation by our staff using original checksums provided by CAA revealed that three files were missing from the transfer. After investigation, it was found that these files had failed to copy to an external USB as the file path was too long (i.e. longer than 256 characters).
The system files, empty folders and duplicates found in the technical analysis were destroyed under GDA7 after consultation with CAA. A test for sensitive, non-business related or draft material did not reveal any files which needed to be removed from the transfer for these reasons. Four PDF files failed the format validation due to an improperly formatted date (e.g. Feb07) and were ingested with an appropriate note assigned to them for future context.
5. Challenges and lessons learned
5.1 Retention period
CAA’s current disposal authority (DA451) has a retention period of 25 years for the rules development files (class 4.1.1) before transfer to our digital archive. The files selected for transfer met the required criteria of being open access and sentenced, but were less than 25 years old.We agreed to an early transfer option under section 21(2)(b) of the Public Records Act 2005. Future transfers of digital public archives may also need more flexibility in terms of age, retention periods and risk. Digital information and records are more at risk of loss or inaccessibility than their physical counterparts, making early transfer prudent in many cases.
During the transfer preparation phase, the CAA had to reopen access to the files. Access had been secured to ensure that there was no further editing or additions to network drives (including to the files for transfer) while some shared drive content was migrated to a new ECMS. It is important for public sector organisations to be aware that up to the point of formal transfer to our archives, they are responsible for the information and records and for ensuring their maintenance, safety and security.
It is also important for organisations to be aware of their post-transfer responsibilities. They must destroy any remaining in-house copies of the transferred digital information and records. They must not download the transferred digital public archives from Archway and save them back into their own systems as this will create more copies of the transferred files.
Each digital transfer is unique and is a learning process both for public sector organisations and for ourselves. While an intermittent transfer process can still ultimately succeed, the loss of resources and expertise in both ours and the transferring organisation during the process is a risk. Staff continuity and capability is particularly important as many transfers require bespoke or tailored solutions. As we gains more skills and experience with digital transfers, we will be able to systematise more processes and incorporate them into our guidance and tools.
This successful live transfer has shown how patience, persistence, and taking everything one step at a time can build the working relationship between a public sector organisation and ourselves that is essential and beneficial for both parties. It provides a model for use with other organisations.
Last updated on 29 July 2021