National Archives of Australia Audiovisual Asset Management And Preservation System (AVAMS)

Introduction

Established by an Act of Parliament, the National Archives of Australia (NAA) is charged with collecting, preserving, and sharing Australia’s most valuable government records and encouraging their use by the public. It also has a responsibility to promote good collection management practices to other government agencies.

In fulfilling these responsibilities, the NAA ensures that Australians have access – both online and in person – to a national archival collection, so they may better understand their heritage and democracy. Through individuals’ interactions with various government departments, aspects of their lives are preserved for future generations, making this an excellent resource for family historians as well as academics.

The collection includes almost one million physical and digital audiovisual assets in a variety of formats, including film, video, audio recordings, and digital formats created by the Australian National Broadcasting Authority and other government bodies. Along with the media itself, is information about the content and context of a production, and detailed physical and technical data about the audiovisual analogue and digital assets, which are regarded as equally significant. Altogether, this constitutes a complete record.

The Challenge

The NAA wanted to achieve a cohesive collections management system for its audiovisual archives which would incorporate one set of functionality for asset management and digital preservation. The project became the highest strategic priority of the NAA from 2012-2014. At the time of implementation, there were nearly one million items in the audiovisual collection, a number which continues to grow.

While new accessions to the collection tend to arrive in digital format, for instance from national broadcaster ABC, which delivers both radio and television content as files, much of the historic material was on analogue tapes and film. These are degrading rapidly, through issues such as “vinegar syndrome”, the degradation of the cellulose film stocks. So the NAA needed to not only catalogue its assets but incorporate information on the state of materials in order to prioritise restoration and preservation work.

As part of the work, the NAA wanted to make the content available as widely as possible. As well as 30 workstations for archivists and researchers at headquarters in Sydney, the requirement included the ability to access the system throughout the country, including all seven NAA offices.

At the start of the project, information on assets was contained on two independent databases: an in-house development called RecordSearch which supports core archive functionality including transfer location and lending (TLL), and serves as the front end in making content widely available for public searches; and PSaMI, (Preservation Sound and Moving Image), an audiovisual database developed using Filemaker Pro software, which was earmarked to be replaced with the enterprise collection management system. Secure and seamless integration of these two databases was a critical part of the challenge.

Design Requirements

Understanding that this was a very large project and would require considerable expertise, the NAA specified that the solution should use existing, proven software which could be demonstrated in the context of an audiovisual archive. It needed to be scalable, robust and capable of being implemented in a cost-effective and timely way.

It had to provide functional links to the existing RecordSearch system, which is the interface for public searches and replace PSaMI Filmaker Pro database in Sydney.  The database had some 600,000 entities already registered. The NAA also operated an electronic record-keeping system for corporate administrative records, including digital copies of paper business records. These physical assets also needed to be integrated.

Tracking And Uniquely Identifying Physical Assets

The central challenge for audiovisual collection management, which sets it apart from digital asset management, is that it also has to include provision for tracking and uniquely identifying physical objects.

The NAA has more than 30km of shelving, repositories of different temperatures, and conditioning rooms for the gradual movement of items into room temperature for access or digitization.

Shelf optimisation to ensure that the available space did not run out, was a key requirement.The NAA has a variety of different shelving configurations for different item sizes and types.

TMD’s Mediaflex-UMS asset management platform allows the management of all this, and adds capacity management, a visual interface showing where spare space is and how full shelves are in real time.

The metadata schema had to include information on the state of physical assets and their location, to allow archivists to priorities digital transfers and physical preservation.

The requirements to manage audiovisual digitisation workflows, storage of physical items, ingest, and digital preservation are much more complex than those for other format types such as photographs or paper.
In order to better manage and search on collection items, a data model with multiple layers was needed.

Typically, libraries have three data layers, and archives have five or six for paper formats. However, in the case of audiovisual, the ideal data model has 12 layers.

To develop a metadata schema this complex, it was a design requirement that the supplier and the NAA would jointly work together to create the detailed software and workflow solution.

Solution

TMD had already implemented a very large audio-visual archive collection management system in Australia, at the National Film and Sound Archive (NFSA).  TMD was able to demonstrate the NFSA system, and show sufficient flexibility to allow it to be closely suited to the NAA’s needs.

The proven functionality of Mediaflex-UMS, TMD’s media asset management platform, together with the experience of the TMD project team in Australia, established that the system could be implemented within the NAA’s budget and in its short timescale of two years.

In addition, the NAA asked TMD to deliver an alternative to a more commercially available HSM system. That subsequent development has become a new product for TMD called Paragon for archiving, a highly configurable archiving solution for companies of any size.

The first phase of the project involved a workflow analysis, focusing on new workflows while monitoring current practices ensuring a complete understanding of operations, assisting in final training and implementation.

The solution was also able to go beyond the minimum requirements, not least by migrating data from the existing RecordSearch and PSaMI applications, making all the metadata available in Mediaflex through one common interface. The TMD team developed a flexible, configurable methodology, which was used to map, clean, and migrate the data.

TMD used the Spiral iterative methodology to ensure that users and developers jointly shape the solution. This gave strong assurance both on the data migration aspect of the project and for the final structure of the new system that all would be as expected. Collaboration at this level also ensured that staff were engaged, clearly understood the new system and were ready to make maximum use of it from day one.

Development of the flexible, configurable data migration methodology was carried out in parallel with the development of the main functionality. Numerous transfers were made from RecordSearch and PSaMI to TMD’s Mediaflex-UMS system, both to prove the data migration and cleaning and to provide real data for workflow developments. Users were thus assured of the security of the final data transfer, at the time of cutting over to the new system.

The system met all the design challenges and delivered effective new solutions in other areas. An immediate benefit Mediaflex-UMS brought is the ability to better manage the storage of audiovisual items.

Despite the enormous challenges, the project was completed on budget and on time, moving from the initial contract to going live in just two years.

David Fricker, director general of the NAA said “We knew that we had a specific set of requirements, and the solution needed to be robust, scalable and flexible as we transition to digital delivery while continuing to control nearly a million individual audiovisual records. TMD worked closely with the NAA in all aspects of the project, including detailed scoping, project management and governance. They put an excellent team on site and using modern agile development techniques, we could clearly understand what was happening and provide feedback throughout the project.

“The result is something I can confidently say will meet the highest standards of audiovisual collection management and digital preservation for the National Archives.”

Collection Management

The experience gained in deploying the collection management function at the NAA in addition to the work done at the NFSA has been developed further to become TMD’s media services application for collections management. Mediaflex focuses entirely on the requirements of National and Cultural AV Archive organisations including:

• Acquisition of articles into a collection, allowing users to identify containers, describe articles, evaluate articles and make decisions about selection or non selection.
• Accessioning of articles into the collection
• Transferring bulk items directly into the collection through rules-based Excel templates or adding them manually.
• Raising of preservation workflows either automatically or manually, providing automated control of media processing tasks such as file copies and moves, checksum creation and validation and as required direct control of 3rd party devices:
• Storage and management of digital articles for security and long term preservation
• Access to articles by cataloging staff to enhance descriptive metadata
• Providing full history and audit trails for all articles while part of the Collection
• Reporting
• Making articles available to a wider audience

KEY FACTS
Go Live: March 2014
Volumes: nearly 1 million articles
Users: 30 researchers, 400 staff in 7 locations

KEY TECHNOLOGY PARTNER INTEGRATIONS:
Storage: Mitsubishi SAN
Transcoder: Telestream Vantage and Amberfin iCR
Robot: IBM TS3310 with 4x LTO6 drives
Content Storage Management: Mediaflex Paragon
Ingest: SAMMA Solo (video tape content)
Quadriga/Audiocube (audio tape collection)
Business Systems: Record Search (bespoke in-house)