Introduction: Digital Preservation as a Relay Race
“Long-term preservation is basically a relay… and the only thing that survives all of those changes is the community of people who are interested in preserving that material.”
Q: Long-term preservation is basically a relay race where material is moved between different hardware, software, systems, and organizations over time. The single constant that survives all of these transitions is the community of people dedicated to preserving that material. That was the voice of Neil Jefferies, our guest for this edition of Epistemicast. Having worked on some of the earliest digital preservation and academic repository projects for the University of Oxford’s Bodleian Libraries, Neil now serves as Head of Innovation at the library’s Centre for Digital Scholarship and lectures on digital humanities. He is also the Executive Director of the Open Preservation Foundation and an author and editor of key standards supporting digital preservation and scholarly communications. If long-term preservation is a relay, technology standards are perhaps the most central part of passing on the baton. I started by asking Neil what drew him into digital academia, digital preservation, and open technologies.
Neil Jefferies’ Career Journey: From Antivirus and IT to Open Source
Early Experience in File Fixity, Data Recovery, and Linux
“Scanning files and identifying them and checking them for fixity is a mainstay of the digital preservation process.”
A: My entry into this space goes back to my early career before joining the academic world. Early on, I was recruited by Dr. Alan Solomon, creator of Dr. Solomon’s Anti-Virus Toolkit. That toolkit included tools like find virus to scan files and check virus to verify checksums and fixity. Anyone working in digital preservation knows that scanning files, identifying them, and checking fixity are core mainstays of the digital preservation process. We also worked in data recovery and digital forensics on damaged media, reassembling files to make them accessible again—techniques that are now widespread across the digital preservation universe.
A: Later, I worked for Mars, the chocolate company, when the commercial internet was brand new. I went on a training course where I was introduced to Linux and ended up writing web servers in Perl. Coming from a Windows environment, Linux was a revelation that recalled my earlier holiday job experiences on Sun workstations, cementing my passion for open-source software.
The Institutional Time-Scale Disconnect: Commercial Sector vs. Memory Institutions
“The average time that companies spent in that index… now it’s more like 15 to 18 years. The big technology organizations are becoming increasingly fragile… sitting in an organization where the university has been around for 900+ years.”
A: When I joined the Bodleian Libraries, I set up their first institutional repository and became increasingly involved in digital preservation. Moving from the corporate sector to an institution like the Bodleian makes you immediately aware of the radically different time scales on which these organizations operate. Having worked for corporate companies that no longer exist, you realize how fragile commercial organizations can be.
A: Research from Innosight on S&P 500 companies shows that in the 1950s, the average tenure on the index was around 30 years, whereas today it has shrunk to 15 to 18 years. Large corporate technology providers are fragile and short-lived. In contrast, Oxford University has existed for over 900 years, and the Bodleian Library for 400 to 500 years. Planning assumptions and infrastructure choices must adapt to these radically different temporal scales.
Defining Standards vs. File Formats in Digital Preservation
Moving Beyond File Storage to Preservation of Services
“What matters in terms of cultural and intellectual discourse is not just the preservation of the information, but the preservation of the services layered on top of it.”
Q: One common area of confusion is what actually constitutes a standard in digital preservation and how it differs from a file format.
A: Some file formats are standards themselves because they are well-documented and structured, allowing software to validate conformance. At the Open Preservation Foundation, tools like veraPDF or Jhove perform validation so you can verify that files adhere to specifications and remain readable across software tools. However, file standards only take you so far. Storing a file and verifying its integrity is necessary, but in today’s world, real utility comes from the services built on top of that content so it can participate in cultural and intellectual discourse.
A: To remain useful, digital content must be findable, provably authentic through provenance metadata, and deliverable in accessible chunks. For large media files, downloading entire files at once is impractical. What matters is preserving the surrounding discovery indexes, presentation tools, annotation capabilities, and analytics—even enabling consumption by AI systems.
Architecture and Abstraction: The Role of APIs in Software Stacks
“Standard APIs allow you to basically draw a line in the sand… you can change the software on either side of that API and as long as they continue to speak the same language, it shouldn’t matter.”
Q: Another important role of standards is helping memory institutions manage inevitable changes in their software stack over time.
A: Standards act as Application Programming Interfaces (APIs) that define standardized interaction points between systems. You can change or replace the software on either side of an API, and as long as both sides maintain standard communication, client applications like web browsers continue to function seamlessly. Standard APIs allow institutions to establish layers of abstraction, isolating core system layers and decoupling components so individual software pieces can be swapped out without breaking the wider environment.
Case Study: The International Image Interoperability Framework (IIIF)
Origins and Technical Motivation
“IIIF’s genesis was really a group of librarians grumbling over some drinks about how awful their image serving platforms were.”
Q: Neil has co-authored and edited key industry standards, including the International Image Interoperability Framework (IIIF).
A: IIIF started when a group of librarians realized that every past digitization project had produced an isolated, incompatible image-serving platform that was expensive and difficult to maintain. It originated as an internal solution to proprietary lock-in, but quickly demonstrated its power for scholarly research.
Unlocking Complex Scholarly Workflows
“The worst material we had was our medieval manuscripts… IIIF gives you the ability to seriously think about virtually reassembling these manuscripts.”
A: To design the standard, the IIIF community focused on solving challenges for the most difficult digitised items: medieval manuscripts. Medieval manuscripts are irregularly shaped, written on animal skins with holes, frequently erased and rewritten as palimpsests, and historically broken up across global collections.
A: With the IIIF Presentation API and manifests, scholars can virtually reassemble fragmented manuscripts or compare disparate copies side-by-side without needing libraries to exchange physical objects or duplicate files. It dynamically requests image fragments directly from authoritative institutional servers, preserving provenance while enabling annotations, analytics, and interactive discovery.
The Oxford Common File Layout (OCFL): Robust, System-Independent Storage
Human-Readable and Machine-Actionable Digital Objects
Pull Quote: “If you lost all of your software, a human could come along and look at your storage and using basic file system tools… work out what’s on that disk.”
Q: Neil also introduced the Oxford Common File Layout (OCFL), a newer standard rapidly becoming essential for academic repositories.
A: OCFL is designed for structured file storage. Complex digital objects consist of multiple files, versions, and administrative metadata. OCFL provides a standardized way to group related files, manage versioning efficiently, and check for file tampering.
A: OCFL localizes all object information directly within the storage directory hierarchy, eliminating reliance on external database indexes. If an institution loses its application software, a human can navigate the storage folders using basic command-line file tools to inspect objects, review versions, and read the embedded OCFL specification file stored at the root.
Eliminating High-Risk System Migrations
“OCFL recognizes that the time when preserved material is most at risk is when you move to a new software system and try to copy all the objects from the old preservation system to the new preservation system.”
A: Software typically reaches end-of-life every 10 to 15 years. Re-ingesting and processing every object during repository software migrations is the single most dangerous event in digital preservation. OCFL decouples stored data from the repository application.
A: For instance, with Fedora Repository version 7, upgrading from version 6 involves turning off version 6, installing version 7, and pointing it directly at existing OCFL storage without undergoing a data migration. Future standards work on OCFL version 2 aims to expand seamless, cross-platform migrations between different systems like Fedora, Invenio, and LibNova.
Community Governance vs. Software Lock-In
Open Source vs. Open Standards
“Open software is great, but it’s not the be-all and end-all… If that software conforms to an open standard, then even if an open software project is superseded, the infrastructure continues to work.”
Q: Standards prevent lock-in to specific software, even within open-source ecosystems.
A: Open-source code is valuable, but individual software projects do not last indefinitely. When software conforms to open standards like IIIF or OCFL, underlying servers can be replaced without breaking downstream user interfaces or scholarly citations.
A: While software and hardware change continuously over decades, the community of archivists, librarians, and technologists remains the persistent backbone.
Principles of Sustainable Standards Design
“From simplicity comes a degree of robustness and a degree of resilience, which are critical things for long-term preservation.”
A: Sustainable standards avoid scope creep, remain tightly focused, and ship alongside reference code implementations and conformance validators. Keeping file layouts simple and localized ensures long-term resilience and ease of disaster recovery.
Strategic Advice for Higher Education and Memory Institutions
“Openness is about making sure you retain control over a longer lifecycle… Take part in those conversations… it’s about control, but it’s about community as well.”
Q: Neil concluded with key advice for institutions managing long-term digital assets.
A: Libraries and cultural heritage institutions are long-term custodians of information. Institutions should actively adopt open standards, open-source software, and open file formats to maintain institutional control and manage risk across multi-century lifespans. Participating directly in standards communities ensures that preservation systems remain adaptable, robust, and community-owned.
Links
Oxford Common File Layout (OCFL): https://ocfl.io/
International Image Interoperability Framework (IIIF): https://iiif.io/
Details of veraPDF and Jpylyzer (referenced by Neil during the podcast) can found alongside a host of other useful tools and standards on the OPF homepage: https://openpreservation.org/
Innosight (rebranded as Huron) 2021 Corporate Longevity Report https://www.huronconsultinggroup.com/en/insights/creative-destruction
Fedora Repository Platform https://fedorarepository.org/
Paper on Harvard’s use of OCFL https://www.cni.org/topics/ci/a-new-storage-paradigm-for-sustainable-digital-stewardship
