Introduction
Last updated
Last updated
The storage service manages the storage of our digital collections, including:
Uploading files to cloud storage providers like Amazon S3 and Azure Blob
Verifying fixity information on our files (checksums, sizes, filenames)
Reporting on the contents of our digital archive through machine-readable APIs and search tools
The storage service is designed to:
Ensure the safe, long-term (i.e. decades) storage of our digital assets
Provide a scalable mechanism for identifying, retrieving, and storing content
To support bulk processing of content, e.g. for file format migrations or batch analysis
Follow industry best-practices around file integrity and audit trails
Enable us to meet for both digitised and assets
This is the basic architecture:
Workflow systems (Goobi, Archivematica) create "bags", which are collections of files stored in the BagIt packaging format. They upload these bags to a temporary S3 bucket, and call the storage service APIs to ask it to store the bags permanently.
The storage service reads the bags, verifies their contents, and replicates the bags to our permanent storage (S3 buckets/Azure containers). It is the only thing which writes to our permanent storage; this ensures everything is stored and labelled consistently.
Delivery systems (e.g. DLCS) can then read objects back out of permanent storage, to provide access to users.
This GitBook space includes:
How-to guides explaining how to do common operations, e.g. upload new files into the storage service
Reference material explaining how the storage service is designed, and why we made those choices
Notes for Wellcome developers who need to administer or debug our storage service deployment
The READMEs in the repo have instructions for specific procedures, e.g. how to create new Docker images. This GitBook is meant to be a bit higher-level.
An ingest is a record of some processing on a bag, such as creating a new bag or adding a new version of a bag.
Once you have a running instance of the storage service, you can use it to store bags. These guides walk you through some basic operations:
Once you're comfortable storing individual bags, you can read about more advanced topics:
[Storing multiple versions of the same bag]
[Sending a partial update to a bag]
[Storing preservation and access copies in different storage classes]
[Reporting on the contents of the storage service]
[Getting callback notifications from the storage service]
and some information about what to do when things go wrong:
[Why ingests fail: understanding ingest errors]
[Operational monitoring of the storage service]
These topics explain how the storage service work, and why it's designed in the way it is:
[Detailed architecture: what do the different services do?]
[How bags are verified]
[How bags are versioned]
These topics are useful for a developer looking to modify or extend the storage service.
[Adding support for another replica location (e.g. Google Cloud)]
[Locking around operations in S3 and Azure Blob]
Developer workflow:
All our storage service code is in
The unit of storage in the storage service is a bag. This is a collection of files packaged together with , which are ingested and stored together.
Each bag is identified with a space (a broad category) an external identifier (a specific identifier) and a version. .
We have that spins up an instance of the storage service. You can use this to try the storage service in your own AWS account.
You can read the for more detailed information about how to use the storage service.
We also have the , the original design document -- although this isn't actively updated, and some of the details have changed in the implementation.