📦
Storage service
  • Introduction
  • How-to: basic operations
    • Ingest a bag into the storage service
    • Look up an already-stored bag in the storage service
    • Look up the versions of a bag in the storage service
  • How to: advanced usage
    • Getting notifications of newly stored bags
  • How to: debugging errors
    • Where to find application logs
    • Manually marking ingests as failed
  • Reference/design decisison
    • The semantics of bags, ingests and ingest types
    • How identifiers work in the storage service
    • How files are laid out in the underlying storage
    • Compressed vs uncompressed bags, and the choice of tar.gz
  • Developer information/workflow
    • An API reference for the user-facing storage service APIs
    • Key technologies
    • Inter-app messaging with SQS and SNS
    • How requests are routed from the API to app containers
    • Repository layout
    • How Docker images are published to ECR
  • Wellcome-specific information
    • Our storage configuration
      • Our three replicas: S3, Glacier, and Azure
      • Using multiple storage tiers for cost-efficiency (A/V, TIFFs)
      • Small fluctuations in our storage bill
      • Delete protection on the production storage service
    • Wellcome-specific debugging
      • Why did my callback to Goobi return a 401 Unauthorized?
    • Recovering files from our Azure replica
    • Awkward files and bags
    • Deleting files or bags bags from the storage service
Powered by GitBook
On this page
  1. Wellcome-specific information
  2. Our storage configuration

Delete protection on the production storage service

PreviousSmall fluctuations in our storage billNextWellcome-specific debugging

Last updated 1 year ago

To prevent accidental deletion or modification of objects in permanent storage, we have a number of deletion protections in place:

  • Explicit deny on our standard IAM roles. Our give us permission to do almost everything in S3. We've added an explicit Deny for DeleteObject and PutObject in our permanent storage buckets, which makes those roles slightly less destructive.

    Where we grant cross-account access to the buckets, we only allow GetObject and ListBucket.

  • We enable S3 versioning on the buckets. This gives us a 30-day grace safety net against any accidental deletions.

  • We don't use . This is an S3 feature meant for objects that must absolutely be retained for a given period, e.g. for legal compliance. We don't enable it because we occasionally need to delete material, and Object Lock would prevent us from doing so.

  • We need to talk to D&T to unlock the Azure replicas. Even if we somehow deleted both copies of an object in S3, nobody in the Platform team has write access to the Azure replica. We have to talk to D&T to get that access, and there's typically a delay of several days to get that approved and set up.

    Additionally, once you're in Azure, we have a temporary legal hold which has to be removed before you can delete anything. This is a weaker variant of S3 Object Lock.

standard admin/developer roles
S3 Object Lock