📦
Storage service
  • Introduction
  • How-to: basic operations
    • Ingest a bag into the storage service
    • Look up an already-stored bag in the storage service
    • Look up the versions of a bag in the storage service
  • How to: advanced usage
    • Getting notifications of newly stored bags
  • How to: debugging errors
    • Where to find application logs
    • Manually marking ingests as failed
  • Reference/design decisison
    • The semantics of bags, ingests and ingest types
    • How identifiers work in the storage service
    • How files are laid out in the underlying storage
    • Compressed vs uncompressed bags, and the choice of tar.gz
  • Developer information/workflow
    • An API reference for the user-facing storage service APIs
    • Key technologies
    • Inter-app messaging with SQS and SNS
    • How requests are routed from the API to app containers
    • Repository layout
    • How Docker images are published to ECR
  • Wellcome-specific information
    • Our storage configuration
      • Our three replicas: S3, Glacier, and Azure
      • Using multiple storage tiers for cost-efficiency (A/V, TIFFs)
      • Small fluctuations in our storage bill
      • Delete protection on the production storage service
    • Wellcome-specific debugging
      • Why did my callback to Goobi return a 401 Unauthorized?
    • Recovering files from our Azure replica
    • Awkward files and bags
    • Deleting files or bags bags from the storage service
Powered by GitBook
On this page
  1. Wellcome-specific information
  2. Our storage configuration

Small fluctuations in our storage bill

PreviousUsing multiple storage tiers for cost-efficiency (A/V, TIFFs)NextDelete protection on the production storage service

Last updated 2 years ago

The storage service runs in a dedicated AWS account, so the account bill only reflects the cost of the storage service.

The dominant cost in this account is the Standard-IA bill, but at a glance it can look a little disconcerting. In particular, our Standard-IA bill goes down as well as up – but why would it ever go down? Why would we ever remove objects in Standard-IA? Should we be concerned about the integrity of the storage service?

No – slight fluctuations in the Standard-IA bill are a normal part of storage service operations.

Why?

In the graph above, you can see a drop in the Standard-IA bill at the beginning of the month – but it's accompanied by a rise in the Glacier/Deep Archive bill. A bunch of objects just got transitioned into a cheaper storage class.

A substantial drop in the Standard-IA bill may be a red flag, but small fluctuations aren't a cause for concern.

When objects in the warm replica , they get removed from Standard-IA. That cost moves from Standard-IA to Glacier.

When objects in the cold replica , they get removed from Standard-IA. That cost moves from Standard-IA to GDA.

To see this with our real numbers, visit . Notice how our StandardStorage bill similarly fluctuates (although it's a much smaller part of our bill) – this is when objects transition from Standard to Standard-IA.

transition to Glacier
transition to Glacier Deep Archive
Cost Explorer filtered to storage costs only
The shape of our storage bill in April 2023. Not drawn to scale; drawn to illustrate the general shape.