📦
Storage service
  • Introduction
  • How-to: basic operations
    • Ingest a bag into the storage service
    • Look up an already-stored bag in the storage service
    • Look up the versions of a bag in the storage service
  • How to: advanced usage
    • Getting notifications of newly stored bags
  • How to: debugging errors
    • Where to find application logs
    • Manually marking ingests as failed
  • Reference/design decisison
    • The semantics of bags, ingests and ingest types
    • How identifiers work in the storage service
    • How files are laid out in the underlying storage
    • Compressed vs uncompressed bags, and the choice of tar.gz
  • Developer information/workflow
    • An API reference for the user-facing storage service APIs
    • Key technologies
    • Inter-app messaging with SQS and SNS
    • How requests are routed from the API to app containers
    • Repository layout
    • How Docker images are published to ECR
  • Wellcome-specific information
    • Our storage configuration
      • Our three replicas: S3, Glacier, and Azure
      • Using multiple storage tiers for cost-efficiency (A/V, TIFFs)
      • Small fluctuations in our storage bill
      • Delete protection on the production storage service
    • Wellcome-specific debugging
      • Why did my callback to Goobi return a 401 Unauthorized?
    • Recovering files from our Azure replica
    • Awkward files and bags
    • Deleting files or bags bags from the storage service
Powered by GitBook
On this page
  1. Developer information/workflow

How Docker images are published to ECR

PreviousRepository layoutNextOur storage configuration

Last updated 2 years ago

We run our services in Docker containers. This page explains how a merge to the main branch becomes a Docker image in ECR that we can use.

  1. Somebody merges a pull request to main.

  2. This merge commit is picked up by BuildKite, our CI runner.

    Our BuildKite instance runs on EC2 instances in our AWS account, and the logs are only visible to Wellcome Collection developers.

  3. BuildKite looks at each project in turn, and decides if it has any changes that need rebuilding. Changes that might trigger a rebuild include:

    • Changes to the source code of an application

    • Changes to the source code of a library that an app depends on

    • Changes to Dependencies.scala, which might mean a new version of an underlying library

    If there are no changes, BuildKite exists early.

  4. If there are changes, BuildKite runs any project tests and then builds a Docker image using the Dockerfile in the project directory.

    The image is tagged with the Git ref of the merge commit (e.g. ref.9ff8df730259938685785337579137e097a9157f) and the tag latest. Note that the latest tag is a tag that moves to reflect the last image pushed by BuildKite.

    BuildKite then publishes the Docker image to two locations: ECR and ECR Public.

Screenshot of BuildKite. There's a large green box titled 'Merge pull request #895 from wellcomecollection/windows-line-endings' with some links to the branch/commit below it. Inside the box are smaller boxes named for each project arranged over a couple of lines: common, display, bags API, and so on.