# RFC 059: Splitting the catalogue pipeline Terraform This RFC proposes a change to how we manage the Terraform for instances of the catalogue pipeline. **Last modified:** 2023-07-03T10:39:33+01:00 ## Background We run multiple instances of the catalogue pipeline.\ We always have at least one pipeline which is populating the currently-live search index, but we may have more than one pipeline running at a time.\ Running multiple pipelines means we can try experiments or breaking changes in a new pipeline, and their data is isolated from the live search index (and the public API). However, we manage every pipeline in a single Terraform configuration, and this breaks some of the isolation.\ If you're trying to change one pipeline, you can inadvertently change another pipeline -- and this has caused a broken API on several occasions. We'd like to extend the isolation to Terraform, and keep each pipeline in a different configuration.\ That way, you can only modify one pipeline at a time, and you know exactly what you're changing. This RFC describes what that looks like in slightly more detail. ## Desirable properties * It should be easy to find the pipeline you want to work in. * It should be easy to create new pipelines. * Each pipeline should have its own Terraform state file and label.\ This should be specified in as few places as possible, to reduce the risk of inconsistency, e.g. a pipeline whose label is `2001-01-01` but the state file is named `2002-02-02`. * This should make sense to somebody who's already an experienced Terraform user – commands like `plan`, `apply`, `import` should all work as they expect (modulo our `run_terraform.sh` wrapper).\ You shouldn't need to learn a new tool to work with the pipeline Terraform. ## The proposal We create per-pipeline folders in the `terraform` directory, something like: ``` . └─ terraform/ ├─ 2001-01-01/ │ ├─ main.tf │ └─ run_terraform.sh ├─ 2002-02-02/ │ ├─ main.tf │ └─ run_terraform.sh └─ modules/ └─ ... ``` Each per-pipeline folder contains: * A single Terraform file that refers to a shared pipeline module in the `modules` directory * A `run_terraform.sh` wrapper which injects the Elastic Cloud secrets, and the name of the current pipeline as a Terraform variable.\ This gets read from the folder name, so it should be possible to use the same copy of `main.tf` across different pipelines. What this should look like for developers: * For a pre-existing pipeline, you `cd` into the folder and run `run_terraform.sh` as usual. * If you want to create a new pipeline, copy an existing folder and rename it.\ When you `cd` into that folder, you run `run_terraform.sh init` to start the new pipeline and then plan/apply as usual. ## Implementation details * **Partial configuration.**\ We can use [partial configuration](https://developer.hashicorp.com/terraform/language/settings/backends/configuration#partial-configuration) for Terraform backends to create a different state file for each pipeline.\ The `run_terraform.sh` script can read the name of the current folder, and pass the pipeline date in as a variable. * **Auto-deleting old copies of the `.terraform` folder.**\ When you copy an existing pipeline folder, you'll get the `.terraform` folder. We'll want to delete this folder in the `run_terraform.sh` script if it points to the wrong state file, to avoid Terraform trying to be "helpful" and inadvertently breaking a different pipeline.\ This is best understood with a worked example: > I create a pipeline `2001-01-01`, then I copy the folder to create `2002-02-02`. > > When I run `run_terraform.sh init` in the `2002-02-02` folder, it notices that the state in `.terraform` points at `s3://…/2001-01-01.tfstate`, but now I'm trying to keep state in `s3://…/2002-02-02.tfstate`. > > It offers to migrate the state to the new location; if I say yes, it will copy the state and then try to destroy the resources labelled `2001-01-01` on the next plan/apply. > > It would be better if our script knew that this was a different pipeline, and that we can't reuse the `.terraform` folder. * The per-pipeline state files should be in a dedicated prefix, so they can be easily identified, e.g. ``` s3://…/catalogue-pipeline/pipeline/2001-01-01.tfstate s3://…/catalogue-pipeline/pipeline/2002-02-02.tfstate s3://…/catalogue-pipeline/pipeline/2003-03-03.tfstate ``` ## Rejected approaches * **Creating an entirely custom tool.**\ We don't want developers to have to learn a new tool just to work on the catalogue pipeline Terraform -- it should be possible to create a familiar workflow with `run_terraform.sh` that's easier for everyone to learn. * **Terraform features: using workspaces, running a script with -chdir.**\ These are both Terraform features that might enable the sort of process we want, but we don't use them anywhere in the platform so they're not a familiar workflow for us.\ Using separate folders should be easier. --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://docs.wellcomecollection.org/request-for-comments-rfcs/059-splitting-pipeline-terraform.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.