# Incident retro - concept pages not available **Incident from:** 2022-11-03 **Incident until:** 2022-11-03 **Retro held:** 2022-10-04 * [Timeline](#timeline) * [Analysis of causes](#analysis-of-causes) * [Actions](#actions) ## Timeline 3 November 2022 See and 14.44 AC doing the 2022-03-11 reindex to: * Fix the snapshot reporter * Flush out some pipeline issues I’m going to kick off by deleting the 2022-10-14 pipeline, which seems like it was never used AC thought it was never used based on 15.05 AC Was concepts looking at the 2022-10-14 index?\ yes\ gonna roll it back 15.19 AC FYI the concepts pages are broken, because the concepts API is broken\ For some reason we had a mixture of API indices: * /works in prod = 2022-10-03 * /concepts in prod = 2022-10-14 * /works in stage = 2022-10-14 15:20 So I thought it was safe to delete 2022-10-14 and spin up 11-03, turns out that was wrong\ I’m rolling everything back to 10-03, and preparing 11-03 for rolling forward 15.24 JP suggests something went wrong with deployment? AC yeah errr | service | old image | new image | Git commit | | ------------------- | --------- | --------------- | ---------- | | concepts | ae604a2 | - | - | | items | ------- | No image found! | | | search | ------- | No image found! | | | snapshot\_generator | ------- | No image found! | | I’m having to deploy locally with weco-deploy, because Buildkite is unhappy at the missing secrets It wants to see 2022-10-14 because that’s what the staging API has, but I’ve deleted that I’m deploying the latest images to staging I wonder if something between our build short-circuiting and weco-deploy has gone wrong, e.g. there is no image tagged ae604a2 because it was only a change to the concepts API 15.34 AC seems to be okay, deploying to prod 15.43 AC Should be back up now ## Analysis of causes Thought same index was being used in works and concepts API Something went wrong with the deployment of the 2022-10-14 index - bug in we-co deploy? ## Actions **Alex** * Remove all the “clever” build logic from all the scala repos [#5626](https://github.com/wellcomecollection/platform/issues/5626) * Simplify the deployment logic in weco deploy so it always deploys a complete set of images --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://docs.wellcomecollection.org/incident-retros/2022-11-03_concept-pages-not-available.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.