> For the complete documentation index, see [llms.txt](https://docs.wellcomecollection.org/incident-retros/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.wellcomecollection.org/incident-retros/2021-01-28_downtime.md). # Incident retro - January downtime **Incident from:** 2021-01-28 **Incident till:** 2021-01-23 **Retro held:** 2021-01-29 * [Timeline](#timeline) * [Analysis of causes](#analysis-of-causes) * [Actions](#actions) ## Timeline 28 January 2021 See 19.00 AC merged a change to add the license aggregation to the /images API 19.08 deployment of catalogue API 19.13 onwards works API search, works api single work, front end works search, Wellcome Images redirect down alerts 19.20 Alex rolls back to previous production release of catalogue API 19.20, 19.27 recovery notification ## Analysis of causes Expected change to not need a reindex, single PR didn’t, but deployment of master had more there than in the single PR. The commit had a green tick so looked fine. Fields added to internal model did need the reindex. Because that reindex hadn’t happened it caused the API to fall over. Integration tests started failing Tue 26 January This wasn’t seen so didn’t know there was an issue. Integration tests are in wc-platform-builds channel. Had been moved out of wc-platform because they were too noisy. ## Actions **JP** Have build badges in Build Kite for: * the main build * integration tests - **AC** Reduce frequency of alerts so that it alerts: * At the start of every day * When state changes * **AC** Move the alert into wc-platform-alerts **AFC** Write RFC about internal model versioning and decoupling API and pipeline --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.wellcomecollection.org/incident-retros/2021-01-28_downtime.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.