> For the complete documentation index, see [llms.txt](https://docs.wellcomecollection.org/incident-retros/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.wellcomecollection.org/incident-retros/2021-06-28_slow-search.md). # Incident retro - slow search **Incident from:** 2021-06-28 **Incident until:** 2021-06-28 **Retro held:** 2021-06-29 * [Timeline](#timeline) * [Analysis of causes](#analysis-of-causes) * [Actions](#actions) ## Timeline 28 June 2021 11.51 AC kicked off reindex into 06-28 index 13.28 Reindex kicked off a second time Restarted later for a third time 15.16 Alert Down: Front End Works Search (Origin) Recovery: Front End Works Search (Origin) CCR was paused, which reduced load on API cluster, which caused the issue to stop 15.21 Comms sent out via status page Waited 5 mins to check the issue was resolved 15.33 Incident resolved ## Analysis of causes * Hard to keep an eye on reindexing * Looked okay in the morning, because the ingestor wasn’t doing anything, which wasn’t obvious * Cross-cluster replication whilst reindexing * How easy is it to find performance metrics? ## Actions **AC** * Improve the following/unfollowing process **RK** * Document the process * Include links to performance metrics that work for anyone who needs them **JG** * Propagate Elastic alerts to Slack: * Alert on CPU load * Alert on CCR --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.wellcomecollection.org/incident-retros/2021-06-28_slow-search.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.