Incident retro - requests not showing in account
Incident from: 2022-08-02
Incident until: 2022-08-02
Retro held: 2022-08-04
Timeline
1 August 2022
Identifying the unidentified #2149 merged
2 August 2022
See https://wellcome.slack.com/archives/C01FBFSDLUA/p1659430616459229
c. 2.00 Increase in requests to ES cluster
c. 7.00 Further increase in latency
9.56 JP catalogue API latency is currently averaging >1.5s, p95 is about 10s imo this is a degradation of service sufficient to be an incident the cluster is having a bad time from a huuuuge load of requests I’m going to bump the cluster size temporarily to give it more CPU credits Then I will investigate where those requests came from and what we should do about it (I don’t think those came via the catalogue API)
9.58 JP this started at about 2am and got quite a lot worse at about 7
9.59 NP message to users sent out
10.05 AC doubled the Elastic cluster size Latency got worse
10.13 NP update to users sent out re identifying cause as high number of searches
AC turned off relation embedder
c. 10.30 latency improved due to relation embedder turned off and increase in Elastic cluster size
12.24 AC looks like the incident has indeed resolved, but now we're not getting any pipeline updates
Analysis of causes
Increase in requests to ES cluster, caused by…
Overnight updates from Sierra. https://github.com/wellcomecollection/catalogue-pipeline/pull/2149 Meant all Sierra documents harvested from CALM were now different, so had to go through the whole pipeline (c 900,000).
This created a need for a proper reindex but the cluster was underpowered, and the batching behaviour impacted it.
If a change to the transformer will change the documents from CALM we need to reindex before an overnight harvest from CALM to Sierra
Turn down transformers
Change batching behaviour if this happens a few more times
Actions
Alex & Paul
Transformers: if more than e.g.10k messages on the queue, turn off the transformer and alert about it
Tidy Slack alert for more focussed information, and have it post to #wc-search, not #wc-platform
Last updated