Incident retro - search not available

Incident from: 2021-09-20

Incident until: 2021-09-20

Retro held: 2021-09-22

• Timeline

• Analysis of causes

• Actions

Timeline

20 September 2021

See https://wellcome.slack.com/archives/C01FBFSDLUA/p1632152123024700

16.32 Updown alert for: Front End Works Search (Origin) Front End Works Search (Cache)

16.35 DM: I’ve just deployed, but server routing for works is broken. I’ve got the fix and think we should be able to roll forward

16.39 JP yeah tempted to say we roll back for now

16.40 NP put a message about the search issue in #wc-platform-feedback

16.41 DM has a PR https://github.com/wellcomecollection/wellcomecollection.org/pull/7062/files

16.43 JP I’ve started a rollback here btw; can’t do any harm even if we get that PR in

16.44 JP But I do think we should wait to check stage before promoting [the PR fix]

16.45 Updown recovery for: Front End Works Search (Origin) Front End Works Search (Cache)

16.46 NP Okay, looks like I can search again. [Fixed by the rollback, not the PR] Also said search was working in #wc-platform-feedback

Analysis of causes

A deployment caused server routing for works to break, and end to end tests didn’t catch the problem on stage.

Actions

DM

• Add render tests for the catalogue app top-level pages #7063 DONE
• Create a runbook for front end incidents

JG

• Investigate why end to end tests didn’t break on all pages