Incident Retros
  • Incident retro - Internal model <=> Elastic index sync question
  • Incident retro - merging
  • Incident retro - Miro images
  • Incident retro - January downtime
  • Incident retro - Elastic Cloud
  • Incident retro - stories and home page down
  • Incident retro - search not available
  • Incident retro - ingestors
  • Incident retro - home page with json
  • Incident retro - slow search
  • Incident retro - search not available
  • Incident retro - search not available
  • Incident retro - cross cluster replication
  • Incident retro - 500'ing on the /images endpoint
  • Incident retro - home page and stories page not available
  • Incident retro - home page and what's on not available
  • Incident retro - requests not showing in account
  • Incident retro - requests not showing in account
  • Incident retro - works search errors
  • Incident retro - date picker
  • Incident retro - requests not showing in account
  • Incident retro - reporting cluster downtime and configuration loss
  • Incident retro - story page appearing then replaced by a 404
  • Incident retro - increased rate of errors in searches on wellcomecollection.org
  • Incident retro - slow search due to 900k messages on the ingestor queue
  • Incident retro - concept pages not available
  • Incident retro - Prismic model changes
  • Incident retro - Images search down
  • Incident retro - wc.org intermittently available
  • Incident retro - web site not available
  • Incident retro - search not available
  • Incident retro - search not available
  • Incident retro - users cannot login to their accounts on wellcomecollection.org
  • Incident retro - users cannot login to their accounts on wellcomecollection.org
  • Incident retro - digital assets not available
Powered by GitBook
On this page
  • Timeline
  • Analysis of causes
  • Actions

Incident retro - date picker

PreviousIncident retro - works search errorsNextIncident retro - requests not showing in account

Last updated 10 months ago

Incident from: 2022-08-08

Incident until: 2022-08-08

Retro held: 2022-08-09

Timeline

8 August 2022

See from https://wellcome.slack.com/archives/CUA669WHH/p1659961075741759

9.07 branch merged to remove the toggle for the date picker

13.14 Email to digital@wc.org: I just noticed that there is no drop down menu for readers to select the date for their requests at the moment. Can this be reinstated please?

13.17 AC: Email to digital@ about the missing date picker – could this be to do with the toggle removal?

13.18 DMC: sounds like a likely candidate I’ll take a look

13.25 DMC: I think possibly the result of me having merged my branch which will have automatically removed the toggle from the json, but I haven’t deployed this morning so the code still bailing out if the toggle isn’t present (which it won’t be

13.36 date picker back on prod

Analysis of causes

Qn: how do we (I) ensure that we don’t remove toggles until after the code that uses them is deployed?

There is only one toggles environment for stage and prod.

Merging a branch with updates to toggles updates the singular toggles environment.

Toggles turned on for the public can be turned off very quickly.

Actions

Jamie, David & Raphaëlle

  • Stop deploying toggles universally on merge

#8249
Timeline
Analysis of causes
Actions