Incident retro - search not available
PreviousIncident retro - search not availableNextIncident retro - users cannot login to their accounts on wellcomecollection.org
Last updated
Last updated
Incident from: 2024-05-07
Incident until: 2024-05-10
Retro held: 2024-05-13
Site was checked against Prismic staging
Decision to not to make changes ahead of a bank holiday weekend
DM: Pushed Slice Machine types to prod and migrated content after checking with Editorial Told editorial about changes to Quote slices and Image gallery slices
: 1st There was an error in the content-pipeline-2023-03-24 Lambda alert in #wc-platform-alert channel Cloudwatch logs indicate the issue is around fetching the data from the Prismic API
14:30ish: DM, RC, RK and AG get together to debug. The issue is identified as the content-api’s articles graph query being incompatible with the new Prismic data model. The graph query is modified to work with the new Prismic data model.
(15:00-16:00 break for review meeting)
16:00: Fix is being implemented and tested locally. PR open for review
16:39: Last alert pop in #wc-platform-alert
16:50: Fix is approved, merged and deployed. Content-pipeline-2023-03-24 alerts stop
Promo images aren’t always using the 16:9 version
‘In Pictures’ articles aren’t showing all images by default
Old ‘webcomics’ (Body Squabbles) are rendering as image galleries with one image
One article from 2017 was somehow listed as the most recently published thing I have delisted the old article.
I’ve got a PR on the way that deals with 2. and 3. impact of 1 is that images aren’t all the same aspect ratio in cards currently (don’t think this is a huge deal?) [aspect ratio via contentUrlSuffix didn’t get taken over in the migration. Reported to Prismic as a bug with migrating assets] Change expected URL for meta image #10859
DM: I’ve just removed and re-added a promo image for one of the cards on the /stories page and it appears to have fixed it for that one so I think it might be a case of doing that for the ones on landing pages I think this is something we can report to Prismic as a bug with migrating assets
What happened that we didn’t anticipate?
Content API affected
Opening times weren’t displaying “name” and “image” as they didn’t have a “label” anymore
Promo images aren’t always using the 16:9 version
‘In Pictures’ articles aren’t showing all images by default
Old ‘webcomics’ (Body Squabbles) are rendering as image galleries with one image
One article from 2017 was somehow listed as the most recently published thing
Drafts weren’t migrated
NB Alerts caught the content API issue
Why didn’t our safeguards catch this?
Mostly these were subtle changes to the site
No end to end tests with the migrated material
DM
Migrate any remaining docs in draft
Investigate how to find drafts
Prismic
To investigate why image weight label crops didn't migrate
NP
Take to planning: Toggle for data from Prismic staging / run e2es on Prismic staging to enable sharing of the site with a wider group before migrating
Future migrations
Share with more people before migrating e.g. devs and editorial
Kickoff with reps from other teams
AG: could it be possible that the slice machine changes are affecting the way we're fetching prismic articles and events to write them into the content index? We're seeing errors in the content pipeline where it's trying to fetch prismic docs. Articles weren’t modified yet
DM Things I’ve noticed after remapping content:
Removed Bodies of Knowledge symposium to help with a bug fix Draft articles not migrated in the same way as published articles, and not easy to find
DM you should be good to add/edit in Prismic now, but there are still old events that you won’t currently be able to see the body content for
17.07 Email reported that the was missing the location in the building - collection venues weren’t migrated (Collection venues weight/label needs to be added )
RK: Am I right in thinking if we'd been using the content API to front all prismic content we'd have avoided this migration issue? The pipeline would have broken but we'd have maintained user facing content. RC: Hmmm that's a good question. We still would have had to migrate the content in order for it to render in the CMS editing side?
[Images] RC: It looks like just the width is now getting passed and anything else is ignored, which explains the difference in height. I feel like it has to do with missing contentUrlSuffix. They should probably contain w=, h= and rect= for crops and they're just empty strings?
Fix for opening times page deployed content for draft event publishing this afternoon (JWM perspective tour and workshop) Fix for change expected URL for meta image
Add publishDate to articles graphQuery deployed; reindex run Monday 13 May [fixed issue with old article that had to be delisted] Override date bug already existed but wasn’t known about - that date wasn’t fetched by the content API