> For the complete documentation index, see [llms.txt](https://docs.wellcomecollection.org/incident-retros/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.wellcomecollection.org/incident-retros/2021-03-02_stories-and-home-page-down.md). # Incident retro - stories and home page down **Incident from:** 2021-03-02 **Incident till:** 2021-03-02 **Retro held:** 2021-03-03 * [Timeline](#timeline) * [Analysis of causes](#analysis-of-causes) * [Actions](#actions) ## Timeline 2 March 2021 See 11.55 updown alerts for: Front End Homepage Front End Stories Front End Articles 12.00 NP can still see home page; JP cannot 12.00 AC confirms they can’t get to not available (I think this is the same URL as checked by Updown) 12.01 AC can get to some stories but not others 12.03 AC suggests rolling back to the release from 11:20 12.03 JP: I don’t really understand enough about the wc.org architecture to know but I guess the issue is with the content app 12.05 AC: notably, CI is failing for the latest commit 12.05 JP restarted the content app 12.07 agreed on rollback 12.08 AC We are rolling back to 5762fe6e-a3b1-40b1-bf51-c399cd2df35c 12.12 deployment done but not back up 12.22 RK, AC, JP, GE, DMc, AN voice called in Slack 12.30 DMc (in #wc-experience) There was a typo in ‘contibutor’ which Prismic was erroring with. Updating this on prismic.io now (but so far doesn’t appear to have solved the problem) Discussion about changing the model or the code 12.52 Updown still down alerts Front End Homepage Front End Stories Front End Articles 12.56 Where we’re at - contibutor has made its way into the Prismic schema and we’re going to stick with it for now to get a fix That means updating the graph query to use that spelling and keeping the Schema in the Prismic JSON editor that way too Use ‘contibutor’ typo in graph query - 1.00 1.02 JG: Once the build is done, the tests are quite quick. We just need to pass the deploy catalogue (ecr image) step, and I can deploy to stage. 1.17 JG: Deploying to stage. Probably 10 - 25 from now 1.21 JG: stage seems up. 1.23 JG deploying to prod 1.28 Updown recovery: Front End Homepage Front End Stories Front End Articles ## Analysis of causes Schemas in Prismic and generated by devs had got out of sync. Possibly was updated by devs but not updated in Prismic. There was no way to rollback to a previous version of the schema that was in Prismic. ## Actions **Gareth E** * Document process for updating the schema * Add contributor / save something on Prismic / deploy fix in the app / remove contibutor / save something on Prismic * Investigate weekly backups so they can be used for this sort of problem in future * Investigate a paid option that will give a development environment (Prismic is currently on the Platinum plan) **David** * Look into Prismic error handling **Robert** * Investigate why errors slipped through CloudFront --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.wellcomecollection.org/incident-retros/2021-03-02_stories-and-home-page-down.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.