# Incident retro - digital assets not available

**Incident from:** 2025-04-10

**Incident until:** 2025-04-10

**Retro held:** 2025-04-10

* [Timeline](#timeline)
* [Analysis of causes](#analysis-of-causes)
* [Actions](#actions)

## Timeline

10 April 2025

See <https://wellcome.slack.com/archives/C02ANCYL90E/p1744276895179669>

10.21 JC reported this work is showing an internal server error, this is a TEI record: <https://wellcomecollection.org/works/axqp36b6> (MS Sinhalese 326). Similar issue reported by an enquirer by email on 10 April at 9.55

10.44 SB it looks like it's not an issue with the catalogue API — the response looks as expected:\
<https://api.wellcomecollection.org/catalogue/v2/works/axqp36b6\\>
<https://api.wellcomecollection.org/catalogue/v2/works/bekbcf63><br>

10.46 AG Looks like it's happening with anything that has digital asset. Could it be the IIIF cloudfront changes applied this morning?

The catalogue API is fine\
<https://api.wellcomecollection.org/catalogue/v2/works/a22qnrb4\\>
for this work that is not fine <https://wellcomecollection.org/works/a22qnrb4>

10.49 AG messaged Digirati

Digirati checked logs but couldn’t see anything because the block occurred before the logging. Recommended reverting.

11.02 AG\
\[10/Apr/2025:10:00:00 +0000] "GET /works/a22ndzm4 HTTP/1.1" 500 90095 "-" "Amazon CloudFront"\
But\
\[10/Apr/2025:09:55:51 +0000] "GET /api/works/items/a22ndzm4 HTTP/1.1" 200 849 "-" "Amazon CloudFront"\
So it's intermittent

11.08 AG I'm checking out the commit on main just before the WAF changes\
<https://github.com/wellcomecollection/platform-infrastructure/commits/main/55180b77af92d388dc97fad5c0990ab271490a6d\\>
and running terraform from there \[to reform the terraform changes to see whether it gets things back to normal]

11.10 AG Plan gives us\
Plan: 0 to add, 7 to change, 6 to destroy.\
Which matches what was changed/created this morning

11.15 AG Apply in progress

11.35 SB It looks like it recovered after the terraform apply. (I think that some pages might still return 500 for a while due to caching)

12.11 incident closed

## Analysis of causes

weco.org firewall block was applied to IIIF firewall and Digirati added to the blocks

## Actions

**Digirati via AG**

* IIIF WAF count rather than block to see how many there are
* Ask Digirati if they have load monitoring

**SB**

* Give Digirati info on IIIF WAF rules

Following answers from above:

* set alam on CF if blocking more than x% requests to the site
* extend CF alerting to IIIF instances


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.wellcomecollection.org/incident-retros/2025-04-10_digital_assets_not_available.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.