> For the complete documentation index, see [llms.txt](https://docs.wellcomecollection.org/catalogue-pipeline/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.wellcomecollection.org/catalogue-pipeline/readme.md).

# Introduction

**The catalogue pipeline populates the search index for** [**our online catalogue search**](https://wellcomecollection.org/collections)**.**

This includes:

* fetching records from source catalogues and keeping them up-to-date
* transforming records into a single, common model
* combining records from multiple sources, where appropriate
* creating an Elasticsearch index which can be queried by [the catalogue API](https://github.com/wellcomecollection/catalogue-api)

## Documentation

This GitBook space is meant to provide a high-level overview of the catalogue pipeline and its design. These docs are meant for Wellcome Collection developers who want to learn about the project, or for colleagues at other institutions who want to build something similar.

It does **not** contain specific operational details, e.g. how to deploy specific services. Those are kept inside the code repository.

## Repo

The catalogue pipeline code is in <https://github.com/wellcomecollection/catalogue-pipeline>


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.wellcomecollection.org/catalogue-pipeline/readme.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.