📇
Catalogue API
  • Catalogue API
  • developers
  • How users request items
  • Search
    • Current queries
      • images
      • Query structure
    • Search
      • Changelog
      • Collecting data
      • Query design
      • Query design
      • wellcomecollection.org query development index
      • Reporting and metrics
      • Work IDs crib sheet
      • Analysis
        • Less than 3-word searches
        • Subsequent searches
        • Searches with 3 words or more
      • Hypotheses
        • Behaviours
        • Concepts, subject, and another field
        • Concepts, subjects with other field
        • Concepts, subjects
        • Contributor with other field
        • Contributors
        • Further research and design considerations
        • Genre with other field
        • Genres
        • Mood
        • Phrases
        • Reference number with other field
        • Reference numbers
        • Search scenarios
        • Synonymous names and subjects
        • Title with other field
        • Titles
      • Relevance tests
        • Test 1 - Explicit feedback
        • Test 2 - Implicit feedback
        • Test 3 - Adding notes
        • Test 4 - AND or OR
        • Test 5 - Scoring Tiers
        • Test 6 - English tokeniser and Contributors
        • Test 7 - BoolBoosted vs ConstScore
        • Test 8 - BoolBoosted vs PhaserBeam
    • Rank
      • Rank cluster
      • Developing with rank
      • Testing
Powered by GitBook
On this page
  1. Search
  2. Rank

Rank cluster

PreviousRankNextDeveloping with rank

Last updated 9 months ago

Our experiments can (and do!) produce long-running queries which can negatively affect the performance of services which real users see.

To avoid affecting the production services, we run the rank tests against a copy of the data in a completely separate cluster, copied using .

CCR is a memory intensive operation for both the leader and follower clusters - so much so that CCR itself can induce serious performance problems in the production cluster which can be tricky to resolve. To avoid that scenario, use the following procedure for refreshing indices in the rank cluster.

Replicating production indices to rank

  1. Make sure you have enough time to do this whole process, and let others know what's happening in case it goes wrong.

  2. Scale up the production pipeline ES cluster, either using scale_up_elastic_cluster in the , or by setting it manually to the 58gb configuration in Elastic Cloud.

  3. Replicate the works index:

    1. Open Kibana for the rank cluster

    2. Open "Cross Cluster Replication" under "Stack Management"

    3. Add the pipeline cluster as a new remote cluster, using the endpoint URL from the Elastic Cloud "Overview" page for the pipeline deployment.

    4. Set the leader and follower indices to works-indexed-${pipeline date}, and create the follower index.

    5. Replication has now begun - even if Kibana is showing that it hasn't! If you wait a while and look in "Index Management", you should see the new index fill up.

    6. Once complete, pause replication for the follower index. You can do this even when the index is still yellow (that just means that not every shard is replicated, and finishing that process does not require CCR).

  4. Repeat step 3 for the images index: images-indexed-${pipeline date}.

  5. Scale the production pipeline ES cluster back down (8gb).

If something goes wrong and the production cluster is overwhelmed, pausing the replication and applying a rolling restart to the production cluster should make things right again.

elasticsearch's CCR feature
pipeline Terraform module