Reporting and metrics

Status: Draft

TODO

• Define a timeframe for data retention (should relate to testing as we should be able to make decisions within this timeframe)

Objectives

Access

A greater breadth of our collection accessed

• Decrease in % of catalogue with 0 views

• Greater spread of works viewed across work types

Precision

People with a specific search intentions can have their expectations met

• Clicks per search (CPS) is measured passively by tracking users' behaviour while they use the search function. This is a variant of a traditional click through rate (CTR), calculated by taking the ratio of the number of items clicked to the number of distinct searches, for each anonymised session id

• Top n clicks per search (CPS-n) is almost exactly the same as the above, but only counts the clicks on works which appear in the top n results.

Metrics without a home

Session segmentation

We occasionally distinguish between 'discerning' and 'non-discerning' sessions where 'discerning' sessions are those which include searches beyond the first page of results. This follows an assumption that the users who work their way through a page of results and decide to keep going are more engaged than those who are satisfied with a single page. This is a very rough proxy for intent and we know that there will be some genuinely discerning users who never make it beyond the first page, either because the results are good enough to satisfy their needs already, or because they're so bad that they lose hope of finding the work(s) they're looking for. Nevertheless, this can be a useful way of splitting metrics to ensure that we're meeting the coarsest of user intentions.