RFC 090: CMS to LMS Sync
Purpose
This RFC proposes an automated pipeline to synchronize data from Axiell Collections (AxC), the new Content Management System (CMS), into FOLIO, the new Library Management System (LMS). The records from Axiell Collections need to exist in FOLIO so they can be requested and circulated in the LMS. It is designed for idempotency (safe to replay without duplication), auditability (every create / update / suppress is recorded), and graceful error isolation (one bad record never halts the batch).
Last modified: 2026-06-30T00:00:00+00:00
Table of Contents
Background
The Wellcome Collection library systems are undergoing migration. They currently operate as distinct layers: CALM (Collections Management), the Content Management System (CMS), serves as the collection metadata source of truth, while Sierra, the Library Management System (LMS), manages patron management, circulation, holds, and requesting. The CALM-to-Sierra harvester is the integration mechanism that synchronizes bibliographic metadata from CALM into Sierra for discovery and requesting workflows.
The library systems migration is replacing both layers. CALM is being migrated to Axiell Collections, and Sierra is being migrated to FOLIO. The current CALM-to-Sierra harvester will be superseded by an Axiell Collections-to-FOLIO integration pipeline. The data must be synchronized between the CMS and LMS so that the circulation of items and patron management can be carried out.
CALM-sourced data in Sierra is not being migrated to FOLIO. Therefore, AxC data must be synced to FOLIO to enable requesting and circulation workflows for items that were not included in the initial FOLIO migration. Library staff rely on FOLIO for real-time item availability and location data. Axiell-to-FOLIO sync is a core data pipeline for the new catalogue system and must be reliable, auditable, and manually recoverable.
System Architecture
Current Data Feeds for AxC and Folio Data
Axiell Collections (AxC) is the source: the Axiell Adapter Platform harvests AxC over OAI-PMH every 15 minutes and writes each record as raw MARCXML into a single Apache Iceberg table on S3, emitting on every run a changeset, the records created, modified, or deleted in that window, identified by changeset_id. FOLIO is the new system of record for library management (instances, holdings, items, patrons) and also exposes its own OAI-PMH feed every 15 minutes.
Volume: 10–500 records per changeset, ~1,000 records/day across ~80 syncs.
Adapter pipeline
Trigger
Compute the next harvest window from WindowStore history
Loader
Harvest OAI-PMH records in the window, write raw MARCXML to Iceberg, emit changeset_ids
Transformer
Read changesets from Iceberg, parse MARCXML (pymarc) into a SourceWork, index to Elasticsearch
Reconciler
Track GUID→ID mapping changes; emit DeletedSourceWork for superseded identifiers
Key characteristics
The Axiell adapter harvests over OAI-PMH using the oai_marcxml metadata prefix and the collect OAI set, authenticating with a custom Token header rather than the standard Authorization header. Record identity comes from the axiell-guid extracted from MARC 001, while visibility is controlled by the InvisibleSourceWork flag (MimsyWorksAreNotVisible). Harvesting runs in 15-minute windows with a 7-day lookback and a 360-minute maximum lag. All records land in a single Iceberg table per adapter (row schema described under Change Detection).
Proposed Change: FOLIO Item Upsert on Every Adapter Run
Fan-out Mechanism
The FOLIO upserter is best understood as a new type of transformer in the existing catalogue pipeline architecture. The architecture already has the concept of a transformer: a step that consumes a changeset from Iceberg and writes the records into a downstream sink. The current AxiellTransformer transforms MARCXML into a SourceWork and indexes it into Elasticsearch (the discovery sink).
The FOLIO upserter follows the same pattern but targets a different sink, the LMS (FOLIO), instead of Elasticsearch, and its write operation is an idempotent upsert of item data (instance → holdings → item) over the FOLIO OKAPI Inventory APIs rather than an index write. In other words: same input (changeset_ids over the same Iceberg table), new transformer, new destination.
So after the loader emits changeset_ids, the event fans out to two independent transformers: the existing ES transformer (AxiellTransformer, unchanged) writing to Elasticsearch for discovery, and the new FOLIO upserter writing to FOLIO Inventory for circulation and requesting in the LMS. Both consume the same changeset_ids independently, and either path can fail and retry without affecting the other, so adding the LMS sink does not put the existing discovery feed at risk.
The FOLIO upserter's internal module flow (ref_cache → read Iceberg → mapping.py → upsert.py → OKAPI) is detailed under Transformation Pipeline and the Lambda execution steps; its AWS deployment wiring is in the System Diagram.
Transformation Design
Approach
The upserter converts raw MARCXML from the AxC adapter into FOLIO payloads using typed Python with Pydantic models (mapping.py). The three payloads (Instance, Holdings, and Item) are Pydantic models with extra="forbid", so the field-mapping rules are ordinary, reviewable Python and the payloads are typed contracts: a malformed payload (missing/ill-typed required field, typo'd key) fails at build time, before any OKAPI call, instead of surfacing as a FOLIO 422 mid-batch.
This replaces an earlier YAML-driven mapper (mapping.yaml + YamlMapper); see Mapping: Pydantic Models vs. YAML Mapper for the trade-off.
Transformation Pipeline
Pydantic Mapping (mapping.py)
mapping.py is the single home for everything that decides what an Axiell record becomes in FOLIO: the MARC source table, normalization tables, defaults, the hrid scheme, the typed payload contracts, and the field-by-field builders.
The flow runs in three steps. parse_marcxml() (in mapper.py) first extracts MARC fields via the MARC_SOURCE table (e.g. title: 245$a, location_code: 852$b, barcode: 949$a) into a typed CanonicalRecord, where MARC 001 is the Axiell GUID and the basis for every hrid. build_instance/holdings/item() then map that record into the Instance, Holdings, and Item Pydantic models, resolving reference data (location, material type, loan type, holdings source, instance type) through RefCache and applying defaults. Finally, Pydantic validates on construction (extra="forbid" rejects unknown keys) and build_payloads() returns {"instance", "holdings", "item", "meta"} as JSON-ready dicts, with meta carrying source_id, the three hrids, and mapping_version (used for audit, rollback, and tracing a record back to the rules that produced it).
hrid scheme (the idempotency key for every upsert): AxC-instance-{guid}, AxC-holding-{guid}, AxC-item-{guid}. Instance.source = "FOLIO" (FOLIO-native, no linked SRS record; see Design Decisions).
Reference Data Cache (ref_cache.py)
FOLIO APIs require UUIDs/IDs, but Axiell supplies human-readable names, so RefCache translates names → FOLIO IDs and caches them to avoid repeated API calls within a run. RefCache.load() runs once per invocation and fetches and indexes six reference sets:
Locations: indexed by both code and name → UUID
Material types: by name → UUID (e.g., "Books", "video recording")
Loan types: by name → UUID (e.g., "Can Circulate", "Reference")
Holdings sources: by name → UUID (e.g., "MARC")
Item note types: by name → UUID (resolves the "Axiell location" note's
itemNoteTypeId)Instance types: the resolved default type id (e.g., "text"), applied to every instance
Resolution: for each record the mapper looks names up against the cache (an in-memory, O(1) lookup, e.g. ref_cache.location["Wellcome Science"]) and embeds the resolved UUIDs directly in the payload; if a name isn't found the record fails with MappingError and is skipped for manual review. Current normalisations: material types map from AxC Object_category to a FOLIO material-type name (e.g. "archives" → "unspecified", audio → "sound recording"); locations from an AxC hierarchical code (e.g. "215;B11;MR;84;3;7") to a location UUID; loan types from AxC OrderingCodes (e.g. "Archives - Requestable") to a loan-type UUID; instance type is typically "text".
Caching decision: the cache is reloaded every run and not carried across invocations — about six reference-endpoint GETs (~1–3 s) is negligible next to the per-record upserts and needs no extra infrastructure (no DynamoDB/ElastiCache/TTL). For when this stops being the right trade-off and the scale-up progression, see Caching Strategy.
Sample Field Mapping
CanonicalRecord field = the attribute parsed from MARC via the MARC_SOURCE table in mapping.py; the hrids are derived from the GUID (MARC 001).
MARC Source (Axiell)
CanonicalRecord field
FOLIO Target
Example
001 (GUID)
source_id → instance_hrid
Instance hrid
AxC-instance-4d8f1208-9812-4bb5-84ef-da436b22d9e2
001 (GUID)
source_id → holdings_hrid
Holdings hrid
AxC-holding-4d8f1208-9812-4bb5-84ef-da436b22d9e2
001 (GUID)
source_id → item_hrid
Item hrid
AxC-item-4d8f1208-9812-4bb5-84ef-da436b22d9e2
245$a
title
Instance title
Daniel Morley, an English Philosopher...
852$b
location_code
Holdings permanentLocationId
resolved FOLIO location UUID
852$c
call_number_prefix
Holdings callNumberPrefix
Arch, Ref
852$h
call_number
Holdings callNumber
(optional)
852$j
shelving_order
Holdings shelvingOrder
(optional)
949$a
barcode
Item barcode
(optional)
949$c
material_type_code
Item materialType.id
Archives - Non-digital
949$l
loan_type_code
Item permanentLoanType.id
Archives - Requestable
876$p
copy_number
Item copyNumber
copy 1, copy 2
876$t
volume
Item volume
v.1, disc 1 of 2
856$u
electronic_access_uri
Item electronicAccess[].uri
(optional)
852$b
location_code
Item notes[], type Axiell location
raw AxC location code: 215;B11;MR;84;3;7
The AxC location code is therefore written to FOLIO in two different fields: the Holdings permanentLocationId is set to the resolved FOLIO location UUID (for shelving/discovery and circulation), while the raw AxC location code is preserved as an item note of type Axiell location (for audit trail and historical reference). On update, the note is refreshed from the incoming 852$b, so the FOLIO item always carries the latest AxC location code. (The note type name resolves to itemNoteTypeId via RefCache before the write; note that this note is not currently surfaced on the OAI-PMH feed, see Item notes not visible on the FOLIO OAI-PMH feed.)
FOLIO Field Mapping Reference
For a comprehensive and detailed mapping of all Axiell Collections fields to FOLIO Inventory API fields, see folio-axc-fields-mapping.md.
That document provides the complete field-by-field mapping for Instance, Holdings, and Item entities, the transformation rules and MARC source information, the required-versus-optional fields for FOLIO, the reference-data requirements (locations, material types, loan types), and the field validation rules and edge cases.
Change Detection Mechanism
The FOLIO upsert step leverages the existing Axiell adapter's OAI-PMH change detection to determine which records to sync.
How It Works
OAI-PMH datestamp windows drive the harvest: the Axiell adapter only returns records whose last_modified falls within [window_start, window_end). Each harvest window produces a unique changeset_id, and the records written to Iceberg in that window are tagged with it. Those Iceberg rows carry namespace (record type, e.g. "location", "item"), id (external identifier), content (the raw MARCXML payload), changeset (the changeset ID from the adapter), last_modified (the OAI datestamp), and deleted (true when OAI returns a tombstone). The FOLIO upserter then receives the changeset_ids and reads only those rows:
Record selection: only harvest-flagged records
Only AxC MARC records that have the harvest flag set in MARC field 980 $a are synced to FOLIO. Before mapping, the upserter filters each changeset row on 980 $a, and records without the flag are skipped entirely (never created, updated, or suppressed in FOLIO). The flag is the source-side switch by which Collection Information controls which Axiell records flow into the LMS, so the sync covers a curated subset rather than the whole changeset. (This selection step is not yet in the prototype, which currently processes every changeset row; it is required behaviour.)
Change Detection Signals
Harvest flag (980 $a)
AxC MARC record
Record is opted in for FOLIO sync; absent means skip
Record in changeset
OAI-PMH datestamp window
Record was created or modified in source
deleted=true
OAI tombstone
Unreliable; best-effort only, not the authoritative delete signal
Payload hash mismatch
XSL output comparison (optional)
FOLIO-relevant fields actually changed
Reconciler GUID remap
Axiell reconciler step
Authoritative delete: old GUID superseded; suppress its FOLIO records
How records are applied: create, update, delete
Each record in a loader changeset is new (the first time this id appears in Iceberg) and is created in FOLIO, or updated (an existing id with a newer last_modified) and is updated in FOLIO.
Deletes are driven by the reconciler, not the loader's deleted=true tombstone (which is unreliable — the very reason the adapter has a separate reconciler step). The reconciler tracks the collectId → guid mapping in its own Iceberg store and, when a collectId is remapped, emits a DeletedSourceWork for the superseded GUID. Two things follow for the upserter: it consumes the loader's changeset, so it never sees these deletes directly, and a reconciler delete is keyed by the old (superseded) GUID, not the changeset row being processed.
Suggested approach: have the reconciler fan out a FOLIO suppression path mirroring the loader's fan-out to the upsert path (reusing the existing reconciler rather than building a parallel mapping store). On a reconciler delete, suppress the records for the old GUID (AxC-instance-{old-guid}) and cascade to its holdings and item; the loader deleted=true stays a best-effort secondary signal only. The exact delete action (suppress vs remove, and cascade scope) is an open policy question — see Delete semantics.
Upsert Key Strategy (Idempotency)
Every entity is matched against its existing FOLIO record by hrid (a GET before each write), then created or updated — no blind creates, which is what makes replay idempotent:
Instance:
GET /inventory/instances?query=(hrid==AxC-instance-{guid})Holdings:
GET /holdings-storage/holdings?query=(hrid==AxC-holding-{guid})Item:
GET /inventory/items?query=(hrid==AxC-item-{guid})
The hrid is derived from the Axiell GUID (MARC 001), not the collectId (object number). Axiell reuses collectIds; keying on GUID prevents a reused id from overwriting the wrong FOLIO record.
Behavior: when a record is found, its mutable fields are updated while FOLIO-internal metadata is preserved; when it isn't found, a new record is created; and when required fields are missing, the record is skipped with a structured error logged. An update applies only if it passes the stale-write guard (incoming Axiell last_modified strictly newer than the watermark on the existing FOLIO record), so out-of-order or replayed changesets never overwrite newer state (see Invocation Pattern).
Replay safety: upserts are idempotent by hrid, so processing the same changeset twice produces the same outcome, and manifest deduplication checks whether a changeset was already processed successfully before running it again.
Proposed AWS Architecture
Key Design Considerations
We propose a Step Functions + Lambda + S3 architecture with event-driven (asynchronous) invocation, per-record error isolation, and 90-day audit retention. This approach balances operational visibility, fault resilience, and simplicity.
Mapping
Typed Pydantic models (mapping.py)
Build-time validation; fail fast before OKAPI, not as a FOLIO 422
Orchestration
AWS Step Functions
Configurable retries + rich execution history for audit
Compute
Lambda (ECR container)
Stateless, scales to 0, integrates with Step Functions
Data Storage
S3 NDJSON manifests (90-day TTL)
Cost-efficient batch writes + queryable (download + jq, or optionally S3 Select)
Trigger
EventBridge on adapter completion
Event-driven (not polling); decoupled from adapter
Invocation
Async event → Step Function (StartExecution); Lambda runs synchronously within it
Decoupled from adapter; ordering via source-timestamp watermark + concurrency=1
Error Handling
Per-record isolation
Batch completes even if individual records fail
Expected outcomes: replay is safe without data corruption (idempotent upserts via FOLIO HRIDs); there is a complete audit trail, with every decision (create/update/suppress/skip) logged in S3 and CloudWatch; operational overhead stays low at ~$3–5/month for typical volume; and the mental model remains clear, with no eventual-consistency puzzles and ordered execution.
System Diagram
1. EventBridge Trigger
Rule: axiell-folio-sync-axiell-adapter-completed
Event Pattern:
Event Payload (emitted by Axiell adapter):
dry_run (default false) makes the Lambda resolve and log every planned create/update/suppress action and still write the manifests, but issue no FOLIO writes. It is the safe way to validate a changeset (or the initial backfill) before going live. sample_limit caps the number of records processed, for smoke-testing against dev FOLIO.
IAM Permissions: states:StartExecution on Step Function ARN
2. Step Function State Machine
Name: axiell-folio-sync-sfn
Type: STANDARD
Flow:
Execution History: Stored in CloudWatch Logs (/aws/states/axiell-folio-sync-sfn, 30-day retention)
IAM Role Permissions:
lambda:InvokeFunctionon sync Lambda ARNlogs:CreateLogGroup,logs:CreateLogStream,logs:PutLogEventsfor CloudWatch
3. Lambda Function: axiell-folio-sync
Docker Image: ECR (uk.ac.wellcome/axiell-folio-sync:TAG)
IAM & secrets (least privilege): the Lambda's execution role needs ssm:GetParameter (+ KMS decrypt) for the OKAPI credential SecureString, read access to the Iceberg/S3 Tables data, s3:PutObject to the manifests bucket, cloudwatch:PutMetricData, and CloudWatch Logs write. FOLIO credentials live only in SSM as a SecureString and are never baked into the image or environment.
Execution Steps:
Step 1: Authenticate with FOLIO
Auth is delegated to the shared folio-client package: a FolioClient is built with a credentials provider that reads SSM, logs in lazily on first use, caches the token (valid 24h) for the invocation, and re-authenticates once automatically on a 401.
Step 2: Scan Iceberg Changesets
Step 3: Map & Validate
Step 4: Upsert to FOLIO (Per Record)
Every entity is matched by its hrid (from meta) and created or updated (no blind creates), which is what makes replay idempotent. On an existing record the update is additionally gated by the stale-write guard: it applies only if the incoming Axiell last_modified is strictly newer than the watermark on the FOLIO record, otherwise the record is skipped (see Invocation Pattern).
Step 5: Batch & Write Manifests
4. S3 Manifest Storage
The NDJSON-manifest output is the existing catalogue-pipeline pattern: the adapter platform already writes per-job NDJSON id manifests to S3, and this step deliberately reuses that convention (one record per line, an .ids.ndjson success file plus a .ids.failures.ndjson error file and a .manifest.json summary per job). Adopting it keeps downstream tooling, backup/recovery, and operational query patterns consistent with the rest of the pipeline.
Bucket: axiell-folio-sync-manifests-{account-id}-{region}
File Structure:
Sample Success Record (NDJSON):
Sample Error Record (NDJSON):
Lifecycle Policy: 90-day expiration (auto-delete old manifests)
Cost Analysis
At ~80 syncs/day the pipeline runs ~2,400 times/month, so the monthly quantities below derive from that:
Lambda
~2,400 invocations × ~60s × 512 MB
~2,400 invocations
$0.0000167/GB-s
~$1.20
Step Functions
~2,400 state transitions
~2,400 transitions
$0.000025/transition
~$0.06
EventBridge
~2,400 events
~2,400 events
$1/M events
~$0.00
S3 (manifests)
~2,400 objects written, 90-day retention
~2,400 objects + storage
$0.005/K PUTs + $0.023/GB/mo
~$1.50
CloudWatch Logs
~2,400 × 5 KB ≈ 12 MB/month
12 MB ingested
$0.50/GB ingested
~$0.20
Total
~$3–5
Design Decisions
The key choices are summarized in Key Design Considerations; the full trade-off analysis for each (alternatives weighed, why this option won) lives in design-rationale.md — covering mapping (Pydantic vs YAML), orchestration (Step Functions), reference caching, manifest storage (S3 NDJSON), ordering safety, error handling, and instance storage type.
The ordering and error-handling mechanisms are load-bearing for correctness, so they are stated as behaviour in Step 4: Upsert to FOLIO and the Upsert Key Strategy; design-rationale.md explains why they take the form they do.
Instance storage type (decision). This sync creates Inventory-native instances (
Instance.source = "FOLIO", no linked SRS record), which keeps the bibliographic fields editable through mod-inventory and keeps the PUT-based update path valid. SRS-backed instances cannot be updated through the mod-inventory API, so a mixed estate is avoided. Confirmed via the prototype: these records are updatable through mod-inventory and are received on the FOLIO adapter in the catalogue pipeline; one gap (item notes not appearing on the OAI-PMH feed) remains open, see Item notes not visible on the FOLIO OAI-PMH feed. Full reasoning: design-rationale.md.
For the cost evidence behind the "~$3–5/month" figure, see Cost Analysis.
Open Questions
Field Mapping from Axc to Folio Instance, Holdings and Items needs to be defined
The field mapping still needs to be finalised with Collection Information; the sample file folio-axc-fields-mapping.md has been shared with them for feedback. A minimum stub record also needs to be defined, and the source of data (MARC vs FOLIO) for inventory types remains open and should be tested with both options.
Delete semantics: what should reconciler-detected deletes do in FOLIO?
Raised here for an answer, not settled. In FOLIO there are different operations: suppress (discoverySuppress=true, optionally staffSuppress=true) hides the record while preserving it and its links, so it is reversible and audit-friendly; remove (a hard DELETE) deletes it outright, which is irreversible, and FOLIO refuses to delete a parent that still has dependents. As a starting point we'd suggest discoverySuppress=true (preserve-and-hide), but that is a recommendation to confirm with Collection Information, not a decision. A second part of the question is whether the action cascades from instance → holdings → item or acts on the item only; this interacts with the 1:1-vs-multi-item question, since an instance shared by other items must not be suppressed or removed when one item is deleted.
Specific points to resolve with Collection Information: whether suppressed items should remain queryable by staff (i.e. whether to also set staffSuppress); how an item-level deletion behaves when its holdings or instance still have other dependents; whether deletion should propagate in either direction (item → holdings → instance, or the reverse); and the retention policy for suppressed records in the audit log.
Watermark storage: where does the source timestamp live?
The stale-write guard (see Invocation Pattern) needs the Axiell last_modified of the last applied change stored on the FOLIO record so the next write can compare against it. FOLIO does not offer an obvious home for this: _version is mod-inventory's optimistic-locking counter (not a source timestamp), and discoverySuppress is a boolean. Candidates are an administrative note, a custom property, or a dedicated field, each with trade-offs for visibility, OAI-PMH leakage, and whether it survives a quickMARC/SRS round-trip. The storage location needs to be decided (and confirmed not to pollute the catalogue feed) before the guard can be implemented.
Reference-data caching: when and how to cache across Lambda runs
Today RefCache reloads all six reference sets on every invocation, which is the right trade-off at current volume (see Reference Data Cache). The open question is what happens if the reference set grows substantially. We would need to decide the size or load-latency threshold at which caching across runs is worth the added complexity. Staleness is unlikely to be the blocker: the reference data changes rarely, so a long TTL would seldom be stale, and a reload-on-miss fallback would keep a newly-added code from failing as a MappingError. So the question is mostly about whether the saved reload justifies the extra infrastructure. We would also need to pick a tier when we get there: the progression runs from a warm-singleton with reload-on-miss, to an S3 snapshot rebuilt by a scheduled refresher, to DynamoDB, and only then to ElastiCache (see Caching Strategy: Reference Data Scaling Options for the trade-offs).
Manifest query mechanism: S3 Select is not an established org pattern
The design mentions S3 Select as a convenient way to query the NDJSON manifests (e.g. list failed records), but S3 Select is not used anywhere else in the org today, so adopting it would be a new pattern to learn, permission, and maintain. The design deliberately does not depend on it: the manifests are plain NDJSON, so the same questions can be answered by downloading the object and using jq/grep, or by pointing Athena at the manifests prefix. Open question: do we standardise on a query mechanism (plain download + local tooling, S3 Select, or Athena/Glue), and is it worth introducing S3 Select solely for this pipeline? Until decided, treat S3 Select as optional and lead with download + jq for operational investigation.
Item creation should likely be gated on record Level
The legacy CALM→Sierra transform (docs/discovery/CalmInnopac.xsl) only attaches an item (its MARC 949) when the record's Level is "Item"; higher levels (Collection, Series, File, …) get a bib record with no item. The current prototype instead builds an instance, holdings, and item for every record, regardless of level. We need to confirm with Collection Information whether the AxC → FOLIO mapping should branch the same way (item only for Item-level records, instance/holdings-only above that) and, if so, how Level arrives in the harvested MARCXML so the mapper can read it. This also backs the 1:1 instance-to-item assumption: CalmInnopac.xsl produces at most one item per record (no for-each over copies/locations, and a multi-location attempt was left disabled), so multiple items per record is not established behaviour and would be a deliberate future extension rather than something to match on day one.
Item notes not visible on the FOLIO OAI-PMH feed
Confirmed via the prototype: FOLIO-native instances (source = "FOLIO") created by this sync can be updated through mod-inventory, and the records are received on the FOLIO adapter in the catalogue pipeline (so the storage-type and catalogue-feed concerns are largely settled for Inventory-native records). One issue remains open: the item notes list does not appear on the OAI-PMH feed (marc21_withholdings). The records otherwise come through, but the notes we set (for example the Axiell location note) are missing from the harvested output. This needs further investigation, likely into how mod-oai-pmh renders Inventory item notes in the MARC output and whether a note type, staffOnly, or suppression setting hides them.
Last updated