Data Lineage in Catalog-Driven Architecture
SID as the Shared Data Model
The TM Forum SID (Shared Information/Data model) is the conceptual glue that threads data entities together across BSS and OSS boundaries. It defines a common vocabulary and entity structure that all systems in the telco stack can align to — regardless of vendor, technology, or deployment pattern.
SID organises entities into domains that map directly to the operational layers of a telco: Customer, Product, Service, and Resource. Each domain owns a set of entities with well-defined attributes, relationships, and lifecycle states. When systems across the BSS/OSS stack share this common model, every entity created at one layer can be unambiguously referenced, traced, and correlated at every other layer.
SID Is Not a Database Schema
SID is a logical data model, not a physical one. No system implements SID tables directly. Instead, SID provides the shared vocabulary and entity relationships that systems align to — so that when CRM says "Customer" and Billing says "Customer", they mean the same thing and can exchange data without translation ambiguity.
What Data Lineage Means in Telco
Data lineage answers a deceptively simple question: where did this data come from, what happened to it, and where did it go? In a telco context, this means tracing how a customer's order creates, transforms, and links entities across every system it touches — from the initial portal click through to the network configuration and back again.
Unlike general-purpose data lineage (which often focuses on ETL pipelines and data warehouses), telco data lineage follows the operational flow of orders and inventory. It traces how a commercial intent becomes a running service on a real network — and how every entity created along the way remains linked, queryable, and auditable.
- Downward lineage (order decomposition): How a commercial order decomposes into service orders, then resource orders, then network actions — each layer creating entities that reference their parent. Product Order → Service Order → Resource Order → Network Configuration.
- Upward lineage (completion and inventory sync): How completion events flow back up the stack — resource activation confirms to ROM, ROM confirms to SOM, SOM confirms to COM — each layer updating its inventory with linked instances that reference the originating order.
- Cross-reference lineage (entity threading): How entities at different layers maintain permanent bidirectional references — a Product Instance links to its CFS Instance, which links to its Resource Instances — enabling traversal in any direction at any time.
Worked Example: Broadband Order End-to-End
To make data lineage concrete, let us trace a single residential broadband order — "Fibre 200 Mbps" — from the customer's portal click through every system layer and back. At each step, we will name the SID entity created, the TMF API used, and the System of Record that owns it.
SID Data Lineage Flow: order decomposition flows down (solid), completion/inventory sync flows up (dashed)
Downward Flow: Order → Activation
Customer Submits Order
Channel → COMThe customer selects "Fibre 200" on the self-service portal. The portal (SoE) submits a ProductOrder via TMF622 to COM. The ProductOrder contains a ProductOrderItem referencing the "Fibre 200" Product Offering from the Product Catalog (TMF620).
COM Creates Product Order
COM (Product Domain)COM (SoR for commercial orders) validates the order against the Product Catalog, performs eligibility and credit checks, and persists the ProductOrder. Each ProductOrderItem carries a productOfferingRef linking back to the catalog. COM then decomposes the order using catalog rules and submits a ServiceOrder via TMF641 to SOM.
SOM Creates Service Order
SOM (Service Domain)SOM (SoR for service orders) receives the ServiceOrder. Each ServiceOrderItem references a CFS specification from the Service Catalog (TMF633) and carries a productOrderItemRef linking back to COM. SOM decomposes CFS into RFS using catalog rules and submits a ResourceOrder via TMF652 to ROM.
ROM Creates Resource Order
ROM (Resource Domain)ROM (SoR for resource orders) receives the ResourceOrder. Each ResourceOrderItem references a Resource specification from the Resource Catalog (TMF634) and carries a serviceOrderItemRef linking back to SOM. ROM translates resource order items into network-level actions — NETCONF commands, CLI scripts, or vendor API calls.
Network Activation
Network InfrastructureROM executes activation: configures the OLT port (VLAN, speed profile), provisions a BNG session (QoS policy, IP assignment), and allocates an IP address via DHCP/IPAM. Each action targets a specific Resource Instance.
Upward Flow: Completion → Inventory Sync
Network Confirms Activation
Network → ROMThe OLT and BNG confirm successful configuration. ROM records the Resource Instances in Resource Inventory (TMF639) — OLT Port, BNG Session, IP Address — each linked to the ResourceOrderItem that created them.
ROM Completes → SOM Notified
ROM → SOMROM marks the ResourceOrder as completed and publishes a ResourceOrderStateChangeEvent. SOM receives the event, correlates it via resourceOrderRef, and records a CFS Instance in Service Inventory (TMF638) — linked to the Resource Instances below and the ServiceOrderItem that created it.
SOM Completes → COM Notified
SOM → COMSOM marks the ServiceOrder as completed and publishes a ServiceOrderStateChangeEvent. COM receives the event, correlates it via serviceOrderRef, and records a Product Instance in Product Inventory (TMF637) — linked to the CFS Instance below and the ProductOrderItem that created it.
Customer Sees Active Service
COM → Channel / BillingCOM updates the ProductOrder state to "completed." The self-service portal queries Product Inventory and shows the customer their active "Fibre 200" subscription. Billing is triggered from the Product Instance lifecycle event. Every entity — from ProductOrder to OLT Port — is now linked and traceable.
Entity Threading: How References Link Layers
The worked example above shows data flowing through layers. But the real power of catalog-driven lineage is not the flow — it is the permanent references that remain after the order completes. Every entity created during the order lifecycle carries explicit references to its parent, its children, and the catalog specification it was instantiated from. These references are the "threads" that enable traversal in any direction, at any time, long after the order is complete.
Entity threading: a single order creates linked entities at every layer with explicit ID references
Entity Threading Reference Chain
| Layer | Entity | SoR | References Up | References Down | Example ID |
|---|---|---|---|---|---|
| Product (BSS) | ProductOrderItem | COM | customerRef, productOfferingRef | serviceOrderItemRef (via decomposition) | PO-2024-001-01 |
| Product (BSS) | Product Instance | Product Inventory | productOrderItemRef | serviceRef (CFS Instance link) | PI-2024-001 |
| Service (OSS) | ServiceOrderItem | SOM | productOrderItemRef | resourceOrderItemRef (via decomposition) | SO-2024-001-01 |
| Service (OSS) | CFS Instance | Service Inventory | serviceOrderItemRef, productRef | resourceRef (Resource Instance link) | SI-2024-001 |
| Resource (OSS) | ResourceOrderItem | ROM | serviceOrderItemRef | resourceRef (target resource) | RO-2024-001-01 |
| Resource (OSS) | Resource Instance | Resource Inventory | resourceOrderItemRef, serviceRef | physicalResourceRef (network element) | RI-2024-001-P04 |
Data Quality Benefits of Catalog-Driven Lineage
The data quality benefits of catalog-driven architecture are not incidental — they are structural. The architecture does not just make data quality easier to achieve; it makes certain categories of data quality problems impossible by design. This is the punchline: catalog-driven lineage prevents the issues that legacy environments spend permanent operational effort managing.
Every entity has exactly one System of Record. Product Inventory is the SoR for subscriptions. Service Inventory is the SoR for CFS/RFS instances. Resource Inventory is the SoR for resources. No ambiguity, no competing truths.
- Eliminates "which system is right?" disputes — the SoR is documented and enforced
- All writes go through the SoR; downstream systems receive events, never modify the master
- Audit trails are authoritative because there is only one place to audit per entity type
Legacy vs Catalog-Driven: Data Quality Outcomes
| Dimension | Legacy (No Lineage) | Catalog-Driven (Full Lineage) |
|---|---|---|
| Entity consistency | Each system defines entities independently. Product in CRM ≠ Product in Billing ≠ Product in OSS. | All systems reference the same catalog-defined entity structure via SID-aligned TMF APIs. |
| Cross-system correlation | Manual — requires matching on customer name, address, or free-text order references. | Automatic — entities carry explicit typed references (productOrderItemRef, serviceRef, resourceRef). |
| Orphaned entities | Common. Services remain active after cancellation. Billing charges persist after deactivation. | Structurally prevented. Lifecycle events propagate through entity references. Termination cascades. |
| Data staleness | Persistent. Nightly batch sync means 24-hour lag. Some systems never sync. | Near-real-time. Event-driven updates propagate within seconds of state changes. |
| Reconciliation effort | Permanent. Dedicated teams run manual reconciliation monthly. Never catches everything. | Eliminated as an operational activity. Monitoring detects event delivery failures, not data drift. |
| Regulatory auditability | Heroic manual effort. Each audit requires reassembling entity relationships from fragmented data. | Structural. Entity threading provides auditable, traversable relationships from contract to resource. |
What Breaks Without Data Lineage
Section 1.6 (Operating Without a Centralised Catalog) describes the broad operational consequences of missing catalogs. Here we focus specifically on the data integrity failures that occur when entities cannot be traced across system boundaries — and why legacy environments cannot retrofit lineage without fundamental architectural change.
- Ghost services — Services running on the network with no corresponding record in billing or CRM. They consume resources, generate no revenue, and are invisible to management reporting.
- Billing orphans — Charges that persist in billing for services that have been deactivated or never existed. Detected only when customers complain.
- Untraceable faults — Network alarms that cannot be correlated to affected customers or products. Engineers investigate in isolation because there is no entity reference chain to follow.
- Modification drift — Changes applied at one layer (e.g., speed upgrade on the network) that are not reflected at other layers (e.g., billing still at old rate). Each change widens the gap.
- Migration impossibility — Legacy data cannot be migrated to a target architecture because there is no canonical entity structure. Migration becomes a reverse-engineering exercise, not a data mapping one.
Legacy environments cannot solve data lineage by adding a "data lineage layer" on top of existing systems. The problem is structural:
- No common entity model — each system uses its own identifiers, entity types, and relationship structures. There is nothing to link.
- No lifecycle events — legacy systems do not emit standardised state change events. Changes happen silently within each system.
- No decomposition chain — without a catalog defining Product → CFS → RFS → Resource, there is no structural relationship to trace. Any lineage would have to be inferred from coincidental data patterns.
- Retroactive linking is unreliable — attempting to correlate legacy entities after the fact (matching by customer name, order date, or free text) produces incomplete and error-prone results.
For architects planning BSS/OSS transformation, SID alignment is not a "nice to have" — it is a prerequisite. Every transformation capability — API-first integration, zero-touch provisioning, event-driven architecture, MANO orchestration — assumes that entities conform to a shared model with explicit cross-layer references.
- TMF Open APIs require SID-aligned payloads — you cannot expose TMF622 (Product Ordering) if your system has no concept of ProductOrderItem with typed references to Product Offering and Service Order.
- ODA components assume SoR boundaries — each ODA component owns specific SID entities and exposes them via TMF APIs. Without SID alignment, ODA decomposition is meaningless.
- Event-driven architectures need typed events — ProductOrderStateChangeEvent, ServiceOrderStateChangeEvent carry SID entity references. Without SID, events are just unstructured notifications.
- Data migration maps to SID — the target data model for any modern BSS/OSS is SID-aligned. Legacy data that does not map to SID entities cannot be migrated programmatically.
Summary
Data lineage in a catalog-driven architecture is not an afterthought or an analytics feature — it is the structural consequence of modelling entities correctly and linking them at creation time. When SID provides the shared vocabulary, TMF APIs provide the exchange format, and catalog-driven decomposition provides the creation chain, lineage is automatic. Every entity knows where it came from, what it relates to, and where to find its upstream and downstream neighbours.
Data Lineage — Key Principles
- SID provides the shared data model that gives every BSS/OSS entity a common vocabulary and structure across system boundaries
- Data lineage has three dimensions: downward (order decomposition), upward (completion/inventory sync), and cross-reference (entity threading)
- In a catalog-driven architecture, every entity carries explicit typed references to its parent, children, and catalog specification — lineage is structural, not reconstructed
- Entity threading enables instant traversal: resource alarm → CFS → product → customer in seconds, not hours
- Catalog-driven lineage structurally prevents orphaned services, billing orphans, modification drift, and untraceable faults
- Event-driven inventory sync replaces manual reconciliation — near-real-time consistency across all layers
- Legacy environments cannot retrofit lineage because they lack a common entity model, lifecycle events, and decomposition chain
- SID alignment is a prerequisite for transformation — TMF APIs, ODA components, and event-driven architectures all assume SID-aligned entities with typed references