Resilience, Platform Engineering and AI Infrastructure

Active/active and active/passive, RTO and RPO, networking and storage, infrastructure as code, the internal platform, and what GPU clusters change.

Resilience: two topologies and two distances

Active/passive and active/active, with RTO and RPO drawn as what they are: two distances from the moment of failure — how much time you lose, and how much data.

Term	Meaning	Order-management example
High availability	No single component failure stops service	Two COM instances behind a load balancer, one database with a replica
Redundancy	Spare capacity that can take over	A second availability zone
Backup	A copy you can restore from	Nightly database snapshot — RPO of up to a day
Disaster recovery	Restoring service after a site is lost	Fail over to site B; replay events since the last sync
Failover	The switch itself: automatic or by runbook	DNS switch plus database promotion
RTO	Recovery time objective: how long until service is back	Orders cannot be placed for two hours
RPO	Recovery point objective: how much data may be lost	Orders placed in the last five minutes are gone

The order in flight

An order is a workflow, not a row. Active/active COM with a synchronously replicated database still loses the in-flight step if the workflow engine was mid-task. RTO and RPO have to be stated for the process, and the workflow must be idempotent so a replayed task does not activate a port twice.

Performance and scale

Lever	What it does	Where it works in BSS/OSS	Where it does not
Horizontal scaling	More instances	Stateless APIs, BFFs, event consumers	The workflow engine’s database
Vertical scaling	A bigger instance	Databases, legacy monoliths	Past the largest machine available
Caching	Serve repeated reads from memory	Published catalog, eligibility, pricing tables	Anything that must be current to the second
Queues	Absorb bursts; smooth downstream load	Order submission spikes at a launch	Requests a person is waiting on
Capacity planning	Sizing for the known peak	Bill run, mass migration, a product launch	Replaced entirely by autoscaling — the database still needs a plan

Platform engineering and infrastructure as code

Infrastructure as code

The environment — networks, clusters, databases, permissions — described in version-controlled files (Terraform, Pulumi, cloud-native templates) and applied by pipeline. GitOps extends it: the repository is the desired state and a controller keeps the estate matching it. A test environment that was set up by hand cannot be rebuilt; one described as code can.

Internal developer platform

A platform team runs Kubernetes, the gateway, observability, CI/CD and the guardrails once, and product teams consume them as a paved road. Without it, every BSS domain team builds its own — five clusters, five monitoring stacks, five ways to fail. Service discovery, ingress and scaling policies are the platform’s job, not each team’s.

Architecture decision

Run one shared Kubernetes platform for all BSS/OSS domains, or one cluster per domain or vendor?

One shared platform
Choose when a platform team exists and is funded, and domains can be isolated by namespace, policy and quota.
Per domain or vendor
Choose when vendors certify only their own cluster configuration, or teams are organisationally separate with no platform function.

Breaks if shared is chosen without a platform team: the cluster becomes everyone’s problem and nobody’s job, and the first upgrade takes down three domains.

AI infrastructure

What an AI workload sits on. The fabric row is the one that surprises BSS/OSS architects: a GPU cluster is a network design problem before it is a compute one.

Term	What it is	Why it matters
GPU	A processor built for massively parallel arithmetic; NVIDIA dominates the accelerator market	The unit of AI capacity, priced and scheduled as such
GPU cluster / scheduling	Many GPU nodes under one scheduler (Kubernetes with a GPU operator, or Slurm)	Jobs queue for GPUs; utilisation is the cost driver
NVLink	High-bandwidth links between GPUs inside one node	Lets several GPUs act as one for a large model
InfiniBand / RoCE, RDMA	Fabrics that let GPUs in different nodes exchange memory directly, bypassing the CPU	Training scales across nodes only if the fabric keeps up
Spine-leaf	A non-blocking network topology: every leaf reaches every other through the spines at full rate	East-west traffic between GPUs is the dominant flow
High-speed storage	Parallel file systems and NVMe feeding data at GPU speed	A starved GPU is an expensive idle GPU
Training vs inference	Building a model vs using one	Telcos mostly need inference — few GPUs, near the data, always on

Architecture decision

Should AI compute be bought as a cloud service or built as dedicated infrastructure?

Cloud AI services
Choose when the use-cases are inference on models you do not train, demand is variable, and the data may leave your footprint with the lineage described in E2E Data Lineage.
Dedicated infrastructure
Choose when data sovereignty forbids the cloud, utilisation will be high and sustained, and the operator can staff a GPU platform — a facility, fabric and scheduling competence most BSS teams do not have.

Breaks if dedicated GPUs are bought for a proof of concept and sit at 8% utilisation, or cloud inference is adopted with customer data and no DPIA.

Key Takeaways

State RTO and RPO for the order process, not the database; make workflows idempotent so replay is safe.
Scale the stateless parts horizontally and plan capacity for the parts that cannot scale.
A platform team is what makes shared infrastructure cheaper than five copies of it.
AI infrastructure is a fabric and storage problem; most BSS/OSS use-cases need inference, not a training cluster.