Resilience, Platform Engineering and AI Infrastructure
Last updated 12 Oct 2026
At a glance
How BSS/OSS platforms survive failure, handle peaks and are run as a shared platform: resilience topologies with RTO and RPO stated for the order process, the scaling levers, infrastructure as code and platform engineering, and what AI infrastructure actually requires.
The picture
How it works
Resilience: two topologies and two distances
| Term | Meaning | Order-management example |
|---|---|---|
| High availability | No single component failure stops service | Two COM instances behind a load balancer, one database with a replica |
| Backup | A copy you can restore from | Nightly database snapshot: an RPO of up to a day |
| Disaster recovery | Restoring service after a site is lost | Fail over to site B; replay events since the last sync |
| Failover | The switch itself, automatic or by runbook | DNS switch plus database promotion |
| RTO | Recovery time objective: how long until service is back | Orders cannot be placed for two hours |
| RPO | Recovery point objective: how much data may be lost | Orders placed in the last five minutes are gone |
The order in flight. An order is a workflow, not a row. Active/active COM with a synchronously replicated database still loses the in-flight step if the workflow engine was mid-task. RTO and RPO have to be stated for the process, and the workflow must be idempotent so a replayed task does not activate a port twice.
Performance and scale
| Lever | Where it works in BSS/OSS | Where it does not |
|---|---|---|
| Horizontal scaling (more instances) | Stateless APIs, BFFs, event consumers | The workflow engine’s database |
| Vertical scaling (a bigger instance) | Databases, legacy monoliths | Past the largest machine available |
| Caching | Published catalog, eligibility, pricing tables | Anything that must be current to the second |
| Queues | Order submission spikes at a launch | Requests a person is waiting on |
| Capacity planning | Bill run, mass migration, a product launch | It is not replaced by autoscaling: the database still needs a plan |
Infrastructure as code and the platform
Infrastructure as code describes the environment (networks, clusters, databases, permissions) in version-controlled files such as Terraform, Pulumi or cloud-native templates, applied by pipeline. GitOps extends it: the repository is the desired state and a controller keeps the estate matching it. An environment set up by hand cannot be rebuilt; one described as code can.
An internal developer platform is run once by a platform team (Kubernetes, the gateway, observability, CI/CD, guardrails) and consumed by product teams as a paved road. Service discovery, ingress and scaling policies are the platform’s job, not each team’s. Without it, every BSS domain team builds its own: five clusters, five monitoring stacks, five ways to fail.
AI infrastructure
| Term | What it is | Why it matters |
|---|---|---|
| GPU | A processor built for massively parallel arithmetic; NVIDIA dominates the accelerator market | The unit of AI capacity, priced and scheduled as such |
| GPU cluster / scheduling | Many GPU nodes under one scheduler (Kubernetes with a GPU operator, or Slurm) | Jobs queue for GPUs; utilisation is the cost driver |
| NVLink | High-bandwidth links between GPUs in one scale-up domain: a server, or a whole rack in NVL72-class systems | Lets many GPUs act as one for a large model |
| InfiniBand / RoCE, RDMA | Fabrics that let GPUs in different nodes exchange memory directly, bypassing the CPU | Training scales across nodes only if the fabric keeps up |
| Spine-leaf | A non-blocking topology: every leaf reaches every other through the spines at full rate | East-west traffic between GPUs is the dominant flow |
| High-speed storage | Parallel file systems and NVMe feeding data at GPU speed | A starved GPU is an expensive idle GPU |
| Training vs inference | Building a model vs using one | Telcos mostly need inference: few GPUs, near the data, always on |
Who owns what
| Entity | System of Record | System of Engagement | System of Reference | Notes |
|---|---|---|---|---|
| Infrastructure definitions (networks, clusters, databases, permissions) | Infrastructure-as-code repository (Git) | CI/CD pipeline | Cloud provider control plane | Under GitOps the repository is the desired state; manual changes are drift to be reverted |
| Running infrastructure state (actual) | Cloud provider control plane / Kubernetes API | — | CMDB / ITSM, observability platform | What actually exists, compared continuously against the repository |
| Shared platform configuration (namespaces, policies, quotas, ingress) | Platform team’s repository | Internal developer platform portal | Product teams | — |
| Order workflow state (in flight) | Workflow engine in COM / SOM | — | Event streaming platform | The part of the order most often lost in a failover |
| RTO and RPO targets, DR runbooks | Service management (ITSM) or business continuity register | — | Platform and application teams | Stated per business process, not per database |
| Backups and snapshots | Backup service or cloud provider snapshot store | — | DR runbooks | A backup is only proven by a tested restore |
| Cost and capacity data (including GPU utilisation) | Cloud provider billing data, reported through a FinOps tool where one exists | — | Finance, platform team | Utilisation is the cost driver for GPUs |
In practice
Resilience is usually specified for infrastructure and tested for infrastructure: the database fails over, the load balancer moves traffic, the dashboard goes green. The order that was mid-activation at the moment of failure is then found days later, stuck or activated twice. Testing DR with orders in flight, not just with an idle system, is what surfaces this.
Many estates contain test environments that were set up by hand years ago and cannot be rebuilt, which blocks parallel testing during a transformation. Bringing them under infrastructure as code is unglamorous work that pays back on every later release.
A shared Kubernetes platform without a funded platform team becomes everyone’s problem and nobody’s job. Some vendors also certify only their own cluster configuration, which can force a per-vendor cluster whatever the platform strategy says.
On AI, the recurring mistake is dedicated GPUs bought for a proof of concept that then sit largely idle, or cloud inference adopted with customer data and no data protection impact assessment. Most telco use-cases are inference, not training.
The trade-off
What it solves
A shared platform run by a platform team, with the estate described as code, gives every BSS/OSS domain the same resilience, scaling and observability once, rather than each team building and failing its own.
What it does not solve
- It does not make the order process resilient: in-flight workflow state and idempotent replay are the application’s responsibility.
- It does not remove the database as the part that cannot autoscale and needs a capacity plan.
What it adds
- A platform team to fund and staff, and a shared upgrade that can affect every domain at once.
- Isolation between domains by namespace, policy and quota has to be designed and enforced.
Architecture decision
Run one shared Kubernetes platform for all BSS/OSS domains, or one cluster per domain or vendor?
One shared platform
Choose when A platform team exists and is funded, and domains can be isolated by namespace, policy and quota.
Per domain or vendor
Choose when Vendors certify only their own cluster configuration, or teams are organisationally separate with no platform function.
Breaks if Shared is chosen without a platform team: the cluster becomes everyone’s problem and nobody’s job, and the first upgrade takes down three domains.
Takeaways
Key Takeaways
- State RTO and RPO for the order process, not the database; make workflows idempotent so replay is safe.
- Scale the stateless parts horizontally and plan capacity for the parts that cannot scale.
- An environment described as code can be rebuilt; one set up by hand cannot.
- A platform team is what makes shared infrastructure cheaper than five copies of it.
- AI infrastructure is a fabric and storage problem; most BSS/OSS use-cases need inference, not a training cluster.
Check yourself
COM runs active/active with a synchronously replicated database. Why can an order still be lost or duplicated in a failover?Show answerHide
The workflow engine may be mid-task when the site fails. Unless the workflow is idempotent and its in-flight state survives, the step is lost or replayed, for example activating a port twice.
An operator wants a GPU cluster to run a customer-service assistant on a model it did not train. What should it consider first?Show answerHide
That this is inference with variable demand, which usually suits a cloud AI service, provided the customer data may leave the footprint and a DPIA is in place. Dedicated GPUs pay back only at high, sustained utilisation with staff to run them.