Building Redundancy: Removing Single Points of Failure in an ISP Network

Customers forgive slow support and the occasional billing error, but they rarely forgive an outage — especially a long one during peak hours. Reliability, though, is not a thing you can purchase as a line item. It is the cumulative result of finding every point in your network where a single failure can interrupt service, then deciding, deliberately, which of those points are worth the money and complexity to eliminate. The goal is not “zero single points of failure everywhere,” which is unaffordable and often counterproductive; it is to spend your redundancy budget where the blast radius — the number of subscribers affected and the duration of the hit — is largest.

That reframing matters because redundancy has a dark side: every redundant component you add is another thing that can be misconfigured, another path that can fail silently, another protocol interaction that can turn a one-device outage into a network-wide one. A poorly designed redundant network can be less reliable than a simple one. The work, then, is twofold: systematically locate single points of failure (SPOFs), and add resilience in a way that genuinely reduces risk rather than just adding moving parts. This article walks the failure domains an ISP actually has — power, links, hardware, upstream, and services — and how to reason about each.

Find the single points of failure first

The most useful exercise is to trace the path a packet takes from a subscriber's home to the wider internet and, at every hop, ask one question: “if this single thing dies right now, who loses service and for how long?” That question turns an abstract architecture diagram into a ranked list of risks. The usual suspects, in roughly descending blast radius:

• Upstream / transit: a single transit provider, or a single border router, means one BGP session or one peering port failing takes your entire customer base offline.
• Core and aggregation: one core router, one aggregation switch, or one BNG terminating thousands of subscribers — a huge blast radius concentrated in one box.
• Power: a single utility feed with no UPS or generator at a core site; the most common cause of multi-hour outages and the easiest to overlook.
• Links: a single fiber span with no diverse path — one backhoe, one rodent, one cut, and a whole region goes dark.
• Shared services: one DNS resolver, one RADIUS server, one DHCP server — small boxes whose failure silently breaks authentication or name resolution for everyone.

Match each SPOF to a mitigation

Once you have the list, the mitigation for each layer is well understood. The discipline is choosing the right one and verifying it actually works.

Diverse paths, not just two paths

The single most expensive lesson in network redundancy is the difference between two links and two diverse links. Buying a second fiber from the same vendor that runs in the same duct, crosses the same bridge, or enters your building through the same conduit gives you the illusion of redundancy and none of the protection — one excavator takes out both. True path diversity means physically separate routes: different ducts, different entry points to the building, ideally different physical infrastructure providers. The same logic applies to power (two feeds from the same substation is not real diversity) and to upstreams (two transit providers who both buy capacity from the same wholesaler share a hidden fate). When you design redundancy, always ask what these two “independent” things have in common — the shared fate you didn't see is exactly where the simultaneous failure will come from.

How redundancy fails in production

Redundant designs fail in characteristic, frustrating ways, and knowing them is half the battle:

• The standby that was never healthy. A backup link or device sits idle for a year, accumulates a config drift or a dead optic, and when the primary fails the “redundant” path is already broken. Untested redundancy is a liability you are paying for.
• Failover that's slower than the SLA. The backup eventually takes over, but the convergence — BGP reconvergence, spanning-tree recalculation, a routing protocol timing out — takes minutes, during which customers are down anyway. Resilience is about how fast you recover, not just whether you do.
• Shared-fate surprises. Two “diverse” paths turn out to share a duct or a power feed, so the redundant component dies in the same event as the primary.
• Redundancy that causes the outage. A misconfigured first-hop redundancy protocol, a loop introduced by a redundant link without proper loop prevention, or a flapping path that destabilizes routing — the added complexity becomes the root cause.
• The thundering herd on recovery. Power returns and thousands of subscriber sessions try to re-establish at once, overwhelming a RADIUS tier that was sized for steady state — the redundancy worked, but the recovery storm took you down anyway.

Don't forget the boring layers

Hardware and link redundancy get the attention, but a large share of real-world recovery speed comes from unglamorous operational redundancy. Configuration backups, taken automatically and stored off-box, turn a dead router into a 30-minute swap instead of a multi-hour rebuild from memory. Out-of-band management — a separate path to reach your devices when the main network is down — is what lets you fix a problem remotely instead of dispatching someone at 3 a.m. Spare hardware on the shelf, clearly labeled, converts a hardware failure from a procurement-lead-time outage into a same-day fix. And clear, current runbooks mean the on-call engineer who didn't design the network can still execute the recovery correctly under pressure. These cost little and shorten every incident.

Redundancy you never test isn't redundancy

This is the rule that separates networks that are resilient from networks that merely look resilient on paper. The backup that is never exercised is the one that fails the day you need it. Build failover testing into your operational rhythm: during planned maintenance windows, deliberately fail a primary link, kill a primary device, or drop an upstream session, and confirm the standby takes over within your target recovery time. Document the expected behavior — which path lights up, how long convergence takes, what the dashboards should show — so that when it happens for real, your on-call staff recognize a working failover instead of panicking. The act of testing also surfaces the silent failures (the dead standby optic, the drifted config) before they matter. Critically, make your monitoring and alerting themselves redundant: if your single monitoring server shares fate with the thing it watches, you will learn about outages from angry customers instead of a dashboard. A clear, always-available view of device and link health — ideally a live network map that shows status at a glance — is part of the resilience story, not separate from it.

How to choose where to spend

Resilience is ultimately a budgeting exercise. For each candidate investment, weigh the cost of the second path or device against the expected cost of the outage it prevents — and outage cost is blast radius (subscribers affected) times duration times frequency, plus the harder-to-quantify reputation damage. That math almost always justifies redundancy at the upstream, border, core and power layers, where one failure hits everyone. It rarely justifies fully duplicating the last-mile drop to a single home, where the blast radius is one customer. Spend top-down: secure the layers whose failure is network-wide before you gold-plate the edges. And remember that operational redundancy — backups, out-of-band access, spares, runbooks — is usually the cheapest resilience you can buy per dollar, because it shortens recovery for failures you couldn't prevent.

The takeaway

Removing single points of failure is not a one-time project but a continuous discipline: trace the packet's path, rank your SPOFs by blast radius, mitigate the widest ones first, insist on genuine path and power diversity rather than its illusion, and — above all — test every failover regularly so your redundancy is real on the day it matters. Add the boring operational layers, keep your monitoring independent of the network it watches, and spend in proportion to the outage each investment prevents. Do that, and you trade the random, reputation-destroying long outage for the controlled, briefly-noticed blip — which is exactly the difference subscribers remember.