Skip to main content

14. Final Boss

This test throws the lab into its darkest scene. Subscribers begin alive and moving real traffic. Then, one by one, three pillars fall:

  1. the OLT -> BNG MASTER access link
  2. the Carrier1 -> BNG2 link, exactly when BNG2 has already become master
  3. the RADIUS server

The question is not whether the network gets hit. The question is whether it survives.

14.1 Goal​

Validate that the lab maintains subscriber service for IPoE, PPPoE, and delegated LAN behind the ONT during a chained failure sequence.

In this real execution, we observed:

  • a successful SRRP switchover to SLAVE
  • loss of Carrier1 on the active node without collapsing the tested traffic
  • complete loss of radius without evicting already authenticated subscribers
  • a single traffic gap during the first switchover, roughly 7.6 to 8.1 seconds depending on the probe

14.2 Prerequisites​

Validate before starting:

show srrp
show service active-subscribers
show router 9999 bgp summary

Expected baseline:

  • MASTER: SRRP master
  • SLAVE: SRRP backupShunt
  • ONT-001 and ONT-002 present in show service active-subscribers
  • Carrier1 and Carrier2 up on the active node

14.3 Determine the current source IPs​

Optional shortcut:

bash configs/cbot/scripts/subscriber-failover-probes.sh final-boss start

If you use this helper, you can skip 14.3 and 14.4 and continue directly with 14.5. The script detects the current source IPs and launches all ATP probes.

To view the ping sequence during the test, leave a second terminal open with:

bash configs/cbot/scripts/subscriber-failover-probes.sh final-boss watch

This lets you watch the latest reply from each ping probe live while you trigger each failure from the main terminal.

Before starting the probes, identify the current lab addresses. These IPs may vary between runs, so fixed values should not be used.

Run from the host:

docker exec ont1 sh -lc "ip -6 addr show dev eth1.150 scope global"
docker exec ont1 sh -lc "ip -6 addr show dev eth3.200 scope global"
docker exec ont1 sh -lc "ip -4 addr show dev eth3.200 scope global"
docker exec ont2 sh -lc "ip -6 addr show dev ppp0 scope global"

Use these IPs in the next steps:

  • ONT1_WAN1_V6: global IPv6 on eth1.150
  • ONT1_WAN2_V6: global IPv6 on eth3.200
  • ONT1_WAN2_V4: global IPv4 on eth3.200
  • ONT2_PPP_V6: global IPv6 on ppp0

Example from a real run:

ONT1_WAN1_V6=2001:db8:100::5
ONT1_WAN2_V6=2001:db8:cccc::1
ONT1_WAN2_V4=100.80.0.4
ONT2_PPP_V6=2001:db8:100::4

14.4 Act I: The Calm Before the Storm​

Launch the continuous probes:

docker exec ont1 sh -lc 'rm -f /tmp/final_boss_*; nohup ping -D -n -i 0.2 -W 1 -I <ONT1_WAN1_V6> 2001:db8:aaaa::2 > /tmp/final_boss_ont1_wan1_v6.log 2>&1 & echo $! > /tmp/final_boss_ont1_wan1_v6.pid; nohup ping -D -n -i 0.2 -W 1 -I <ONT1_WAN2_V6> 2001:db8:aaaa::2 > /tmp/final_boss_ont1_wan2_v6.log 2>&1 & echo $! > /tmp/final_boss_ont1_wan2_v6.pid; nohup ping -D -n -i 0.2 -W 1 -I <ONT1_WAN2_V4> 99.99.99.99 > /tmp/final_boss_ont1_wan2_v4.log 2>&1 & echo $! > /tmp/final_boss_ont1_wan2_v4.pid'
docker exec ont2 sh -lc 'rm -f /tmp/final_boss_*; nohup ping -D -n -i 0.2 -W 1 -I <ONT2_PPP_V6> 2001:db8:aaaa::2 > /tmp/final_boss_ont2_ppp_v6.log 2>&1 & echo $! > /tmp/final_boss_ont2_ppp_v6.pid'
docker exec pc1 sh -lc 'rm -f /tmp/final_boss_*; nohup ping -D -n -i 0.2 -W 1 2001:db8:aaaa::2 > /tmp/final_boss_pc1_lan_v6.log 2>&1 & echo $! > /tmp/final_boss_pc1_lan_v6.pid'

Replace <ONT1_WAN1_V6>, <ONT1_WAN2_V6>, <ONT1_WAN2_V4>, and <ONT2_PPP_V6> with the values discovered in 14.3.

These probes cover:

  • ONT1 WAN1 IPv6-only -> DNS64
  • ONT1 WAN2 IPv6 -> DNS64
  • ONT1 WAN2 IPv4 -> 99.99.99.99
  • ONT2 PPPoE IPv6 -> DNS64
  • PC1 behind PD -> DNS64

To inspect the logs generated by the probes:

Option 1, helper:

bash configs/cbot/scripts/subscriber-failover-probes.sh final-boss tail

Option 2, manual commands:

docker exec ont1 sh -lc 'tail -n 12 /tmp/final_boss_ont1_wan1_v6.log'
docker exec ont1 sh -lc 'tail -n 12 /tmp/final_boss_ont1_wan2_v6.log'
docker exec ont1 sh -lc 'tail -n 12 /tmp/final_boss_ont1_wan2_v4.log'
docker exec ont2 sh -lc 'tail -n 12 /tmp/final_boss_ont2_ppp_v6.log'
docker exec pc1 sh -lc 'tail -n 12 /tmp/final_boss_pc1_lan_v6.log'

Observed baseline samples:

[1773326051.535098] 64 bytes from 2001:db8:aaaa::2: icmp_seq=43 ttl=62 time=3.97 ms
[1773326051.543326] 64 bytes from 99.99.99.99: icmp_seq=43 ttl=63 time=11.2 ms
[1773326051.648545] 64 bytes from 2001:db8:aaaa::2: icmp_seq=43 ttl=62 time=1.73 ms
[1773326051.667407] 64 bytes from 2001:db8:aaaa::2: icmp_seq=43 ttl=61 time=1.83 ms

14.5 Act II: The First Impact​

Cut OLT -> BNG MASTER:

docker exec containerbot sh -lc '/app/scripts/olt-to-bng1-down.sh'

Validate:

show srrp
show service active-subscribers

Observed result:

A:admin@MASTER# show srrp
2  9998  dual-stack  Up  initialize
1  9998  ipv6-only   Up  initialize
3  9998  vip         Up  initialize
A:admin@SLAVE# show srrp
2  9998  dual-stack  Up  master
1  9998  ipv6-only   Up  master
3  9998  vip         Up  master
A:admin@MASTER# show service active-subscribers
...
00:d0:f6:01:01:01  IPoE   DHCP6        9998  N
00:d0:f6:01:01:02  IPoE   DHCP         9998  N
00:d0:f6:01:01:04  PPP 1  DHCP6        9998  N
A:admin@SLAVE# show service active-subscribers
...
00:d0:f6:01:01:01  IPoE   DHCP6        9998  Y
00:d0:f6:01:01:02  IPoE   DHCP         9998  Y
00:d0:f6:01:01:04  PPP 1  DHCP6        9998  Y

This was the hardest hit. The probes did not die, but there was a real pause during the switchover.

Observed example on ONT1 WAN1:

[1773326055.947060] 64 bytes from 2001:db8:aaaa::2: icmp_seq=65 ttl=62 time=3.00 ms
[1773326063.996298] 64 bytes from 2001:db8:aaaa::2: icmp_seq=103 ttl=63 time=1.92 ms

Interpretation:

  • there was a single gap of roughly 8 seconds during the first failover
  • when traffic returned, ttl=62 became ttl=63, proving the path had moved through SLAVE

14.6 Act III: Progressive Isolation​

With SLAVE already acting as master, cut Carrier1 -> BNG2:

docker exec containerbot sh -lc '/app/scripts/carrier1-to-bng2-down.sh'

Validate on SLAVE:

show router 9999 bgp summary
show srrp

Observed result:

A:admin@SLAVE# show router 9999 bgp summary
...
172.16.1.3
to_CARRIER1
                65501       0    0 00h00m01s Connect
                            0    0
172.16.2.3
to_CARRIER2
                65502     181    0 01h27m08s 2/2/3 (IPv4)
                          191    0           2/2/4 (IPv6)
A:admin@SLAVE# show srrp
2  9998  dual-stack  Up  master
1  9998  ipv6-only   Up  master
3  9998  vip         Up  master

Observed probe signals:

  • no new icmp_seq gaps appeared
  • ONT1 WAN2 IPv4 -> 99.99.99.99 kept replying
  • internal IPv6 flows (ONT1, ONT2, PC1) remained alive

Real sample after this second failure:

[1773326087.851562] 64 bytes from 99.99.99.99: icmp_seq=222 ttl=63 time=16.3 ms
[1773326088.052683] 64 bytes from 99.99.99.99: icmp_seq=223 ttl=63 time=16.8 ms

14.7 Act IV: Brain Death​

Stop RADIUS:

docker stop radius

Validate:

docker ps -a --format '{{.Names}}\t{{.Status}}' | rg '^radius\b'
show service active-subscribers

Observed result:

radius    Exited (0) 32 seconds ago

Already active subscribers remained present and forwarding on the active node:

A:admin@SLAVE# show service active-subscribers
...
2001:db8:100::5/128
              00:d0:f6:01:01:01  IPoE               DHCP6        9998       Y
100.80.0.4
              00:d0:f6:01:01:02  IPoE               DHCP         9998       Y
2001:db8:100::4/128
              00:d0:f6:01:01:04  PPP 1              DHCP6        9998       Y

And the probes kept moving:

[1773326125.789863] 64 bytes from 99.99.99.99: icmp_seq=411 ttl=63 time=6.39 ms
[1773326125.737325] 64 bytes from 2001:db8:aaaa::2: icmp_seq=410 ttl=63 time=1.79 ms
[1773326125.757919] 64 bytes from 2001:db8:aaaa::2: icmp_seq=410 ttl=62 time=1.85 ms

Operational note:

  • with Test account disabled in the health-check policy, show aaa radius-server-policy "radius_policy" may take time to declare the server down even if the container is already gone
  • this test did not force new authentication; it validated continuity for already established sessions

14.8 Verdict​

The lab survives the “Final Boss”.

But the truth matters:

  • it was not hitless
  • the first switchover (OLT -> BNG MASTER down) caused a visible interruption of around 8 seconds
  • after that initial blow, the network absorbed the loss of Carrier1 on the active node and the loss of RADIUS without losing the tested flows

Conclusion:

  • if the criterion is service survival, the test passes
  • if the criterion is zero loss during the first failover, the test does not pass yet

14.9 Restore the realm​

docker exec containerbot sh -lc '/app/scripts/carrier1-to-bng2-up.sh'
docker exec containerbot sh -lc '/app/scripts/olt-to-bng1-up.sh'
docker start radius

Note:

  • carrier1-to-bng2-up.sh must enable ethernet-1/2 on Carrier1, which is the interface affected by the failure step

Validate nominal state:

show srrp
show router 9999 bgp summary

Expected state:

  • MASTER returns to master
  • SLAVE returns to backupShunt
  • Carrier1 and Carrier2 return
  • radius returns to Up

14.10 Clean up the probes​

Use only one of these two options:

  • if you started the probes with the optional helper, stop them with the helper
  • if you started the probes manually with docker exec, stop them manually with the kill commands

There is no need to run both options. If you use the helper first and then run the manual kill commands, you may see No such process because the probes were already stopped.

Option 1, helper:

bash configs/cbot/scripts/subscriber-failover-probes.sh final-boss stop

Option 2, manual commands:

docker exec ont1 sh -lc 'kill -INT $(cat /tmp/final_boss_ont1_wan1_v6.pid) $(cat /tmp/final_boss_ont1_wan2_v6.pid) $(cat /tmp/final_boss_ont1_wan2_v4.pid)'
docker exec ont2 sh -lc 'kill -INT $(cat /tmp/final_boss_ont2_ppp_v6.pid)'
docker exec pc1 sh -lc 'kill -INT $(cat /tmp/final_boss_pc1_lan_v6.pid)'