PAN-FSR-2091-054

Fault Description

At 02:14 local on 2091-05-22, the Synaptic Data Fabric monitoring layer detected a link-state transition on relay NC-04's primary encrypted trunk (trunk-A, 10GbE fibre, connecting NC-04 to the core aggregation switch in Sub-level B2). The trunk oscillated between UP and DOWN states three times in 90 seconds before settling into a DOWN condition at 02:16. Traffic automatically failed over to trunk-B (the redundant path), maintaining end-to-end Fabric connectivity — but the single-trunk operation increased latency on NC-04's downstream segment by 23ms (from 78ms baseline to 101ms) and triggered PANACEA alert PAN-ALT-2091-0522-001 for degraded metric-feed performance on the Watch Floor's real-time dashboard. The alert auto-generated this FSR at Tier 2 due to the mission-affecting latency impact.

Root Cause Analysis

K. Yeo attended the Sub-level B1 relay room at 02:40 local. Physical inspection of relay NC-04 revealed that the SFP+ transceiver in trunk-A port 1 (slot NC-04-P1) was reporting degraded optical power: received signal at −18.2 dBm against a minimum receiver sensitivity of −14.4 dBm. The transceiver's transmit power was nominal, indicating the fault was in the receive path. Fibre-end inspection (using a fibre microscope) of the LC connector on the relay side showed no contamination or damage; the patch-panel connector at the aggregation switch was also clean. The transceiver was swapped to a known-good spare: trunk-A immediately restored to full link with received power at −6.1 dBm — well within operating range. The failed transceiver (S/N SFP-10G-3284) was logged for manufacturer return. Failure mode: internal photodiode degradation consistent with age (transceiver in continuous service since 2088, approximately 26,000 powered-on hours).

Work Performed

K. Yeo attended under SOP PAN-ENG-009 (Synaptic Data Fabric Relay Maintenance):

1. Fault isolation. Confirmed trunk-A DOWN via the Fabric management console. Verified trunk-B was carrying full traffic load without errors. Inspected relay NC-04's physical indicators: power, fan, and chassis health LEDs all nominal. Trunk-A port LED showed amber (link-down).

2. Transceiver diagnostics. Read SFP+ DOM (Digital Optical Monitoring) data from the failed transceiver via CLI. Received optical power: −18.2 dBm (threshold: −14.4 dBm). Transmit power: −2.3 dBm (nominal: −1.0 to −4.0 dBm). Temperature: 41°C (nominal). Bias current: 8.1 mA (nominal). Diagnosis: receive-path degradation only.

3. Fibre inspection. Both LC connector end-faces (relay side and patch-panel side) inspected with a 400x fibre microscope. Both clean — no contamination, scratching, or chipping. Fibre path ruled out as a contributing factor.

4. Transceiver swap. Failed SFP+ transceiver (S/N SFP-10G-3284) removed from slot NC-04-P1. Replacement transceiver (S/N SFP-10G-4917, from engineering spares) installed. Trunk-A restored to UP state within 8 seconds of insertion. Received power: −6.1 dBm. Link-state stable.

5. Traffic re-balancing. Both trunks confirmed operational. Fabric load-balancing algorithm redistributed traffic across trunk-A and trunk-B within 45 seconds. Watch Floor latency returned to 78ms baseline. PANACEA alert PAN-ALT-2091-0522-001 auto-cleared at 03:07.

6. Monitoring hold. NC-04 placed on 48-hour enhanced monitoring (5-minute DOM polling on both trunks). No anomalies detected during the hold period.

Parts Consumed

1x SFP+ 10GbE transceiver, single-mode (P/N DF-SFP10-SM) — drawn from engineering spares, Sub-level B1 rack store. Stock level dropped to 3 units; automatic reorder triggered. Replacement batch received 2091-05-28.

Failed transceiver (S/N SFP-10G-3284) — logged for manufacturer return under warranty-replacement programme. Failure analysis report requested.

Sign-off Chain

Service performed by: K. Yeo, Fabric Infrastructure Engineer — 2091-05-22

48h monitoring verified by: K. Yeo — 2091-05-24

Report closed by: Kieran Yeo, Director of Engineering — 2091-05-24

Post-Action Notes

The failed transceiver (S/N SFP-10G-3284) was one of the original batch installed during the Synaptic Data Fabric deployment in 2088. At approximately 26,000 hours of continuous operation, the photodiode degradation is within the manufacturer's quoted MTBF range (30,000 hours typical) but below the 40,000-hour design target. Engineering has identified 14 additional original-batch transceivers still in service across the Fabric relay infrastructure. A proactive replacement programme has been scheduled for Q3 2091: all original-batch units will be swapped during their next relay maintenance window, before they approach the failure-probability zone. The Fabric's redundant-trunk architecture performed as designed — the failover was automatic, traffic loss was zero, and the latency impact (23ms) was within the Tier 2 threshold but did not reach the Tier 3 service-affecting level.