31st March 2021

Seattle Network Network outage

Seattle network is currently offline and being investigated.

UPDATE @ 10:04 BST - network came back up at 09:58. Fix for this issue will be implemented later today.

UPDATE @ 10:31 BST - network went down again at 10:21 and came back up at 10:30.

UPDATE @ 10:54 BST - Outage is currently in progress, we're working on implementing the fix now.

UPDATE @ 12:34 BST - Software update is currently underway, we hope to restore stability very soon.

UPDATE @13:50 BST - Router is now running on the new software, we're monitoring

Summary of the cause: A DoS exploit in the version of our router's software (Junos) meant that if a certain BGP packet was sent to our router, it caused all routing processes to crash and as a result the network would become unavailable. The fix was to update to a Junos version that did not have this exploit. We were planning scheduled maintenance to perform the update however as a result of today's major incident, we went ahead with the update as emergency maintenance to restore network stability.