We're observing intermittent network outages for our Seattle network, lasting up to around 1 minute, multiple times per day. The cause is issues with our network transport provider, we've been discussing it with them but unfortunately they haven't found the cause yet, in part due to how short the outages are. We are continuing to discuss it with them so we can reach a resolution as soon as possible.
Our Seattle DDoS protected network saw instability starting at 20:37 UTC and ending at 20:45 UTC, this was caused by one of our DDoS mitigation providers, Neoprotect, suffering a full network outage, our backup network automatically took over within minutes and this will remain active until Neoprotect is stable again.
See the update below for full details:
Neoprotect has stated that they are not going to continue their service. We are investigating alternative DDoS mitigation providers and hope to have one available soon, this will also return latency to normal levels.
Neoprotect is still offline, they use datapacket/CDN77 as a sole upstream and they disabled Neoprotect's service which resulted in the full global outage of Neoprotect's network. You can read more about the cause here: https://status.neoprotect.net
We are currently routing DDoS protected inbound traffic through Cnservers Los Angeles so there will be a latency increase for locations that are geographically close to Seattle, if this is a problem for you in the short term, please contact us and we'll discuss the options available with you.
Longer term, we will onboard another ddos mitigation provider, potentially GSL, that has a local Seattle POP, this will ensure latency is normal.
Our Seattle network is seeing at least TCP traffic dropped for inbound routes via Telia, we're working on removing Telia routing temporarily.
No incidents reported
No incidents reported
No incidents reported
No incidents reported
No incidents reported
We are currently seeing degraded performance for traffic to China caused by a faulty router on the NCP submarine cable which is used by the AS4837 (China Unicom) line which started on 26th December.
We're currently evaluating the AS4134 line as a temporary solution as it has recently had considerable capacity added.
AS4837 has now been enabled for routes to China Unicom
China Unicom has said the issue with AS4837 has been resolved, we have enabled AS4837 for China Telecom and will enable it for China Unicom within 24 hours if it is stable.
China Unicom has still not provided an estimated fix date for AS4837. We're continuing to test daily and are seeing very low bandwidth throughput for AS4837 at peak time, below the level of AS4134, therefore we believe it's still in everyone's best interest to continue using AS4134 for the time being.
All outbound traffic to China is now using AS4134 (China Telecom) temporarily until AS4837 (China Unicom) resolves the congestion caused by the fiber cut within their network.
We're currently seeing good performance from AS4134 and will continue to monitor.