We're currently seeing a network outage in Seattle due to issues with our transport provider. We're currently investigating the situation and discussing it with them.
This incident started at 07:15 UTC
All issues have now been resolved and latency is back to normal. Thank you for your patience throughout this incident, we apologise greatly for the inconvenience caused.
Routing is now fully back to the normal routes. Latency is increased by around 50ms due to our transport provider having to re-route around fiber cuts likely in Portland Oregon and maintenance being carried out on the Seattle -> Salt Lake City route - this route should be used later today when that maintenance concludes around 11am UTC - once this route is used, additional latency will reduce to around 20ms, until the original fiber cut is resolved to reduce the additional latency to 0.
This incident is caused by multiple fiber cuts with our transport provider, we have since moved to backup routing via our GSL network. There will be some packet loss to/from China due to less than optimal routing, we're currently working to optimise further where possible until our transport to Ceranetworks is restored.
Our transport provider are working to resolve the issues with their network. The network has partially recovered with a small amount of packet loss continuing.
We are currently working with our transport provider to resolve the issues. We are seeing a partial recovery in the network with a full recovery expected within 10-15 minutes.
We're currently seeing thermal issues with Seattle Premium KVM server: SEAKVM19. This server will suffer periodic downtime while we diagnose and resolve.
Our Seattle DDoS protected network saw instability starting at 20:37 UTC and ending at 20:45 UTC, this was caused by one of our DDoS mitigation providers, Neoprotect, suffering a full network outage, our backup network automatically took over within minutes and this will remain active until Neoprotect is stable again.
See the update below for full details:
Neoprotect has stated that they are not going to continue their service. We are investigating alternative DDoS mitigation providers and hope to have one available soon, this will also return latency to normal levels.
Neoprotect is still offline, they use datapacket/CDN77 as a sole upstream and they disabled Neoprotect's service which resulted in the full global outage of Neoprotect's network. You can read more about the cause here: https://status.neoprotect.net
We are currently routing DDoS protected inbound traffic through Cnservers Los Angeles so there will be a latency increase for locations that are geographically close to Seattle, if this is a problem for you in the short term, please contact us and we'll discuss the options available with you.
Longer term, we will onboard another ddos mitigation provider, potentially GSL, that has a local Seattle POP, this will ensure latency is normal.
No incidents reported
No incidents reported
No incidents reported
No incidents reported
Intermittent packet loss of 10% on average, often 100% loss for up to 20 seconds, to China Telecom and Unicom. This is caused by the network transport provider we use for AS4837 having problems, we have sent them a message about it and are awaiting an update.
No incidents reported