We're currently seeing a network outage in Seattle due to issues with our transport provider. We're currently investigating the situation and discussing it with them.
This incident started at 07:15 UTC
All issues have now been resolved and latency is back to normal. Thank you for your patience throughout this incident, we apologise greatly for the inconvenience caused.
Routing is now fully back to the normal routes. Latency is increased by around 50ms due to our transport provider having to re-route around fiber cuts likely in Portland Oregon and maintenance being carried out on the Seattle -> Salt Lake City route - this route should be used later today when that maintenance concludes around 11am UTC - once this route is used, additional latency will reduce to around 20ms, until the original fiber cut is resolved to reduce the additional latency to 0.
This incident is caused by multiple fiber cuts with our transport provider, we have since moved to backup routing via our GSL network. There will be some packet loss to/from China due to less than optimal routing, we're currently working to optimise further where possible until our transport to Ceranetworks is restored.
Our transport provider are working to resolve the issues with their network. The network has partially recovered with a small amount of packet loss continuing.
We are currently working with our transport provider to resolve the issues. We are seeing a partial recovery in the network with a full recovery expected within 10-15 minutes.
We're currently seeing thermal issues with Seattle Premium KVM server: SEAKVM19. This server will suffer periodic downtime while we diagnose and resolve.
Our Seattle DDoS protected network saw instability starting at 20:37 UTC and ending at 20:45 UTC, this was caused by one of our DDoS mitigation providers, Neoprotect, suffering a full network outage, our backup network automatically took over within minutes and this will remain active until Neoprotect is stable again.
See the update below for full details:
Neoprotect has stated that they are not going to continue their service. We are investigating alternative DDoS mitigation providers and hope to have one available soon, this will also return latency to normal levels.
Neoprotect is still offline, they use datapacket/CDN77 as a sole upstream and they disabled Neoprotect's service which resulted in the full global outage of Neoprotect's network. You can read more about the cause here: https://status.neoprotect.net
We are currently routing DDoS protected inbound traffic through Cnservers Los Angeles so there will be a latency increase for locations that are geographically close to Seattle, if this is a problem for you in the short term, please contact us and we'll discuss the options available with you.
Longer term, we will onboard another ddos mitigation provider, potentially GSL, that has a local Seattle POP, this will ensure latency is normal.
We're currently seeing degraded performance to China via AS4837 (China Unicom) on our Seattle network at peak China time. The issue appears to be at China Unicom's side, we've contacted our upstream to ask about this and will provide any further updates on this page.
Our Dallas location went offline at 11:15 UTC (6:15 am local time) due to a major power outage in the area.
All services have since come back online as of 14:45 UTC.
We received the following update from our Data Center provider, Hivelocity, during the outage:
At roughly 6:15am EDT the facility where Hivelocity operates its DAL1 data center (Prime, formerly Carrier-1 and Evocative) lost utility power. Redundant power sources, UPS and generators, did not operate as designed thus causing the entire facility to lose power. Prime Data Centers has electrical engineers on site at this time working to solve the problem. Hivelocity has technicians on site standing by and ready to get all impacted servers back online the moment power is restored to the facility. More information will be forthcoming as solutions progress.
No incidents reported
No incidents reported
No incidents reported
No incidents reported
No incidents reported