We're currently seeing network disruption in Seattle, China Unicom is currently suffering the most issues. It appears there has likely been major fiber cuts, we're investigating.
Transport provider's route has gone back to a long distance re-route so latency will have increased again, hopefully won't take too long to reduce back to normal again.
Our transport provider has confirmed they suffered multiple fiber cuts in a very short time period which is what caused the outages and high latency. A path has since come back online so latency has reduced to only 17ms above normal, this will be resolved later today as soon as the fiber cut can be repaired. China Unicom routing via AS4134 has been restored.
China Unicom connectivity has been restored, AS4134 stopped routing traffic to AS4837 - likely related to the major suspected fiber cut event on US west coast. We're still checking the quality of the network overall. Latency will be temporarily relevated due to the fiber cuts.
Our Seattle DDoS protected network saw instability starting at 20:37 UTC and ending at 20:45 UTC, this was caused by one of our DDoS mitigation providers, Neoprotect, suffering a full network outage, our backup network automatically took over within minutes and this will remain active until Neoprotect is stable again.
See the update below for full details:
Neoprotect has stated that they are not going to continue their service. We are investigating alternative DDoS mitigation providers and hope to have one available soon, this will also return latency to normal levels.
Neoprotect is still offline, they use datapacket/CDN77 as a sole upstream and they disabled Neoprotect's service which resulted in the full global outage of Neoprotect's network. You can read more about the cause here: https://status.neoprotect.net
We are currently routing DDoS protected inbound traffic through Cnservers Los Angeles so there will be a latency increase for locations that are geographically close to Seattle, if this is a problem for you in the short term, please contact us and we'll discuss the options available with you.
Longer term, we will onboard another ddos mitigation provider, potentially GSL, that has a local Seattle POP, this will ensure latency is normal.
Our Dallas DDoS protection provider, CosmicGuard, is suffering global network issues due to very large ddos attacks coming from a known 40Tb/s botnet, they're working on resolving the situation as soon as possible, no exact ETA currently available.
Stability returned at 12:01 UTC, CosmicGuard has posted a RFO here: https://status.as30456.net/cmiivbxm6004wt518aafiq5dd
We're seeing ongoing long outages, we're still awaiting an update from CosmicGuard.
CosmicGuard saw a brief global outage from 09:49 - 09:53 UTC, issues with specific POPs are ongoing.
CosmicGuard started global maintenance at 17:58 UTC and it appears it caused a global outage rather than only in Los Angeles, things have since came back starting at 18:04 UTC.
No incidents reported
Cloudflare is currently suffering issues, you may see a 500 error if using Cloudflare for your website.
You can follow the issue on their status page: https://www.cloudflarestatus.com
No incidents reported
No incidents reported
Our Dallas DDoS mitigation provider, CosmicGuard, is seeing issues with their Los Angeles POP, this means any traffic routing through it is currrntlt being dropped. We've informed CosmicGuard and await an update from them.
No incidents reported