We're seeing intermittent network outages for our Dallas and Ashburn network due to issues with CosmicGuard's network, they're working on a resolution.
Issues have started again, CosmicGuard has said they have a plan to resolve the disruption, caused by large attacks leaking past Cloudflare.
We haven't seen any disruption for the past 35 minutes so it's likely this has now been resolved.
CosmicGuard has now updated their status page: https://status.as30456.net/en-gb/cmhhpokrs00s1omomxdm6ppa4
CosmicGuard has confirmed they're actively working on a resolution with Cloudflare to stabilise the network.
The network instability has been continuing, with disruptions every ~5 minutes affecting almost the whole of CosmicGuard's global network. Our current assumption is that ddos attacks are likely the cause, we haven't yet received an update from CosmicGuard.
Our Seattle DDoS protected network saw instability starting at 20:37 UTC and ending at 20:45 UTC, this was caused by one of our DDoS mitigation providers, Neoprotect, suffering a full network outage, our backup network automatically took over within minutes and this will remain active until Neoprotect is stable again.
See the update below for full details:
Neoprotect has stated that they are not going to continue their service. We are investigating alternative DDoS mitigation providers and hope to have one available soon, this will also return latency to normal levels.
Neoprotect is still offline, they use datapacket/CDN77 as a sole upstream and they disabled Neoprotect's service which resulted in the full global outage of Neoprotect's network. You can read more about the cause here: https://status.neoprotect.net
We are currently routing DDoS protected inbound traffic through Cnservers Los Angeles so there will be a latency increase for locations that are geographically close to Seattle, if this is a problem for you in the short term, please contact us and we'll discuss the options available with you.
Longer term, we will onboard another ddos mitigation provider, potentially GSL, that has a local Seattle POP, this will ensure latency is normal.
Our DDoS protected Dallas network is currently offline due to an issue on CosmicGuard's side, we are talking to them.
It has since become known that Cloudflare is currently performing large-scale maintenance in many of their locations, their Dallas location went completely offline for the period we were offline, so this explains why we saw an outage. There was likely something that went wrong on Cloudflare's side in another of their locations that somehow then impacted Dallas, CosmicGuard is setup to automatically fail us over to another of their locations in the event of BGP routes being pulled - something that Cloudflare did not do in Dallas for an unknown reason. CosmicGuard will be talking to Cloudflare about this in the hope of avoiding a repeat.
Dallas came back online at 08:40 UTC. CosmicGuard has been using Cloudflare Magic Transit recently to help absorb the large attacks that DDoS mitigation providers are being hit with nowadays, 20Tb/s+. Unfortunately Cloudflare suffered a connectivity issue to CosmicGuard in Dallas, which resulted in traffic being blackholed, essentially dropped. CosmicGuard is currently investigating with Cloudflare why this happened, to hopefully prevent a repeat. We apologise for the inconvenience caused.
A temporary change has been implemented by our Dallas DDoS mitigation provider, CosmicGuard, that means that any IP that didn't already have rules on it, now has rules automatically applied to it based on the ports used within the last 24 hours, any other ports will be dropped (including those with very low traffic levels). ICMP (ping) has been dropped. If an IP did have rules already on it (such as allowing/dropping ports/applying filters) then this IP is unaffected by these changes. If you need any ports whitelisted or ICMP allowed, let us know via the usual contact methods. This will only be a temporary measure during large attacks and will be lifted automatically afterwards, it will likely no longer be needed in the coming days.
No incidents reported
No incidents reported
No incidents reported
No incidents reported
Our DDoS protected Seattle network is seeing ongoing intermittent disruption due to issues with our DDoS mitigation provider, Neoprotect. There is ongoing disruption with most DDoS mitigation providers currently and they are unfortunately affected too - they're working on solutions to the causes to get things stable again as soon as possible. This only impacts non China routes.
We're seeing intermittent connectivity issues for our Dallas DDoS protected IPs (CosmicGuard), this is likely related to recent disruptions. Issues started at 00:07 UTC and are ongoing. We're in contact with CosmicGuard and are awaiting an update from them.
CosmicGuard is still working on this, they have made progress towards a more permanent solution.
CosmicGuard has informed us that they now have a strategy to prevent the ongoing intermittent disruption, they will be implementing this gradually, with the aim to have it completed by the end of Sunday.
CosmicGuard provided us with a detailed report that included mitigation they intended to permanently resolve this but unfortunately we've started seeing intermittent outages again, they're continuing to investigate and apply mitigations.
CosmicGuard routing has now been fully restored.
Network is now back up, unfortunately something related to the CosmicGuard disruption caused a complete block on TCP/UDP traffic, we're continuing to debug with them.
Wr've been seeing a complete TCP outage since 02:19 UTC, we're talking to CosmicGuard about it.
CosmicGuard has shared an update: "We are aware of the issues affecting services globally for the last few hours, and have been hard at work. We do know what the issue is, and are in the process of applying remedies to fix it. We should have the situation under control soon."
No incidents reported
We're currently seeing significant instability in Seattle due to issues with our transit provider, Cogent, we're working on it currently.
This was fully resolved at 09:10 UTC on 30th September and has been stable since except for 01:38 - 01:41 UTC on 1st October.
We've determined the issue is related to an issue with Ceranetworks' Los Angeles POP, we're awaiting them resolving the issue, in the meantime, bursts of packet loss will be seen for traffic coming from China. If it's not resolved shortly, we'll enable alternative routes temporarily.
All non China traffic returned to normal at 06:40 UTC, we're still seeing some packet loss with China connectivity due to an issue with our connection to Ceranetworks, we're still investigating and monitoring.