Last Thursday at about 6:40 a.m. we received an email threatening a Distributed Denial of Service (DDoS) attack on our systems, unless we paid a ransom of 2.5 bitcoins. Within minutes an attack was launched and hootsuite.com became inaccesible. Our OnCall team immediately went to work to mitigate the attack.
In a DDoS attack, a large number of compromised computers (collectively called a botnet) repeatedly sends many simultaneous web requests to a website, which overwhelms the site’s ability to process regular traffic.
To a regular hootsuite.com customer, it appears like the site is unresponsive or in the best case really, really slow. Though such an attack is annoying and potentially costly to the customer, it’s important to note that in no way was any hootsuite.com customer data compromised. The attack was directed at the system that handles incoming web requests – our load balancer grid. Our databases and other internal systems were unaffected. In fact, during the attack scheduled messages continued to be sent.
After verifying the attack was legitimate, HootSuite’s OnCall team focused on getting our load balancer (LB) grid back online. We first attempted to identify and isolate the malicious traffic so that we could block it before it reached the LBs. However, our LBs were unable to perform diagnostic activity because they were maxed out handling incoming traffic. Next, we turned our attention to scaling up our capacity to handle the incoming traffic. Fortunately, we have a solid configuration management system in place powered by ansible, which allows us to spin up new LBs and add them to our grid quickly. That combined with the elasticity provided by AWS enabled us to ramp up the capacity of our LB grid to handle the traffic from the attack and process regular traffic at the same time – bringing hootsuite.com back online for our users at approximately 9:40am. We then continued to triage the malicious traffic in an attempt to block it, however about an hour after we brought the site back up, the attack stopped.
We recognize that HootSuite is an essential tool for our customers, and as such we invest a lot of effort in making sure it is available 24/7. We’re taking the following steps to reduce downtimes and improve resiliency to DDoS attacks in the future:
- Hardening the outer layer of our architecture so malicious traffic gets dropped instead of bogging down servers
- Improve effectiveness of our monitoring systems which will allow us to pinpoint the problem quickly
- Improve on our auto scaling infrastructure to make it even faster and easier to add capacity to handle attacks
You may be wondering why we didn’t just fork over the 2.5 Bitcoins (about $1200) to pay off the attacker. For one, we are not interested in negotiating with criminals. Additionally, paying the ransom would likely lead to future attacks.
It’s probable that the attacker was testing our willingness to bargain – 2.5 Bitcoins is high enough to sound legitimate, but low enough that there was a reasonable chance we would pay. If we handed over the money it’s likely another ransom would follow with an even higher price tag.
It’s important to us that our customers know what’s going on during an outage – we commit to posting an update to status.hootsuite.com as soon as we detect an issue and at least every 15 minutes after that until it’s resolved. The steps outlined above will ensure we can diagnose issues even faster and get updates out to customers as quickly as possible.
I would like to thank the folks at meetup.com who were eager to help and provided valuable information to our team, having suffered a similar attack a week before. Our team would be happy to collaborate with anyone suffering a similar attack – feel free to reach out to me on twitter at @sedsimon if you have information. In closing, I would like to offer my sincere apologies to all customers affected by this attack, and to thank you for your patience and continued support.