Between November 4-6, Runbox experienced powerful DDoS attacks by a group calling themselves “Armada Collective”. Other security oriented email services such as ProtonMail, Hushmail, and Neomailbox were also attacked, as recently reported by Forbes.
The initial threats and attacks that attempted to extort money were withdrawn by the attackers on Saturday morning, when they offered an apology.
During the attacks we were focused on coordinating with our partners, putting in place countermeasures, and providing our customers with necessary information. Since the situation was unclear and evolved quickly, we decided to not publish any details that could inform the attackers in any way.
The situation is now under control and we are publishing this summary of the events as it may help shed light on what transpired against both Runbox and the other services that were attacked.
Timeline
Wed. 04-Nov-2015
- Shortly after 16:00 CET we received a message demanding payment in bitcoins before 07-Nov-2015 otherwise there would be a sustained Distributed Denial of Service (DDoS) attack on our servers. The message said they would also begin an attack on our servers to demonstrate that this was not a hoax.
- Within a few minutes the attack begun, and we took action to mitigate this. This was a DNS amplification attack which peaked at 34 Gbps, and our email services were down for around 10 minutes before we restored service. The total duration of the attack at various levels was for around 1 hour and there was a resulting mail queue that we cleared shortly after that.
- We immediately began discussing what measures we would take should this happen again, and along with our partners developed what we believe to be an appropriate and responsible plan of action. Paying the demand to avoid future attacks was not an option that was on the table at any point. We informed our partners we would not pay, and they confirmed without reservation that this was the right course of action.
- Once we had gathered sufficient details regarding the attack, including all IP addresses used, we reported the attack to NorCERT (The Norwegian Computer Emergency Response Team) and Økokrim (The Norwegian National Authority for Investigation and Prosecution of Economic and Environmental Crime).
Thu. 05-Nov-2015
- Throughout the day we continued to monitor the situation, and detected a short low level attack of 2-3 Gbps on our servers that didn’t cause us any problems and was dealt with by the countermeasures that we put in place following the first attack.
- Later in the afternoon we recorded another DNS amplification attack which peaked around 18:25 CET at 42 Gbps, which we dealt with effectively with no downtime.
- We also continued to monitor the problems being experienced by other secure email providers that were affected, and made contact with some to share information that might be of use to the Internet community.
- Talks continued through the day on various strategies we could adopt in different future scenarios depending on the type and duration of any further attack.
Fri. 06-Nov-2015
- Now that we had a plan of action in place we informed our customers via a further blog post and information on social media. In addition we sent an email to all customers.
- The immediate response we got via email, our blog and on social media was overwhelmingly positive and supported our stance not to pay the money demanded. In fact, 100% of the messages we received about this supported our stance on the payment, and we were offered support and help from around the world. We are extremely grateful for the huge amount of support from our customers and others as it enabled us to confidently make difficult decisions in what is often an ever changing situation.
- During the morning we had a further demand from attackers for payment. The message quoted an extract directly from the blog post made by ProtonMail where they explain that to protect themselves they would need a solution that would cost them $100,000 per year. The attackers said that if we didn’t want to end up like ProtonMail, then we should pay.
- At 17:42 CET we suffered a further attack. This attack was more sophisticated and sustained than the first ones. The attackers employed both DNS amplification, NTP amplification, and TCP SYN flooding, and exceeded 50 Gbps. We adapted a more radical strategy in order to protect ourselves and our partners. Due to the countermeasures we put in place many of our services ran normally during the attack, and depending on location some customers were able to access them. We continued to respond to support requests during the attack.
- At 22:30 CET we decided that the level of the attack had dropped enough that we could adjust our countermeasures in order to bring up more services. Incoming mail from servers that were not able to deliver to us during the attack started arriving, and we monitored the situation closely to ensure that the incoming servers were not overwhelmed by the large amount of mail that was arriving for our customers. Mail delivery was back to normal at around 03:00 CET.
- Among the messages that arrived following the attack, was a message apparently from the attackers. They described their message as an apology, and said that attacking Runbox was a mistake and that we should not pay them. They were keen to point out that they were not behind the sustained attack on ProtonMail. They insisted that while they were responsible for the initial 15 minute attack on ProtonMail, the ongoing attack was not of their doing. The message was very much at odds with the previous messages, and while we do believe it was genuinely from the attackers, we have no way to verify this for sure.
Sat. 07-Nov-2015
- As already mentioned, mail delivery was back to normal at around 03:00 CET, and as of the time of writing we have no reason to believe we will be attacked again. We are pleased to say we have some excellent people working with us who are monitoring the situation closely. We remain on alert.
Commentary
The ability to keep Runbox running and accessible for all our customers is essential to us. The support we had both publicly and privately from customers and our partners Copyleft Solutions AS and Blix Solutions AS was extremely important in allowing us to make difficult decisions in dealing with these attacks.
Moreover, the response from our customers after we informed them that we would not pay any ransom was overwhelmingly positive, and we are sincerely grateful for the support we received via email and other comments.
Regardless of the apparent apology we finally received from the attackers, DDoS attacks are a criminal act. We built our company on the core values of ethics, security, and privacy, and we will continue to work with our partners, customers, and others to defend those values.
Thank you RUNBOX for maintaining service during what must have been a very difficult and anxious time for you; your reputation as a secure service provider has been increased as a result of your response and control to the situation. Well done!
Wow! Great job Runbox Team! It sounds like I chose the correct email provider indeed. I for one, am very appreciative of the communication regarding the events. Just enough to let us know what was happening. If I can’t live without my email for an hour then something is wrong with me.
Outstanding response and service!
What can I say? I’m impressed, both by your response and the above detailed account of the incidents.
Too often people assume this is the work of a couple of losers in their mothers basement, nothing could be further from the truth.
The people behind these kinds of attacks are working for big business operations and I would hope the government tracks these people down and demands their extradition to Norway for criminal prosecution.
I am impressed with how runbox handled this assault and I fear this kind of crime will be common in the future.
The NPR broadcast Radiolab in the U.S., and on itunes, has a shocking show on this very subject.
Excellent response Runbox.
Being kept updated during the attack was very much appreciated, as is the summary.
I suppose we must all get used to the fact that criminals will continue to try to extort payment from service suppliers.
Any that surrender to criminal demands increase the risk to others and I hope, be shunned by their clients thereafter!
You did a great job and can only hope the criminal investigators perform equally well.
Thank you.
I wouldn’t have even noticed the DDoS attacks if it weren’t for the email you sent round! Great job.
Thank you all for your praise and support!
It’s been a hectic few days at Runbox, but you can be sure we are filing a detailed report to the police and will be evaluating the events together with our partners to prepare for possible future attacks.
I’m proud to be a runbox customer! Thank you for all you’ve done to keep our email up and secure.
Well done Runbox, absolutely the right approach.
Thanks
Great job Runbox Team, especially the transparency and continuous communication with which you have handled the whole situation!
So proud to be a Runbox client! Compliments to the team in every way. You all are awesome. I love your service.
I am not a Runbox customer (yet).
I like what I hear about Runbox based on your active responses and care as related by comments above. However, I still remain troubled by these same comments. The general attitude as expressed here is that the attack involved “criminal elements” and “big business operations”– which is not what was described by you and other email servers (ProtonMail for example) within the attack description.
I am informed that the “second attackers exhibit[ed] capabilities more commonly possessed by state-sponsored actors.” This means to me that whole governments are involved in order to foment confusion and disinterest in privacy measures.
I also am aware that many corporations (Hulu, Microsoft, Google, and financial institutions, email servers, among others) utilize geo-location as a means of tracking individuals, ostensibly for “marketing purposes,” but more realistically as against their privacy and reporting storage and data to the government at the whim of a subpoena.
I would like to find out from Runbox what it is doing as countermeasures to government scrutiny and “state-sponsored attacks” before I fully commit to Runbox. I welcome any feedback or comment.
These kinds of attacks are criminal regardless of who carries them out, but the comments about “second attackers” were not made by Runbox, and we have no reason to believe there was more than one group involved in the DDoS attacks against us.
You can find a lot of information on our website about our privacy policy and also on the procedures required for law enforcement and governments to gain access to Runbox accounts. I hope the links below are useful.
https://runbox.com/about/privacy-policy/
https://runbox.com/why-runbox/email-privacy/runbox-norway-important/
https://runbox.com/why-runbox/email-privacy/email-privacy-regulations/
Thank you Dave…
a couple of questions then:
– I understand that your servers in Norway are protected and secure, based on your policies. Great. However, that protection is afforded at the end of the “pipe.” What about the source or beginning of the “pipe?” Is my so-called “private” email secure over the web emanating from the USA, for example–can these emails be extracted electronically “over the air?” i.e., encrypted end-to-end? Am I saying that correctly??
– I understand you have a great security policy for private emails, and require court orders, etc. for access to data–superb. However, my question above was in regards to “state-sponsored” hacking (not mere requests) of data, as the original hackers in Proton Mail discussions suggested. You are correct that Runbox “…made no comments…” related to state sponsorship, but we learned that possibility existed in proton mail as expressed by Proton. Thus my question as to hacking versus front-end privacy—I pray for and therefore believe I am confident in Runbox’s sophistication against likely DDNS overload by state (and corporate) sponsorships??
Thanks again…
I believe you will find the answers to your questions at this link:
https://runbox.com/why-runbox/security-reliability/secure-transfer-email/
Encryption during delivery relies on both the sending and receiving servers both supporting encryption. End-to-end encryption is best achieved by using an email program and something like PGP or S/MIME encryption. I hope that helps.
Doc,
I don’t know if this is still relevant to you, but there are programs and/or browser extension that can be combined with Runbox to provide privacy. There are many, but you can check Mailvelope as an excellent and user-friendly option to get the features want. For now, this patchwork solution is probably the best you can do.
There is a project underway by the founder of Lavabit to fundamentally reorganize the way e-mail operates with an emphasis on privacy. It’s called “Dark Mail”. As far as I am aware, it’s more of a standard than a service provider, so Runbox may be able to add support for Dark Mail when the project publishes a standard.
I hope that’s helpful. Good luck!