Due to network maintenance there will be some periods of downtime on 2014-10-06, between 00:01 and 07:59 CET.
This maintenance will require rebooting routers, which will cause about 10 minutes of downtime for
each router.
We apologize for any inconvenience caused.
We are currently experiencing problems with either our network or our firewall, and all Runbox email services are currently unavailable.
Our system administrators are working to correct the problem. Meanwhile, any incoming email will be queued on other servers for later delivery.
Our web and domain hosting services are not affected by this problem.
UPDATE 23:50 CET: A power strip inside a server rack had died, causing the firewall server to go down. We expect this to be resolved shortly.
UPDATE 23:51 CET: Problem has been resolved and email services are back online.
We’re having problems with an unresponsive central server which is affecting much of the Runbox email system and causing the email service to be inaccessible. We’re working to resolve it and expect to be operating normally within 30-45 minutes.
Update 01:15 CET: We are back online and investigating what happened. In any event this was a server that is soon to be replaced.
On April 8, it was revealed in the media that a vulnerability in the internet encryption standard OpenSSL had been discovered. This vulnerability could potentially allow someone to access additional parts of the memory of servers protected by the OpenSSL software.
As stated in the OpenSSL Security Advisory:
A missing bounds check in the handling of the TLS heartbeat extension can be used to reveal up to 64k of memory to a connected client or server.
This could potentially compromise sensitive data such as the secret keys used to identify the service providers and to encrypt the traffic, the names and passwords of users, and actual content.
Runbox immediately upgraded our installations of OpenSSL on April 8 upon learning about this vulnerability. We have also reissued and reinstalled all our SSL certificates for both Web, POP, IMAP, and SMTP services.
Additionally Runbox web services already supports Perfect Forward Secrecy, which issues unique SSL key pairs for each connection. This prevents an unlikely eavesdropper from retroactively decrypting communications between server and client even if they managed to get the private key.
We have no indications that any information has leaked from our systems, and our assessment is that the risk of such leaks is very small. Client computers and software are not affected by this vulnerability.
However, we recommend that you change your Runbox password to be entirely certain that no one else can access your account. It’s a good idea to change your password regularly, and use different passwords for different services. Please see Tips for choosing and protecting passwords for some useful rules about password generation and usage.
More information about Heartbleed from the security company Codenomicon is available at http://heartbleed.com/.
Runbox has seen a tremendous growth in our user base over the past months following the NSA revelations in the press. As a consequence of this we started executing our plans in January to acquire and install new and powerful virtualization servers and storage units.
After substantial preparation of our server infrastructure we started moving data to the new ZFS based storage servers this week. The new storage servers are substantially faster, more reliable, and adds a lot more capacity than the current ones, and this process is moving forward steadily.
We are also deploying new, IMAP servers as an intermediate step towards completely replacing our application server infrastructure. The IMAP servers we are currently deploying will improve IMAP performance while we complete the process of installing new, physical application servers that will replace both our current IMAP, POP, and web servers.
Some of our POP users started experiencing connection problems after being moved to the new storage servers. These users have now been moved back to the old storage servers until we resolve these problems. Update 13:00 CET 27.03.2014: This has probably been solved and we are waiting for feedback from everyone that was affected previously.
Additionally, the interaction between new storage, old storage and the new IMAP servers did not work exactly as predicted, so we rolled back the changes on Wednesday. We had done extensive testing over a long period of time before we deployed this solution, but with some differences (NIC, OS versions) We have now done further testing and will attempt deployment again shortly .
We have reviewed the process thus far in detail and uncovered the likely cause of the problems between the new and old servers. We are making the required system changes to ensure a smooth transition next time.
We would like to apologize to those of you who have experienced connection problems with Runbox recently with IMAP and POP, and assure you that we, along with our team of system administrators, will work to resolve these problems over the next few days so that we can provide fast and reliable services to everyone who cares about online privacy, security and sustainable services.
Update 01.04.2014:
We have gathered and analyzed data from the previous attempt at deploying the new servers and will make another attempt Wednesday (02.04.2014) morning CET, this time using a new set of virtualized servers. We will test new combinations of hardware and software between 8-10 AM CET until we have found the configuration that performs best. Meanwhile we have adjusted the configurations of the current IMAP servers to allow more concurrent connections and stop the connection errors some of our customers have seen throughout the day.
Update 03.04.2014:
Generally IMAP should now operate normally. Between 9 and 11 AM CET when we carry out configuration work with the new IMAP servers some users may experience intermittent connection problems. This work will ensure that the new servers perform at their optimum reliability when we complete their configuration.
The new IMAP servers have performed perfectly during our test phase while emulating a large number of users, but something causes them to slow down when communicating with the new ZFS based storage units. We are working systematically to eliminate the causes and are excited about offering this superior storage technology to all our customers.
Update 08.04.2014:
After several days of testing we have narrowed down the problem to the new ZFS based storage units; not the IMAP servers as was indicated earlier. There are two main issues we are looking at and we expect to have a permanently deployed solution after a couple more days of work.
We plan to do the work outside of European and US business hours to avoid service disruptions for as many customers as possible. We are also looking at contingency plans in case this does not turn out as expected.
If you experience connection errors with Runbox IMAP, please contact Support as the symptoms can vary from account to account. We can then take steps to improve the situation for your account specifically.
Update 11.04.2014:
We have confirmed that the problem with the new ZFS storage was related to deadlocks in certain NFS threads in its operating system. A patch for this error was recently released, and after applying this upgrade the server has been operating perfectly for a full working day.
We therefore believe the problem to be resolved. We will continue to monitor its performance closely over the next few days.
The plan is then to continue moving user accounts to the new ZFS storage and our new IMAP servers, which is likely to improve IMAP performance for all our customers.
Our payment processor is currently having problems processing credit card payments. While they are working to resolve them you can use PayPal to pay with your credit card, even if you don’t have a PayPal account.
Just go to https://runbox.com/payment, select the desired product(s), and choose PayPal as your payment method on the next page.
Thank you, and our apologies for the inconvenience.
Our sysadmins will replace two switches in our network Thursday January 9 at around 18:00 CET (12 noon EST). This will probably cause some short periods of downtime for the Runbox email services while traffic shifts from one path to the other.
No email will be lost or rejected as a consequence of this maintenance. Our US based web hosting services will not be affected.
We’re planning to deploy an extra POP server Monday May 27 at 6-7 AM CET to improve performance for POP email clients.
This deployment should not affect services negatively, but please let us know if you do experience any irregularities using POP during or after the specified time.
Over the past 8-10 days we have seen a significant surge in incoming spam, especially about weight loss.
We are working to reject more of this type of spam centrally, and hope to have achieved this as of today.
But if your account is still severely affected, there are things you can do to improve the situation meanwhile:
Most of our customers report that their spam filter does catch the spam, but that going through the Spam folder looking for legitimate messages becomes too big a task due to the large number of spam messages.
If this is affecting you, please see the following article on how to set up an extra filter to separate the very obvious spam from the messages the spam filter is less certain about and that you should review manually: https://help.runbox.com/avoid-sorting-through-spam-for-legitimate-email/
If your spam filter doesn’t catch this type of spam, please make sure that your Trainable Spam Filter is enabled in Manager:Filter, and that you train any missed spam mails using the Report Spam button.
Many Apple Mail users have been having problems connecting to Runbox via POP recently due to Apple changing the supported authentication methods in Mail.
Apple Mail POP users can now use the server name apop.runbox.com to connect to Runbox.
If you use Apple Mail with POP (not IMAP), please do this:
Your Runbox account should now be able to receive email. If not, please try restarting Apple Mail.
