Due to necessary reorganization of our email server infrastructure, all Runbox email services will be inaccessible on Sunday 01.04.2018 between approximately 10:00 CEST (08:00 GMT, 4:00 AM EDT) to 14:00 CEST (12 GMT, 8 AM EDT).
The reorganization involves consolidating our servers to simplify management and maintenance, and modernizes and improves server power management.
Any updates during the downtime will be posted on https://status.runbox.com.
We apologize for any inconvenience caused.
Yesterday we deployed our new web servers, which are powering the Runbox web app at https://runbox.com. There are a few changes and improvements that were deployed at the same time, and that we would like to tell you about.
Among other things you may have noticed that the login procedure has changed. This is related to the roll-out of our new Account Security features, which include Two-Factor Authentication. We will post more about this soon, but the important thing to note is that the new login regime is more secure than before. This also completes our transition to a new, global authentication system which we have described previously.
If you are experiencing problems logging in, please make sure that your browser has the latest version of the login screen. You can do this by pressing Ctrl + F5 on Windows and Cmd + R on macOS. If this doesn’t help, please try to clear your browser’s cache and restart it. If this doesn’t help or if you are unsure how to accomplish this, please contact Runbox Support.
There are a few other wrinkles on the new web servers that we are currently ironing out, and besides a more powerful and reliable webmail service we have also deployed a new spam filter.
The new spam filter is powered by Cloudmark, which is one of the strongest authorities on spam analysis in the world. You can try out the new spam filter by going to Manager > Filter and selecting “Cloudmark (beta)” under “Detect junk mail”. If you are already using Dspam (the trainable spam filter) you can select “Both” to activate Cloudmark and Dspam.
The Cloudmark spam filter will automatically catch more spam by comparing spam signatures (fingerprints) with the central Cloudmark database. If you click “Not spam” or “Report spam” to correct spam filter behavior in the webmail, a report will be sent encrypted to the central Cloudmark service. Select “Train using reduced email details” to only send a message signature instead of the full message when reporting misclassified messages.
And if you haven’t already done so, we recommend that you try out the Runbox Aero webmail theme, which you can find in Webmail > Preferences. This theme has a more modern design and includes larger and more legible fonts.
Finally, with the new web servers we have also established a streamlined deployment system that makes the path from development to production much more efficient. We won’t bore you with details, but we can say that you can expect more exciting features from Runbox going forward.
On Monday, October 12 at 0600 CEST we will replace an email storage unit, and our email services will unfortunately not be accessible while we switch servers.
The downtime will start at approximately 0600 CEST (0400 GMT, 0000 EDT) and last for 30-45 minutes.
Runbox will accept incoming email to your account during the downtime, and those messages will be delivered when the operation is complete. It will however not be possible to send email.
We apologize for any inconvenience caused, and recommend that you check your email before or after the scheduled maintenance window.
To find the local time where you are for this maintenance, please see the time conversion at timeanddate.com
We have worked hard on upgrading our email server park to accommodate all our new customers — and things will only get better: Based on our recent success, our shareholders decided to invest fresh capital to start several new projects that will take Runbox to the next level.
To boost IMAP performance and improve the experience especially for iPhone and iPad users we will soon deploy new Dovecot-based IMAP software.
We have started implementing an SSD (Solid-state Drive) based storage strategy to dramatically improve the performance of our email storage units. SSDs are up to 100 times faster than regular, rotating disks, so this will improve the speed of our IMAP services.
We are also expanding our server park with new servers and upgrades to continue improving reliability and overall performance.
We will continue working hard to ensure the privacy and security of your data, and have the following projects lined up for next year:
We look forward to continue providing great email services in the year to come.
Runbox has seen a tremendous growth in our user base over the past months following the NSA revelations in the press. As a consequence of this we started executing our plans in January to acquire and install new and powerful virtualization servers and storage units.
After substantial preparation of our server infrastructure we started moving data to the new ZFS based storage servers this week. The new storage servers are substantially faster, more reliable, and adds a lot more capacity than the current ones, and this process is moving forward steadily.
We are also deploying new, IMAP servers as an intermediate step towards completely replacing our application server infrastructure. The IMAP servers we are currently deploying will improve IMAP performance while we complete the process of installing new, physical application servers that will replace both our current IMAP, POP, and web servers.
Some of our POP users started experiencing connection problems after being moved to the new storage servers. These users have now been moved back to the old storage servers until we resolve these problems. Update 13:00 CET 27.03.2014: This has probably been solved and we are waiting for feedback from everyone that was affected previously.
Additionally, the interaction between new storage, old storage and the new IMAP servers did not work exactly as predicted, so we rolled back the changes on Wednesday. We had done extensive testing over a long period of time before we deployed this solution, but with some differences (NIC, OS versions) We have now done further testing and will attempt deployment again shortly .
We have reviewed the process thus far in detail and uncovered the likely cause of the problems between the new and old servers. We are making the required system changes to ensure a smooth transition next time.
We would like to apologize to those of you who have experienced connection problems with Runbox recently with IMAP and POP, and assure you that we, along with our team of system administrators, will work to resolve these problems over the next few days so that we can provide fast and reliable services to everyone who cares about online privacy, security and sustainable services.
Update 01.04.2014:
We have gathered and analyzed data from the previous attempt at deploying the new servers and will make another attempt Wednesday (02.04.2014) morning CET, this time using a new set of virtualized servers. We will test new combinations of hardware and software between 8-10 AM CET until we have found the configuration that performs best. Meanwhile we have adjusted the configurations of the current IMAP servers to allow more concurrent connections and stop the connection errors some of our customers have seen throughout the day.
Update 03.04.2014:
Generally IMAP should now operate normally. Between 9 and 11 AM CET when we carry out configuration work with the new IMAP servers some users may experience intermittent connection problems. This work will ensure that the new servers perform at their optimum reliability when we complete their configuration.
The new IMAP servers have performed perfectly during our test phase while emulating a large number of users, but something causes them to slow down when communicating with the new ZFS based storage units. We are working systematically to eliminate the causes and are excited about offering this superior storage technology to all our customers.
Update 08.04.2014:
After several days of testing we have narrowed down the problem to the new ZFS based storage units; not the IMAP servers as was indicated earlier. There are two main issues we are looking at and we expect to have a permanently deployed solution after a couple more days of work.
We plan to do the work outside of European and US business hours to avoid service disruptions for as many customers as possible. We are also looking at contingency plans in case this does not turn out as expected.
If you experience connection errors with Runbox IMAP, please contact Support as the symptoms can vary from account to account. We can then take steps to improve the situation for your account specifically.
Update 11.04.2014:
We have confirmed that the problem with the new ZFS storage was related to deadlocks in certain NFS threads in its operating system. A patch for this error was recently released, and after applying this upgrade the server has been operating perfectly for a full working day.
We therefore believe the problem to be resolved. We will continue to monitor its performance closely over the next few days.
The plan is then to continue moving user accounts to the new ZFS storage and our new IMAP servers, which is likely to improve IMAP performance for all our customers.
In order to complete improvements to our email services we are scheduling the following downtime:
*** Thursday, December 20 at 6:30 AM to 8:30 AM CET ***
(5:30-7:30 GMT / 12:30-2:30 AM EST / 9:30-11:30 PM Dec 11 PST)
This means that the Runbox email services to be unavailable for up to 2 hours while the work takes place.
The downtime should be outside business hours for most of our customers, but we know you depend on your Runbox email and we will work as efficiently as possible.
Incoming email will be queued on other servers and will be delivered once the system is fully operational again.
If you usually check your email during this time, please log on either well before or after that interval on Thursday.
To keep updated you can check our Blog at http://blog.runbox.com and Twitter at https://twitter.com/Runbox.
We are completing work started last week, switching to new and powerful database servers. This replacement will improve both the performance and reliability of our email services.
This is part of our server upgrade plan in which most of our servers are being replaced and/or virtualized.
By replacing or virtualizing our servers we greatly improve the reliability of our email services since we will not only be running better hardware, but will be able to move services across our servers much more easily.
We apologize if the upgrade causes any inconvenience for you and appreciate your understanding!
In order to improve our email services we will perform server replacements with the scheduled downtime:
*** Wednesday, December 12 at 6:30 AM to 8:30 AM CET ***
(5:30-7:30 GMT / 12:30-2:30 AM EST / 9:30-11:30 PM Dec 11 PST)
This means that the Runbox email services to be unavailable for up to 2 hours while the replacement take place.
The downtime should be outside business hours for most of our customers, but we know you depend on your Runbox email and we will work as efficiently as possible.
Incoming email will be queued on our servers and will be delivered once our system is fully operational again.
If you usually check your email during this time, please log on either well before or after that interval on Wednesday.
To keep updated you can check our Blog at http://blog.runbox.com and Twitter at https://twitter.com/Runbox.
We are switching to two much more powerful database servers which will improve both performance and reliability of our email services, since the second server can take over in case the first one goes down.
This server replacement is one of the last on our system that will need extended downtime.
This is part of our server upgrade plan in which most of our servers are being replaced and/or virtualized.
By replacing or virtualizing our servers we greatly improve the reliability of our email services since we will not only be running better hardware, but will be able to move services across our servers much more easily.
We apologize if the upgrade causes any inconvenience for you and appreciate your understanding!
Thank you,
The Runbox Team
UPDATE 12.12.2012 7 AM CET: Please go to the main page at http://blog.runbox.com for updates.
Here are some pictures from the move showing our old and new Data Center, our servers, and some of our sysadmins.
All in all the process involved around 2 weeks of planning, 10 people, 3 cars, a few hundred feet of cable, 10 hours of deracking, driving, and installing, and a week cleaning up afterwards.
It’s good to know that Perdita, Pongo, Patch, Penny, Pepper, Taishi, Rambo, Takara, Tinkerbell, Chernushka, Strelka, Bars, Pink, Oscar, Fenris, Greyhound, Odie, Marmaduke, and Sirius are all running smoothly in their new pound!
