[Resolved] Transition to new servers and storage

March 27th, 2014  |  Published in Status

Runbox has seen a tremendous growth in our user base over the past months following the NSA revelations in the press. As a consequence of this we started executing our plans in January to acquire and install new and powerful virtualization servers and storage units.

Moving to the new servers

After substantial preparation of our server infrastructure we started moving data to the new ZFS based storage servers this week. The new storage servers are substantially faster, more reliable, and adds a lot more capacity than the current ones, and this process is moving forward steadily.

We are also deploying new, IMAP servers as an intermediate step towards completely replacing our  application server infrastructure. The IMAP servers we are currently deploying will improve IMAP performance while we complete the process of installing new, physical application servers that will replace both our current IMAP, POP, and web servers.

Some bumps in the road…

Some of our POP users started experiencing connection problems after being moved to the new storage servers. These users have now been moved back to the old storage servers until we resolve these problems. Update 13:00 CET 27.03.2014: This has probably been solved and we are waiting for feedback from everyone that was affected previously.

Additionally, the interaction between new storage, old storage and the new IMAP servers did not work exactly as predicted, so we rolled back the changes on Wednesday. We had done extensive testing over a long period of time before we deployed this solution, but with some differences (NIC, OS versions) We have now done further testing and will attempt deployment again shortly .

What we’re doing to resolve the problems

We have reviewed the process thus far in detail and uncovered the likely cause of the problems between the new and old servers. We are making the required system changes to ensure a smooth transition next time.

We would like to apologize to those of you who have experienced connection problems with Runbox recently with IMAP and POP, and assure you that we, along with our team of system administrators, will work to resolve these problems over the next few days so that we can provide fast and reliable services to everyone who cares about online privacy, security and sustainable services.

Update 01.04.2014:

We have gathered and analyzed data from the previous attempt at deploying the new servers and will make another attempt Wednesday (02.04.2014) morning CET, this time using a new set of virtualized servers. We will test new combinations of hardware and software between 8-10 AM CET until we have found the configuration that performs best. Meanwhile we have adjusted the configurations of the current IMAP servers to allow more concurrent connections and stop the connection errors some of our customers have seen throughout the day.

Update 03.04.2014: 

Generally IMAP should now operate normally. Between 9 and 11 AM CET when we carry out configuration work with the new IMAP servers some users may experience intermittent connection problems. This work will ensure that the new servers perform at their optimum reliability when we complete their configuration.

The new IMAP servers have performed perfectly during our test phase while emulating a large number of users, but something causes them to slow down when communicating with the new ZFS based storage units. We are working systematically to eliminate the causes and are excited about offering this superior storage technology to all our customers.

Update 08.04.2014:

After several days of testing we have narrowed down the problem to the new ZFS based storage units; not the IMAP servers as was indicated earlier. There are two main issues we are looking at and we expect to have a permanently deployed solution after a couple more days of work.

We plan to do the work outside of European and US business hours to avoid service disruptions for as many customers as possible. We are also looking at contingency plans in case this does not turn out as expected.

If you experience connection errors with Runbox IMAP, please contact Support as the symptoms can vary from account to account. We can then take steps to improve the situation for your account specifically.

Update 11.04.2014:

We have confirmed that the problem with the new ZFS storage was related to deadlocks in certain NFS threads in its operating system. A patch for this error was recently released, and after applying this upgrade the server has been operating perfectly for a full working day.

We therefore believe the problem to be resolved. We will continue to monitor its performance closely over the next few days.

The plan is then to continue moving user accounts to the new ZFS storage and our new IMAP servers, which is likely to improve IMAP performance for all our customers.

Tags: ,

Server replacements and scheduled downtime

December 19th, 2012  |  Published in Status

In order to complete improvements to our email services we are scheduling the following downtime:

*** Thursday, December 20 at 6:30 AM to 8:30 AM CET ***

(5:30-7:30 GMT / 12:30-2:30 AM EST / 9:30-11:30 PM Dec 11 PST)

This means that the Runbox email services to be unavailable for up to 2 hours while the work takes place.

The downtime should be outside business hours for most of our customers, but we know you depend on your Runbox email and we will work as efficiently as possible.

Incoming email will be queued on other servers and will be delivered once the system is fully operational again.

What you can do

If you usually check your email during this time, please log on either well before or after that interval on Thursday.

To keep updated you can check our Blog at http://blog.runbox.com and Twitter at https://twitter.com/Runbox.

What we are doing

We are completing work started last week, switching to new and powerful database servers. This replacement will improve both the performance and reliability of our email services.

Why it is being done

This is part of our server upgrade plan in which most of our servers are being replaced and/or virtualized.

By replacing or virtualizing our servers we greatly improve the reliability of our email services since we will not only be running better hardware, but will be able to move services across our servers much more easily.

We apologize if the upgrade causes any inconvenience for you and appreciate your understanding!

Tags: ,

Server upgrades and scheduled downtime

December 5th, 2012  |  Published in Status

In order to improve our email services we will perform server replacements with the scheduled downtime:

*** Wednesday, December 12 at 6:30 AM to 8:30 AM CET ***

(5:30-7:30 GMT / 12:30-2:30 AM EST / 9:30-11:30 PM Dec 11 PST)

This means that the Runbox email services to be unavailable for up to 2 hours while the replacement take place.

The downtime should be outside business hours for most of our customers, but we know you depend on your Runbox email and we will work as efficiently as possible.

Incoming email will be queued on our servers and will be delivered once our system is fully operational again.

What you can do

If you usually check your email during this time, please log on either well before or after that interval on Wednesday.

To keep updated you can check our Blog at http://blog.runbox.com and Twitter at https://twitter.com/Runbox.

What we are doing

We are switching to two much more powerful database servers which will improve both performance and reliability of our email services, since the second server can take over in case the first one goes down.

This server replacement is one of the last on our system that will need extended downtime.

Why it is being done

This is part of our server upgrade plan in which most of our servers are being replaced and/or virtualized.

By replacing or virtualizing our servers we greatly improve the reliability of our email services since we will not only be running better hardware, but will be able to move services across our servers much more easily.

We apologize if the upgrade causes any inconvenience for you and appreciate your understanding!

Thank you,

The Runbox Team

UPDATE 12.12.2012 7 AM CET: Please go to the main page at http://blog.runbox.com for updates.

Tags: ,

Pictures from the Move

June 21st, 2011  |  Published in News

Here are some pictures from the move showing our old and new Data Center, our servers, and some of our sysadmins.

All in all the process involved around 2 weeks of planning, 10 people, 3 cars, a few hundred feet of cable, 10 hours of deracking, driving, and installing, and a week cleaning up afterwards.

It’s good to know that Perdita, Pongo, Patch, Penny, Pepper, Taishi, Rambo, Takara, Tinkerbell, Chernushka, Strelka, Bars, Pink, Oscar, Fenris, Greyhound, Odie, Marmaduke, and Sirius are all running smoothly in their new pound!

Tags: ,