The Informatikdienste are upgrading the routers in our HIT/D/13 server room causing a downtime of the network of about 1 hour on Thursday morning, 15th September, between 05:30 and 07:30. Please note that various services will not be available during that time.
sometimes it just has to work, and fast!
UPDATE Thu 00:50 - a broken valve blocked the cooling water in the HIT D 13 server room and all 14 water cooled racks severely overheated (not just D-PHYS). We managed to revive almost all services with the exception of the GGL file shares (this server is dead). We'll post updates later today when we have more information.
complete loss of cooling in the server room. We have yet to assess the damage.
Apparently this time they mean it:
On Thursday, February 14, ETH facility services will conduct extensive power network tests in the HPT building, where ISG (and hence the helpdesk) is located. Power will be gone for ~ 2 hours starting around 13:30. During this time we will not be able to answer the helpdesk phone or work on your tickets. We'll post an update when power is back.
Update, 10:30h: We're already offline since in other floors of the building power has already been cut and caused a network outage on other floors.
UPDATE: the power test has been cancelled. Helpdesk duty as usual.
On Thursday, December 20, ETH facility services will conduct extensive power network tests in the HPT building, where ISG (and hence the helpdesk) is located. We were told to expect at least one power cut lasting at least 15 min, possibly longer. During this time we will not be able to answer the helpdesk phone or work on your tickets. We'll post an update when power is back.
Due to maintenance work relating the electric power supply of the HIT building there will be a planed interruption from 5:00am to 8:00am on Wednesday the 25th of July.
Please note that the whole HIT building will be without electric power during this time (The server room HIT D 13 is excepted from this interruption). Shutdown your computer and switch off (use main switch if available or unplug) your electrical devices in advance to avoid local data loss and help prevent start-up peaks when electric power is switched back on.
Note: this is a purely anecdotal posting about our struggles with some performance bottlenecks in the last few months. If you're not interested in such background information, just skip.
You might have noticed that since about January 2012 using our file and mail servers hasn't been as smooth as usual. This posting will give you some background information concerning the challenges we encountered and why it took so long to fix them. Let's begin with the file server.
Way back in the days (i.e. 5 years ago), when the total file server data volume at D-PHYS was about 10 TB, we used individual file server to store this data. When one server was full, we got a bigger one, copied all the data and life was good for another year or two. Today, the file server data volume (home and group shares) is above 150 TB and growing fast and this strategy doesn't work any longer: individual servers don't scale and copying this amount of data alone takes weeks. That's why in 2009 we started migrating the 'many individual servers' setup to a SAN architecture in which the file servers are just huge hard drives (iSCSI over Infiniband, for the technically inclined) connected to a frontend server that manages space allocation and the file system. The same is true for the backup infrastructure, where the data volume is even bigger.
This new setup had to be developed, tested and put in place as seamlessly and unobtrusively as possible while ensuring data access at all times (apart from single hour-long migrations). The SAN architecture was implemented for Astro in December 2010 and has been running beautifully ever since. In 2011 we laid the groundwork to adopt this system for the rest of D-PHYS's home and group shares and after a long and thorough testing period the rollout happened on January 5, 2012. Unfortunately, that's when things got ugly.
At first, we noticed some exotic file access problems on 32bit workstations. It took us some time to understand that the underlying issue was an incompatibility with the new filesystem using 64-bit addresses for the data blocks. As a consequence we had to replace the filesystem of the home shares. Independently we ran into serious I/O issues with the installed operating system, so we had to upgrade the kernel of the frontend server and move the home directories onto a dedicated server. In parallel, we had to incorporate some huge chunks of group data while always making sure that nightly backups were available. All this necessitated a few more migrations until we finally achieved a stable system on March 28.
The upshot: what we had hoped to be a fast and easy migration turned out to cause a lot of problems and take much longer than anticipated, but now we have a stable and solid setup that will scale up to hundreds or even thousands of TB of data.
See live volume management and usage graphs for our file servers.
As for the mail server, matters are to some extent related and partly just coincidental in time. The IMAP server does need access to the home directories and hence also suffered when their performance was impaired. But even after having solved the file server issues, we still saw single load peaks on the IMAP server that prevented our users from working with their email. Again, we put a lot of time and effort into finding the reason. As of April 13, we're back to good performance and arrive at the following set of conclusions:
Particular issues:
General mail server load:
Bottom line:
We'd like to thank you for your patience during the last 4 months and apologize for any inconvenience you might have had to endure. In all likelihood the systems will be a lot more stable in the future, but of course we're constantly working to ensure the D-PHYS IT infrastructure is able to keep up with the fast growing demand of disk space (the data volume has tripled in the last year alone). We've learned a lot and we'll put it to good use.
Yesterday's outage could be traced to a flaky voltage controller on one of our RAID adapters. We schedule a short maintenance downtime on
Sunday, Feb 12, around 13:00
in order to replace the faulty controller. Most services will be affected.
Update 14:57 Cleanup took a bit longer than expected, but now all system are back again.
Severe hardware failure on one of our core infrastructure servers. We're working on it.
Update 11:44 The hardware problem could be fixed and all services are recovering now. Sorry for the downtime.
On Jan 5, after weeks of thorough planning and rigorous testing, we performed a migration of the home directories and group shares to our new SAN system. Soon afterwards, the first phone calls started coming in. The initial problem was very exotic and affected very few people (that's why we had no chance to detect it during the testing period), but the action we took to address it unfortunately caused a cascade of consecutive faults that led to the instabilities you had to endure for one week now and for which we are truly sorry. We now know how to fix the underlying problem, but we cannot operate on the running server. That's why we have to schedule an
emergency file server migration on Sat, Jan 14, starting at 07:00 and lasting well into the afternoon probably.
During this time, you will not have access to your home or group directories, and also email will only work intermittently. Please stop all running jobs and log out before Saturday morning.
We apologize for the suboptimal performance since Jan 5. You have every right to expect better, but this caught us completely off guard. Thank you for your understanding.
Update, Sat 14:15: mounts and email are up and running again. The problem on 32bit machines still persists, but we have an idea how to fix it on Monday.
Update Fri 20.01: we (hence you) are still suffering from severe stability problems on the file server. We are very hard at work and now have a plan that we really really hope will solve the problems. There will be another migration sometime next week. We're truly sorry for the inconvenience you have to endure.
This Thursday, the 7th of July 2011, around 7am, there will be a short network interruption in the whole Department of Physics. The central ETH IT Services ("Informatikdienste") will move our network zone to new hardware, necessary for some future services.
Additionally, the WLAN Landing Page of the "public" network will have a maintenance downtime from 7am to 8am.
