Severe server failure
UPDATE Thu 09:30 - all systems should be back to normal. Please let us know if you still encounter problems. Thanks to Axel and Paddy for their commitment and the incredible Dalco service for fixing it within 6h (at 8am, mind you).
UPDATE Thu 00:50 - a broken valve blocked the cooling water in the HIT D 13 server room and all 14 water cooled racks severely overheated (not just D-PHYS). We managed to revive almost all services with the exception of the GGL file shares (this server is dead). We'll post updates later today when we have more information.
complete loss of cooling in the server room. We have yet to assess the damage.