Web services downtime

Update 07:00 All web services are back online.

Tomorrow Wednesday 2021-01-20 starting at 06:00 we will upgrade the server hardware hosting most of our web services. We expect them to be back by 08:00 at the latest.

Affected web services

The following services are unavailable during the downtime:

Our Debian, Ubuntu and Raspbian mirror as well as Grafana, InfluxDB and Webmail will not be affected.

We will not be able to send any status updates via our news blog or via our Matrix news and status rooms.

2020 in review – Corona edition

oh boy, what a year.

This post is meant to give you a short overview of what has been accomplished in D-PHYS IT by ISG this year. We’ve been hard at work to further improve and extend our services for you, our customers. Some highlights of 2020:

  • Home office: on March 12, due to rapidly rising Covid-19 numbers, ISG was sent to work from home, along with most of the department. While we had somewhat anticipated this step and were prepared for it, the first two weeks were very busy because we had to assist a lot of people who weren't. In the end I believe we got everyone set up and we have been fully operational from home with only occasional individual visits to the office since then.
  • Matrix/Element/Riot: one of the most pressing issues with everyone working from home was an efficient and versatile tool for team communication. We had started internal tests of our Matrix chat system in late 2019, but then intensified our efforts in February and were able to release the system for general D-PHYS availability in home office week (HOW) 2. During the course of 2020, we continually kept working on the system and added new exciting features.
    We also run a Jitsi instance for privacy-aware video conferencing.
  • New laptop backup: our traditional BackupPC backup system for laptops and lab computers relies on each backup client to be reachable in the D-PHYS network, which obviously didn't work any longer in the home office regime. In HOW 17, we released a new backup system for laptops that works from any internet connection worldwide. Unfortunately, only very few of you have signed up for the service so far. Please make sure you have a backup of your laptop!
  • Ansible deployment: more servers and finally also the managed Linux workstations have been added to our ansible configuration management, allowing for completely automated installation of our systems.
  • Network migration: the extensive Hönggerberg network reorganization we reported two years ago hasn't seen much progress by Informatikdienste, but we have been working on our side to make the first steps. In early 2020 we migrated the dhcp.phys DNS service from our servers to ID's as a prerequisite for the eventual Gebäudezonen split.
  • Storage: in 2020 the disk space occupied by data and backup grew from 2.7 PiB to 3.2 PiB, continuing the obvious trend of ever-growing data. We have now also started the process of phasing out the oldest disk backends in order to replace them with fewer, bigger disks.
  • Software licenses: in the past 12 months, both Adobe and Microsoft decided to switch to a new license system in which each installation requires a license tied to a personal user account. In future, we can't create or extend your Adobe or Microsoft licenses for you, no matter how often you ask us to. You have to do it yourself, according to our instructions for Adobe and Microsoft (you might also want to think about switching to less oppressive software alternatives).
  • Outages: apart from two pre-announced storage migration windows (one of which took a bit longer than expected), 4 h of mail server hardware issues and some short-term network interruptions, our systems have been very stable in 2020. We are aware of the fact that wifi is quite often an issue, and we're trying to convince Informatikdienste to take it seriously.
  • OS upgrades: The Windows team was active migrating the Windows 2016 servers to 2019 while on the Linux side the first workstations were upgraded to Ubuntu 20.04 and most servers are now running Debian buster.
  • Software upgrades: mostly incremental upgrades in our Windows and Linux software list this year.
  • UCC: in February, the old non-VoIP phones in HPT, HPF and HPK were replaced by shiny new ones, just a few weeks before we were all sent home...
  • ISG staff changes: Patrick Schmid left us at the end of 2019 and was replaced by Maciej Bonin in February. Christian Schneider was replaced by Stephan Müller in September. And finally, Sukash Sugumaran superseded Janosch Bühler as our apprentice.

I would like to take this opportunity to thank my whole team for their hard and dedicated work all year long.

Happy Holidays and see you in 2021!

Hardware maintenance of storage front-end server igp-data.

 Update 00:30:

All shares coming back online now. My sincerest apologies for the delay. And good night to all.

Update 22:40: We have run into hardware problems with one of the backends. The shares remain offline as we continue to diagnose it. A further update will be posted as soon as possible.

On Wednesday, 02 XII 2020, between 18:00 and 20:00, access to
igp-data shares will be interrupted for scheduled maintenance.
The shares need to be taken entirely offline for a network upgrade.
At the same time we will be adding more space to the underlying SAN.
This affects all shares on the igp-data storage gateway (ggl, pf and gsg).

Thank you for your patience, and kindest regards.

Update on Matrix (chat)

Since our launch of Matrix at the D-PHYS in March a few things have changed.

Important changes

  • The subdomain of the D-PHYS hosted Riot Element Web App changed and is now at element.phys.ethz.ch. If you still use the old subdomain riot.phys.ethz.ch, please move over to the new domain and verify your new session. We will start redirecting the domain in 2021. If you have no other way of getting access to your encryption keys, you could lose access to your encrypted messages
  • New documentation is available at readme.phys.ethz.ch/chat

Other changes

  • Federation with other Matrix instances at ETH and the rest of the Matrix network, giving us access to bridged networks like IRC, Slack, etc. as well
  • We now auto-invite new users to two more ISG moderated rooms (News/Status). Opt-out by leaving if you do not wish to receive updates from us (discouraged!). All existing members of the D-PHYS Lobby have been joined last night
  • End to end encryption support (enabled by default). Element Web does not provide a search function in encrypted rooms as that would be a security issue, use the Desktop App instead
  • Secure key backup method has been set to show one option only ('Security Passphrase') to new users, which will still provide the secondary recovery method using a 'Security Key'
  • We opened up public rooms and room aliases for everyone. Publish your room in the public room directory to be discoverable by others on the Matrix network
  • We enabled quota of 2 GiB (initial) quota per user for media (uploads) to protect from accidental or abusive over-usage
  • Bots for gitlab, reminders, RSS feeds and more
  • Bridges for webhooks and Slack
  • Integrations (widgets)
  • Import of the Slack workspace history of 3 groups at the D-PHYS
  • An Element Web instance with experimental (Labs) features like tags
  • A tag manager to organize many rooms in large accounts
  • Jitsi video conferencing and screen sharing solution (still experimental but used by ISG on a daily basis)
  • URL redirecting service to invite people via websites or email


  • We will focus on improving performance and less on features
  • Stabilizing 1:1 calls and video (TURN server) and Jitsi (group calls)

For help and discussion about Matrix and Element join #matrix:phys.ethz.ch

new ISG staff member

It is my pleasure to welcome Stephan Müller into our group. He joins us to replace Christian Schneider in the Linux team.

Welcome Stephan!

Hardware failure on mail server

Update 09:45: the mail server could be recovered. It'll take some time for all queued incoming mail to be delivered.

Early this morning, a RAID controller in the D-PHYS mail server died. We're trying to get replacement parts later today. In the meantime, there's no access to email.
We apologize for the inconvenience.

Hardware maintenance of storage front-end servers.

Update 23:50: we ran into severe problems and the migration took longer than expected. Everything is back online now. Sorry we're late.

Planned maintenance will be taking place on all shared-storage front-end servers on Thursday, August 6th, starting at 17:00. The service will be down for approximately 2-3 hours. This post will be updated as soon as work is completed, were we to finish earlier than expected. We will be upgrading the network switch and replacing hardware in several machines.

All group shares will be affected, i.e. group-data, IPA, IGP and Galaxy. Only the home and backup servers will be accessible during this time.

For emergency cases, there will be read-only access to last night’s backup as described here.

New backup service (beta)

TL;DR we have a new backup solution mainly for laptops that frequently leave campus and are looking for testers.

Among the many things we had to learn during the Corona-induced home office stint, one particularly interesting one was: which of our services do work from outside the ETH network, and which do not. I'd like to believe we fared rather well in general, but one glaring counter-example is our BackupPC service for lab and laptop computers. It relies on the client machine having a .dhcp.phys.ethz.ch host name, which only works on wired campus connections. We understand this meant missing backups for those of you who had to take machines off-campus.

In order to be prepared for a second wave, we've partnered up with our colleagues of isginf to provide a lab/laptop backup system that works with whatever network connection you have available. It's based on the tried and tested Open Source restic backup solution that runs on Windows, Linux and macOS clients.

At the current stage we're looking for early beta testers. Since lab computers are typically still in the lab and continue to work fine with BackupPC, laptop users are currently most interesting for us. So if you'd like to test out our road-warrior compatible backup solution, please get in touch. Be aware that this service still has some rough edges, but we'll walk you through all necessary steps and the backup runs themselves have been working fine for weeks now.

Group-data server hardware maintenance.

Update 18:15 group-data is back!

Planned maintenance will be taking place on our group-data.phys.ethz.ch server on Friday, June 19, starting at 17:00. The service will be down for approximately 2 hours. We will be replacing the network interface card to improve service stability.

All group shares will be affected except IPA, IGP and Galaxy.

For emergency cases, there will be read-only access to last night’s backup as described here.

New D-PHYS collaboration/chat service (Slack replacement)

Now that we're all forced to work from home, one of the biggest challenges is effective communication. Some of you might have heard about Slack, a popular proprietary collaboration/chat solution for group communication. At ISG, we have worked on a D-PHYS-hosted alternative for quite some time, and have now redoubled all efforts to get it out to you as soon as possible. Today we're launching our Matrix homeserver and Element chat client to all D-PHYS employees and students. In contrast to Slack, which is a proprietary communication silo that holds your chat history hostage, Matrix is an open source project that publishes the Matrix open standard for secure, decentralized, real-time communication (chat, VoIP, AR/VR, IoT). All parts of your communication (chats, uploaded documents etc) will be hosted at D-PHYS allowing data self-sovereignty (making it compatible with ETH regulations).

If you're interested to use this service to communicate and collaborate in your group or with us, please see this readme page.

Please note that we're starting with a basic set of functionality (group and direct chats, image upload). Internally we've already tested features like video calling and custom bots which will be rolled out incrementally in the coming weeks.

Informatikdienste will soon announce the availability of Microsoft Teams, which tries to achieve the same goal but which we neither recommend nor support in D-PHYS.