Archive for the ‘Downtime’ Category

groupdata downtime

Monday, December 1st, 2025

Upcoming downtime: XFSv5 on groupdata network shares

There will be a larger maitainance downtime for all D-PHYS grouphares. Other shares like Web, IPA, ... are not affected. The downtime is scheduled for:

  • Saturday, January 3rd 2026 20:00 -- Sunday, January 4th 2026 10:00

You will not be able to access any shares on groupdata during this time. Afterwards everything will just work as before. If this will disrupt any essential operation, let us know asap! There will be another reminder closer to the downtime.

Technical background: We need to upgrade the XFS filesystem metadata. This cannot be done in place, so we have to backup, reformat and restore all shares. We need to sync roughly 180TB so it will just take some time.

Linux Workstations Debian

Wednesday, July 24th, 2024

Starting in mid-August, we will migrate the OS of all managed Linux workstations from Ubuntu to Debian during summer/fall of 2024.

The exact migration dates for specific workstations will be coordinated individually in close collaboration with the respective user(s). Please note that due to the complexity of the migration it is not possible to delay this indefinitely. The migration is mandatory for all Ubuntu workstations (incl. older 20.04 Focal).

Hardware owners and users are asked to check the required steps before the migration [1] now, which includes possibly needed hardware upgrades.

All users should be aware of the (breaking) changes [2] introduced with the new setup.

Please also refer to our readme page on why [3] we do the switch to Debian.

Beta test

Please get in touch with us if you have a spare workstation to participate in beta testing, starting now. Note that any such workstation will be reinstalled again in August 2024 after completion of beta testing.

Mailing list server upgrade

Friday, May 31st, 2024

Our D-PHYS mailing lists will be offline for maintenance on Thursday 6th June from early morning to approximately 10h00. No emails to @lists.phys.ethz.ch addresses will be accepted during that downtime.

Under the hood, the mailing list software will be migrated from mailman2 to mailman3. After the migration, the web page lists.phys.ethz.ch will have a new look with a modern web interface. You will be able to see and manage all your mailing lists right from the landing page. But note that all other URLs will have a new structure, so any browser bookmarks to individual mailing list configurations or archives will need to be updated.

For users that only receive messages from these mailing lists, nothing should change.

When sending messages to our mailing lists, pay attention, already now, to always use the <somelist>@lists.phys.ethz.ch domain. The obsolete <somelist>@phys.ethz.ch alias exceptions will no longer be accepted after the migration.

Mailing list owners and moderators will be contacted in a separate email with additional information, right after the migration. In particular, the new system will have personal accounts for all users, so that passwords must no longer be shared among list administrators. Also note that any held messages pending for moderation will not be migrated. So please accept or discard moderation requests the day before the migration.

Further documentation is available in our readme.

Matrix (chat) server maintenance

Wednesday, August 30th, 2023

All Matrix services will be offline for maintenance starting on Thursday 31th Aug 2023 in the morning around 06h00. Minimal downtime is 1h, but some bots/bridges may take longer.

The host system will be upgraded from Debian 10 to 12, followed by an upgrade of the database (PostgreSQL 11 to 15) and ~20 application servers.

Upgrade schedule:

phys.ethz.ch homeserver (people accounts)

First priority is the phys.ethz.ch homeserver hosting our accounts and rooms. Estimated (best case) downtime ~1h.

Your Matrix clients (Element) will show connectivity errors during the downtime:

Matrix homeserver offline

mbot.ethz.ch homeserver (bots, bridges)

Second priority is the mbot.ethz.ch homeserver, all bots, bridges present in the #mbot:phys.ethz.ch room and anything else. I expect most to be back after another few hours but Thursday evening at the latest.

The deprecated webhook bridge will be put out of service.

Not affected

The ETH homeservers staffchat.ethz.ch and studentchat.ethz.ch are not affected by this downtime.

Alternative: jitsi.phys.ethz.ch video conferencing with integrated chat.

home server maintenance

Wednesday, July 5th, 2023

Scheduled maintenance will be taking place on our home.phys.ethz.ch file server on Wednesday, July 12, starting at 16:00. The service will be down for approximately 4 hours. We will be replacing the hardware with all-flash storage and upgrade the base system.

Update 18:15: the new home server is open for business. Most SMB + NFS clients will not have survived the 2h downtime and will have to be rebooted. We'll go through the most obvious ones, but if yours won't work, try restarting.

All home directories (Linux, Windows and Mac, SMB and NFS) will be unavailable during this time.

For emergency cases, you'll have read-only access to the backups as described here.

This migration will mark the end of the huge storage migration project of 2023. Thanks for your patience.

group-data server maintenance

Wednesday, May 31st, 2023

Scheduled maintenance will be taking place on our group-data.phys.ethz.ch server on Wednesday, June 7, starting at 16:00. The service will be down for approximately 4 hours. We will be replacing some hardware and upgrade the base system.

All group shares will be affected except IPA, IGP and Galaxy.

For emergency cases, you'll have read-only access to the backups as described here.

Web server upgrade

Tuesday, February 8th, 2022

This Thursday 2022-02-10 starting at 07:00 we will upgrade the server hosting most of our websites.

Affected websites

The following websites are unavailable during the downtime:

Important changes for website owners

All website owners: If you are a website owner/admin, please join our new Matrix room #web:phys.ethz.ch, to get support and news. After the upgrade, please check your websites for problems.

Python WSGI app owners: All WSGI apps have been switched to use a virtual environment to pin the currently used Python package versions. We encourage you to review and upgrade your dependency versions (via requirements.txt) after the server upgrade. Please read our new WSGI documentation for details.

Versions

  • OS: Debian 10 -> 11
  • Python: 3.7 -> 3.9
  • PHP: 7.3 -> 7.4

Partial Network Downtime on Mon 6th Dec after 19h00

Monday, November 29th, 2021

The central Informatikdienste will have a scheduled downtime of all networking (cable and wireless) in the buildings HPK, HEZ, HPM, HPL and HPW on Monday 6th Dec 2021 in the evening between 19h00 and 23h00.

This is the second of three downtimes for the ongoing project to split the current networks into smaller chunks. This major undertaking will also induce a short downtime for some computers in the dynamic DHCP pool in other buildings (as some of our IP ranges are being moved to the listed buildings).

Users don’t need to do anything and their computers should come back online automatically. Otherwise try to reboot or get in touch with us.

In order to prepare for the migration, Informatikdienste will forbid all changes to their DHCP servers between Friday 3th Dec 13:00 and Tuesday morning. As a consequence we will not be able to register new devices or hostnames during this period.

Partial Network Downtime on Mon 8th Nov after 19h00

Monday, November 1st, 2021

The central Informatikdienste will have a scheduled downtime of all networking (cable and wireless) in the buildings HPH, HPP, HPR, HPS, HPV and HPZ on Monday 8th Nov 2021 in the evening between 19h00 and 23h00.

This is the first of three downtimes for the ongoing project to split the current networks into smaller chunks. This major undertaking will also induce a short downtime for some computers in the dynamic DHCP pool in other buildings (as some of our IP ranges are being moved to the listed buildings).

Users don't need to do anything and their computers should come back online automatically. Otherwise try to reboot or get in touch with us.

In order to prepare for the migration, Informatikdienste will forbid all changes to their DHCP servers between Friday 5th Nov 13:00 and Tuesday morning. As a consequence we will not be able to register new devices or hostnames during this period.

Web services downtime

Tuesday, January 19th, 2021

Update 07:00 All web services are back online.

Tomorrow Wednesday 2021-01-20 starting at 06:00 we will upgrade the server hardware hosting most of our web services. We expect them to be back by 08:00 at the latest.

Affected web services

The following services are unavailable during the downtime:

Our Debian, Ubuntu and Raspbian mirror as well as Grafana, InfluxDB and Webmail will not be affected.

We will not be able to send any status updates via our news blog or via our Matrix news and status rooms.