ISG Helpdesk this week:
This Thursday 2022-02-10 starting at 07:00 we will upgrade the server hosting most of our websites.
The following websites are unavailable during the downtime:
- Most ISG websites (news, readme, wiki)
- All customer webshares (dedicated websites and webapps)
- D-PHYS shop and Experimente
Important changes for website owners
All website owners: If you are a website owner/admin, please join our new Matrix room #web:phys.ethz.ch, to get support and news. After the upgrade, please check your websites for problems.
Python WSGI app owners: All WSGI apps have been switched to use a virtual environment to pin the currently used Python package versions. We encourage you to review and upgrade your dependency versions (via requirements.txt) after the server upgrade. Please read our new WSGI documentation for details.
- OS: Debian 10 -> 11
- Python: 3.7 -> 3.9
- PHP: 7.3 -> 7.4
This post is meant to give you a short overview of what has been accomplished in D-PHYS IT by ISG this year. We’ve been hard at work to further improve and extend our services for you, our customers. Some highlights of 2021:
- Network migration: as first announced in 2018 and later detailed in July of this year, we had to completely restructure the D-PHYS network this fall. This reorganization was prompted by a segmentation of the router infrastructure at Hönggerberg and will render the network more redundant and resilient. Visible changes include a NAT network, new DHCP/DynDNS technology and the foundation for IPv6 in all network zones.
- Hypervisor setup: we run a lot of virtual machines at ISG and this spring we remodeled our hypervisor infrastructure to make it more flexible and capable. Hourly snapshots now give us the possibility to roll back if something goes wrong in a VM. It also allowed us to move our InfluxDB server to an SSD backed hypervisor, increasing performance and stability.
- Office 365 migration: the Microsoft Office suite was upgraded to M365 on all managed Windows workstations this year.
- Proprietary software woes: two major software companies caused us (and you!) a lot of headaches this year: on July 5, Microsoft broke Windows printing while trying to fix a security problem and it took them until the end of November to really repair it for everyone. Good job. Meanwhile, Adobe managed to break Acrobat logins for months on end and there's no general solution yet.
- Windows configuration synchronization: the technology used to sync your desktop settings between managed Windows workstations was migrated from 'roaming profiles' to UE-V this year for greater speed and better reliability.
- New lab PC backup solution: after we've had a good experience with our 2020 laptop backup system based on restic, we set up a similar system for lab PCs in 2021. We're currently migrating the last machines from the old BackupPC server.
- 2021 Hardware Crisis: you might have noticed that a lot of hardware components are only available at outrageous prices, lead times measured in months or just not at all. The situation is especially bad for graphics cards and storage components.
- ISG lecture series: reacting to a growing demand for IT-related knowledge in the department, we established the Basics of Computing Environments for Scientists lecture series that we'll repeat each semester.
- Matrix/Element: in 2021 we continued to extend the feature set of our popular chat & collaboration system. We contributed bug fixes and lots of time in bringing usable maths support into Element (our supported Matrix client) as this was our number one most wanted feature. The second most wanted was better support for managing groups, which was added this year with spaces. Behind the scenes we have been scaling out our homeserver to keep up with the demand and continue to be stable and responsive. This year we counted 702 active users, who sent 927'123 messages in 4'571 rooms that were created on our server. Our users also participated in 396 rooms that were not created on our server where 731'451 messages were sent.
- Storage: in 2021 the disk space occupied by data and backup grew from 3.2 PiB to 3.7 PiB, continuing the obvious trend of ever-growing data. In spring (just in time before the 2021 Hardware Crisis) we replaced the older disk backends in our SAN with fewer, bigger disks.
- Outages: apart from some short-term network interruptions, the only noteworthy service interruptions this year were two update-induced storage hiccups on June 10 and December 7.
- OS upgrades: most managed Linux workstations were upgraded to Ubuntu 20.04 and a first batch of servers are now running Debian bullseye.
- Software upgrades: mostly incremental upgrades in our Windows and Linux software list this year.
I would like to take this opportunity to thank my whole team for their hard and dedicated work all year long.
Happy Holidays and see you in 2022!
The central Informatikdienste will have a scheduled downtime of all networking (cable and wireless) in the buildings HPK, HEZ, HPM, HPL and HPW on Monday 6th Dec 2021 in the evening between 19h00 and 23h00.
This is the second of three downtimes for the ongoing project to split the current networks into smaller chunks. This major undertaking will also induce a short downtime for some computers in the dynamic DHCP pool in other buildings (as some of our IP ranges are being moved to the listed buildings).
Users don’t need to do anything and their computers should come back online automatically. Otherwise try to reboot or get in touch with us.
In order to prepare for the migration, Informatikdienste will forbid all changes to their DHCP servers between Friday 3th Dec 13:00 and Tuesday morning. As a consequence we will not be able to register new devices or hostnames during this period.
Between yesterday Tue 23rd Nov 07:27 and tonight Wed 24th Nov 01:48 several hundred emails have erroneously been flagged as spam by our mail server. Please check your SpamBox folder for potential false positives during that period.
While we are constantly updating our filtering rules to catch the nasty spam, we are always doing our best to avoid flagging real emails as spam. We apologize for this and are still investigating the details of the root cause. Because the bulk of the messages were being correctly delivered, it took us several hours before we noticed the problem and could fix it.
The central Informatikdienste will have a scheduled downtime of all networking (cable and wireless) in the buildings HPH, HPP, HPR, HPS, HPV and HPZ on Monday 8th Nov 2021 in the evening between 19h00 and 23h00.
This is the first of three downtimes for the ongoing project to split the current networks into smaller chunks. This major undertaking will also induce a short downtime for some computers in the dynamic DHCP pool in other buildings (as some of our IP ranges are being moved to the listed buildings).
Users don't need to do anything and their computers should come back online automatically. Otherwise try to reboot or get in touch with us.
In order to prepare for the migration, Informatikdienste will forbid all changes to their DHCP servers between Friday 5th Nov 13:00 and Tuesday morning. As a consequence we will not be able to register new devices or hostnames during this period.
In August of 2010, we introduced our groupware solution for D-PHYS. In the last 11 years the system has served the Department well, but we believe now would be a good time to think about the future of groupware at D-PHYS. Here's an incomplete list of things we noticed over the years:
- the groupware system seems to get used almost exclusively as a calendar
- for distributed calendars, the world and people's expectations have changed
- file and sync formats have come and gone
- 17 people will have 634 completely orthogonal and incompatible use cases for calendars
- the product that won our evaluation round in 2010 is not necessarily the best system for 2021
We have sampled the market and test-installed several candidates, but since our humble ISG-internal calendar only covers one very specific use case, we strongly encourage you to give us your feedback so that the next D-PHYS calendar solution will suit you well. In particular, we're interested in learning
- are you using any other egroupware module(s) aside from the calendar? If you don't tell us about it now, a potential replacement may not have this functionality!
- what's your current calendar use case? Just a personal calendar? A group calendar? In which configuration?
- what's your desired calendar use case? This might be the most interesting thing to learn...
- which sync protocols/devices are you using?
- what other software/products/services have you been/are you using?
If you would like to help make sure the next evolution in distributed D-PHYS calendars is a success, please join our Matrix room and participate! Thanks a lot.
Several network migrations will take place over the next months that will have an impact on the design and inner workings of the ethernet network at D-PHYS. Even though all hosts will be affected at a technical level, we believe that most changes will not require any involvement from your side. By the end of the year this should further increase the fault-tolerance of the cabled network infrastructure and enhance the security of the bulk of the computers at D-PHYS.
The central Informatikdienste are splitting several networks into smaller chunks to increase the overall stability and fault-tolerance. Unfortunately the details are flagged as confidential, prohibiting us from exposing the precise structure of this segmentation. The main repercussion is that our D-PHYS networks will no longer be able to span across all current buildings at once. So depending on the building, we will have to introduce new subnets and assign new IP addresses to the computers inside.
Motivated by the above-mentioned segmentation as well as security considerations, we are planning to migrate a large number of hosts to a NAT network. This means that the computer will only get an ETH-internal IP address and will no longer be directly reachable from outside of ETH. From inside ETH or VPN, all communication with that computer remains unaffected. However, while the host can still communicate with all of the internet, it will no longer be exposed to direct attacks from the outside. We believe that this is a very sensible default for most computers and laptops. Of course it will still be possible to assign a public IP to selected hosts in order to provide a specific service to the outside. The new NAT network also provides DynDNS with sentname.dhcp-int.phys.ethz.ch hostnames and full IPv6 connectivity. So if you rely on DNS entries for dynamic IP addresses, make sure to use the domain dhcp.phys.ethz.ch for public subnets and dhcp-int.phys.ethz.ch for internal subnets.
Right now, some of our networks are serviced by our own D-PHYS DHCP servers, while others use the DHCP servers of central IT services. We are now consolidating all networks and migrating the remaining ones step-by-step to the DHCP servers of Informatikdienste. This change is mostly technical and should remain unnoticed by most users.
For further details and up-to-date information please refer to our readme page.
Last year, the license agreement between ETH and Microsoft has changed. There are some new terms and conditions which are important if you use a Microsoft product from the ETH IT Shop.
If your hardware came with Windows preinstalled from the hardware vendor, you have your own license for MS Office, or you are using non-Microsoft products like Libre Office, then you are not affected by this new license agreement. No further action is needed in this case.
However, if you use any of the following software products from the ETH IT Shop you have to take action:
- Windows 10 Education
- Windows 10 LTSC
- MS Office XXXX (for example MS Office 2016)
- Microsoft 365
- Microsoft Teams
- ETH Exchange Mail (mail.ethz.ch)
either on a managed Windows workstation from ISG or a self-managed workstation or notebook.
In this case you need to request access for a free Microsoft Cloud Subscription in the IT Shop. See here for instructions. If you already use MS 365 or MS Teams, maybe you already have the Cloud Subscription and no additional step is necessary. This can also be verified in the IT Shop. Remember that you need renew the subscription every year.
The new Microsoft license agreement allows only 1 MS Office XXXX (for example MS Office 2016 or Office 2019) installation per user. If you need installations on multiple computers for example in labs, use MS 365 instead.
Note that you will have to take action if you're affected. You cannot sit this one out. The Informatikdienste license team will be coerced by Microsoft to enforce license compliance, and they will contact you if you don't get your Cloud Subscription.
Thank you for your attention, and kind regards.
Update 07:00 All web services are back online.
Tomorrow Wednesday 2021-01-20 starting at 06:00 we will upgrade the server hardware hosting most of our web services. We expect them to be back by 08:00 at the latest.
Affected web services
The following services are unavailable during the downtime:
- Most ISG websites (news, readme, wiki)
- All customer webshares (dedicated websites and webapps)
- All groupshare and personal homepages (public_html)
- Web databases (on sqlweb.phys.ethz.ch)
- D-PHYS shop and Experimente
- D-PHYS GitLab
- Chat (Matrix, Element, Jitsi)