Author Archive

2021 in review

Friday, December 17th, 2021

This post is meant to give you a short overview of what has been accomplished in D-PHYS IT by ISG this year. We’ve been hard at work to further improve and extend our services for you, our customers. Some highlights of 2021:

  • Network migration: as first announced in 2018 and later detailed in July of this year, we had to completely restructure the D-PHYS network this fall. This reorganization was prompted by a segmentation of the router infrastructure at Hönggerberg and will render the network more redundant and resilient. Visible changes include a NAT network, new DHCP/DynDNS technology and the foundation for IPv6 in all network zones.
  • Hypervisor setup: we run a lot of virtual machines at ISG and this spring we remodeled our hypervisor infrastructure to make it more flexible and capable. Hourly snapshots now give us the possibility to roll back if something goes wrong in a VM. It also allowed us to move our InfluxDB server to an SSD backed hypervisor, increasing performance and stability.
  • Office 365 migration: the Microsoft Office suite was upgraded to M365 on all managed Windows workstations this year.
  • Proprietary software woes: two major software companies caused us (and you!) a lot of headaches this year: on July 5, Microsoft broke Windows printing while trying to fix a security problem and it took them until the end of November to really repair it for everyone. Good job. Meanwhile, Adobe managed to break Acrobat logins for months on end and there's no general solution yet.
  • Windows configuration synchronization: the technology used to sync your desktop settings between managed Windows workstations was migrated from 'roaming profiles' to UE-V this year for greater speed and better reliability.
  • New lab PC backup solution: after we've had a good experience with our 2020 laptop backup system based on restic, we set up a similar system for lab PCs in 2021. We're currently migrating the last machines from the old BackupPC server.
  • 2021 Hardware Crisis: you might have noticed that a lot of hardware components are only available at outrageous prices, lead times measured in months or just not at all. The situation is especially bad for graphics cards and storage components.
  • ISG lecture series: reacting to a growing demand for IT-related knowledge in the department, we established the Basics of Computing Environments for Scientists lecture series that we'll repeat each semester.
  • Matrix/Element: in 2021 we continued to extend the feature set of our popular chat & collaboration system. We contributed bug fixes and lots of time in bringing usable maths support into Element (our supported Matrix client) as this was our number one most wanted feature. The second most wanted was better support for managing groups, which was added this year with spaces. Behind the scenes we have been scaling out our homeserver to keep up with the demand and continue to be stable and responsive. This year we counted 702 active users, who sent 927'123 messages in 4'571 rooms that were created on our server. Our users also participated in 396 rooms that were not created on our server where 731'451 messages were sent.
  • Storage: in 2021 the disk space occupied by data and backup grew from 3.2 PiB to 3.7 PiB, continuing the obvious trend of ever-growing data. In spring (just in time before the 2021 Hardware Crisis) we replaced the older disk backends in our SAN with fewer, bigger disks.
  • Outages: apart from some short-term network interruptions, the only noteworthy service interruptions this year were two update-induced storage hiccups on June 10 and December 7.
  • OS upgrades: most managed Linux workstations were upgraded to Ubuntu 20.04 and a first batch of servers are now running Debian bullseye.
  • Software upgrades: mostly incremental upgrades in our Windows and Linux software list this year.

I would like to take this opportunity to thank my whole team for their hard and dedicated work all year long.

Happy Holidays and see you in 2022!

Rethinking groupware

Tuesday, October 12th, 2021

In August of 2010, we introduced our groupware solution for D-PHYS. In the last 11 years the system has served the Department well, but we believe now would be a good time to think about the future of groupware at D-PHYS. Here's an incomplete list of things we noticed over the years:

  • the groupware system seems to get used almost exclusively as a calendar
  • for distributed calendars, the world and people's expectations have changed
  • file and sync formats have come and gone
  • 17 people will have 634 completely orthogonal and incompatible use cases for calendars
  • the product that won our evaluation round in 2010 is not necessarily the best system for 2021

We have sampled the market and test-installed several candidates, but since our humble ISG-internal calendar only covers one very specific use case, we strongly encourage you to give us your feedback so that the next D-PHYS calendar solution will suit you well. In particular, we're interested in learning

  • are you using any other egroupware module(s) aside from the calendar? If you don't tell us about it now, a potential replacement may not have this functionality!
  • what's your current calendar use case? Just a personal calendar? A group calendar? In which configuration?
  • what's your desired calendar use case? This might be the most interesting thing to learn...
  • which sync protocols/devices are you using?
  • what other software/products/services have you been/are you using?

If you would like to help make sure the next evolution in distributed D-PHYS calendars is a success, please join our Matrix room and participate! Thanks a lot.

2020 in review – Corona edition

Thursday, December 17th, 2020

oh boy, what a year.

This post is meant to give you a short overview of what has been accomplished in D-PHYS IT by ISG this year. We’ve been hard at work to further improve and extend our services for you, our customers. Some highlights of 2020:

  • Home office: on March 12, due to rapidly rising Covid-19 numbers, ISG was sent to work from home, along with most of the department. While we had somewhat anticipated this step and were prepared for it, the first two weeks were very busy because we had to assist a lot of people who weren't. In the end I believe we got everyone set up and we have been fully operational from home with only occasional individual visits to the office since then.
  • Matrix/Element/Riot: one of the most pressing issues with everyone working from home was an efficient and versatile tool for team communication. We had started internal tests of our Matrix chat system in late 2019, but then intensified our efforts in February and were able to release the system for general D-PHYS availability in home office week (HOW) 2. During the course of 2020, we continually kept working on the system and added new exciting features.
    We also run a Jitsi instance for privacy-aware video conferencing.
  • New laptop backup: our traditional BackupPC backup system for laptops and lab computers relies on each backup client to be reachable in the D-PHYS network, which obviously didn't work any longer in the home office regime. In HOW 17, we released a new backup system for laptops that works from any internet connection worldwide. Unfortunately, only very few of you have signed up for the service so far. Please make sure you have a backup of your laptop!
  • Ansible deployment: more servers and finally also the managed Linux workstations have been added to our ansible configuration management, allowing for completely automated installation of our systems.
  • Network migration: the extensive Hönggerberg network reorganization we reported two years ago hasn't seen much progress by Informatikdienste, but we have been working on our side to make the first steps. In early 2020 we migrated the dhcp.phys DNS service from our servers to ID's as a prerequisite for the eventual Gebäudezonen split.
  • Storage: in 2020 the disk space occupied by data and backup grew from 2.7 PiB to 3.2 PiB, continuing the obvious trend of ever-growing data. We have now also started the process of phasing out the oldest disk backends in order to replace them with fewer, bigger disks.
  • Software licenses: in the past 12 months, both Adobe and Microsoft decided to switch to a new license system in which each installation requires a license tied to a personal user account. In future, we can't create or extend your Adobe or Microsoft licenses for you, no matter how often you ask us to. You have to do it yourself, according to our instructions for Adobe and Microsoft (you might also want to think about switching to less oppressive software alternatives).
  • Outages: apart from two pre-announced storage migration windows (one of which took a bit longer than expected), 4 h of mail server hardware issues and some short-term network interruptions, our systems have been very stable in 2020. We are aware of the fact that wifi is quite often an issue, and we're trying to convince Informatikdienste to take it seriously.
  • OS upgrades: The Windows team was active migrating the Windows 2016 servers to 2019 while on the Linux side the first workstations were upgraded to Ubuntu 20.04 and most servers are now running Debian buster.
  • Software upgrades: mostly incremental upgrades in our Windows and Linux software list this year.
  • UCC: in February, the old non-VoIP phones in HPT, HPF and HPK were replaced by shiny new ones, just a few weeks before we were all sent home...
  • ISG staff changes: Patrick Schmid left us at the end of 2019 and was replaced by Maciej Bonin in February. Christian Schneider was replaced by Stephan Müller in September. And finally, Sukash Sugumaran superseded Janosch Bühler as our apprentice.

I would like to take this opportunity to thank my whole team for their hard and dedicated work all year long.

Happy Holidays and see you in 2021!

new ISG staff member

Tuesday, September 22nd, 2020

It is my pleasure to welcome Stephan Müller into our group. He joins us to replace Christian Schneider in the Linux team.

Welcome Stephan!

Hardware failure on mail server

Thursday, August 27th, 2020

Update 09:45: the mail server could be recovered. It'll take some time for all queued incoming mail to be delivered.

Early this morning, a RAID controller in the D-PHYS mail server died. We're trying to get replacement parts later today. In the meantime, there's no access to email.
We apologize for the inconvenience.

New backup service (beta)

Friday, July 10th, 2020

TL;DR we have a new backup solution mainly for laptops that frequently leave campus and are looking for testers.

Among the many things we had to learn during the Corona-induced home office stint, one particularly interesting one was: which of our services do work from outside the ETH network, and which do not. I'd like to believe we fared rather well in general, but one glaring counter-example is our BackupPC service for lab and laptop computers. It relies on the client machine having a .dhcp.phys.ethz.ch host name, which only works on wired campus connections. We understand this meant missing backups for those of you who had to take machines off-campus.

In order to be prepared for a second wave, we've partnered up with our colleagues of isginf to provide a lab/laptop backup system that works with whatever network connection you have available. It's based on the tried and tested Open Source restic backup solution that runs on Windows, Linux and macOS clients.

At the current stage we're looking for early beta testers. Since lab computers are typically still in the lab and continue to work fine with BackupPC, laptop users are currently most interesting for us. So if you'd like to test out our road-warrior compatible backup solution, please get in touch. Be aware that this service still has some rough edges, but we'll walk you through all necessary steps and the backup runs themselves have been working fine for weeks now.

Corona and ISG

Thursday, March 12th, 2020

ISG is working from home. We will not be able to pick up the phone. If you have any helpdesk request, please use isg@phys.ethz.ch

As ETH and D-PHYS are working hard on every level to prepare for all conceivable situations related to Covid-19, also here at ISG we are trying to deal with the situation as it evolves. Some information from our side:

  • we're currently dealing with a great number of requests. Please bear with us if reaction time is somewhat longer than usual
  • please always send requests to isg@phys.ethz.ch and not to individual people (fun fact: this also applies in non-Corona times!)
  • we have created a readme page that helps you prepare for home office
  • there is a D-PHYS internal wiki for easy sharing of information
  • please don't take your managed Windows or Linux workstation home. It won't work.
  • if you feel that you need some IT assistance or major changes to your setup, please get in touch with us at an early stage
  • we have taken precautions to be able to maintain D-PHYS IT Services and ISG Helpdesk even if we were being requested to stay home

new ISG staff member

Friday, February 7th, 2020

It is my pleasure to welcome Maciej Bonin into our group. He joins us to replace Patrick Schmid in the Linux team.

Welcome Maciej!

2019 in review

Friday, December 13th, 2019

This post is meant to give you a short overview of what has been accomplished in D-PHYS IT by ISG this year. We’ve been hard at work to further improve and extend our services for you, our customers. Some highlights of 2019:

  • Ansible deployment: while we had already started to deploy servers using ansible as early as 2015, it was in 2019 that we consolidated and migrated almost all server configuration to this system and now have a common base for the D-PHYS server infrastructure.
  • Storage server separation: in the past years a constant growth in both volume and bandwidth of our SAN storage system caused occasional performance issues for some users. To alleviate this, we split our single SAN frontend file server into 4 individual machines (D-PHYS general, IPA, IGP and galaxy) in order to distribute the load.
  • New web server: at the end of 2018 we purchased a new D-PHYS web server to replace the previous 10-year-old system. In 2019 we devised a completely new and upgraded web server setup on this new machine and migrated all D-PHYS hosted web shares to the new system. If you are the owner of one of our web shares, please make sure to read the updated documentation for things that have changed.
  • Network migration: the extensive Hönggerberg network reorganization we reported last year is even more complex than we initially thought, so there's no end-user-tangible progress this year - which doesn't mean there hasn't been a lot of behind-the-scenes work.
  • Storage: in 2019 the disk space occupied by data and backup grew from 2.1 PiB to 2.7 PiB, continuing the obvious trend of ever-growing data. The end of 2019 also saw a substantial expansion of the available disk capacity.
  • Clusters: we inherited two HPC clusters from CSCS that we're now running locally.
  • InfluxDB / Grafana: we included this popular time-series database / visualization combination into our service catalog.
  • Outages: apart from a pre-announced migration window and some short-term network interruptions, our systems have been very stable in 2019.
  • OS upgrades: The Windows team was active in getting rid of the remaining Windows 7 machines and upgrading Windows 10 to the 1809 build, while on the Linux side workstations were upgraded to Ubuntu 18.04 and a first batch of servers to Debian buster.
  • Software upgrades: the FileMaker server has been upgraded.
  • UCC: the UCC project of Informatikdienste was stopped due to nonfulfillment of the technical requirements and all deployed services and devices have been rolled back. The whole project will be reevaluated from scratch.
  • IT security: we participate in and support the ETH-wide IT security initiative.

I would like to take this opportunity to thank my whole team for their hard and dedicated work all year long.

Happy Holidays and see you in 2020!

Phishing and malware emails

Sunday, September 29th, 2019

In light of the recent surge of malware waves, we have decided to quarantine all incoming emails containing Microsoft Office documents with macros enabled - actually, we have been doing so for a week already. Unfortunately way too many of you still open those documents and risk (or succeed in) infecting your computer. Emails containing such dangerous documents will be quarantined and are never shown to the user. Emails with static office documents (no macros) will be delivered unaffected. We're aware of the fact that this policy might create the occasional false positive, but the benefits for all of D-PHYS far outweigh the downside and real use cases for macro documents via email are in fact very rare. In the 8 days of operation so far, we've detected ~850 infected office documents and only 1 false positive. Quarantined emails will be deleted after 30 days, so you have ample time to contact us in case a valid document gets flagged by accident.

Please get in contact if you have any questions.