Posts Tagged ‘milestones’

2023 in review

Thursday, December 21st, 2023

This post is meant to give you a short overview of what has been accomplished in D-PHYS IT by ISG this year. We’ve been hard at work to further improve and extend our services for you, our customers. Unfortunately, two of our colleagues had to take extended sick leaves this year, so we didn't have as much capacity for innovation and had to focus on system maintenance.

Some highlights of 2023:

  • Storage: in the first half of 2023, we performed a major overhaul on our storage and backup infrastructure in both hardware and software. The disk space occupied by data and backup grew from 4.8 PiB to 5.1 PiB.
  • Infrastructure work: our Ansible deployment setup was further extended and refined. All Windows servers have been added as well as the macOS clients.
    All eGroupware users were migrated to the new SOGo calendar. We also enabled DKIM signing on our D-PHYS mailserver. The Windows HyperV cluster has been upgraded for better performance.
  • Matrix/Element: We upgraded both the OS and the Matrix server software. This year we counted 748 active users on 2579 devices (1019 Windows, 517 Linux, 424 Mac, 619 Mobile), who sent 1'086'537 messages in 5'510 rooms created on our server. Our users also participated in 763 other rooms with 1'157'466 messages.
  • ISG lecture series: our Basics of Computing Environments for Scientists lecture series was held twice in 2023 with shockingly low attendance.
  • Outages: apart from some short-term network interruptions, our systems were pretty stable in 2023.
  • OS upgrades: work is well underway to migrate the managed Linux workstations from Ubuntu to Debian and the majority of servers are now running Debian bookworm. The Windows team started the Windows 11 rollout. Most of the managed Macs were upgraded to macOS Ventura.
  • Software upgrades: mostly incremental upgrades in our Windows and Linux software list this year.
  • IT security: with the world being what it is, IT security plays an ever increasing role in our work and permeates all our plans and projects. We also take part in the current rewrite of ETH's IT security regulations. On the Windows virus scanner front, Sophos was EOL'd and replaced by Windows Defender.

I would like to take this opportunity to thank my whole team for their hard and dedicated work all year long.

Happy Holidays and see you in 2024!

2022 in review

Tuesday, December 13th, 2022

This post is meant to give you a short overview of what has been accomplished in D-PHYS IT by ISG this year. We’ve been hard at work to further improve and extend our services for you, our customers. Since it took us almost exactly one year to fill our vacant Linux System Engineer position (once again: welcome Sascha!), we didn't have as much capacity for innovation as in previous years and had to focus more on system maintenance.

Some highlights of 2022:

  • Mail server: the D-PHYS mail server got an OS upgrade in spring and was migrated to our general hypervisor setup, which adds redundancy and facilitates maintenance. In addition, work is in progress to support DKIM and further tighten our SPF and DMARC settings.
  • Web server setup: the main D-PHYS web server got an OS upgrade in spring, test and staging environments and optional ssh access for power users.
  • Infrastructure work: our Ansible deployment setup was further extended and refined and the first Windows servers have been added.
    Work has started to replace the Sophos virus scanner on managed Windows workstations.
    We migrated our floating licenses from three servers to a single high availability server.
    Within the next year, we'll migrate all eGroupware users to the new SOGo calendar.
  • Storage: in 2022 the disk space occupied by data and backup grew from 3.7 PiB to 4.8 PiB, marking a significant annual growth in storage volume. A major storage migration is due in early 2023.
  • Matrix/Element: This year we counted 737 active users, who sent 1'019'205 messages in 5'259 rooms that were created on our server. Our users also participated in 423 other rooms where 1'190'446 messages were sent. Two additional research groups migrated from Slack to Matrix.
  • ISG lecture series: our Basics of Computing Environments for Scientists lecture series was held twice in 2022 with surprisingly low attendance.
  • Outages: apart from some short-term network interruptions, our systems were pretty stable in 2022, with the notable exception of a localized "3 dead disks in a RAID6" disaster in September.
  • OS upgrades: most managed Linux workstations were upgraded to Ubuntu 22.04 and the majority of servers are now running Debian bullseye. The Windows team prepared a new LTSC release and a Windows 11 setup. The managed Macs were all upgraded to macOS Monterey.
  • Software upgrades: mostly incremental upgrades in our Windows and Linux software list this year.
  • IT security: with the world being what it is, IT security plays an ever increasing role in our work and permeates all our plans and projects. We also take part in the current rewrite of ETH's IT security regulations.

I would like to take this opportunity to thank my whole team for their hard and dedicated work all year long.

Happy Holidays and see you in 2023!

2021 in review

Friday, December 17th, 2021

This post is meant to give you a short overview of what has been accomplished in D-PHYS IT by ISG this year. We’ve been hard at work to further improve and extend our services for you, our customers. Some highlights of 2021:

  • Network migration: as first announced in 2018 and later detailed in July of this year, we had to completely restructure the D-PHYS network this fall. This reorganization was prompted by a segmentation of the router infrastructure at Hönggerberg and will render the network more redundant and resilient. Visible changes include a NAT network, new DHCP/DynDNS technology and the foundation for IPv6 in all network zones.
  • Hypervisor setup: we run a lot of virtual machines at ISG and this spring we remodeled our hypervisor infrastructure to make it more flexible and capable. Hourly snapshots now give us the possibility to roll back if something goes wrong in a VM. It also allowed us to move our InfluxDB server to an SSD backed hypervisor, increasing performance and stability.
  • Office 365 migration: the Microsoft Office suite was upgraded to M365 on all managed Windows workstations this year.
  • Proprietary software woes: two major software companies caused us (and you!) a lot of headaches this year: on July 5, Microsoft broke Windows printing while trying to fix a security problem and it took them until the end of November to really repair it for everyone. Good job. Meanwhile, Adobe managed to break Acrobat logins for months on end and there's no general solution yet.
  • Windows configuration synchronization: the technology used to sync your desktop settings between managed Windows workstations was migrated from 'roaming profiles' to UE-V this year for greater speed and better reliability.
  • New lab PC backup solution: after we've had a good experience with our 2020 laptop backup system based on restic, we set up a similar system for lab PCs in 2021. We're currently migrating the last machines from the old BackupPC server.
  • 2021 Hardware Crisis: you might have noticed that a lot of hardware components are only available at outrageous prices, lead times measured in months or just not at all. The situation is especially bad for graphics cards and storage components.
  • ISG lecture series: reacting to a growing demand for IT-related knowledge in the department, we established the Basics of Computing Environments for Scientists lecture series that we'll repeat each semester.
  • Matrix/Element: in 2021 we continued to extend the feature set of our popular chat & collaboration system. We contributed bug fixes and lots of time in bringing usable maths support into Element (our supported Matrix client) as this was our number one most wanted feature. The second most wanted was better support for managing groups, which was added this year with spaces. Behind the scenes we have been scaling out our homeserver to keep up with the demand and continue to be stable and responsive. This year we counted 702 active users, who sent 927'123 messages in 4'571 rooms that were created on our server. Our users also participated in 396 rooms that were not created on our server where 731'451 messages were sent.
  • Storage: in 2021 the disk space occupied by data and backup grew from 3.2 PiB to 3.7 PiB, continuing the obvious trend of ever-growing data. In spring (just in time before the 2021 Hardware Crisis) we replaced the older disk backends in our SAN with fewer, bigger disks.
  • Outages: apart from some short-term network interruptions, the only noteworthy service interruptions this year were two update-induced storage hiccups on June 10 and December 7.
  • OS upgrades: most managed Linux workstations were upgraded to Ubuntu 20.04 and a first batch of servers are now running Debian bullseye.
  • Software upgrades: mostly incremental upgrades in our Windows and Linux software list this year.

I would like to take this opportunity to thank my whole team for their hard and dedicated work all year long.

Happy Holidays and see you in 2022!

2020 in review – Corona edition

Thursday, December 17th, 2020

oh boy, what a year.

This post is meant to give you a short overview of what has been accomplished in D-PHYS IT by ISG this year. We’ve been hard at work to further improve and extend our services for you, our customers. Some highlights of 2020:

  • Home office: on March 12, due to rapidly rising Covid-19 numbers, ISG was sent to work from home, along with most of the department. While we had somewhat anticipated this step and were prepared for it, the first two weeks were very busy because we had to assist a lot of people who weren't. In the end I believe we got everyone set up and we have been fully operational from home with only occasional individual visits to the office since then.
  • Matrix/Element/Riot: one of the most pressing issues with everyone working from home was an efficient and versatile tool for team communication. We had started internal tests of our Matrix chat system in late 2019, but then intensified our efforts in February and were able to release the system for general D-PHYS availability in home office week (HOW) 2. During the course of 2020, we continually kept working on the system and added new exciting features.
    We also run a Jitsi instance for privacy-aware video conferencing.
  • New laptop backup: our traditional BackupPC backup system for laptops and lab computers relies on each backup client to be reachable in the D-PHYS network, which obviously didn't work any longer in the home office regime. In HOW 17, we released a new backup system for laptops that works from any internet connection worldwide. Unfortunately, only very few of you have signed up for the service so far. Please make sure you have a backup of your laptop!
  • Ansible deployment: more servers and finally also the managed Linux workstations have been added to our ansible configuration management, allowing for completely automated installation of our systems.
  • Network migration: the extensive Hönggerberg network reorganization we reported two years ago hasn't seen much progress by Informatikdienste, but we have been working on our side to make the first steps. In early 2020 we migrated the dhcp.phys DNS service from our servers to ID's as a prerequisite for the eventual Gebäudezonen split.
  • Storage: in 2020 the disk space occupied by data and backup grew from 2.7 PiB to 3.2 PiB, continuing the obvious trend of ever-growing data. We have now also started the process of phasing out the oldest disk backends in order to replace them with fewer, bigger disks.
  • Software licenses: in the past 12 months, both Adobe and Microsoft decided to switch to a new license system in which each installation requires a license tied to a personal user account. In future, we can't create or extend your Adobe or Microsoft licenses for you, no matter how often you ask us to. You have to do it yourself, according to our instructions for Adobe and Microsoft (you might also want to think about switching to less oppressive software alternatives).
  • Outages: apart from two pre-announced storage migration windows (one of which took a bit longer than expected), 4 h of mail server hardware issues and some short-term network interruptions, our systems have been very stable in 2020. We are aware of the fact that wifi is quite often an issue, and we're trying to convince Informatikdienste to take it seriously.
  • OS upgrades: The Windows team was active migrating the Windows 2016 servers to 2019 while on the Linux side the first workstations were upgraded to Ubuntu 20.04 and most servers are now running Debian buster.
  • Software upgrades: mostly incremental upgrades in our Windows and Linux software list this year.
  • UCC: in February, the old non-VoIP phones in HPT, HPF and HPK were replaced by shiny new ones, just a few weeks before we were all sent home...
  • ISG staff changes: Patrick Schmid left us at the end of 2019 and was replaced by Maciej Bonin in February. Christian Schneider was replaced by Stephan Müller in September. And finally, Sukash Sugumaran superseded Janosch Bühler as our apprentice.

I would like to take this opportunity to thank my whole team for their hard and dedicated work all year long.

Happy Holidays and see you in 2021!

2019 in review

Friday, December 13th, 2019

This post is meant to give you a short overview of what has been accomplished in D-PHYS IT by ISG this year. We’ve been hard at work to further improve and extend our services for you, our customers. Some highlights of 2019:

  • Ansible deployment: while we had already started to deploy servers using ansible as early as 2015, it was in 2019 that we consolidated and migrated almost all server configuration to this system and now have a common base for the D-PHYS server infrastructure.
  • Storage server separation: in the past years a constant growth in both volume and bandwidth of our SAN storage system caused occasional performance issues for some users. To alleviate this, we split our single SAN frontend file server into 4 individual machines (D-PHYS general, IPA, IGP and galaxy) in order to distribute the load.
  • New web server: at the end of 2018 we purchased a new D-PHYS web server to replace the previous 10-year-old system. In 2019 we devised a completely new and upgraded web server setup on this new machine and migrated all D-PHYS hosted web shares to the new system. If you are the owner of one of our web shares, please make sure to read the updated documentation for things that have changed.
  • Network migration: the extensive Hönggerberg network reorganization we reported last year is even more complex than we initially thought, so there's no end-user-tangible progress this year - which doesn't mean there hasn't been a lot of behind-the-scenes work.
  • Storage: in 2019 the disk space occupied by data and backup grew from 2.1 PiB to 2.7 PiB, continuing the obvious trend of ever-growing data. The end of 2019 also saw a substantial expansion of the available disk capacity.
  • Clusters: we inherited two HPC clusters from CSCS that we're now running locally.
  • InfluxDB / Grafana: we included this popular time-series database / visualization combination into our service catalog.
  • Outages: apart from a pre-announced migration window and some short-term network interruptions, our systems have been very stable in 2019.
  • OS upgrades: The Windows team was active in getting rid of the remaining Windows 7 machines and upgrading Windows 10 to the 1809 build, while on the Linux side workstations were upgraded to Ubuntu 18.04 and a first batch of servers to Debian buster.
  • Software upgrades: the FileMaker server has been upgraded.
  • UCC: the UCC project of Informatikdienste was stopped due to nonfulfillment of the technical requirements and all deployed services and devices have been rolled back. The whole project will be reevaluated from scratch.
  • IT security: we participate in and support the ETH-wide IT security initiative.

I would like to take this opportunity to thank my whole team for their hard and dedicated work all year long.

Happy Holidays and see you in 2020!

2018 in review

Tuesday, December 18th, 2018

This post is meant to give you a short overview of what has been accomplished in D-PHYS IT by ISG this year. We’ve been hard at work to further improve and extend our services for you, our customers. Some highlights of 2018:

  • New mail server: between January and March, the virtual machines that make up the D-PHYS mail server were migrated to new hardware. We're now running on a state-of-the-art server with SSD storage that will serve the department's needs for many years to come.
  • New LDAP servers: in late 2017 we started a big migration to a cluster of new LDAP servers. This move was completed in the spring of 2018 and the old server turned off.
  • group membership edit: one of the benefits of the LDAP migration is that group memberships can now be managed directly by dedicated owners of a group. If you feel responsible for one such group and would like to be able to perform member management yourself without having to go through us each time, please get in touch.
  • New web server: we purchased new D-PHYS web server hardware to replace the old 10-year-old system. Since we're also planning to change the setup of your web hosting, migrating the existing web sites to the new hardware will be a long process that will extend well into 2019.
  • Network migration: while we were in an advanced planning stage of a segmentation of the D-PHYS network and had already started to implement the first changes, Informatikdienste announced that the underlying network layout of the whole Hönggerberg campus would be redesigned in 2018/19 which deeply influences and impacts our work as well. We're now on hold until we know details of ID's technical implementation.
  • Storage: in 2018 the disk space occupied by data and backup grew from 1.6 PiB to 2.1 PiB, which means that growth in storage has picked up steam again after two slow years.
  • Outages: apart from the above-mentioned pre-announced migration windows and some short-term network interruptions, our systems have been very stable in 2018.
  • OS upgrades: the Windows 10 rollout has been largely completed and most Linux workstations have been upgraded to Ubuntu 18.04.
  • WiFi change: we accompanied and supported ETH's wifi change project in November.
  • UCC: the UCC rollout which will replace the existing ETH telephony system with an all-IP based solution has been put on hold by Informatikdienste since the service quality was severely lacking. We'll know more in 2019.Q2.
  • IT security: we participate in and support the ETH-wide IT security initiative.

I would like to take this opportunity to thank my whole team for their hard and dedicated work all year long.

Happy Holidays and see you in 2019!

2017 in review

Monday, December 18th, 2017

This post is meant to give you a short overview of what has been accomplished in D-PHYS IT by ISG this year. We’ve been hard at work to further improve and extend our services for you, our customers. Some highlights of 2017:

  • Account expiry: in early 2017 we finished assessing all ~7600 D-PHYS accounts and blocked the expired ones. We also tied all D-PHYS accounts to their nethz counterparts wherever possible. This allows us to make use of ETH's employment information from now on. While we were at it:
  • New LDAP servers: Since implementing account expiration meant touching most aspects of our identity management infrastructure anyway, we decided to completely overhaul our LDAP user database. We reworked the LDAP schema (the original one dating back to the early 90s) and set up a 3-way replicating OpenLDAP cluster.
  • Windows Server Cluster: Several mission critical Windows Server instances have been moved to a newly created Windows Cluster. This complements last year's Linux cluster.
  • Storage: in 2017 the disk space occupied by data and backup grew from 1.3 PiB to 1.6 PiB, making this a very slow year as far as storage growth is concerned.
  • Server room migration: in August we had to move most of D-PHYS's servers three rack rows down in the HIT D 13 server room. We now have a solid foundation for our servers for the next years.
  • Outages: apart from the above-mentioned migration, some short-term network interruptions and the unfortunate file server issues of late our systems have been very stable in 2017.
  • Web server upgrade: in January we upgraded the operating system on the D-PHYS web server. We also used the occasion to clean up a lot of legacy cruft.
  • OS upgrades: 2017 brought new OS versions for almost every system: the Windows 10 rollout picked up steam, High Sierra arrived on the Macs and Ubuntu 16.04 on the remaining Linux workstations.
  • eXile: we migrated the configuration management from Puppet to Ansible and then re-installed all eXile gateways in a fully automated way with the latest Debian release.
  • UCC: we laid the technical groundwork and performed implementation tests for the upcoming UCC rollout which will replace the existing ETH telephony system with an all-IP based solution.
  • IT security: we participate in and support the ETH-wide IT security initiative.

I would like to take this opportunity to thank my whole team for their hard and dedicated work all year long.

Happy Holidays and see you in 2018!

2016 in review

Friday, December 16th, 2016

This post is meant to give you a short overview of what has been accomplished in D-PHYS IT by ISG this year. We’ve been hard at work to further improve and extend our services for you, our customers. Some highlights of 2016:

  • New team member: Sven Mäder joined ISG this year to replace Axel in our Linux server team.
  • Account expiry: you might have heard that D-PHYS decided to phase out old accounts in the future. We spent the last year laying the technical groundwork for a smooth and painless implementation of this policy change. One first visible result is our new account portal.
  • Printing: in summer we integrated student printing into the pia printing system which means that we now have a comprehensive printing solution for the whole department. The D-PHYS print server will be shut down in early 2017.
  • Storage: in 2016 the disk space occupied by data and backup grew from 929 TiB to 1.3 PiB, again increasing the yearly growth rate. We are now using 60-disk toploader chassis to maximize storage space-per-volume.
  • Outages: we scheduled two maintenance windows, on April 14 and September 5, in order to perform hardware and system upgrades. Together with a network upgrade by Informatikdienste on September 15, these were the only noteworthy downtimes in 2016.
  • Docking network: in fall 2016 we migrated most of the department's network sockets to the 802.1x-enabled docking network. While there is little immediate benefit for most of us, this is a prerequisite for future network projects like the upcoming Unified Collaboration & Communication (UCC) project.
  • Wifi: in early 2016 we developed and installed a portable wifi probe that eventually led to the discovery of one of the underlying problems causing ETH's wifi woes. Since then, wifi has been much more stable.
  • OS upgrades: 2016 brought new OS versions for almost every system: the Windows 10 rollout picked up steam, Sierra arrived on the Macs and Ubuntu 16.04 on the Linux workstations.
  • Cluster: we built and deployed a new high-availability cluster setup for our virtual servers this year.
  • Core services: a lot of infrastructure work has happened in the background to ensure smooth operation and seamless growth of our services in the future. Examples are: new ActiveDirectory servers for our Windows users, migrating our webserver certificates to Let's Encrypt, a facelift for most of our websites to match the AEM design and an upgrade of our iPXE boot screen.
  • IT security: we participate in and support the ETH-wide IT security initiative and also worked hard to make the mandated n.ethz password change as humane as possible.

I would like to take this opportunity to thank my whole team for their hard and dedicated work all year long.

Happy Holidays and see you in 2017!

2015 in review

Thursday, December 17th, 2015

This post is meant to give you a short overview of what has been accomplished in D-PHYS IT by ISG this year. We’ve been hard at work to further improve and extend our services for you, our customers. Some highlights of 2015:

  • new team members: both Christian Ringger and Christian Schneider joined ISG this year and have already made significant contributions to our setup.
  • new D-PHYS website: the department website moved from the self-hosted Zope system into the ETH-wide AEM/CQ5 content management system. While the hard work of migrating all the content was done by Andreas Trabesinger, we had to sort out a lot of technical details to ensure a smooth transition and to keep D-PHYS's various web services operational.
  • Printing: the majority of the department's printers have been migrated to the new pia printing system. We are now waiting for Informatikdienste to support student printing in order to complete the project.
  • Storage: in 2015 the disk space occupied by data and backup grew from 685 TiB to 929 TiB, further increasing the yearly growth rate. We are also preparing to keep an off-site disaster recovery copy of D-PHYS data on tape.
  • Outages: apart from a hardware failure of our mail server on December 2nd and a short interruption on July 2nd our system have been very stable this year.
  • System upgrades: 2015 brought OS upgrades for almost every system: Debian Jessie on many servers, OS X 10.11 for the Macs, Ubuntu 14.04 on the Linux workstations and the first pilot installations of our new Windows 10 setup.
  • Core services: a lot of infrastructure work has happened in the background to ensure smooth operation and seamless growth of our services in the future. Examples are: more IPv6 work, 802.1x / NAC in our network, a new network zone in the server rooms, an upgrade of our iPXE boot screen and enhanced monitoring.
  • IT security: we participate in and support the ETH-wide IT security initiative.

Happy Holidays and see you in 2016!

2014 in review

Thursday, December 18th, 2014

This post is meant to give you a short overview of what has been accomplished in D-PHYS IT by ISG this year. We’ve been hard at work to further improve and extend our services for you, our customers. Some highlights of 2014:

  • eXile: in order to be able to keep Windows XP machines that cannot be upgraded connected to the network, we have created the exile system of dedicated virtual firewalls. Currently there are 57 computers safely hidden in this network.
  • Security flaws: 2014 saw the disclosure of three rather severe and widespread security problems in quick succession: Heartbleed, Shellshock and Poodle. We patched all affected systems within hours of the announcements and also scanned the network for hosts that had been overlooked. If you're managing any networked machines (not just servers!) yourself, please make sure those are not vulnerable.
  • Outages: we had a major incident on August 27 due to a failure of the server room cooling system. Fortunately we were able to repair the damage within hours. Other than that, our systems have been very stable in 2014 and we only had minor issues.
  • Storage: in 2014 the disk space occupied by data and backup grew from 535 TiB to 685 TiB, further increasing the yearly growth rate. Another 120 TiB are already in the pipeline.
  • Printing: in cooperation with Informatikdienste we prepared and introduced the new ETH printing system in D-PHYS. Several groups have migrated already, the rest of D-PHYS will follow in 2015.
  • IPv6: during the last 12 months we prepared the D-PHYS network for dual stack (IPv4 + IPv6) operation. The biggest step towards a working IPv6 infrastructure was the deployment of an IPv6-ready DHCP server. Beginning next January we will incrementally hand out IPv6 addresses in the D-PHYS network. Later on, we'll make our services IPv6-ready.
  • Brain drain: two ISG group members decided to take on new challenges this year. In November, Thomas Berchtold left us after 3 successful years to become the new Head of IT of D-BAUG, and Elmar Heeb, the founding father of ISG D-PHYS, will start his new job in Informatikdienste in February. We thank both Thomas and Elmar for their dedicated work and contribution to the team and hope to stay in regular contact with them in the future. Christian Ringger will replace Thomas in January, while Elmar's succession is still work in progress.

Happy Holidays and see you in 2015!