My Switch Stopped Responding, Taking Half the Lab Down With It: HP 2520-PoE and a Console Restart

The evening started innocently enough: I just wanted to peek into the HP 2520-PoE switch's web interface to check port utilization. The page spun indefinitely and eventually timed out. Refreshing didn't help, nor did trying a different browser.

First, Verify It's Just the Web Interface

The first lesson I learned from these kinds of issues is not to assume that if the interface is down, the device itself is down. Pinging the switch worked. Traffic through the ports was also flowing, as machines on the network could still see each other. So, it was just the management web server that failed, not the frame switching. These are two different things, and it's worth distinguishing them before panicking.

SSH was also unresponsive, leaving the last resort that always works: the console cable. This is one of those situations where you're grateful you didn't stash it deep in a drawer.

Restarting via the Console Port

Connecting via console and entering privileged mode gave me access to the device without network connectivity. I checked the logs, saved the configuration, and only then restarted the switch. The order here is crucial: if I had started with a restart, I would have lost any trace of what actually happened.

After the restart, the web interface came back up normally. The problem itself was minor. The side effects, however, were worse.

Proxmox Doesn't Appreciate Losing Its Network

Restarting a switch means a temporary loss of connectivity on all ports simultaneously for a virtualization host. Proxmox didn't take this well: its management interface became unresponsive, and virtual machines lost their connections. Some services in containers didn't come back up automatically because they started faster than the network returned.

The recovery process went step-by-step, from bottom to top: first checking if the host's bridges had connectivity, then bringing up the network on the host, and only finally the VMs and services. Doing it in reverse just leads to endlessly restarting something that has no network to communicate over anyway.

What I Would Do Differently

First: the console cable and a note with the password should be kept next to the rack, not wherever they happened to land. I spent more time looking for them than on the actual fix.

Second: services that need to survive a network restart require a delayed start or a condition to wait for network connectivity. Otherwise, after every outage, you have to bring them up manually.

Third: before touching anything, it's wise to dump the configuration. It takes a minute and can save your evening.

The biggest takeaway, however, is different. The failure was trivial, but it manifested across three layers: the device, the network, and virtualization. Without breaking them down into separate issues, I would have looked for the problem in entirely the wrong place.