My Switch Went Down, Taking Half the Lab With It: HP 2520-PoE and a Console Restart

The evening started innocently enough: I just wanted to check the HP 2520-PoE switch's web panel to see port utilization. The page spun indefinitely and eventually timed out. Refreshing didn't help, and neither did trying a different browser.

First, Verify It's Only the Web Panel

The first thing I learned from such incidents: don't assume that if the interface isn't working, the device itself is down. Pinging the switch worked. Traffic through the ports was also flowing, as machines on the network could still see each other. So, it was just the management web server that was down, not the frame switching. These are two different things, and it's worth distinguishing them before panicking.

SSH also didn't respond, so the last resort, which always works, was the console cable. This is one of those situations where you're grateful you didn't stash it deep in a drawer.

Restarting Through the Console Port

Connecting via console and entering privileged mode gave me network-independent access to the device. I checked the logs, saved the configuration, and only then restarted the switch. The order here is crucial: if I had started with a restart, I would have lost any trace of what actually happened.

After the restart, the panel came back up normally. The problem itself was minor. The side effects, however, were worse.

Proxmox Doesn't Appreciate the Network Disappearing Under Its Feet

Restarting a switch means a momentary loss of connectivity on all ports at once for a virtualization host. Proxmox didn't take it well: the management interface stopped responding, and virtual machines lost their connections. Some services within containers didn't come back up on their own because they started faster than the network returned.

The recovery proceeded step-by-step, from bottom to top: first checking if the bridges on the host had connectivity, then bringing up the network on the host, and only finally, the VMs and services. Doing it in reverse order just leads to repeatedly restarting something that has no way to communicate anyway.

What I Would Do Differently

First: the console cable and a note with the password should be kept near the rack, not wherever they happen to have landed. I spent more time looking for them than on the actual fix.

Second: services that need to survive a network restart require a delayed start or a condition to wait for connectivity. Otherwise, you'll have to bring them up manually after every outage.

Third: before touching anything, it's worth dumping the configuration. It takes a minute and can save your evening.

The biggest takeaway, however, is different. The outage itself was trivial, but it spanned three layers: the device, the network, and virtualization. Without separating them into distinct questions, I would have looked for the problem in entirely the wrong place.