Switch took a dive today

This morning my 8 port switch stopped working. No logs notifications (other than email) nothing. Had to power reset the switch to get the network back up and running.

Is there anyway to pull logs on the device or from Alta to see what happened?

Unfortunately, it’s unlikely. However, if you like, you can send me a PM with the MAC address of the switch and we can check server side logs to see if anything was reported.

Have had a similar experience a couple of times with my S16-PoE :frowning:

When this occurs, has anyone attempted to ping the switch itself? If so, do pings fail or respond?

Have you tried SSH? This would likely have to be setup prior to the switch ceasing communications to the cloud.

I’m pretty sure I tried pinging the switch and gateway and that’s what led me to reboot the switch.

I’ve got SSH enabled but I imagine it wouldn’t have worked if I couldn’t ping it.

Ideally, of course, we’d have SSH access and I agree that if you can’t ping the switch, you likely won’t be able to SSH. As you’d imagine, an a perfect world, we’d have the switch in this bad state and have SSH access so we can check the logs. Normally, logs don’t survive reboot/reset but there’s a trick we can use to get them to survive reboot. You simply need to SSH into the switch and execute touch /cfg/.persistent.log. But that would then require SSH access in the future to be able to get in and grab the logs.

Let me ask you both this. What do you have plugged in to the switch that’s PoE?

1 Like

Completely understand, I do IT support with other brands and nothing worse than “it’s faulty” “I can’t replicate it” :joy: If it does happen again, I’ll try and jump into SSH.

I’m powering an AP6-Pro, Grandstream phone and a cough UniFi Flex Mini Switch cough. I’ve also got my Apple TV, VSSL A.1x and PS5 connected to it.

Weird thing though, I’m running everything through a UDM-SE and I assigned them static DHCP leases from there and I logged in yesterday and the AP was showing the same IP address as the switch. I’ve set them static IPs on the Alta Cloud for now to see if this stops the issue.

Connected to my switch is:

  • Port 1: AP 6 Pro (PoE)
  • Port 2: AP 6 Pro (PoE)
  • Port 3: Wyze Cam (PoE)
  • Port 4: Wyze Cam (PoE)
  • Port 5: Wyze Camera (non-PoE)
  • Port 6: patch port, not utilized
  • Port 7: Uplink to another PoE switch
  • Port 8: Uplink to router

My switch and all devices lost all traffic from devices connected to the switch,. Assuming if we could ping it would only be unidirectional and would assume SSH would fail as there is no ACK along the way. Should we be trying to SSH in another method or via a different connection?

@donsandro it’s possible you’re exceeding the PoE budget of the switch. The 2 AP6 Pro’s alone are going to be pushing it, then add two more PoE devices, it’s fully possible you’re exceeding the budget.

@SimonNZ91 yeah, IP conflict could’ve definitely done it, although it shouldn’t be possible as each device would have unique MAC and then should get the unique sticky IP that you set up. If the switch crashes again, please let me know.

1 Like

I have 4 x AP6 Pros on my S8 switch - are you saying this isn’t recommended and I should move them back to my main switch as they will exceeed the POE budget of 60w? Is there an easy way to see how much power they are using?

@Alta-Matt_v2 looking at active stats…
AP6 Pro peak was ~10W, currently 6.6W
Wyze camera is ~3W
For the devices I had that’s about 26W (10+10+3+3). being generous and doubling that amount should still be less than the budget. Feels unlikely I exceeded the budget.
However, assume I did, are you suggesting the switch locks up when we exceed power budget?

I am passing logs to a syslog server, is it safe to say the valuable details would get passed over there or will that not be of value?

Side note had 3 Unifi APs (UAP-Pro, UAP-LR, U6Lite), and 3 wyze cameras over PoE connected to a GS308EP (62W budget).

Could we get a feature to map power consumption over time?

Thanks

I wouldn’t say it’s “not recommended” so much as the math doesn’t add up.
Per the spec sheet, the AP6 Pro can pull a maximum of 25W. 25 * 4 == 100, so you have a distinct potential to exceed to the PoE budget of the S8 switch by a maximum of 40W.

While a typical installation will not hit 25W, it’s still a possibility and therefore is ill advised.

When an Alta switch detects that the PoE budget has been exceeded, it will shut off power to the last PoE interface supported and work backwards. In the case of the S8, port 4 goes first, then port 3 and so on.

To be clear, I’m not suggesting that the PoE budget was exceeded. I’m considering the possibilities.

When an Alta switch detects that the PoE budget has been exceeded, it will shut off power to the last PoE interface supported and work backwards. In the case of the S8, port 4 goes first, then port 3 and so on. That’s the expectation.

As we all know, power can fluctuate greatly. I had a call a couple weeks back that presented a perfect example of why the PoE budget could play a role. The user reported that APs were dropping offline at a school. Well, as school has fairly defined hours of operation and the user noted that this only occurred during the day; no alerts at night. The more clients associated, the harder the AP is working and thus, the more power it’s drawing.

That’s neither here nor there as we cannot say that the PoE budget was exceeded and, if it was, the behavior of the switch is not consistent with what it should do, so the topic of PoE budget can be tabled for the moment.

At this time, there are no messages that get logged if the PoE budget is exceeded…yet.

For your power consumption over time request, please post this as a feature request as development is generally community driven.

@donsandro I would suggest you do the same as I recommended earlier in this thread. Without a definitive way to reproduce the switch lock up, it will be difficult to pin down the root cause. If you have logs (syslog in your case) and a date/time that the switch froze, I’d be happy to look at the logs.

Thanks @Alta-Matt_v2
Based on how over consumption of the PoE budget works it’s safe to say that’s not what happened here as all devices on my switch and the other daisy chained off it stopped transmitting. I have put another S8PoE and balanced the devices across the two, both are directly connected to the router.

As for the recommendation…
Assuming if we could ping it would only be unidirectional and would assume SSH would fail as there is no ACK along the way. Should we be trying to SSH in another method or via a different connection?

Thanks in advance

When the switch detects PoE budget has been exceeded and shuts power off to the last PoE interface, will a notification be sent so we know this is going on?

At present, there is no feedback at all. However, I’ll be putting in a ticket to increase the feedback for events pertaining to PoE budget reporting. As our development is very much community driven, I would strongly recommend creating a feature request for this as well, providing details one what it is you’d like to see. I’ll put the alerts and logging in my ticket, but perhaps there’s another level that you folks are thinking that I am not.

At the same time, PoE budgets can mostly be taken care of in the planning phase. Obviously not all PoE devices will pull their maximum power at all times, but most reputable devices do state their max PoE consumption in their documentation so that should be taken into account when planning the network to ensure no disruption of services.