Anyone else seeing their devices as disconnected when looking at manage.alta.inc? Having no issues but just noticed it. I did upgrade to 2.2u yesterday as well.
No, looking pretty normal and connected from my end unfortunately (or fortunately for me I guess). What devices are you running?
Thanks for checking. Mine just started showing up as connected. I’m running two AP6-Pros, one AP6-Outdoor, and one AP6W. What’s strange is that they didn’t all go offline in the controller at the same time.
No problem! I had a minute to check so figured I’d chime in.
If you go into the Logs section of the controller you might be able to see if the APs logged losing connectivity to the controller, assuming they didn’t lose power entirely.
I’m using the cloud controller. They didn’t loose power as all are POE and the AP6-Outdoor is on a seperate POE switch. Are you able to share where the log option is?
Sure! I’ll just link the help article since it has some screenshots and details on all the stuff you can do with it:
https://help.alta.inc/hc/en-us/articles/39964980287003-System-Logs
Is the network itself working as expected?
Yes, the network at the time was working fine. Looking through the logs I see that all access points had failed curl attempts during the “outage” time.
This is happening again and I’m wondering if it’s tied to additional endpoint needing to be allowed. Since setup I’ve only allowed the following endpoints, minus DNS and NTP. @Alta-MikeD have endpoint been added or removed? Connectivity is restored once I allow the following addresses below and I show an established session from the access point.
netstat: showing only processes with your user ID
tcp 0 296 172.16.100.6:38052 3.161.136.40:443 ESTABLISHED 24199/rc
3.161.136.108
3.161.136.26
3.161.136.40
Previously allowed:
”dl.alta.inc”
”manage.alta.inc”
”ping.alta.inc”
Source: https://help.alta.inc/hc/en-us/articles/29110546984475-Outbound-Firewall-Rules-Required
I’m not aware of any new endpoints, but I will ask and follow up just in case. Our infra is on AWS and load-balanced, so I could only guess that the IPs rotate periodically, but I’m not personally sure. That block would fall under this:
{
"ip_prefix": "3.160.0.0/14",
"region": "GLOBAL",
"service": "AMAZON",
"network_border_group": "GLOBAL"
},
Got that from here: https://ip-ranges.amazonaws.com/ip-ranges.json (from THIS article)
Do you have a strict ACL that is matching static IPv4 addresses?
This doc covers how ELB handles IP address rotations: How ELB works - Elastic Load Balancing
I have zero knowledge of our configuration, so I do not know what does or doesn’t apply to us. I’ll circle back.
Thanks for the reply, Mike. Yes, I have a strict security policy that allows connections to those three URLs above. I think something changed on the Alta side and the document I linked was not updated.
No problem! I understand you noticed a change, I’m just not sure if this is expected or unexpected. The IPs themselves point back to where our infra is hosted, so this could be expected periodic IP rotation, or maybe emergency re-routing due to infra outages on AWS side, etc, or it could be new IPs off the endpoints.
AFAIK we’re leveraging ELB, so I am suspecting this one of the former cases, more than the latter.
But ultimately, I don’ t know personally/don’t have access to check, so I’ve asked, and I’ll follow up once I hear back.
1 Like
Are the access points hardcoded to those addresses from a recent update? When you perform a dig on the the new addresses, they do not come back to anything *.alta.inc. Ideally the DNS records would be updated dynamically when there is a change.
No, the access points are not hardcoded to any service IPs on our side, and nothing in our infrastructure depends on fixed addresses. The APs include a set of DoH resolvers (OpenDNS, Google DNS, Cloudflare) whose resolver IPs are intentionally pinned but this applies only to those DNS providers. For all other lookups, the AP will use whichever upstream DNS source is available with the best response time (ISP-provided DNS or DoH), and all connectivity to Alta services remains hostname-based, not IP-based.
We use an AWS Elastic Load Balancer, and AWS owns and manages the underlying IP ranges. Because those IPs belong to AWS, reverse DNS will never return anything under *.alta.inc; that’s expected behavior.
I’ve confirmed there have been no changes on our side. So as suspected, this behavior is simply how AWS Elastic Load Balancers operate: their IPs rotate periodically.
If your ACL is pinned to specific IPv4 addresses, it will break whenever AWS performs an IP rotation. AWS does publish its IP space in a JSON file, but it contains many large ranges across multiple AWS services and regions. Allowing all of them would be overly broad and would defeat the purpose of a restrictive ACL. Can your ACL be updated to allow by domain/hostname or FQDN-based objects?
Mike, my security policy is using the three FQDNs I shared above. Those FQDNs do not return to 3.161.136.108, 3.161.136.26, or 3.161.136.40. For what it’s worth, it appears this became an issue after installing firmware 2.2t.
~ # nslookup ping.alta.inc
Server: 127.0.0.1
Address: 127.0.0.1#53
Name: ping.alta.inc
Address 1: 75.2.70.75
Address 2: 2600:9000:2341:fa00:11:18e6:d580:93a1
Address 3: 2600:9000:2341:6000:11:18e6:d580:93a1
Address 4: 2600:9000:2341:4a00:11:18e6:d580:93a1
Address 5: 2600:9000:2341:ca00:11:18e6:d580:93a1
Address 6: 2600:9000:2341:7000:11:18e6:d580:93a1
Address 7: 2600:9000:2341:bc00:11:18e6:d580:93a1
Address 8: 2600:9000:2341:d400:11:18e6:d580:93a1
Address 9: 2600:9000:2341:f000:11:18e6:d580:93a1
~ # nslookup dl.alta.inc
Server: 127.0.0.1
Address: 127.0.0.1#53
Name: dl.alta.inc
Address 1: 52.84.199.102
Address 2: 52.84.199.99
Address 3: 52.84.199.34
Address 4: 52.84.199.129
*** Can’t find dl.alta.inc: No answer
~ # nslookup manage.alta.inc
Server: 127.0.0.1
Address: 127.0.0.1#53
Name: manage.alta.inc
Address 1: 108.156.245.3
Address 2: 108.156.245.8
Address 3: 108.156.245.40
Address 4: 108.156.245.26
Address 5: 2600:9000:2341:3000:11:18e6:d580:93a1
Address 6: 2600:9000:2341:4000:11:18e6:d580:93a1
Address 7: 2600:9000:2341:d400:11:18e6:d580:93a1
Address 8: 2600:9000:2341:9600:11:18e6:d580:93a1
Address 9: 2600:9000:2341:8400:11:18e6:d580:93a1
Address 10: 2600:9000:2341:ca00:11:18e6:d580:93a1
Address 11: 2600:9000:2341:e600:11:18e6:d580:93a1
Address 12: 2600:9000:2341:cc00:11:18e6:d580:93a1
~ # cat /etc/resolv.conf
Interface lan
nameserver 127.0.0.1
nameserver 8.8.8.8
nameserver 8.8.4.4
1 Like
Issue appears to be with Cloudflare. When pointing my Palo Alto firewall to 1.1.1.1 for DNS resolution for the address objects that use a FQDN, it comes back with what appears to be stale records. When I point to 8.8.8.8 for the FQDN resolution, it comes back with accurate records and connectivity to the controller is restored.
1 Like
One thing I’ll highlight with the difference between 1.1.1.1 and 8.8.8.8, especially when AWS services like ELB are involved, is how they handle DNS-based geolocation routing (GeoDNS). Google DNS passes location hints (EDNS Client Subnet), so AWS can return an IP from the correct regional pool. Cloudflare DNS does not send those hints for privacy reasons, so AWS may fall back to a more generic or global pool.
By design this likely explains part of the discrepancy; 1.1.1.1 is giving a less region-aware answer than 8.8.8.8. The pool referenced above is from the global pool rather than a US-specific pool, which lines up with that behavior. That is of course outside of the variance in update times that sometimes occurs between major providers, albeit those aren’t overly common in most cases.
EDIT: I’m also not sure why there is a discrepancy between device side and what’s happening personally. I’m not ignoring that. Just some of the parts are behaving just like LB traffic, whereas others aren’t. Maybe this indicates that the AP is using the fallback DoH service for resolution as CF probably has the lowest average latency, which ties back into your observation with it being on CloudFlare’s side… Sorry for the uncertainty in my response. I do appreciate your testing here.