Hello,
Not trying to be adversarial or anything here, but why do the APs work DNS so hard? They are reaching out to Cloudflare, Cisco and Google DNS rapid fire non-stop. I have internal DNS and prefer to use that. I added my internal DNS servers to the APs when I set them up, but they continue to hammer on the others mentioned above. I would very much like to disable this. As it is, I have blocked all but Cloudflare, because we use their ZeroTrust setup for several of our systems that remote workers need to access, and those are used extensively in the ZT setup. The internal DNS is pretty solid, so I don’t worry much about losing one of them (we have 3). Google DNS is a privacy issue, Cisco much the same, so can we please get the ability to disable these?
I’m glad you mentioned this as it was also a major concern of mine as well from a business security standpoint. I was about to contact support regarding it, but this forum will work for others who might have the same questions.
The AP is constantly making queries to the DNS servers you mentioned using DoT and DoH. At least once every minute. I looks like it has to do with the AP trying to make sure their cloud is reachable/pingable? Security conscious individuals do NOT like any device that tries to contact any DNS server you have not given it permission to. Especially encrypted, as that further evades threat detections that may be in place.
Related to the above, it also appears that the AP does not even use the DNS servers you supply it? It is set to use its own localhost resolver listening on 127.0.0.1 as the primary resolver.
In addition to the constant DNS lookups, I’ve observed it’s also contacting Let’s Encrypt SSL certificate servers every time you make a change to a connected client “device” in their cloud, like changing its name or icon. This originally threw all sorts of red flags with my firewall, but looking into it it’s nothing nefarious, but also something I have never seen any other AP do and looking for clarification on why.
I only mention these things because over my career I have never seen any AP “chatter” as much as this. For example, I just looked at the past firewall logs for an AP from a different manufacturer and it has tried contacting the Internet ZERO times over the last 6 hours, vs the Alta sending traffic every minute. Both AP’s are “cloud controlled” so why does Alta’s need to chatter so much?
I do want to note that I do NOT think Alta is doing anything nefarious. I must also note though that it appears Alta is using OpenWRT for their base software, and I have never examined one before. So it’s possible that’s just the way things work on OpenWRT?
We are still working to decrease the periodicity of the DNS requests going out for ping.alta.inc, but it’s a relatively insignificant amount of traffic from our perspective. Allow me to explain why it works the way it does.
The APs and switches will send an ICMP to ping.alta.inc every 30 seconds to ensure they have a valid Internet connection. They will also send an ICMP to the gateway, and if both of these fail, then (if enabled) the device may enter “fallback upon failure” mode, where it will try to recover its connection (by switching to DHCP+VLAN1), or in the case of an AP (and if enabled), start meshing.
The reason the DNS requests occur as much as they do is because ping.alta.inc is a CNAME for dl.alta.inc, and dl.alta.inc is tied to an AWS CloudFront Distribution, and the DNS TTL on CloudFront Distributions is very low (like 30 seconds). We may tie it to a different hostname at some point, but, like I said, it is a relatively low amount of traffic compared to everything else going on in a network.
The Alta devices also need to be able to look up local controller dynamic DNS hostnames, regardless of whether the local router has DNS rebinding protection active (local controller will not work if those queries are dropped). That is why they all employ secure DNS in addition to the local DNS server supplied by DHCP.
Pulling a report from my SIEM, it shows 64,410 attempts in the last hour for a single AP reaching out to Google DNS and OpenDNS (Cisco), this doesn’t include the Cloudflare DNS requests. That’s not insignificant. Now, it may be my fault it is so high, because I am blocking both of those and it may cause them to increase the attempts, but going back before the block was put in place, it was 55,340 in an hour from a single AP. So, in our location, we have 3 AP6-Pros, so multiply that by 3. It is actually 0 (zero) data because I am blocking it, but if I weren’t, it would be enough to set off alarms. My thoughts are: I understand the explanation, but I want the APs and switches to use the DNS servers I tell them to. I can’t tell you the number of times I have run an incident where data is being exfiltrated using DNS, either UDP or TCP and in the past couple of years, DOH, etc. I, perhaps suffer from a bit of professional paranoia, but it is what it is and I want to be in control of data and requests that leave networks I am responsible for. If you guys feel that strongly about using external DNS for this, pick one and use it, or give us a drop down list of them to choose from, because I don’t want any of my systems using Google DNS. Using all three of them and running through all of them for each request is not great. I want to repeat what I said in my first post, I am not trying to be adversarial, but I also don’t feel this is acceptable.
@scatterday It feels like there is something wrong in your site or configuration that is causing excessive requests. There should only be ~120 requests per hour maximum, one every 30 seconds. If you can share the logs with me privately, I’d be happy to try to understand why there are so many requests going out.
@user19 I’ve just monitored an AP here, and it looks like a single AP will typically use the DHCP-supplied DNS by default, and send 2 requests per 30 seconds per protocol (one for IPv4 and one for IPv6), for a total of 4 requests per 30 seconds, or 480 per hour. Keep in mind that this is through the locally supplied DNS server which can cache the responses, anyway. Compared to the amount of DNS traffic at a typical home/office where there are hundreds of requests per minute, depending on the traffic, this is a relatively small number of requests.
If you are truly concerned about these limited number of requests, you are free to disable the fallback-upon-failure and/or mesh features.
I have the APs set with static IP addresses with 2 local DNS servers set. I also turned off the “mesh” when I first installed them and last night, turned off the fallback on failure. I have been pretty busy today so haven’t looked at it yet, but have received alerts from our SOC about them again. I will ask them to do a count again and see where we are so far today.
As I’ve explained to both of you privately, if you really want to disable the connection daemon, you can ssh to the AP, and add this to /cfg/rc.local:
mv /lib/sh/netd /lib/sh/netd.no
killall netd
Of course this is not supported and will break mesh and failover functionality, but will definitely eliminate the requests.
I don’t think the problem is the amount of times DNS is being queried to lookup ping.alta.inc and dl.alta.inc. It can do it every 30 seconds, I don’t care, but it needs to be using ONLY the DNS servers that are provided to it via DHCP or assigned statically.
Instead, the AP’s are using trying to use Google, OpenDNS and Cloudflare DNS servers to make the queries every 30 seconds, and each one of those requests also appears to try to use ports 53, 853 (DoT) and 443 (DoH) each time as well. That doesn’t bode well for security/privacy conscious users/businesses and each one will have their own reasons. This will fill up the firewall logs with “blocks” and set off alerts, as it should. Basically, the firewalls are doing their job and detecting a device on the network trying to use DNS servers other than the one’s assigned to it.
Do the Google, OpenDNS and Cloudflare DNS servers NEED to be used for some particular reason? If not, remove them because this problem is not going to go away for business users. And again, I’m have not seen this behavior on other manufacturer’s AP’s.
I’m attaching a little snippet of one of my logs so you can better understand what we’re seeing. (I don’t see a way to attach a CSV file but assume I can email you one if needed)
Thanks, Jeff. I am going to look at changing the DOH proxy first, then if that doesn’t work, I will do this. I don’t want to break the things, just make them behave a bit better.
Thank you again for your great responsiveness when it comes to all of my questions, I really do appreciate your help.
I modified the https-dns-proxy and am happy to report I didn’t brick anything and my DNS woes are resolved. I have to say, it is very rare that I have been provided the level of support I have received here, and I very much appreciate it.
Hopefully it will survive updates, but if not, it ultimately took about 10 minutes to do and I have saved my changes in our internal Git repo, so it will be very easy to apply again if firmware updates default it to the previous behavior.
Thanks again (Alta-Matt) and (Alta-Jeff) are my heros!
The DOH requests are required when using a local controller in case DNS rebind protection is active on the local router, and can also simply be a backup in case the provided DNS does not work. Our priority is to ensure that the product works, even under non-ideal circumstances. However, if you’d really like to disable that, you can disable the https-dns-proxy daemon by adding this to your /cfg/rc.local file:
/etc/init.d/https-dns-proxy stop
You can also edit the /etc/config/https-dns-proxy file in rc.local on-the-fly if you’d like to use different DOH servers.
Well, the changes didn’t survive a reboot. It is odd though, because when it reverted, it didn’t have the Google DNS as the Bootstrap servers, so I assume Alta made some kind of change and I got an updated version of the config file…or at least that is the only thing that makes sense to me currently.
If you share your /cfg/rc.local file privately, I can tell you if it should survive a reboot. That file is just run once at boot, right after the saved configuration takes effect.
I don’t have an rc.local under the /cfg directory…assuming you mean the /cfg directory off of the root filesystem, so if you are logged in, you do a “cd /” and then “cd /cfg”.
The only one I have found so far is under /etc. The contents of that are basically nothing.
# Put your custom commands here that should be executed once
# the system init finished. By default this file does nothing.
if ! grep -q rescue /proc/cmdline && [ -e /cfg/rc.local ]; then
. /cfg/rc.local
fi
exit 0
Just to reiterate, I am not trying to do away with the https-dns-proxy, I am trying to configure it to use specific servers. I don’t want to use GoogleDNS or OpenDNS (Cisco) or Cloudflare DNS.
I am fairly familiar with OpenWRT, so I used UCI to set the configuration of https-dns-proxy.
