Hardware Acceleration issues

Routers
1

I’ve had the Route 10 for over a month now and seem to have a strange issue that’s connected to the hardware acceleration feature.

At first I thought it was a compatibility issue with my Access Point (TP-Link EAP670). Connectivity to the AP establishes at 2.5gb and wifi devices connect at a transmit speed of 1200mbps, but speed tests are less than 200 down and 900 up. Aside from the speed, DHCP is painfully slow and devices constantly lose connectivity to the internet even when wifi signal is excellent. Disabling hardware acceleration resolves all the issues.

I purchased a AP6-PRO with hopes of resolving this compatibility issue, but it seems to have the same identical problems with the same limit on speed. I tried switching cables, switch ports, unplugging all other devices, complete reset of all devices and removing them from manage.alta.inc. Nothing seemed to change other than disabling hardware acceleration.

I could keep hardware acceleration off, but I have 10gbit internet and 10gbit devices connected to the Route10. They suffer in performance with the feature disabled. Forget about enabling IDS/IPS as well.

Does anyone have any ideas? I couldn’t find a forum post where others had similar issues around hardware acceleration…

2

We’ve never heard or reproduced anything like this, but if you give me an invite to your site, I can take a look at your logs to see what’s going on there.

3

Thank you I appreciate the help Jeff. I have the logs forwarding to a graylog server but haven’t been able to find any clues myself.

4

This is exactly what I’ve experienced as well. I have the same access point as you, and like you mentioned, the issue with hardware acceleration affects all devices. For example, the direct LAN connection to my MacBook Pro encounters the same problem. For speed tests, I get similar results to yours, but when I disable hardware acceleration, I’m able to fully utilize the port speed. On some servers, with hardware acceleration enabled, I can reach maximum speeds, but on others, the speeds are extremely low, much like your experience. However, a direct SFP connection to other devices allows me to max out the 10 Gbps speed regardless which servers I chose. Which means, I can isolate out the speedtest servers being the congested, but pinpoint the issue with Route10’s hardware acceleration.

1 Like
5

Khaled,

I use EAP 660HD’s and EAP670’s. My issue is different in that it is focused around Smart devices connecting to the EAP’s on the 2.4GHz bandwidth. Its across smart devices from different manufacturers too. They get DHCP addresses but the mobile Apps show them offline. Turning off acceleration done the tick for me too.

Richard

6

Have you explored the alternate acceleration setting as well?

satty-2025-02-18_09:45:36
satty-2025-02-18_09:45:36422×697 48.1 KB

7

Can you please clearly explain exactly what “alternative” does please?

8

@rutman286 It simply just another hardware acceleration method. Our chipset offers two different methods, and we’ve found good results in some scenarios with this alternate method.

9

Hi Alta-Josh,

Alternate acceleration mode does not fix the issue.

Richard

10

I have the route 10, and I’m seeing that throughput is severely impacted by enabling any acceleration option. It doesn’t matter whether packet inspection is enabled or not.

Throughput is about half, and this is only with a 2GBPS WAN. Not 10GBPS. Why??? Hardware offloading is supposed to be faster, not slower!!!

WAN connection is via SFP port, LAN is via 2.5G ethernet, and the ports report connectivity at the expected full-rate for the port being used.

There are some firewall rules (the defaults) enabled, but that’s it.

IPv4 and IPv6 are enabled.

What the what???

Oh, and it’s running 1.4b software.

11

Do you experience the same bottleneck if you only use the RJ45 copper ports in the Route10, or does it result in line rate for your 2Gbps uplink?

12

Well, it’s very interesting. I am currently unable to tell you.

Reason: the device providing the network connectivity is a WAS-110 from Azores, which is an XGS-PON on an SFP+ stick.

The WAS-110 has OpenWrt-based software running there, and offers DHCP for IP connectivity negotiation.

For some reason that’s unknown to me currently, this negotiation happens just fine when the WAS-110 is plugged directly into the SFP+ port on the route 10.

But, if I plug the WAS-110 into an unmanaged switch (having SFP+ and 2.5GBE wired available), it seems that the route 10 brings up the port to the switch, but never manages to complete the negotiation for an IP address, and so, the wired WAN connection is totally unusable.

I should note here that the setup above is known to work perfectly fine with other routers, so, it’s not something that isn’t known to work!!

It’s very strange, because I can connect my computer directly to the switch having the WAS-110, and it negotiates an IP address just fine.

Both of the WAN ports are configured identically, and only one of them is connected at a time.

Even if I have an SFP+ in WAN2 on the route 10 that’s providing 2.5GBE ethernet and have that connected to the unmanaged switch having the WAS-110, the SFP+ port still manages to get an IP, while the WAN1 does not!!!

And, I have to say, while the route 10 is trying to negotiate with the WAN over ethernet, there’s absolutely no visibility into what’s going on on the router because it’s lost WAN connectiity to the management website, so it’s impossible to get any information!!!

This, to me, is the one major downfall of having the router managed in the cloud. Even if I host it myself, in my own network, if I have multiple sites, unless I’m on the site that has the route 10 with good connectivity, I simply have no visibility as to what’s going on if the route 10 is remote. Kinda hope for the best and cry when it’s not happy, because there’s absolutely no way to figure out what’s happening.

I think there needs to be some level of ability to get into the device if it’s lost connectivity with the management service. Even if it just displays some status information that would be helpful to be able to diagnose what the issue is, but, currently, there’s absolutely no information available.

Imagine making a configuration change that makes the device not have WAN connectivity available? Ouch!!!

13

As a follow-up, I have tried switching to a different XGS-PON device, one that offers 5Gbit wired ethernet, and tried connecting via (one at a time) both the wired (WAN1) and SFP+ (WAN2) ports.

I tried enabling hardware acceleration, both modes, and found that in both WAN ports, I still get best throughput when acceleration is disabled.

Throughput is about 50% better with acceleration turned off.

Interestingly, the download speed is the thing that’s particularly negatively affected. The upload speed seems to truck along at the max speed in both cases.

I am still using just the basic suggested rules. Nothing added or changed.

As for the wired ethernet versus the SFP+ port, I would say that the SFP+ port beats the wired ethernet every time. I’d say the speed delta is about 20% slower via built-in wired ethernet compared to the SFP+ port.

I also tried enabling the download & upload limit values in the WAN settings. It doesn’t seem to make a lot of difference as far as performance goes.

So…

The WAN1 does not like to play nice with regards to requesting DNS when it’s connected to an unmanaged switch being used as a media converter for the SFP+ XGS-PON on-a-stick.

SFP+ (WAN2) is faster than wired ethernet (WAN1) by a healthy margin.

Acceleration does not accelerate inbound traffic at all, in my case, download speeds jump up about 50% when acceleration is entirely disabled. Yes, I tried both modes.

And, as far as I can tell, yes, the wired ethernet port (WAN1) is connecting faster than 1GB speed when it’s being used.

And, yes, I did try swapping out the wiring used for the WAN connection. No differences were observed as a result. Both patch cables are rated as CAT6.

14

It may be worth trying disabling flow control, at least on the 10Gbps PHY (which is where the impact is most noticeable). Please see my comment HERE for the commands. You can try disabling it on all ports, or just the SFP+ port(s) in use, or any combination you prefer. Please let me know if I can clarify anything.

I’m not saying this is going to fix what you’re seeing, but I’m curious if it brings an improvement. If it does help then it can be made permanent.

15

As it happens, I did see that post, and I tried the change for both the WAN ports, and saw no difference in general.

ie: ingress was still 2/3 the speed (roughly) when acceleration is enabled vs being disabled. outgoing traffic rate appears to be full-rate and unaffected.

I see that I’m not alone in finding that performance is not improved by using acceleration.

But:

I do happen to have another route 10 here that I’m curious to swap over to, to see if it’s peculiar to this particular example or not.

Is there a way to transfer all configuration details from one device to another, so I can swap it out?

Perhaps I’m not seeing it, but, I didn’t see any kind of ‘backup/restore’ functionality.

If I need to swap out the router, and essentially move the entire config from one to the other, how is that done? I can’t imagine it’s not possible? Please advise.

16

Local instances support backup and restore for the site, but the cloud version does not yet offer this functionality in a traditional, exportable, and downloadable format. However, you can copy a site’s settings to a new site using the Site Manager.

At this time, there’s no way to directly copy a device configuration. Devices inherit settings from their assigned site, so if you add the new device to the existing site, it will inherit those site-level settings—this provides a starting point, at least. That said, device-specific overrides may be lost in either approach that is currently available.

17

That’s a bummer, dude.

19

I did hook up the 2nd route 10.

OK. Well, the XGS-PON device has multiple ports, but, only one of them is 5Gbit/s, the others are 1Gbit/s.

So, I plugged the 5Gbit/s into the same unmanaged switch that I was using as a media converter using ethernet (it has 2 SFP+ and 4 2.5Gbit ethernet ports), so now both route 10 each have 2.5Gbit/s wired ethernet available simultaneously.

Each route 10 is assigned it’s own 192.168.1.x WAN address.

They both appear to be behaving the same with respect to throughput.

The WAN port configs options are all set to ‘default’ (as for speed and such), and there’s no bufferbloat settings configured.

If I use this setup, with acceleration enabled, I’m seeing full rate down & up, which, for me, is around 2Gbit/s.

If I set bufferbloat values, I now see about 900Mbit/s throughput (acceleration is enabled here).

If I set acceleration to be disabled, I see the ingress throughput increase about 30% over the acceleration-enabled throughput.

Regardless of the settings, it seems that the egress throughput is able to run at full speed.

Given the way this is going, I have to wonder: how do you guys test and validate that the route 10 throughput is working as expected?

Perhaps I should be asking: what throughput numbers should I be expecting to see, given what I have shared?

20

I got to wondering… Bufferbloat. I’m wondering if setting a limit on both the inbound and outbound side of things will make a practical difference?

If the WAN is slower than the LAN connection, typically, your ISP is going to limit the ingress of data to somewhere around what you’ve contracted them to provide. If they’re unaware of anything you’re doing in your own router, you don’t really want your router to become an artificial limit to the flow of incoming data do you?

On the other hand, as far as sending data, you definitely don’t want to push more than what you’ve contracted to have available, as the carrier/ISP is not likely to pass unlimited amount of traffic. So, if you impose a limit on the outgoing data rate, so that you don’t over-fill / give reason for anything on the ISP side to drop packets, and you leave the incoming limit unset, is’nt that the best way to go?

You limit your outgoing traffic to stay within the limits.

The ISP isn’t going to want to send you more than they want to, but, you’re not in control of that anyway, you just want to provide as clear a path for the data they send to arrive on your LAN.

?

Part of why I raise this is that if I don’t set an ingress limit, but do set an egress limit, I’m seeing full rate incoming data and near maximum outgoing data.

Using the above setup, I’m seeing full-rate download speeds with & without acceleration, and the upload speeds are being kept close to the max rate I’ve configured, just shy of 2Gbit/s.

?

And, this is with 1.4e firmware.

Did the 1.4e firmware do anything that might fix my SFP+ XGS-PON stick being able to be used via an unmanaged switch as a media converter to wired ethernet?

I wonder if the lan tagging issue was impacting me?

21

I guess it’s a trade-off, speed vs. responsiveness/latency. I’ve got a 600/600 Mbps contract which turns out as 625/625 Mbps with latency of 6/12/12 ms. I assume this is very good to start with. Tuning CAKE-settings to approximately 600-605 up and down gives 575/575 Mbps and 6/6/6 ms latency. Speed is then above 90% of nominal and latency reduced to half. See some results below. Though, when looking at the corresponding results from Waveform for an Android device over WiFi, I found the download latency reduced from appr. 25-30 to. 12-13, while the upload did not change much at all. Consecutive tests shows that it remains around 15-17 ms during upload. The majority of numbers are still very good though. The conclusion being, as you put it, that the limit on upload might not actually do that much good except from limiting the upload max speed.

Results from Waveform, nominal as well as 602/602 CAKE limits...

1000007078
10000070781440×3216 170 KB

1000007079
10000070791440×3216 158 KB

Speedtest from Route10, nominal...

Nominal:

Idle Latency:     
5.72 ms (jitter: 0.16ms, low: 5.64ms, high: 6.16ms)
  
Download:   
625.72 Mbps (data used: 289.8 MB)
11.68 ms   (jitter: 0.72ms, low: 5.84ms, high: 13.34ms)

Upload:
626.66 Mbps (data used: 298.6 MB)
12.09 ms (jitter: 0.83ms, low: 5.80ms, high: 13.49ms)

Packet Loss:     0.0%
Speedtest from Route10, download limit 602 (upload blank)...

Download limit only (602):

Idle Latency:
5.67 ms   (jitter: 0.01ms, low: 5.63ms, high: 5.68ms)

Download:
577.21 Mbps (data used: 288.8 MB)
5.90 ms   (jitter: 0.40ms, low: 5.69ms, high: 10.65ms)

Upload:
626.58 Mbps (data used: 284.6 MB)
11.93 ms   (jitter: 0.82ms, low: 5.68ms, high: 13.48ms)

Packet Loss:     0.0%
Speedtest from Route10, 602/602...

Download and upload limit (602/602):

Idle Latency:
5.75 ms   (jitter: 0.17ms, low: 5.67ms, high: 5.89ms)

Download: 
571.86 Mbps (data used: 293.7 MB)
5.94 ms   (jitter: 0.17ms, low: 5.76ms, high: 7.12ms)

Upload:
574.07 Mbps (data used: 280.1 MB)
5.82 ms   (jitter: 0.14ms, low: 5.71ms, high: 7.68ms)

Packet Loss:     0.0%

Edit: While trying out your suggested approach, I found this potential bug, which I reported here:

Edit 2: Seems like this was related to the Hardware acceleration setting for the Route10 WAN port. Toggling the setting, Enabled → Alternate → Disabled → Enabled, seems to provide sensible results again.