It may be worth trying disabling flow control, at least on the 10Gbps PHY (which is where the impact is most noticeable). Please see my comment HERE for the commands. You can try disabling it on all ports, or just the SFP+ port(s) in use, or any combination you prefer. Please let me know if I can clarify anything.
I’m not saying this is going to fix what you’re seeing, but I’m curious if it brings an improvement. If it does help then it can be made permanent.