FOSDEM presentation and my ISP

[jschwartzenberg]

Hi all, especially @FrantisekBorsik

I watched the video from FOSDEM (https://fosdem.org/2025/schedule/event/fosdem-2025-6051-every-isp-needs-to-use-a-qoe-middle-box-on-their-network/) and asked my ISP to see what they think about this.
The original Dutch forum thread is here: https://community.freedom.nl/t/libre-quality-of-experience-libreqos-every-isp-needs-to-use-a-qoe-middle-box-on-their-network/5122

Essentially the response is that they are using RED (Random Early Dectection) instead of CoDel (Controlled Delay) from RFC8290. RED would be less effective than CoDel, but does have higher throughput. Apparently their connections aren't filled up either to a point where these algorithms would be relevant, their routers aren't dropping anything from their queues. What do you think of this?

It's not completely clear what "bandwidth is a lie" refers to. Would that mean that ISPs are oversubscribed or the idea that more bandwidth cannot always solve latency issues?

Many thanks for presenting at FOSDEM! It's great I could share the video to my ISP. I hope it is also interesting to see their response. I'm curious if it might still be helpful to deploy anything related to this on my Turris Omnia.

[dtaht]

RED would be less effective than CoDel, but does have higher throughput. Apparently their connections aren't filled up either to a point where these algorithms would be relevant, their routers aren't dropping anything from their queues. What do you think of this?

RED actually has odd behaviors that codel doesn't, and throughput achieved is roughly comparable at a consistently lower latency. That said, I am happy an ISP is using RED.

I don't believe that they are not dropping packets from their queues (that's what RED does, too), but I can believe that on the backhaul they are not congested much. The biggest place of congestion an ISP has
is to the customer and up from them, and if they are also using RED there, great. Most ISPs have no AQM or FQ there.

(fq_codel or cake are better, Imho)

[jschwartzenberg]

I don't believe that they are not dropping packets from their queues

You can see the thread, but this is really the case. There is also command output from one of their core routers. I am not surprised they have this much better under control than any other ISP would have. They are also owned by a foundation with a construct that avoids they would be striving to focus on excess profitability.

My subscription with them is without any router, which is also why I can use a Turris Omnia without having to explain myself if there would be any issues and my monthly tariff is reduced.

I'm curious if you'd advice to look at running any QoE tooling on my Omnia. Maybe having a dashboard there could still be helpful and indeed show how my ISP handles things?

@FrantisekBorsik, I see you use this phrase "Bandwidth is a lie" more often :) :) I'm quite curious whether you could elaborate on this. I would say that more bandwidth is helpful when there are more consumers, but indeed not to speed interactive things up significantly with small households. Many thanks in advance!!

[FrantisekBorsik]

Hey @jschwartzenberg - happy to see that you have watched our talk at FOSDEM.
With Turris Omnia and any router running OpenWrt, you should be good, as for your "behind CPE from ISP" network, your Wi-Fi/LAN goes. Obviously, you can always upgrade your Omnia to Wi-Fi 6: https://www.discomp.cz/turris-omnia-wi-fi-6-upgrade-kit_d120048.html?action=setlng&lngid=1

As for improving your bufferbloat/latency/jitter setting on Omnia/OpenWrt, this your option: https://openwrt.org/docs/guide-user/network/traffic-shaping/sqm, but you know that most probably. Another option for you would be "CAKE with Adaptive Bandwidth - "cake-autorate" https://github.com/lynxthecat/cake-autorate if you have a variable bandwidth connection from your ISP (like LTE, 5G.)

QoE like LibreQoS/Bequnat/Preseem/Paraqum/Sandvine/Cambium QoE is not for routers, even though people are asking for an option to run it on Raspberry Pi at home, small office or similar.

But besides a typical ISP use case, we have already seen an use case in mining: https://libreqos.io/2024/06/16/network-measurements-at-callio-futuremine/ as well as deployments on cruise ships, university campuses, in hospitality and so on. More use cases will come up in the future. We hope to create case studies presenting it, too.

If you want to check how your are doing regarding bufferbloat/latency/jitter, this is a great test to use:
https://www.waveform.com/tools/bufferbloat

Now, let's address that thing you were asking for, why "Bandwidth is a LIE!" Beyond a certain point, bandwidth really doesn't matter, except for big download or upload. Does bandwidth help? Yes, it does. But the Internet of today is mostly interactive, and it's bufferbloat/latency/jitter that matters the most for our experience using it. We see speedtests with 100s of Mbps Down/Up and terrible bufferbloat/latency/jitter in the US or Europe, while and an ISP in Global South, say in Africa, with 10/1 Mbps Down/Up is offering a better experience on the internet to the customers and yet, still, ISPs worldwide as well as vendors manufacturing antennas, radios Wi-Fi routers or regulators like FCC or CE are selling "speed", the newest Wi-Fi standard and so on. It's nonsense.

Dave et al, we were are addressing it in a comment to Notice of Inquiry from FCC, in 2023. There is a lot of good points, notes and links to studies showing that beyond certain point (and that's not 100s of Mbps, but way less), bandwidth really doesn't matter:
https://www.fcc.gov/ecfs/document/12010651418616/1 - a shorter, article version describing it is here: https://circleid.com/posts/20231211-its-the-latency-fcc

Now, there is a hope. A new FCC chairman might be willing to listen to our arguments more and there is one particular commissioner that was willing to listen to us in the past and wants to work on improving of a broadband definition that would include bufferbloat, latency and jitter so let's see.

One other good news for customers is that another vendor announced a plan to add a support of CAKE in 2025 - https://www.alta.inc and we are looking forward to test it once it's done.

[jschwartzenberg]

As for improving your bufferbloat/latency/jitter setting on Omnia/OpenWrt, this your option: https://openwrt.org/docs/guide-user/network/traffic-shaping/sqm, but you know that most probably. Another option for you would be "CAKE with Adaptive Bandwidth - "cake-autorate" https://github.com/lynxthecat/cake-autorate if you have a variable bandwidth connection from your ISP (like LTE, 5G.)

Many thanks for this! I have read about this, but I was never sure what would be the best approach to start with. I'll look into sqm.

If you want to check how your are doing regarding bufferbloat/latency/jitter, this is a great test to use:
https://www.waveform.com/tools/bufferbloat

Currently scoring a C, so I have some work to do it seems...

Now, let's address that thing you were asking for, why "Bandwidth is a LIE!" Beyond a certain point, bandwidth really doesn't matter, except for big download or upload. Does bandwidth help? Yes, it does. But the Internet of today is mostly interactive, and it's bufferbloat/latency/jitter that matters the most for our experience using it.

Ah yes that makes a lot of sense. It's exactly why I only have a 50 MBit/s (both down&up) FTTH connection as I don't get less latency when I pay more, only more bandwidth (which some of my equipment might end up just struggling with anyhow).

@FrantisekBorsik many thanks for your talk and for all the information! This is very helpful!

[FrantisekBorsik]

Sure thing, @jschwartzenberg!