Modify

Opened 3 years ago

Last modified 3 years ago

#17807 new defect

qos codel patch noecn to eth1 (wan) and qos-scripts question

Reported by: robnitro@… Owned by: developers
Priority: normal Milestone:
Component: packages Version: Trunk
Keywords: Cc:

Description

In CeroWRT and it's scripts, they make it clear to run noecn on outbound, because some internet routers have issues with it.

So, a simple patch to /usr/lib/tcrules.awk to fix this

 # leaf qdisc
        avpkt = 1200
        for (i = 1; i <= n; i++) {
+                if (device == "eth1") {
+                        print "tc qdisc add dev "device" parent 1:"class[i]"0 handle "class[i]"00: fq_codel limit 800 quantum 300 noecn"
+                } else {
                        print "tc qdisc add dev "device" parent 1:"class[i]"0 handle "class[i]"00: fq_codel limit 800 quantum 300"
+                }
        }

Another question, what does pktsize and pktdelay actually do in qos-scripts?
Also, why are some classes missing _down by default? (example why no bulk_down or priority_down?)
Forgive me, I'm new to openwrt, been using gargoyle and they have a more clear explanation of their qos system.

Attachments (0)

Change History (23)

comment:1 Changed 3 years ago by dtaht

Um, we don't use ECN on outbound because it hurts latency at 4Mbit and below.

Above that, the jury is out. It seems like it works, and works well, against every server we've tested, but it is a rather large internet. There *were* problems a decade back that seem to be resolved.

So in cero we make it optional to leave on or off, and at the time we'd first written the SQM system it seemed safest to turn it off on outbound. Given the success of the free.fr deployment (on by default on outbound) it seems safe to leave it on and get some more results from the field.

Note that on or off at the router is only half the problem, you also have to turn it on the client and server to see any benefit. see: http://www.ietf.org/proceedings/89/slides/slides-89-tsvarea-1.pdf

comment:2 follow-up: Changed 3 years ago by dtaht

I think avpkt is a holdover from when qos-scripts used RED.

I note that I keep trying to find the time to make sqm-scripts readily available in openwrt mainline, the packages come right over from ceropackages with no mods needed. You can even install the cerowrt version on top of nearly anything by forcing the opkg install, as they are just scripts.

Notable features include correct ipv6 support, and support for dealing with PPPoe and ATM encapsulation better that qos-scripts.

comment:3 Changed 3 years ago by anonymous

Where to get those sqm-scripts ipks and how to configure them in OpenWRT? Does it work with luci-qos? Or where to also get luci-app-sqm?

comment:4 Changed 3 years ago by anonymous

http://snapon.lab.bufferbloat.net/~cero2/cerowrt/wndr/3.10.50-1/packages/ has the most current sqm-scripts and luci-app-sqm.

comment:5 Changed 3 years ago by anonymous

But will they also work with trunk OpenWRT because of different kernel/mods? I remember when I wanted to install qos-script from bb on trunk it didnt work because of newer kernel version than what bb version was compiled toward.

comment:6 Changed 3 years ago by robnitro@…

Thanks dtaht for joining this thread! I appreciate your testing, it's pretty fun to mess around. So far I settled with 375 for quantum... any benefits to change flows?

/changeset/42479.html
The new patch removed it on both interfaces??

I also back engineered debloat, changing the BQL to around 3000 for all interfaces (which use around 80-150 mbits) and txqueuelen 16, but it doesn't seem to change much on ar71xx WNDR3800 and ag300h, which share similar interfaces (ag71xx)

comment:7 in reply to: ↑ 2 Changed 3 years ago by robnitro@…

Replying to dtaht:

I think avpkt is a holdover from when qos-scripts used RED.

I note that I keep trying to find the time to make sqm-scripts readily available in openwrt mainline, the packages come right over from ceropackages with no mods needed. You can even install the cerowrt version on top of nearly anything by forcing the opkg install, as they are just scripts.

Notable features include correct ipv6 support, and support for dealing with PPPoe and ATM encapsulation better that qos-scripts.

It's interesting, but I don't like that I can't set my torrents to bulk... there's no set priorities by port and/or IP....
I need to do that, otherwise a couple of torrents can kill the video on demand (VOD) that our set top boxes use MOCA coax through the wan.

comment:8 Changed 3 years ago by dtaht

OK, will try to comment on 3 comments

(please note that I pay WAY more attention to email on the cerowrt-devel list than here)

1) I would hope sqm-scripts remains compatible with bb for a while until I (or someone) gets around to polishing it up for mainline openwrt.

2) Well, we've settled on 300 as a decent value overall. If you look at the packet curve distributions for most traffic you will see a sharp knee in the curve starting about there and rapidly going to 1280, then 1500, so 300 seemed ideal.

2a) Turning off ecn by default is the safe choice. Although we've had extensive testing by now, notably in mosh, and free.fr, and in cerowrt - I don't blame nbd for turning it off. That said, I have no reports of leaving it on hurting anyone for over 2 years now, and certainly feel it does no harm on the ingress interface, so, feel free to bug him to put it back in.

2c) debloat really needs to get debloated. I mean, it's enormous, needs lua and is mostly broken, with tons of ideas that didn't work out (like the qfq stuff). I have a shell script that does almost all that is needed. Fiddling with BQL is less necessary now, I generally let it autotune at 100Mbit and up, and measure whatever a box comes up with. Certainly 3000 seemed about right at 100Mbit on everything I tried.

A possibly more useful thing to fiddle with is the size of the tx ring. I have had it set as low as 4 packets with sometimes useful results, sometimes with slightly higher overhead, but better mixing.

2d) as for txqueuelen, using nearly any other qdisc than pfifo_fast, means it ignores the txqueuelen setting entirely. No need to bother.

3) As for torrent, what I have generally done has turned on CS1 marking in my torrent *client(s)*, and walla! no more work needed at the router, sqm-scripts tosses it in the background queue. For example, the transmission client supports this, you set a rule in the .config/transmission/settings.json for it.

I agree that a more elaborate classifier section would be useful in some instances, however my overall goal in developing fq_codel, sqm, etc, was to try and have something as simple as possible, that worked across a broad a range of traffic as possible, before getting elaborate.

You can, rather than doing it in sqm, add an iptables rule to the outbound torrent port if you like, also, in /etc/firewall.user, something like

iptables -I POSTROUTING -t mangle -p the_torrent_port -o ge00 -j DSCP --set-class CS1
ip6tables the_same_stuff # and the above line might be a bit wrong, but you get the idea...

Or is it the download torrent killing you? In that case a hook into upnp might be worthwhile, to generate the right lines.

We regarded torrent as a huge problem when the work started, and the only solution we came up with back then was this sort of client specification of the diffserv value, or doing port numbers, or dpi:

http://perso.telecom-paristech.fr/~drossi/paper/rossi13tma-b.pdf

Lastly...

I have asked Bram to mark bittorrent's stuff by default as CS1, also.

comment:9 Changed 3 years ago by dtaht

OK, will try to comment on 3 comments

(please note that I pay WAY more attention to email on the cerowrt-devel list than here)

1) I would hope sqm-scripts remains compatible with bb for a while until I (or someone) gets around to polishing it up for mainline openwrt.

2) Well, we've settled on 300 as a decent value overall. If you look at the packet curve distributions for most traffic you will see a sharp knee in the curve starting about there and rapidly going to 1280, then 1500, so 300 seemed ideal.

2a) Turning off ecn by default is the safe choice. Although we've had extensive testing by now, notably in mosh, and free.fr, and in cerowrt - I don't blame nbd for turning it off. That said, I have no reports of leaving it on hurting anyone for over 2 years now, and certainly feel it does no harm on the ingress interface, so, feel free to bug him to put it back in.

2c) debloat really needs to get debloated. I mean, it's enormous, needs lua and is mostly broken, with tons of ideas that didn't work out (like the qfq stuff). I have a shell script that does almost all that is needed. Fiddling with BQL is less necessary now, I generally let it autotune at 100Mbit and up, and measure whatever a box comes up with. Certainly 3000 seemed about right at 100Mbit on everything I tried.

A possibly more useful thing to fiddle with is the size of the tx ring. I have had it set as low as 4 packets with sometimes useful results, sometimes with slightly higher overhead, but better mixing.

2d) as for txqueuelen, using nearly any other qdisc than pfifo_fast, means it ignores the txqueuelen setting entirely. No need to bother.

3) As for torrent, what I have generally done has turned on CS1 marking in my torrent *client(s)*, and walla! no more work needed at the router, sqm-scripts tosses it in the background queue. For example, the transmission client supports this, you set a rule in the .config/transmission/settings.json for it.

I agree that a more elaborate classifier section would be useful in some instances, however my overall goal in developing fq_codel, sqm, etc, was to try and have something as simple as possible, that worked across a broad a range of traffic as possible, before getting elaborate.

You can, rather than doing it in sqm, add an iptables rule to the outbound torrent port if you like, also, in /etc/firewall.user, something like

iptables -I POSTROUTING -t mangle -p the_torrent_port -o ge00 -j DSCP --set-class CS1
ip6tables the_same_stuff # and the above line might be a bit wrong, but you get the idea...

Or is it the download torrent killing you? In that case a hook into upnp might be worthwhile, to generate the right lines.

We regarded torrent as a huge problem when the work started, and the only solution we came up with back then was this sort of client specification of the diffserv value, or doing port numbers, or dpi:

http://perso.telecom-paristech.fr/~drossi/paper/rossi13tma-b.pdf

Lastly...

I have asked Bram to mark bittorrent's stuff by default as CS1, also.

comment:10 Changed 3 years ago by anonymous

A good reason to have something like "debloat.sh" in openwrt one day is that it disables tso gso and gro offloads, and many x86 platforms have those turned on by default, which is undesirable in a router.

comment:11 Changed 3 years ago by dtaht

Gah, I missed a question. The prospect of changing the flows variable is that at very low values (say, 16 or 32), the AQM portion of the code dominates. (at flows "1", it degrades to pure codel). At the higher values, for edge devices, the "fq" portion of the code dominates. We did a bunch of simulation and tests, put a finger in the wind and decided that 1024 queues was "enough" for up to gigE along the edge for everything we looked at. There is some thinking that a larger number of flows makes sense for servers running at that speed and at 10GigE, but those ideas are still baking in the new "fq" and pacing code at google, and if you are building a 10GigE openwrt router, please come talk to me about the rangely processor. :)

And I forgot that "debloat" at least used to, trim the wifi aggregation queue length down to reasonable values for typical bandwidths (instead of) STA right next to the AP bandwidths. Depending on what you commit you pulled,

We had changed it back to the default queue length after 5 months of battling with bufferbloat.net bug 442, which I am deeply grateful to felix for finding and fixing. Reducing the aggregation buffer size to 12 for BE, BK, and VO was often helpful for multistation access (it's still the wrong answer long term, but feel free to try it)

comment:12 Changed 3 years ago by robnitro@…

dthaht, I'm surprised you never reported the wndr routers having the issue when saturating upload , the download would be slow as crap : /ticket/17569.html and the fix developed here: /ticket/13072.html
But maybe if you run 100 on wan, it doesn't show up? My fiber isp uses gigabit, and setting 100 fd on eth1 would make it crawl (turns out to be driver issue).

I have no idea about cs1, I'm using transmission 2.76 cygwin compiled... it's a bit strange but I couldn't stand deluge, and utorrent is garbage, no useful gui application.

How would you mark highest prio traffic, cs3?

BTW, I tried sqm scripts on openwrt, and it ended up working on the uplink but not the downlink (it wasn't being "throttled" to the max setting). It's strange, you have a setting for wan interface, but where is the setting to say what is the LAN, or was it hardcoded to the cero lan? In openwrt, we have eth0, br-lan, and ifq0 (no idea what the ifb0 is there for?)

Btw, does fq_codel on the wlan use up a lot of cpu? I might disable that, openwrt does the fq_codel rules on the ifb0

The tx buffer setting can cause issues... set it too low and it ends up crashing the interface.

As for gro/gso/tso, our drivers have half that off already and can't be changed. I do disable gro, however.

Default settings:

OpenWrt:/etc# ethtool -k eth1
Features for eth1:
rx-checksumming: off [fixed]
tx-checksumming: off
        tx-checksum-ipv4: off [fixed]
        tx-checksum-ip-generic: off [fixed]
        tx-checksum-ipv6: off [fixed]
        tx-checksum-fcoe-crc: off [fixed]
        tx-checksum-sctp: off [fixed]
scatter-gather: off
        tx-scatter-gather: off [fixed]
        tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: off
        tx-tcp-segmentation: off [fixed]
        tx-tcp-ecn-segmentation: off [fixed]
        tx-tcp6-segmentation: off [fixed]
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: off [requested on]
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: off [fixed]
tx-vlan-offload: off [fixed]
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: off [fixed]
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-ipip-segmentation: off [fixed]
tx-sit-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-mpls-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]

comment:13 Changed 3 years ago by dtaht

And incidentally, fq_codel works pretty well even with flows 16 in the sqm system. This has potential to better enable a pure hardware implementation of the code one day in the future.

Feel free to experiment! Give netperf-wrappers a shot, while you are at it.

comment:14 Changed 3 years ago by dtaht

1) It is possible that sqm is not inserting all the modules it needs, try running it by hand and checking for errors. Notably act_mirred and ifb needs to be in lsmod.

As I noted, I (or someone) needs to sit down and make sure sqm works right always on an openwrt system, rather than just on cerowrt. That involves doing a complete dependency check, getting rid of the experimental qdiscs, and using the openwrt default naming schemes.

2) I have no idea what from those threads was relevant. I have not checked 10Mbit speeds at all, prefering to do that sort of speed in software.

3) the ifb0 device is for ingress.

comment:15 Changed 3 years ago by dtaht

and log bugs for sqm over in the ceropackages-3.10 on github, please, not here.

the sqm-on-openwrt bug is there: https://github.com/dtaht/ceropackages-3.10/issues

comment:16 Changed 3 years ago by anonymous

The magic for telling transmission to mark packets with lower priority is here:

https://lists.bufferbloat.net/pipermail/bloat/2012-May/000976.html

"peer-socket-tos": "lowcost",

comment:17 Changed 3 years ago by dtaht

too many threads of convo this late at night:

"Btw, does fq_codel on the wlan use up a lot of cpu? I might disable that, openwrt does the fq_codel rules on the ifb0"

No. fq_codel barely registers on most cpu traces.

comment:18 Changed 3 years ago by dtaht

The real cpu-eater is the rate limiter. (htb or hfsc). We have hit a fairly hard limit in the 60mbit range on inbound with the current code there on most of the sub 700mhz mips platforms. Some work is going on investigating it. So if your inbound is >60mbit, measure cpu usage, and turn off inbound if it's excessive. Fixing outbound is the most critical thing anyway.

comment:19 Changed 3 years ago by robnitro@…

I'll try someday, right now just happy the driver got fixed...
I tried cero for a bit, but wanted an overclocked build and Arokh in openwrt forum was making em. Then I learned how to build my own.

The router is a wndr3800, oc'ed to 800 mhz (memtest stable overnight with a t shirt wrapped around to make it run hot!- everything I overclock has to be stable when hot- I don't trust cooling to always run right hehe).
netperfrunner gets 67/35 on my fiber 75/75 (tests 80/90- qos set to 74/80)

Other router I've been setting up for a friend which also had the ethernet driver bug, buffalo ag300h, same cpu but only does 760 max (the ram is the issue on some).
Similar tests on my line, double natted (and for testing disabling qos on my primary router), about 66/32 or so.

comment:20 Changed 3 years ago by anonymous

How do I setup the sqm-scripts in OpenWrt? I tried changing option interface to pppoe-wan because it was set to @wan which didnt work, though pppoe-wan wasnt in the luci interface drop down list either, doing /ect/init.d/sqm start I get:

/etc/init.d/sqm restart
SQM: /var/run/SQM/SQM_active_on_pppoe-wan
SQM: Queue Setup Script: /usr/lib/sqm/simple.qos
SQM: ifb associated with interface pppoe-wan: ifb1
Failed to find sch_fq_codel. Maybe it is a built in module ?
SQM: Squashing differentiad services code points (DSCP) from ingress.
SQM: STAB: stab mtu 2047 tsize 512 mpu 0 overhead 0 linklayer atm
SQM: get_limit: CURLIMIT: 1001
SQM: cur_target: auto cur_bandwidth: 1000
SQM: get_limit: CURLIMIT: 1001
SQM: cur_target: auto cur_bandwidth: 1000
SQM: get_limit: CURLIMIT: 1001
SQM: cur_target: auto cur_bandwidth: 1000
SQM: egress shaping activated
SQM: Do not perform DSCP based filtering on ingress. (1-tier classification)
SQM: STAB: stab mtu 2047 tsize 512 mpu 0 overhead 0 linklayer atm
SQM: get_limit: CURLIMIT: 1001
SQM: cur_target: auto cur_bandwidth: 11500
SQM: ingress shaping activated

My config:

config queue 'ge00'
	option qdisc 'fq_codel'
	option script 'simple.qos'
	option qdisc_advanced '1'
	option ingress_ecn 'ECN'
	option egress_ecn 'NOECN'
	option qdisc_really_really_advanced '1'
	option itarget 'auto'
	option etarget 'auto'
	option enabled '1'
	option squash_dscp '1'
	option squash_ingress '1'
	option linklayer 'atm'
	option overhead '0'
	option interface 'pppoe-wan'
	option download '11500'
	option upload '1000'
ifconfig
br-lan    Link encap:Ethernet  HWaddr ...
          inet addr:10.0.1.203  Bcast:10.0.1.255  Mask:255.255.255.0
          inet6 addr: fe80::200:83ff:fe8a:1000/64 Scope:Link
          inet6 addr: fdae:f741:ee79::1/60 Scope:Global
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:46938 errors:0 dropped:0 overruns:0 frame:0
          TX packets:78012 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:3268442 (3.1 MiB)  TX bytes:110290391 (105.1 MiB)

eth0      Link encap:Ethernet  HWaddr ...
          inet6 addr: fe80::200:83ff:fe8a:1000/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:47016 errors:0 dropped:0 overruns:0 frame:0
          TX packets:77970 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:4136761 (3.9 MiB)  TX bytes:110598507 (105.4 MiB)

eth0.1    Link encap:Ethernet  HWaddr ...
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:46984 errors:0 dropped:0 overruns:0 frame:0
          TX packets:77966 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:3281522 (3.1 MiB)  TX bytes:110286195 (105.1 MiB)

ifb0      Link encap:Ethernet  HWaddr ...
          inet6 addr: fe80::84c3:bfff:feb5:8ffa/64 Scope:Link
          UP BROADCAST RUNNING NOARP  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:32
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

ifb1      Link encap:Ethernet  HWaddr ...
          inet6 addr: fe80::1407:ceff:fe4c:db9e/64 Scope:Link
          UP BROADCAST RUNNING NOARP  MTU:1500  Metric:1
          RX packets:72601 errors:0 dropped:0 overruns:0 frame:0
          TX packets:72601 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:32
          RX bytes:108137374 (103.1 MiB)  TX bytes:108137374 (103.1 MiB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:314 errors:0 dropped:0 overruns:0 frame:0
          TX packets:314 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:23314 (22.7 KiB)  TX bytes:23314 (22.7 KiB)

nas0      Link encap:Ethernet  HWaddr 00:00:01:00:00:00
          inet6 addr: fe80::200:1ff:fe00:0/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:77289 errors:0 dropped:0 overruns:0 frame:0
          TX packets:46064 errors:12 dropped:0 overruns:12 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:109804707 (104.7 MiB)  TX bytes:4441744 (4.2 MiB)

pppoe-wan Link encap:Point-to-Point Protocol
          inet addr:...  P-t-P:...  Mask:255.255.255.255
          UP POINTOPOINT RUNNING NOARP MULTICAST  MTU:1492  Metric:1
          RX packets:76681 errors:0 dropped:0 overruns:0 frame:0
          TX packets:45463 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:3
          RX bytes:109159738 (104.1 MiB)  TX bytes:2962194 (2.8 MiB)

Also dont know why there are two ifb devices. /etc/init.d/qos is disabled.

comment:21 Changed 3 years ago by fgimenez

I posted my patch on the forums: https://forum.openwrt.org/viewtopic.php?pid=258254#p258254

It does the ECN part and it also calculates target and interval using some math I borrowed from CeroWRT.

This is what I'm using at home:

--- tcrules.awk.orig    2014-09-20 16:47:39.000000000 -0300
+++ tcrules.awk    2014-10-26 21:18:22.000000000 -0300
@@ -15,6 +15,9 @@
     maxrate[n] = ($6 * linespeed / 100)
     qdisc[n] = $7
     filter[n] = $8
+    target = (12320 / linespeed)
+    if (target < 5) { target = 5 }
+    if (target > 10) { interval = 5 + (10 * target) } else { interval = (target + 95) }
 }
 
 END {
@@ -79,7 +82,10 @@
     # leaf qdisc
     avpkt = 1200
     for (i = 1; i <= n; i++) {
-        print "tc qdisc add dev "device" parent 1:"class[i]"0 handle "class[i]"00: fq_codel limit 800 quantum 300"
+       if (device != "ifb0") {
+        noecn="noecn"
+       }
+       print "tc qdisc add dev "device" parent 1:"class[i]"0 handle "class[i]"00: fq_codel limit 800 quantum 300 target "target"ms interval "interval"ms "noecn
     }
 
     # filter rule

comment:22 Changed 3 years ago by dtaht

@robnitro could you get ahold of me via email please?

comment:23 Changed 3 years ago by robnitro@…

Hi, I emailed you at your github address.. Mine is robnitro AT yahoo DOT com

I no longer am using openwrt. I have gargoyle now that they fixed the ethernet driver issues. I have dl qos through gargoyle and my ul qos done by the isp router with all the ports forwarded to the gargoyle router.

Add Comment

Modify Ticket

Action
as new .
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.