Queues and Bandwidth Management
Document revision 17-Jan-2003
This document applies to the MikroTik RouterOS V2.6
The queuing can be used for limiting the bandwidth for certain IP addresses, protocols or ports. The queuing is performed for packets leaving the router through a physical interface. It means that the queues should always be configured on the outgoing interface regarding the traffic flow. If there is a desire to limit the traffic arriving at the router, then it should be done at the outgoing interface of some other router. But in some cases you can use firewall rule that simply drop packets when traffic matching this rule exceeds some value.
Installation
The queue management feature is included in the 'system' software package.
No additional software package installation is needed for this feature.
How Queues Work
There are four types of simple queues implemented in RouterOS: PFIFO, BFIFO, SFQ and RED. This
chapter explains difference between these types and introduces queue trees.
With Bytes First-In First-Out (BFIFO) and Packets First-In First-Out (PFIFO) packets are served in the same order as they are received. The only difference between BFIFO and PFIFO is that PFIFO has a length measured in packets, BFIFO in bytes. Generally, you do not want to use BFIFO or PFIFO as traffic shapers. It's better to use them just for statistics as they are pretty fast. The only exception is when you are running out of resources with RED and/or with complicated queue tree.
Stochastic Fair Queuing (SFQ) cannot limit traffic at all. Its main idea is to equalize sessions (not computer traffic, but session traffic, it is sometimes mentioned as SFQ drawback) when your link is completely full. It works in round-robin fashion, giving each session a chance to send sfq-allot bytes. Its algorithm can distinguish only 1024 sessions, and that is why several sessions can be treated as one. Each sfq-perturb seconds it drops internal table mixing all the connections and creates a new table. As it is very fast, you may want to use it as a child queue.
The normal behavior of queues is called tail-drop. Tail-drop works by queuing up to a certain amount, then dropping all traffic that 'spills over'. Random Early Detection (RED is also known as Random Early Drop because it actually works that way) statistically drops packets from flows before it reaches its hard limit. This causes a congested backbone link to slow more gracefully. It starts dropping packets when threshold reaches red-min-threshold mark randomly with increasing probability as threshold rising. Maximum probability is used when traffic reaches red-max-threshold mark. Then packets are simply thrown away. burst parameter is the number of packets allowed to burst through the interface when the link is empty (generally value of (min+min+max)/3 works fine). The minimum value that can be used here is equal to the value of red-min-threshold.
Classful queues are very useful if you have different kinds of traffic which should have differing treatment. Generally, we can set only one queue on the interface, but in RouterOS even simple queues (known as classless queues) are attached to the main (attached to the root, which represent physical interface) Class Based Queue (CBQ) and thus have some properties derived from that parent queue. With classful queues it is possible to deploy hierarchical queue trees. For example, we can set a maximum bandwidth for a workgroup and then distribute that amount of traffic between the members of that group as we can do with simple queues attached to the main CBQ, but with upper limit.
Each queue represents a virtual interface with the allowed bandwidth. It can be borrowed from sibling queues (queues that are children of one queue) if we set bounded to no. If we set bounded to yes, the queue can not occupy bandwidth of other queues. If set to no, the queue would use over the allocated bandwidth whenever possible. Only when other queues are getting too long and a connection is not to be satisfied, then the 'not-bounded' queues would be limited at their allocated bandwidth. When the parent is allowed to send some amount of traffic, it asks its inner queues in order of priority (priorities are processed one after another, from 1 to 8, where 1 means the highest priority). When there are some queues with the same priority value, they are asked in Weighted Round Robin (WRR) fashion. In each WRR round the queue can send the amount of data equal to weight*allot, where allot is the amount of data sent in one turn, and weight shows the number of allowed transmittings in one Weighted Round Robin round (for example, if there are two queues, but weight for the second is two times higher then for the first, then the second queue gets its data sent two times in a round, while the first queue - only one time). That is why allot should be bigger than interface MTU (MTU+14 works fine in most cases).
max-burst parameter specifies the maximal number of packets that can burst when there are no packets
in the queue. In other words, when current data rate is below the limit, max-burst packets
may spillover before the actual limiting will be applied. CBQ algorithm obviates the possibility
of exceeding the allowed average data rate.
Configuring Simple Queues
Simple queues can be used to set up bandwidth management for the
whole traffic leaving an interface, or for certain source and/or destination addresses.
For more sophisticated queue setup use the queue trees described further on.
To add simple queues, use the /queue simple add command:
[admin@MikroTik] queue simple> add dst-address=192.168.0.0/24 interface=ether1 \
limit-at=128000
[admin@MikroTik] queue simple> print
Flags: X - disabled, I - invalid
0 name="" src-address=0.0.0.0/0 dst-address=192.168.0.0/24
interface=ether1 limit-at=128000 queue=default priority=8 bounded=yes
[admin@MikroTik] queue simple>
Argument description:
name - descriptive name for the queue
src-address - Source IP address. Can be set in the form a.b.c.d/n, where n is network mask
src-netmask - Source netmask in decimal form a.b.c.d
dst-address - Destination IP address. Can be in the form a.b.c.d/n
dst-netmask - Destination netmask in decimal form a.b.c.d
interface - Outgoing interface of the traffic flow
limit-at - Maximum stream bandwidth (bits/s). 0 means no limit (default for the interface).
queue - queue type. If you specify the queue type other than default, then it overrides the default queue type set for the interface under /queue interface. See the /queue type for available types.
priority - Flow priority (1..8), 1 is the highest.
bounded - Queue is bounded.
To track how the rules are processed, see the bytes and packets counters for the queues:
[admin@MikroTik] queue simple> .. tree print
Flags: X - disabled, I - invalid, D - dynamic
0 D name="" parent=ether1 flow="" limit-at=128000 max-burst=20
queue=default priority=8 weight=1 allot=1514 bounded=yes
[admin@MikroTik] queue simple>
Queue rules are processed in the order they appear in the /queue tree print list. If some packet matches the queue rule, then the queuing mechanism specified in that rule is applied to it, and no more rules are processed for that packet.
The queue types are used to specify some common argument values for queues. There are four default built-in queue types: default, ethernet-default, wireless-default, and synchronous-default. The built-in queue types cannot be removed. You can add your own queue types by specifying the argument values, for example:
[admin@MikroTik] queue type> add name=CUSTOMER-def kind=red \
\... red-min-threshold=0 red-burst=0
[admin@MikroTik] queue type> print
0 name=default kind=none bfifo-limit=15000 pfifo-limit=10 red-limit=60
red-min-threshold=10 red-max-threshold=50 red-burst=20 sfq-perturb=5
sfq-allot=1514
1 name=ethernet-default kind=none bfifo-limit=15000 pfifo-limit=10
red-limit=60 red-min-threshold=10 red-max-threshold=50 red-burst=20
sfq-perturb=5 sfq-allot=1514
2 name=wireless-default kind=sfq bfifo-limit=15000 pfifo-limit=10
red-limit=60 red-min-threshold=10 red-max-threshold=50 red-burst=20
sfq-perturb=5 sfq-allot=1514
3 name=synchronous-default kind=red bfifo-limit=15000 pfifo-limit=10
red-limit=60 red-min-threshold=10 red-max-threshold=50 red-burst=20
sfq-perturb=5 sfq-allot=1514
4 name=CUSTOMER-def kind=red bfifo-limit=15000 pfifo-limit=10 red-limit=60
red-min-threshold=0 red-max-threshold=50 red-burst=0 sfq-perturb=5
sfq-allot=1514
[admin@MikroTik] queue type>
Argument description:
name - name for the queue type
kind - kind of the queuing algorithm used:
pfifo - Packets First-In First-Out
bfifo-limit - BFIFO queue limit. Maximum packet number that queue can hold.
bfifo - Bytes First-In First-Out
red - Random Early Detection
sfq - Stochastic Fair Queuing
none - (same as default) The queue type as it is by default for the specific interface.
pfifo-limit - PFIFO queue limit. Maximum byte number that queue can hold.
red-limit - RED queue limit
red-min-threshold - RED minimum threshold
red-max-threshold - RED maximum threshold
red-burst - RED burst
sfq-perturb - amount of data in bytes that can be sent in one round-robin round
sfq-allot - how often to change hash function
For small limitations (64kbps, 128kbps) RED is more preferable. For larger speeds PFIFO will be as good as RED. RED consumes much more memory and CPU than PFIFO & BFIFO.
Setting Default Queue Type for the Interface
To change the default queue type for the interface,
use the /queue interface set command, e.g.:
[admin@MikroTik] queue interface> print # INTERFACE QUEUE 0 ether1 ethernet-default 1 prism1 default [admin@MikroTik] queue interface> set prism1 queue=wireless-default [admin@MikroTik] queue interface> print # INTERFACE QUEUE 0 ether1 ethernet-default 1 prism1 wireless-default [admin@MikroTik] queue interface>
Configuring Queue Trees
The queue trees should be used when you want to use sophisticated
bandwidth allocation based on protocols, ports, groups of IP addresses, etc.
If you have added a simple queue, it is listed as dynamic one under the /queue tree print, e.g.:
[admin@MikroTik] queue simple> print
Flags: X - disabled, I - invalid
0 name="A_Simple" src-address=0.0.0.0/0 dst-address=192.168.0.0/24
interface=ether1 limit-at=128000 queue=default priority=8 bounded=yes
[admin@MikroTik] queue simple> .. tree
[admin@MikroTik] queue tree> print
Flags: X - disabled, I - invalid, D - dynamic
0 D name="A_Simple" parent=ether1 flow="" limit-at=128000 max-burst=20
queue=default priority=8 weight=1 allot=1514 bounded=yes
[admin@MikroTik] queue tree>
Argument description:
name - descriptive name for the queue
parent - name of the parent queue. The top-level parents are the available interfaces (actually, main CBQ). Lower level parents can be other queues. Dynamic queues (created with the simple queue tool) cannot be used as parents.
flow - flow mark of the packets to be queued. Flow marks can be assigned to the packets under /ip firewall mangle when the packets enter the router through the incoming interface
limit-at - Maximum stream bandwidth (bits/s). 0 means no limit (default for the interface).
max-burst - Maximal number of packets allowed for bursts of packets when there are no packets in the queue. Set to 0 for no burst.
queue - queue type. See the /queue type for available types.
priority - Flow priority (1..8), 1 is the highest.
weight - Flow weight in the Weighted Round Robin process
allot - Number of bytes allocated for the bandwidth. Should not be less than the MTU for the interface.
bounded - Queue is bounded.
To apply queues on flows, the mangle feature should be used first to mark incomming packets:
[admin@MikroTik] ip firewall mangle> add action=passthrough mark-flow=abc-http \
\... protocol=tcp src-port=80
[admin@MikroTik] ip firewall mangle> print
Flags: X - disabled, I - invalid
0 src-address=0.0.0.0/0:80 in-interface=all dst-address=0.0.0.0/0:0-65535
protocol=tcp tcp-options=any icmp-options=any:any flow=""
src-mac-address=00:00:00:00:00:00 limit-count=0 limit-burst=0
limit-time=0s action=passthrough mark-flow=abc-http tcp-mss=dont-change
[admin@MikroTik] ip firewall mangle>
See the Firewall Filters and Network Address Translation (NAT) Manual for details on how to mark the packets.
You can add queue using the /queue tree add command:
[admin@MikroTik] queue tree> add name=HTTP parent=ether1 flow=abc-http \
limit-at=128000 max-burst=0 bounded=yes
[admin@MikroTik] queue tree> print
Flags: X - disabled, I - invalid, D - dynamic
0 D name="A_Simple" parent=ether1 flow="" limit-at=128000 max-burst=20
queue=default priority=8 weight=1 allot=1514 bounded=yes
1 name="HTTP" parent=ether1 flow=abc-http limit-at=128000 max-burst=0
queue=default priority=8 weight=1 allot=1514 bounded=yes
[admin@MikroTik] queue tree> print brief
Flags: X - disabled, I - invalid, D - dynamic
# NAME PARENT FLOW LIMIT-AT PACKETS BYTES
1 D A_Simple ether1 128000 0 0
0 HTTP ether1 abc-http 128000 0 0
[admin@MikroTik] queue tree>
Queue Applications
One of the ways to avoid network traffic ‘jams’ is usage of traffic shaping in large networks.
Traffic shaping and bandwidth allocation is implemented in the MikroTik RouterOS as queuing mechanism.
Thus, the network administrator is able to allocate a definite portion of the total bandwidth
and grant it to a particular network segment or interface.
Also the bandwidth of particular nodes can be limited by using this mechanism.
Further on, several examples of using bandwidth management are given arranged according to complexity:
The IP addresses and routes of the MikroTik router are as follows:
[admin@MikroTik] > ip address print
Flags: X - disabled, I - invalid, D - dynamic
# ADDRESS NETWORK BROADCAST INTERFACE
0 10.0.0.217/24 10.0.0.217 10.0.0.255 Public
1 192.168.0.254/24 192.168.0.0 192.168.0.255 Local
[admin@MikroTik] > ip route print
Flags: X - disabled, I - invalid, D - dynamic, J - rejected,
C - connect, S - static, R - rip, O - ospf, B - bgp
# DST-ADDRESS G GATEWAY DISTANCE INTERFACE
0 S 0.0.0.0/0 r 10.0.0.1 1 Public
1 DC 192.168.0.0/24 r 0.0.0.0 0 Local
2 DC 10.0.0.0/24 r 0.0.0.0 0 Public
[admin@MikroTik] >
Assume you want to limit the bandwidth to 128kbps on downloads and 64kbps on uploads for all hosts on the LAN. Bandwidth limitation is done by applying queues for outgoing interfaces regarding the traffic flow. It is enough to add two queues at the MikroTik router:
[admin@MikroTik] queue simple> add name=Down interface Local limit-at 128000
[admin@MikroTik] queue simple> add name=UP interface Public limit-at 64000
[admin@MikroTik] queue simple> print
Flags: X - disabled, I - invalid
0 name="Down" src-address=0.0.0.0/0 dst-address=0.0.0.0/0 interface=Local
limit-at=128000 queue=default priority=8 bounded=yes
1 name="UP" src-address=0.0.0.0/0 dst-address=0.0.0.0/0 interface=Public
limit-at=64000 queue=default priority=8 bounded=yes
[admin@MikroTik] queue simple> .. tree print
Flags: X - disabled, I - invalid, D - dynamic
0 D name="Down" parent=Local flow="" limit-at=128000 max-burst=20
queue=default priority=8 weight=1 allot=1514 bounded=yes
1 D name="UP" parent=Public flow="" limit-at=64000 max-burst=20
queue=default priority=8 weight=1 allot=1514 bounded=yes
[admin@MikroTik] queue simple>
Leave all other parameters as set by default. The limit is approximately 128kbps going to the LAN and 64kbps leaving the client's LAN. Please note, that the queues have been added for the outgoing interfaces regarding the traffic flow.
To monitor the traffic flow through the interface while doing file transfer, use the /interface monitor-traffic command:
[admin@MikroTik] interface> monitor-traffic Public once
received-packets-per-second: 9
received-bits-per-second: 4.32kbps
sent-packets-per-second: 6
sent-bits-per-second: 65.58kbps
[admin@MikroTik] interface> monitor-traffic Public once
received-packets-per-second: 7
received-bits-per-second: 3.36kbps
sent-packets-per-second: 10
sent-bits-per-second: 65.15kbps
[admin@MikroTik] interface> monitor-traffic Public once
received-packets-per-second: 11
received-bits-per-second: 5.66kbps
sent-packets-per-second: 7
sent-bits-per-second: 52.70kbps
[admin@MikroTik] interface>
If you want to exclude the server from being limited, add two queues for it with limit-at=0 (no limit) and move them to the top:
[admin@MikroTik] queue simple> add name=Serv_D interface=Local \
\... dst-address=192.168.0.17/32 limit-at=0
[admin@MikroTik] queue simple> add name=Serv_U interface Public \
\... src-address=192.168.0.17/32 limit-at=0
[admin@MikroTik] queue simple> print
Flags: X - disabled, I - invalid
0 name=Down src-address=0.0.0.0/0 dst-address=0.0.0.0/0 interface=Local
limit-at=128000 queue=default priority=8 bounded=yes
1 name=UP src-address=0.0.0.0/0 dst-address=0.0.0.0/0 interface=Public
limit-at=64000 queue=default priority=8 bounded=yes
2 name=Serv_D src-address=0.0.0.0/0 dst-address=192.168.0.17/32
interface=Local limit-at=0 queue=default priority=8 bounded=yes
3 name=Serv_U src-address=192.168.0.17/32 dst-address=0.0.0.0/0
interface=Public limit-at=0 queue=default priority=8 bounded=yes
[admin@MikroTik] queue simple> move 2 0
[admin@MikroTik] queue simple> move 3 1
[admin@MikroTik] queue simple> print
Flags: X - disabled, I - invalid
0 name=Serv_D src-address=0.0.0.0/0 dst-address=192.168.0.17/32
interface=Local limit-at=0 queue=default priority=8 bounded=yes
1 name=Serv_U src-address=192.168.0.17/32 dst-address=0.0.0.0/0
interface=Public limit-at=0 queue=default priority=8 bounded=yes
2 name=Down src-address=0.0.0.0/0 dst-address=0.0.0.0/0 interface=Local
limit-at=128000 queue=default priority=8 bounded=yes
3 name=UP src-address=0.0.0.0/0 dst-address=0.0.0.0/0 interface=Public
limit-at=64000 queue=default priority=8 bounded=yes
[admin@MikroTik] queue simple>
Example of Using Masquerading
If masquerading is used for the local address space 192.168.0.0/24 of
the client computers in the previous example setup,
then the outgoing traffic has masqueraded source address 10.0.0.217, i.e.,
the outgoing packets have external address of the router as the source.
If you use simple queues, as in the previous example, the queuing rule for incoming traffic should match the customer's local addresses, whereas the rule for outgoing traffic should match the router's external address as the source address. The previous example would work fine, but you cannot exclude the server from being limited.
To apply specific queuing for the server, use /ip firewall mangle to mark the packets originated from the server:
[admin@MikroTik] ip firewall mangle> add src-address=192.168.0.17/32 \
\... action=passthrough mark-flow=Serv_Up
[admin@MikroTik] ip firewall mangle> add in-interface=Local action=passthrough \
\... mark-flow=Local-all
[admin@MikroTik] ip firewall mangle> print
Flags: X - disabled, I - invalid
0 src-address=192.168.0.17/32:0-65535 in-interface=all
dst-address=0.0.0.0/0:0-65535 protocol=all tcp-options=any
icmp-options=any:any src-mac-address=00:00:00:00:00:00 limit-count=0
limit-burst=0 limit-time=0s action=passthrough mark-flow=Serv_Up
tcp-mss=dont-change
1 src-address=0.0.0.0/0:0-65535 in-interface=Local
dst-address=0.0.0.0/0:0-65535 protocol=all tcp-options=any
icmp-options=any:any src-mac-address=00:00:00:00:00:00 limit-count=0
limit-burst=0 limit-time=0s action=passthrough mark-flow=Local-all
tcp-mss=dont-change
[admin@MikroTik] ip firewall mangle>
Add a queue to the queue tree, which uses the flow mark:
[admin@MikroTik] queue tree> add name=Server parent=Public flow=Serv_Up
[admin@MikroTik] queue tree> add name=Workst parent=Public flow=Local-all \
\... limit-at=64000 bounded=yes max-burst=0
[admin@MikroTik] queue tree> print
Flags: X - disabled, I - invalid, D - dynamic
0 name=Server parent=Public flow=Serv_Up limit-at=0 max-burst=20
queue=default priority=8 weight=1 allot=1514 bounded=no
1 name=Workst parent=Public flow=Local-all limit-at=64000 max-burst=0
queue=default priority=8 weight=1 allot=1514 bounded=yes
[admin@MikroTik] queue tree>
Thus, we used queue trees for limiting the upload. Use the same simple queues as in the previous example for limiting the download.
Assume we want to emulate a 128k download and 64k upload line connecting IP network 192.168.0.0/24 as in the previous examples. But if these speeds are the best that you can get from your Internet connection, you may want to guarantee certain speeds to the 192.168.0.17 server so that your customers could download from and upload to this server with the speeds not dependent on the other traffic using the same channel (for example, we will guarantee this server the speed of 32k for each flow direction).
First of all, you should limit the interface speed:
[admin@MikroTik] queue tree> add name=Up parent=Public limit-at=64000 \
\... max-burst=0 bounded=yes
[admin@MikroTik] queue tree> print
Flags: X - disabled, I - invalid, D - dynamic
0 name="Up" parent=Public flow="" limit-at=64000 max-burst=0
queue=default priority=8 weight=1 allot=1514 bounded=yes
[admin@MikroTik] queue tree>
Next, mark the traffic from the FTP server. We will mark only TCP port 20
because that port is used to send and receive FTP data.
[admin@MikroTik] ip firewall mangle> add src-address=192.168.0.17/32:20 \
\... protocol=tcp mark-flow=Server_Up in-interface=Local
[admin@MikroTik] ip firewall mangle> print
Flags: X - disabled, I - invalid, D - dynamic
0 src-address=192.168.0.17/32:20 in-interface=Local
dst-address=0.0.0.0/0:0-65535 protocol=tcp tcp-options=any
icmp-options=any:any flow="" src-mac-address=00:00:00:00:00:00
limit-count=0 limit-burst=0 limit-time=0s action=accept
mark-flow=Server_Up tcp-mss=dont-change
[admin@MikroTik] ip firewall mangle>
The second mangle rule will match the rest of the traffic from the network:
[admin@MikroTik] ip firewall mangle> add src-address=0.0.0.0/0 \
\... mark-flow=Local_Up in-interface=Local
[admin@MikroTik] ip firewall mangle> print
Flags: X - disabled, I - invalid, D - dynamic
0 src-address=0.0.0.0/0 in-interface=Local
dst-address=0.0.0.0/0:0-65535 protocol=tcp tcp-options=any
icmp-options=any:any flow="" src-mac-address=00:00:00:00:00:00
limit-count=0 limit-burst=0 limit-time=0s action=accept
mark-flow=Local_Up tcp-mss=dont-change
[admin@MikroTik] ip firewall mangle>
Finally shaping the traffic:
[admin@MikroTik] queue tree> add name=Server_Up parent=Up limit-at=32000 \
\... max-burst=0 bounded=no flow=Server_Up
[admin@MikroTik] queue tree> add name=Local_Up parent=Up limit-at=0 flow=Local_Up
[admin@MikroTik] queue tree> print
Flags: X - disabled, I - invalid, D - dynamic
0 name="Up" parent=Public flow="" limit-at=64000 max-burst=0
queue=default priority=8 weight=1 allot=1514 bounded=yes
1 name="Server_Up" parent=Up flow="Server_Up" limit-at=32000 max-burst=0
queue=default priority=8 weight=1 allot=1514 bounded=no
2 name="Local_Up" parent=Up flow="Local_Up" limit-at=0 max-burst=0
queue=default priority=8 weight=1 allot=1514 bounded=yes
[admin@MikroTik] queue tree>
Thus, we used queue trees for limiting the upload. The download speed can be
limited the same way simply changing the interface names and matching the packets
destinated to the server (use 'external' server address if you are using DST-NAT)