Table of Contents

Introduction and requirements

NiceShaper works to give you optimal bandwidth utilization and simplify of configure traffic shaping in Linux Operating System which can be really complex. I always seek when creating solutions to makes them flexible, intuitive and clear. To makes them be tool for simplify what is to much complex and at once gives ability to configure everything what is worth to have control on.

It's possible to create an entry configuration without reading this documentation. It's thanks to example configuration files included in program package. Solid knowledge of networks is prerequisite. Keep in mind, when you get first look at included example configuration files next it is worth to look on this documentation unless you don't want to create optimal traffic shaping. NiceShaper really seeks but can't outguess all your needs so knowledge of IP network and Linux System is helpful. Still don't be fear when you do not understand everything here, keep in mind that this documentation is far beyond your needs for the first time. After all NiceShaper goal is simplify what in HTB is hard and at once to gives you additional features.

In minimal configuration you need router with your favorite Linux System, with installed iptables and compiled in kernel HTB and SFQ packet scheduling algorithms and U32 kernel filter.

Hereinafter HTB class means HTB packet scheduling class and NiceShaper class is in NiceShaper level. Queuing algorithms means SFQ and others classless queueing disciplines. Kernel filters means U32 and FW and it is not the same as NiceShaper filters which are on NiceShaper level of configuration.

In most popular Linux Distributions HTB, SFQ, U32 and FW filter are compiled in kernel or as modules by default. Iptables program too but apart from IMQ.

For purpose of this documentation we assume that we have a router equipped with two interfaces. WAN uplink is connected to interface eth0 and local LAN to eth1. Public address of the router is 198.51.100.100 and LAN network is 192.168.0.0/24.

As we discount poor ingress qdisc, it's fundamental that egress shaping occurs on outgoing interface, so, shaping of download traffic has to be placed on eth1 interface. On the other hand, shaping of upload traffic has to be placed on eth0. This is because the downloaded packets incomes via eth0 interface and outgoes to the local network via eth1 interface (for above assumptions!). Respectively, the uploaded packets incomes from local network via eth1 and outgoes to the internet via eth0 interface. It's all valid for above assumptions and may be required to be adjusted in configuration prepared for the other environment!

Proper choose of interface is crucial to define hosts, classes, and some of configuration directives. For example, if LAN network is hidden behind NAT and we need to shape upload by packet marking method then we need to put it on eth0 by mark-on-iface eth0 directive.

I would like to add that upload shaping of NATed network is also possible with IMQ interfaces. Both, packets marking and IMQ interfaces are documented in hereafter.

Remember as these examples above are valid for previous assumes not necessarily for your router.

For the end of the introduction. Iptables and Iproute do not understand concept of interface aliases, so in NiceShaper configurations you should omit colon and alias number from interface. It's safe. On the other side, VLANs written as ethX.vid are fully supported.

Complementing about the VLANs in Linux. If you create the tagged sub-interface and run traffic shaping on it, you should completely abandon the untagged interface. Otherwise, you may find that the kernel queues tagged data frames twice. Once on tagged sub-interface and second time on physical one.

NiceShaper has good scalability on the routers used by no more than half thousand of hosts - although there are notices about the NiceShaper usage in bigger networks. In such bigger networks there is rather a hash table requirement. Hash tables are unsupported so far, because of being too inflexible for NiceShaper needs.

Please be aware that NiceShaper has been written for polish Network Administrators since start of the project, so there is a forum on the web page but it is useless for you because of polish language. Also you should realise that this documentation can be strongly language incorrect and even incomplete. If you have corrections and share with me i will be appreciate.

Installation

Compilation and installation dependencies are: c++ compiler from gcc package, C and C++ standard libraries, and make utility. Unpacked package have to be compiled with omitted ./configure, because NiceShaper is Linux only software.
$ bunzip2 niceshaper-%{wersja}.tar.bz2
$ tar xf niceshaper-%{wersja}.tar
$ cd niceshaper-%{wersja}
$ make
$ su
# make install
Make install command copies the compiled binary to the /usr/local/bin directory. The binary path may be changed using BINDIR environment variable. Make install command creates important directories: /etc/niceshaper, /var/lib/niceshaper, and /usr/share/doc/niceshaper. Example configuration files are copied to the /etc/niceshaper directory. If configuration files are found there then the actual files are copied with "-dist" postfix. The documentation and the rest of package content is copied to /usr/share/doc/niceshaper.

Configuration

Configuration files syntax.

Common and required configuration files are /etc/niceshaper/config.conf and /etc/niceshaper/class.conf.
include file path - used in these files can include next ones. This directive accepts absolute and relative paths (by default to /etc/niceshaper/).
There are a few different syntax types for directives:
Syntax type #1) directive value - It is directive with one value only, e.g.:
rate 128kB/s
Syntax type #2) directive value [value] - It is directive with one or number of values separated with whitespaces, e.g.:
mark-on-ifaces eth0 eth1
Syntax type #3) directive parameter value [parameter value] - It is directive with one or number of parameter-value pairs, e.g.:
status file /var/www/niceshaper/status.txt unit kB/s file-mode 644
Order of above directives in a scope does not matter.

When NiceShaper is starting, configuration parser splits type #2 and #3 directives into separated lines, so if you prefer you can write these directives also like that:

Syntax type #4) match directive looks and works like type 3, but it's additional type. This is because you can not split it as it may change logic. Only order changing is permitted.
This is example of filter which classify packets with source address 10.10.10.5 and destination in network 192.168.0.0/29.
match srcip 10.10.10.5 dstip 192.168.0.0/29
After split, gives 2 autonomous filters:
These two filter are properly in the mining of NiceShaper configuration syntax, but they are not identical with source filter.
Syntax type #5) Directives class is the most inflexible one, it can not be split and order changes at all, e.g.:
class dl eth1 pc55
In addition to differences in syntax, directives are also divided according to the scope of operation, for directives of global section, directives of functional sections and directives of classes. When it comes to the directives of classes, all except the class header and filters can be placed in the functional sections configurations, providing default values for all classes included in the section. In the rest of the documentation, the three groups are clearly separated in groups.

The basic units of capacity is b/s (bits per second), and B/s (Bytes per second). The prefixes are k (kilo) and M (mega). The suffix '/s' is not compulsory and the distinction between rate and the amount of transferred data is based on the context of use. Default unit of bandwidth is b/s (bits per second).

NiceShaper gives you some special chars: Char hash "#" is using for comment out the rest of line. Comment block is "<# comment #>" and is using to affect some piece of configuration line. Commented configuration is useless. Char ";" means end of line, it gives you way to minimize length of config files by write a lot of semicolon separated directives in one line.

Each change in configuration require to restart NiceShaper.

Remember, all examples are only examples and might not be optimal for you, although they are good point to start.

Main configuration file - with example

Main configuration file /etc/niceshaper/config.conf is divided into sections. One called global, and minimum one, and in practice two or even more functional sections. Functional section is NiceShaper classes container configured for grouping classes and shape packages on appointed interfaces. Functional section reflect one direction of traffic so each the internet access uplink needs 2 functional sections to be fully controlled. Each functional section has got individual configuration, contained classes and defaults for these classes. Section can contain classes that works on various interfaces. Any number of functional sections may be placed on an interface.

An example of main configuration file:

Global section directives:

Functional section directives:

You can place any classes parameters in functional section configuration, these becomes to be defaults for whole classes in a section.

Hosts and classes in NiceShaper

File /etc/niceshaper/class.conf holds list of hosts definitions and/or classes with their configuration. We can assume that NiceShaper class is an extended HTB class. Each NiceShaper class is built by a header, at least one filter and eventually additional options. Corresponding HTB class is created when NiceShaper notices activity and removed after a set period of inactivity. This gives optimal value of the rate parameter as only active HTB classes exists at the same time. Each packet leaving the interface is classified using filters to the first matching class.

It is vitally important to define classes for whole traffic potentially observed on a link. If not there will appear traffic impossible to control, e.g, if not all hosts in local network will be assigned to classes, these will steal bandwidth from known classes.

Network host:

NiceShaper classes are advanced and flexible tool but simple directive named 'host' is in most cases sufficient and clear way to create fairly traffic shaping for all or most of common hosts in local network. It's the one of element thanks that NiceShaper offers such comfort. Host directive can give you a way to configure traffic shaping in NiceShaper without get to know classes.

To define host you need: list of section in which host must be placed paired with interface on which traffic shaping occurred, ip address and assigned name. Host directive may be placed into one or more of running sections. NiceShaper translate host directive to classes automatically for you.

We can assume that host directive is some kind of NiceShaper macro.

NiceShaper host is placed in one line with syntax given below:

host section interface [section interface] ip name
For example two host:
These host directives will be translate, by NiceShaper configuration parser, to these classes:
NiceShaper inserts filter test dstip or srcip depending on section mode.

Host are pointed out only by ip address. When you need another filtering tool or overwrite default parameters use classes instead of hosts.

Be carefully as above example of host pc11, which is included only in one section, is only example of how host directive syntax works. You should not leave any host traffic without control especially on upload.

Class structure

In the current version NiceShaper provides 4 types of classes. These are the standard class and a classes for special purposes: virtual, wrapper and do-not-shape. Last 2 types of classes are described in "Router's self generated traffic shaping".
Class definition starts with a header and ends with a next class header or end of file.

Header of standard class and virtual class is as follows:

class|class-virtual section interface name
Where:

section - section to whose a class belongs.
interface - network interface on which a HTB class resides.
name - class name.

Each class must contains one or more filters:

match test <test> [test <test>]
Filters classifies packets to classes. This applies to packets outgoing from class interface only (egress queue). Kernel filters are created when NiceShaper starts and, as you know after read of "Cooperation with iptables", iptables rules if needed. More than one test can be joined in one filter to be more specific on assign traffic to classes. Classes can contains many filters to aggregate more traffic.

An example of the simplest class is as follows:

Everywhere in configuration you can use semi-colon instead of new line:

Class parameters:

Each option of classes can be placed in a functional section configuration, and becomes to be defaults to not repeat the same values for each class. Next if you wish these defaults can be override in classes with individual values.

Basics filters tests:

srcip and dstip can match single ip address or network with subnet mask. In second case allows bits counter notation e.g. /24 or doted notation e.g. 255.255.255.0. Mask can be non-continuous (e.g. 255.255.128.255) but in this case needs mark-on-ifaces because of it needs fw filter instead of u32.

Example filters:

Tests requiring packets marking on a class interface:

Iptables is equipped with a huge number of filters that are unfortunately not feasible using u32 filter. Therefore, these filters require packet marking by mark-on-ifaces directive. Each packet captured and marked by iptables can now easily be enqueued to the appropriate HTB class thanks to received mark value.

CAUTION! Some of these filters may need additional kernel and iptables features compiled in.

Macros of the class.conf file

Macros in a classes file are introduced to improve create of a lot of classes if they are similar. Macro iterates in a loop to duplicate specified area. Macro is built with a header, content and ending mark. Header is built by curly brackets that contains macro type and parameters. Content is ordinary class file area. In content there you may put some special chars, like $ (dollar) and % (percent). These chars are replaced by expected number or string. Macro ending mark is curly brackets that contains slash.

In actual NiceShape version are implemented 3 macro types - sequence, foreach-elem, foreach-pair.

sequence macro:

{sequence from to} content {/} - sequence macro generates ascending numeric values in specified range. Values "from" and "to" must be positive integers, in range from 0 to 65535. Values are inserted in place of dollar special character.
Example of sequence usage:
Macro generates:

foreach-elem macro:

{foreach-elem list} content {/} - foreach-elem macro gets each element from given list and inserts it in place of dollar special char. List must contain numeric or text values separated by whitespaces.
Example of foreach-elem usage:
Macro generates:

foreach-pair macro:

{foreach-pair list_of_pairs} content {/} - foreach-pair macro gets pairs of values from list. List must contain numeric or text values. Pairs are separated by commas, elements in pair are separated by whitespace. First element in pair is named a key and is inserted in place of percent char. Second element in pair is inserted in place of dollar.
Example of foreach-pair usage:
Macro generates:

Packet marking

Packet marking is a procedure to assign a virtual tag to a packet. Thanks to that it is still possible to identify packet sender after change the source address by SNAT (IP masquerade). The value is assigned by iptables, maintained by the Linux kernel and recognizable by iptables and the kernel FW filters. FW kernel filters classifies packets into a appropriate HTB class using test based on a virtual mark instead of a packet header.

Marking packages mostly do not require patching iptables, iproute or kernel. NiceShaper manages all completely automatically so it's very easy to use this mechanism. Therefore, packet marking is a good way to avoid use of IMQ interfaces, which are comfortable but required patching the operating system important components. Marking allows you to comfortable control upload of your masqueraded computers.

So if packet marking on interface is turned on by mark-on-ifaces directive then NiceShaper for each class working on this interface will assign a unique mark value and U32 kernel filter on that interface will be replaced with FW. All this is done completely automatically. However if necessary it is possible to make intervention in this process thanks to a set-mark parameter. The value of set-mark must be unique for each class.

Iptables so NiceShaper too accepts mark values from 0 to 4294967295 (32bit unsigned value) written in decimal or hex started with 0x.

Usage of IMQ interfaces

NiceShaper supports IMQ compiled in AB mode (after NAT in PREROUTING chain and before NAT in POSTROUTING chain).

From the configuration of NiceShaper IMQ interfaces are the same way as physical ones. You can forget about their virtual, mostly, with complete freedom to mix classes on physical and IMQ interfaces.

Niceshaper filters require to indicate a physical interface by out-iface test (or in-iface if iptables hook is changed to PREROUTING).

Example snip from class.conf file:

NiceShaper automatically redirects traffic to IMQ interface. This behavior is configurable by iptables imq-autoredirect directive in global section. If you disable the automatic redirect to the IMQ you need to make it your self for NiceShaper by something like iptables ... -j IMQ --todev imqX.

Triggers

Trigger is mechanism that allows you to automatically change the values of some class when a defined case occurs. In the current version NiceShaper two triggers types are implemented, alter and quota.

Controllable parameters of the class:

Control parameters of triggers:

Triggers are class directives, most convenient way is to define them within the section, such as:
The quota trigger have higher priority than the alter trigger if both have to be enabled. Triggers works only for standard-class. Counters of trigger quota between running the program are stored in files section_name.quota in /var/lib/niceshaper directory. These files are updated when NiceShaper is stopping and during his work at 5 minutes intervals. If this file write fails, counters will be irretrievably lost!

Router's self generated traffic shaping

Traffic generated by the router for his own needs is mostly negligibly small and often doesn't need to be shaped. But, in real world, router with Linux often serves file shares to the local network (simpler scenario) or even serves some network services to the internet (harder scenario).

Due to the fact that the traffic shaping takes place at the interface from which packets outgoings a machine, shaping incoming traffic (both on Internet and LAN side) is difficult and requires usage of IMQ interfaces.

To practically deal with this topic, the most convenient is the breakdown of the issues on four separate scenarios.

In all scenarios it's required to add from-local or to-local test to filters. In combination with IMQ interfaces there is additional in-iface test requirement, eventually out-iface for outgoing traffic. It's important to let NiceShaper know physical interface at which packet arrives or leaves and to distinguish WAN side from LAN side traffic.

Keep in mind that in most cases this purpose classes should be on the top of the list of all your classes.

Remind assumption that WAN is connected to eth0 and LAN is connected to eth1. Public address of router is 198.51.100.100, second public address for services is 198.51.100.101, private address is 192.168.0.1, and private network is 192.168.0.0/24.

Traffic between the router and the local network (uplink bandwidth is not involved).

Traffic between a router and a local network requires to be not shaped or shaping should be minimised. This traffic shouldn't be shaped to the fact that local ethernet connection is mostly fast. Another big mistake would be accounting this traffic as uplink usage with the traffic generated by hosts. Classes in #1 and #2 scenarios (router with local network traffic related) have to be wrapper or do-not-shape types.
Wrapper class and do-not-shape class doesn't belong to any section so their header is not the same as standard class has:
class-wrapper|class-do-not-shape interface name

1. Router -> Localnet - Router sends traffic to the local network.

Traffic in this scenario is for example, local file or ftp server, local imap or pop3 server, etc.

Using do-not-shape type class - traffic is local so you do not want to shaping and accounting:

Using wrapper type class - traffic is local but we have WiFi bridge on LAN and we are afraid to overload, so we process shaping but with statically allocated bandwidth and still do not account to any section:

2. Router <- Localnet - Router gets traffic from the local network.

Traffic from the LAN to the router. For example, sending e-mail via the local SMTP server, place the files on a local file server, etc.

Using do-not-shape type class - NiceShaper use POSTROUTING chain at iptables and doesn't use ingress shaping at kernel QOS. So, there is no requirement to create any do-not-shape type classes as traffic described in this scenario is uncontrolled in default configuration.

Using wrapper type class - but, if we want to use wrapper type class (analogically to scenario #1), we need to use the IMQ device.

Traffic between the router and the internet (using uplink bandwidth).

Putting services on WAN side should be avoided. If SNAT and internet services on WAN side are used, services should use another public IP address. In this situation requirements of two public addresses is forced because there is no way to distinguish local from forwarded (NATed) traffic in the PREROUTING chain of mangle table. This distinguish is possible in INPUT chain (simplifying), but there is no way to redirect traffic to IMQ devices.

3. Router -> Internet - Router sends traffic to the internet.

Packets generated locally while sending to the internet e.g., mail service, web pages and other services on the router.

In this examples, two classes are created to separate traffic of two different services.

Usage of class of mode upload and filter with a from-local test does not require extensive commentary here. Provide belief that traffic from the router to the internet is properly accounted.

4. Internet => Router - downloading traffic from the internet to the router.

This scenario occurs when a router retrieves traffic from the internet for e.g. the web pages retrieves by a proxy server or system updates that may steal a bandwidth from clients.
Using a class at download mode section and a filter to-local don't require extensive commentary here too.

By default, the classes that belong to download mode puts their filters in the chain targeted from POSTROUTING where our packets will never arrive, so a to-local test creates a cloned rule in the PREROUTING chain.

Advanced topics

Cooperation with iptables

In some cases NiceShaper uses iptables. In that case it creates rules in mangle table on starting and flushes them on stopping. You should avoid this cases whenever possible because rules for many NiceShaper classes generates additional system load. If at least one NiceShaper class in section required iptables rule, then rules are created for all NiceShaper classes in this section. And furthermore worst, iptables rules are created for each section with the same mode parameter value.

To be precise, iptables rules are required and created for the following cases:

Reading any of the chains not held more than 10 times a second. If a section is reloaded again before this time it will use the previous cached values.

Until version 1.0 NiceShaper used iptables rules as fundamental. Rules was always created and NiceShaper looked for activity and accounted traffic. From version 1.2pre1 iptables stopped to be fundamental.

Cooperation with HTB

NiceShaper creates QOS framework on each controlled interface. This framework looks like on diagram below. It contains HTB classes, queuing algorithms and kernel filters. HTB classes creates tree hierarchy by which traffic is propagated as you see.

NiceShaper doesn't use the tc program. It communicates, using a Netlink protocol, directly with the Linux kernel (the code that provides this functionality is partially based on iproute code). It's the most efficient way to manage QOS objects.

HTB Diagram
On bottom level of tree there are HTB classes which becomes to be parents for NiceShaper classes defined by you. These HTB classes share full speed of interface but do not borrow from each other.

These special HTB classes act as:

1) Each section which works on an interface gets HTB class with throughput equal to section speed parameter value. This HTB class becomes parent for each child HTB classes within that section. One child HTB class is automatically created as Waiting Room. Traffic classified by filters in first step is directing into Waiting Room. In second step when section is reloading filter are modified to directing traffic to newly created HTB class. After some time of inactivity this newly created HTB class is removed and kernel filter directing traffic to Waiting Room again. This time of inactivity is set by hold parameter which has 30 seconds default. In this solution in one time there are only us much HTB classes as activated NiceShaper classes, thanks for that HTB class rate parameter value is optimal.

2) This HTB class as parent for do-not-shape and wrapper classes is created when on interface works one of or both of specified class types. In case of do-not-shape class it is true only with iface do-not-shape-method safe option. This HTB class throughput is calculated from iface speed minus sections speed minus fallback class.

3) HTB fallback class works with whole traffic outgoing from interface and unclassified by filters. In proper configuration this class should be idle, otherwise you should looking for lack in defined classes and filters.