syslog-ng reference manual

One of the new features of syslog-ng 2.0 is the support for messages originating from various timezones. Although this problem is not simple as the original syslog protocol does not include timezone information, syslog-ng tries to solve the problem first by extending the syslog protocol to include this information and second by giving finegrained control to the administrator to supply timezone information for legacy devices which do not support the protocol extension.

Timezone information is associated with messages entering syslog-ng using the following algorithm:

There is another case when message timestamps are formatted: when using macro expansion in destination filenames, in this case the local timezone is used.

A source is a collection of source drivers, which collect messages using a given method. For instance there's a source driver for AF_UNIX, SOCK_STREAM style sockets, which is used by the Linux syslog() call.

To declare a source, you'll need to use the source statement in the configuration file with the following syntax:

	  source <identifier> { source-driver(params); source-driver(params); ... };
	

The identifier has to uniquely identify this given source and of course may not clash with any of the reserved words (in case you had a nameclash, simply enclose the identifier in quotation marks)

You can control exactly which drivers are used to gather log messages, thus you'll have to know how your system and its native syslogd communicate. Here's a introduction to the inner workings of syslogd on some of the platforms I tested:

Each possible communication mechanism has the corresponding source driver in syslog-ng. For instance to open a unix socket with SOCK_DGRAM style communication you use the driver unix-dgram, the same with SOCK_STREAM style - as used under Linux - is called unix-stream.

	  source src { unix-stream("/dev/log"); internal(); udp(ip(0.0.0.0) port(514)); };
	

Each driver may take parameters, some of them are required, others are optional. The required parameters are positional, meaning that they must be specified in a defined order. A unix-stream() driver has a single required argument, the name of the socket to listen to, and several optional parameters, which follow the socket name. Optional arguments can be specified in any order and must have the form option(value).

For a complete descriptions on the above drivers, see Chapter 3, Reference

Filters perform log routing inside syslog-ng. You can write a boolean expression using internal functions, which has to evaluate to true for the message to pass.

Filters have also a uniquely identifying name, so you can refer to filters in your log statements.

Syntax for the filter statement:

	  filter <identifier> { expression; };
	

An expression may contain parentheses, the boolean operators "and", "or" and "not", and any of the functions listen in Table 3.12, “Available filter functions in syslog-ng”.

	  filter f_blurp_deny { host("blurp") and match("deny"); };
	

For a complete description on the above functions, see Chapter 3, Reference.

In earlier revisions of syslog-ng there was a special filter identifier, "DEFAULT", which matched all not-yet-matched messages. This could make your configuration much simpler and easier to manage. This feature was removed in syslog-ng 1.5.x, and a more powerful idea was introduced. For more details consult Section 5, “Log paths”.

A destination is where log is sent if filtering rules match. Similarly to sources, destinations are comprised of one or more drivers, each of which define how messages are handled. To declare a destination in the configuration file, you'll need a destination statement, whose syntax is as following:

	  destination <identifier> { destination-driver(params); destination-driver(params); ... };	
	

For detailed list of the supported drivers, see Chapter 3, Reference.

Unlike in previous versions of syslog-ng, where template formats had to be defined for every destination, syslog-ng 2.0 allows you to define common templates in advance, and refer to them from every object that needs a template to operate. For example:

          template t_filetmpl { template("$ISODATE $HOST $MSG\n"); template_escape(no)); };
          
          destination d_file { file("/var/log/messages" template(t_filetmpl); };
        

Templates can reference one or more macros as described in Section 4, “Macros”

In the previous chapters we learnt how to define sources, filters and destinations. We'll need to connect those components together, which is accomplished by the log statement. The needed syntax is here:

	  log { source(s1); source(s2); ... 
	        filter(f1); filter(f2); ... 
	        destination(d1); destination(d2); ... 
	        flags(flag1[, flag2...]); };
	

Any message coming from any of the listed sources, matching the all the filters are sent to all listed destinations. Log statements are processed in the order they appear in the config file.

By default all matching log statements are processed, therefore a single log message might be sent to the same destination several times, given that destination is listed on several log statements.

This default behaviour can be changed by the flags() parameter.

There are several options you can specify, which modifies the behaviour of syslog-ng. For an exact list of possible options see the Chapter 3, Reference. The general syntax is here:

	  options { option1(params); option2(params); ... };
	

Each option may have parameters, just like in driver specification.

The following drivers may be used in the source statement, as described in the previous chapter.

Some of the parameters are common for all sources which affect the way messages are parsed.

	  Declaration: internal()
	

	    source s_local { internal(); };
	  

	  Declaration: 
	    unix-stream(filename [options]);
	    unix-dgram(filename [options]); 
	

	    # source declaration on Linux
	    source s_stream { unix-stream("/dev/log" max-connections(10)); };

	    # source declaration on BSDs
	    source s_dgram { unix-dgram("/var/run/log"); };

	  

	  Declaration:
	    tcp([options]);
	    udp([options]);
	

	    source s_tcp { tcp(ip(127.0.0.1) port(1999) max-connections(10)); };
	    source s_udp { udp(); };
	  

	    #!/bin/sh
	    tail -f logfile | logger -p local4.info
	    

	  Declaration:
	    file(filename);
	

	    source s_file { file("/proc/kmsg" flags(kernel); log_prefix("kernel: "); };
	    source s_follow { file("/var/log/apache/access.log" flags(no-parse) log_prefix("apache: " follow-freq(1))); };
	  

	  Declaration:
	    pipe(filename);
	

	    source s_pipe { pipe("/dev/log" pad_size(2048)); };
	  

	      source s_stream { sun-streams("/dev/log" door("/etc/.syslog_door"); };
	    

Destination drivers output log messages to somewhere outside syslog-ng: a file or a network socket.

Some of the parameters are common for all destinations which affect the way messages are formatted, sent.

	    destination d_file { file("/var/log/messages" ); };
	  

	    destination d_file {
	            file("/var/log/$YEAR.$MONTH.$DAY/messages"
	                    template("$HOUR:$MIN:$SEC $TZ $HOST [$LEVEL] $MSG $MSG\n")
	                    template_escape(no)
	            );
	    };
	  

	  Declaration:
	    pipe(filename);
	

	    destination d_pipe { pipe("/dev/xconsole"); };
	  

	  Declaration: 
	    unix-stream(filename [options]);
	    unix-dgram(filename [options]); 
	

	    destination d_unix_stream { unix-stream("/var/run/logs"); };
	  

	  Declaration:
	    tcp(host [options]);
	    udp(host [options]);
	

	    destination d_tcp { tcp("10.1.2.3" port(1999); localport(999)); };
	  

	  Declaration: 
	    usertty(username);
	

	    destination d_usertty { usertty("root"); };
	  

	  Declaration: 
	    program(commandtorun);
	

	    destination d_prg { program("/bin/cat >/dev/null"); };
	  

The following functions may be used in the filter statement, as described in the previous chapter.

There are some parts in syslog-ng (destination filenames and message content templates) that might refer to one or more macros, which get expanded as a message is processed. The table below summarizes the available macros in syslog-ng.

Macros can be included by prefixing the macro name with a '$' sign (without the quotes), just like in Bourne compatible shells. A new syntax of using braces around macro names is also added to syslog-ng 2.0 ("$MSG" and "${MSG}" is equivalent).

The following options can be specified in the options statement, as described in the previous chapter.

Syslog-ng 2.0 is a complete reimplementation of syslog-ng 1.6, and does not use the mark and sweep garbage collector at all. So the garbage collector parameters (gc_idle_threshold, gc_busy_threshold) are still accepted but are ignored.

When there are a lot of parallel connections to syslog-ng the amount of time required to prepare for a single poll loop iteration is significant. Since the arrival of every single log message triggers a new iteration, the CPU usage of syslog-ng might increase significantly even if the number of messages received is otherwise low. The solution is to add a fixed latency to message processing to wait some messages to arrive and to process them in blocks. This can be accomplished by changing the value of time_sleep() to a non-zero value which specifies this latency in milliseconds. It is not recommended to make it larger than 100ms, as that might skew timestamps and might slow syslog-ng too much and cause messages to be dropped. An alternative solution is to avoid tcp() and unix-stream() sources and use the DGRAM versions, udp() and unix-dgram() respectively.

The value of fetch_limit() and log_fifo_size() must also be sized accordingly when changing this value.

Syslog-ng always reads its incoming log channels to prevent your running daemons from blocking. This may result in lost messages if the output queue is full.

It's therefore important to set the output queue size (termed in number of messages), which you can do globally, or on a per destination basis.

	  options { log_fifo_size(1000); };
	

or

	  destination d_messages { file("/var/log/messages" log_fifo_size(1000)); };
	

You should set your fifo size to the estimated number of messages in a message burst. If bursts extend the bandwidth of your destination pipe, syslog-ng can feed messages into the destination pipe after the burst has collapsed.

Of course syslog-ng cannot widen your network bandwidth, so if your destination host lives on a noisy network, and your logtraffic extends the bandwidth of this network, syslog-ng can't do anything. It'll do its best however.

The sync parameter doesn't exactly do what you might expect. As you have seen messages to be sent are buffered in an output queue. The sync parameter specifies the number of messages held in this buffer before anything is written.

Note that it doesn't write all buffered messages in one single chunk, it writes each distinct message with a single write() system call.