iptables NFLOG support in Vuurmuur

Overview

Vuurmuur is a firewalling application which utilizes Linux iptables. Vuurmuur consists of two daemon processes:

• vuurmuur: The daemon that initializes the firewall rules and updates them when required to do so by the configuration program
• vuurmuur_log: The daemon that translates logging data to human readable form.

Both daemons can be controlled with a user front end called vuurmuur_conf, which can also display the human readable logging data in an easily interpretable way.

Logging

iptables is the command through which administrators can add or delete firewall rules to the iptables subsystem in the Linux kernel. Optionally, the kernel can be instructed to ‘log’ when a specific rule gets triggered. Quite frequently this logging is done through the syslog daemon, which in turn will write a line in the syslog files.

The syslog files are not always the best place to store information as syslog clutters a lot of information in these files. The loglines in syslog are also not always very easy to read. The vuurmuur_log daemon parses the syslog lines and translates IP addresses and portnumbers to vuurmuur zones and service names, aggregating lots of numbers into data that’s easily visualized.

Netfilter Logging

Netfilter is the packet filtering framework in the Linux kernel since the 2.4.x version. With netfilter it’s also possible to access network packets from userspace instead of only inside the kernel. This can be done for any packets but for Vuurmuur we’re mostly interested in packets that would be logged after they’ve hit an iptables rule with a logging target. The library to deal with packets that would be logged is the libnetfilter_log library and most of this article will describe the technique used to use the information contained in the packets to build human readable vuurmuur loglines.

LOG/ULOG/NFLOG targets

Using the syntax ‘-j LOG’ an iptables administrator can cause the iptables framework to ‘jump’ to a specific target should a packet match the criteria in a rule. The keyword ‘LOG’ is in this case a special, non-terminating target which means to ‘write a line to syslog and continue processing the packet’. There are two other non-terminating targets used for logging purposes:

• ULOG: A first implementation of of sending packets to userspace.
• NFLOG: A newer implementation of sending packets to userspace

By logging packets that meet the logging rule’s criteria to this intermediate userspace buffer, a program could attach to this buffer, interpret the packets as they come in, and perform any neccesary action. For vuurmuur we’ve chosen to use the NFLOG target as an alternative to the already implemented LOG target.

Documentation for Netfilter

On the subject of the userspace programming of netfilter, the existing documentation is sparse. Some excellent documentation has been written on how to write kernel modules for the netfilter core but parts of that documentation are simply not applicable for userspace programming. That’s why I decided to try and make a tutorial of this article, hoping that other programmers would be encouraged to do more in userspace.

Doing this without documentation was a difficult process and I’m sure I made some mistakes along the line, so feel free to comment where neccesary so I can make this document (and perhaps my own code) better and more useful.

Using the Proper Libraries

Vuurmuur uses the autoconf tooling to determine what capabilities an operating system has before compiling the software. Not being very familiar anymore with the latest and greatest autoconf macro’s, I dug into the ulogd2 source code to find some good way to test for, and link in, the existence of the required libraries. The required libraries are:

• libnfnetlink: low level kernel / userspace interface used by libnetfilter_log
• libnetfilter_log: programmer’s library

All I did was to add the following lines to vuurmuur’s configure.in script (borrowed from ulogd2′s configure.in):

dnl Check for the right nfnetlink version
LIBNFNETLINK_REQUIRED=0.0.39
LIBNETFILTER_CONNTRACK_REQUIRED=0.0.95
LIBNETFILTER_LOG_REQUIRED=0.0.15

PKG_CHECK_MODULES(LIBNFNETLINK, libnfnetlink >= $LIBNFNETLINK_REQUIRED,, AC_MSG_ERROR(Cannot find libnfnetlink >= $LIBNFNETLINK_REQUIRED))

PKG_CHECK_MODULES(LIBNETFILTER_CONNTRACK, libnetfilter_conntrack >= $LIBNETFILTER_CONNTRACK_REQUIRED,, AC_MSG_ERROR(Cannot find libnetfilter_conntrack >= $LIBNETFILTER_CONNTRACK_REQUIRED))

PKG_CHECK_MODULES(LIBNETFILTER_LOG, libnetfilter_log >= $LIBNETFILTER_LOG_REQUIRED,, AC_MSG_ERROR(Cannot find libnetfilter_log >= $LIBNETFILTER_LOG_REQUIRED))

Re-running autoconf again on my system then results in -lnfnetlog -lnetfilter_log to be added to the link command through the autoconf magic.

Restructuring vuurmuur_log

Parsing the syslog lines was at first the only method vuurmuur uses to generate its own human readable loglines. As a consequence, most of the coding logic was fixed in the main vuurmuur_log.c file although some code had been put in its own file logfile.c. I moved large chunks of code to more logical places, bundling logical functionality together in C files, so I could add the nflog functionality as its own logical block next to the syslogging functionality. While this is not so relevant for hacking with libnetfilter, it creates a proper context. I will include a full listing of nflog.c and refer to line numbers to document what I’ve done and why I thought that was neccesary.

I added a few new files to vuurmuur:

• nflog.c and nflog.h: nflog.c contains the functionality for dealing with the NFLOG target successfully while nflog.h contains the relevant function prototypes.
• stats.c and stats.h: While working on restructuring the main program, I found that there are some statistics being collected that could potentially be used in somewhat more meaningfull ways so I made a start to put this functionality in its own distinct files.

Source Code Review

You can view the source code in a new window by clicking on this link. The very first thing you need to play with libnetfilter_log is to include the header file the way we see it done in line 32:

   32 #include <libnetfilter_log/libnetfilter_log.h>

Even though this is quite common for any library use, it is not documented anywhere. The libnetfilter_log.h contains all the function prototypes for the library. libnetfilter_log itself uses the nfnetlink library so the header files for this (lowlevel) library are included inside libnetfilter_log.h. To be able to include the libnetfilter_log.h header file, you also need to actually have this file! On Debian (which I run at my build system), header files for libraries are often part of separate development packages. For libnetfilter_log the development package is called:

omega:~# dpkg -S /usr/include/libnetfilter_log/libnetfilter_log.h
libnetfilter-log-dev: /usr/include/libnetfilter_log/libnetfilter_log.h

So make sure you have the libnetfilter-log-dev package installed before you try your own hacking.

Binding to the nflog group

On line 250 is the initialisation function. When an NFLOG target is added to an iptables rule it is possible to include a prefix similar to adding a prefix to a regular LOG target. It is however also possible to include an ‘nflog-group’ on the iptables command line, here’s an example of a line vuurmuur generates:

/sbin/iptables -t filter -A FORWARD -m limit --limit 20/s --limit-burst 40 -j NFLOG --nflog-prefix "vrmr: DROP fw policy "  --nflog-group 8

Using this –nflog-group option, we can let the netfilter core route log messages to different applications, each listening to messages for a specific group. vuurmuur_log reads the configured group from the same configuration file vuurmuur reads it, so vuurmuur_log always listens to netfilter messages generated by rules created by vuurmuur. It is important to realize this as apparently only one application can bind to a specific nflog group. I’ve been experimenting with ulogd2 a lot as well, mostly since it was the only well written nflog client I could find, but when setting up ulogd2 to listen to messages at the same nflog-group as vuurmuur_log, I found that mysteriously the nflog_bind_group() call in vuurmuur_log would fail. In this case, nflog_bind_group() simply returns a 0, which often is not considered an error return value, but it’s also most definitely not a struct nflog_g_handle * like we expect to get back in case of no error. So when you have any other processes listening to the nflog-group in use by vuurmuur, you will find an error message “nflog_bind_group error, other process attached?” in your vuurmuur_log error log.

Another thing worth mentioning here is that in vuurmuur_log we want to see the whole packet in the callback handler (line 273):

if (nflog_set_mode (qh, NFULNL_COPY_PACKET, 0xffff) < 0)

The NFULNL_COPY_PACKET option specifies to the netfilter core that it will copy the whole packet, data and header, into the buffer read with recv() in line 292.

Finally the subscribe_nflog() function does an nflog_callback_register() call to add our createlogrule_callback() function as a callback for nflog_handle_packet(). The term ‘callback’ sounds to me like some sort of asynchronous packet processing going on in libnetfilter_log. This is not the case, the callback function gets called only as part of nflog_handle_packet() in a polling type message handling. I have not tested it myself but I believe that the term callback is used to indicate that multiple callback functions can be stacked on top of each other, letting each callback be processed in sequence by nflog_handle_packet().

Non-blocking I/O

The vuurmuur_log daemon currently doesn’t do a whole lot. On the developer IRC channel we’ve even discussed merging the vuurmuur and vuurmuur_log daemons into one daemon. However, it is imaginable that we would let vuurmuur_log do some more things in the main loop and as it stands it does do some things. Therefore it would be very annoying to start a recv() on the netfilter layer and stall the program until it gets a log message. So in line 292:

if ((rv = recv (fd, buf, sizeof (buf), MSG_DONTWAIT)) == -1)

I chose to use non-blocking I/O. Normally I’m not a big fan of this as using blocking I/O instead leaves the scheduling of other work to be done to the OS instead of doing your own. In this case however, we cannot simply block vuurmuur_log as it also interacts with the vuurmuur_conf reloading functionality through IPC (shared memory and signals). This also means that the logic needs to include something like a ‘readnflog was fine, except no data was processed’ return code (0 in this case). If there is data to be processed after the recv() call, readnflog() will call libnetfilter_log’s nflog_handle_packet() function and return a 1 to the main program to indicate it has processed data.

Passing back and forth output

Something I found a bit confusing to implement is passing output from the callback function to the main program. As we can see, the nflog_handle_packet() only returns an integer value. The struct nflog_handle *h, the char *buf and the int len arguments to the call are all input parameters. The earlier method of syslog processing in vuurmuur_log would process a text line and return its results in two structures, each of which was used to generate a ‘vuurmuur log’ line. I wanted to have something similar from the readnflog() function but had to do this through some (in my opinion) hackish ways.

The key to getting control over some of the output of the callback function is to use the third argument to the nflog_callback_register() function. This is a void * I’ve seen used in ulogd2 to pass input to the callback instead, however I pass nflog_callback_register() the logrule_ptr here. The callback function also gets a void * as its 4th argument when it’s being called by nflog_handle_packet() and it is the same pointer passed in nflog_callback_register(). I’m not sure if the library developers intended this use but “It Works for Me ™”.

Since I can only pass one single pointer to the callback this way, I also had to consolidate the two structures the earlier method of syslog processing used into one logrule structure.

Timestamps in packets

As it turns out, there isn’t always a timestamp in the packet we get from libnetfilter_log. While you’re also not guaranteed to get an interface back when using nflog_get_indev() / nflog_get_outdev() which is okay for vuurmuur, the timestamp is something that’s always in syslog, therefore also always in vuurmuur’s traffic log, and therefore something we want to have in nflog!

So in line 169 we attempt to get the timestamp from the packet itself as it was generated by the kernel. If this timestamp is not set properly for whichever undocumented reason, we just calculate one ourselves using a gettimeofday() call. While this is probably exactly the same syslogd does, I don’t think much of this appearingly random skipping of timestamp calculation. Rumor has it that timestamp calculus in the kernel is expensive and with the netfilter core being subject of constant performance debates, under some conditions the timestamp isn’t calculated.

Parsing the L3 header and data

The hw_protocol field we get in the struct nfulnl_msg_packet_hdr * returned by nflog_get_msg_packet_hdr() maps to the internet protocols as maintained by IANA. Personally I think ‘hw_protocol’ is an extremely confusing name as it makes me think of ‘hardware’. If anybody can explain the logic behind this to me, by all means go ahead.

So after we get the packet’s payload in line 182, we can analyse the packet’s contents based on the hw_protocol. Currently, nflog.c only massages IPv4 packets but as you can see a hook for IPv6 is also already there as IPv6 support is being built into vuurmuur as we speak.

The IP transport protocol in the logrule_pts is set from iph->protocol in the L3 header. These IP protocols are described in /etc/protocols on any Linux box. Using this information together with the IP header length iph->ihl, we neatly cast the relevant variable to either a struct tcphdr *, a struct icmphdr * or a struct udph * so get to a few other data fields we need to fill up the logrule_ptr:

• src_ip: The source IP address
• dst_ip: The destination IP address
• src_port: The source portnumber
• dst_port: The destination portnumber
• syn/fin/rst/ack/psh/urg: flags in case of a TCP connection