Installing ntop with nprobe
When I restructured my network I wanted to test some more tools. I had ntop on my list for some time so I thought I could finally give it a shot.
It quickly turned out that newer versions of ntop do not anymore include a flow collector you need to save samples sent to your analyzer from your routing devices. ntop now works only bundled with nprobe, which works as a collector and/or proxy in front of ntop. Too bad, since nprobe is not available for free so just playing around was not an option..
Searching for an alternative to ntop is not easy, since ntop clearly has a very good and tailored interface.
The OpenWRT wiki has a list of possible software to be used for this. But their frontends are either generic data aggregators or the package is not available for OpenWRT. Meh.
Scrolling through multiple forums (e.g. /r/networking) brought me to nfsen. I used a cubieboard tutorial to get a rough orientation and fired it up in a VM. Some customization later I set it aside, since I was still searching for a maintained OpenWRT solution - this was not it.
I kinda gave it a last lucky shot and entered
~# opkg find *flow*. And boom, it gave me softflowd.
softflowd is available on OpenWRT in version 0.9.9 (come on, make this a stable 1.0!).
It comes bundled with a test collector script written in Perl. Use this to test the basic connection between your router and the collector (most probably your machine):
perl collector.pl -p 9001 -4 on the collector and
softflowd -v 5 -d -n COLLECTOR_IP:9001 -i INTERFACE on the router, where
COLLECTOR_IP is the IP of the machine the collector.pl script is listening and
INTERFACE is the network interface you want to capture packet flows from.
If you don't want to wait until the first flows expire, open a second connection to OpenWRT and use
softflowctl expire-all. This flushes all flows regardless of their status and sends everything to the collector.
You can btw also use
tcpdump -i INTERFACE -w /tmp/flow.dump to capture packets and let softflowd later use this capture file to generate the flows with
softflowd -r /tmp/flow.dump -n COLLECTOR_IP:9001. This way you can debug your collector with a reproducable input.
Now I partly had what I searched for - a packaged program which exports flows on OpenWRT.
What was still missing was the collector part. I got curious about how the packets softflowd exports work and dug deeper to find specs about NetFlow. I kind of knew what the aim of NetFlow and sFlow were, as we use it in another project with pmacct to count traffic. But I had never seen it in action nor looked into any specifications. The collector.pl script gave me first hints what I had to expect. There's also another Python testing script from the softflowd developers, but it only supports version 1!
Well, I am not sure what happened then but the end result is an implementation of a version 9 parser in Python. I published it under MIT license on Github: https://github.com/cooox/python-netflow-v9-softflowd.
Version 9 is the first version to fully support IPv6. It's also the first version to introduce templates which make the structure of flow records dynamic. Some more references:
- Cisco: NetFlow Version 9 Flow-Record Format
- RFC 3954: Cisco Systems NetFlow Services Export Version 9
- Solarwinds PDF with nice grafics
In conclusion: I (you?) can now use softflowd from the OpenWRT package repository to export NetFlow v9 flows to a collector. This collector is a UDP listener started by with my implemented parser. Of course this needs some more work to produce visualizations, but since the parser spits out lists of dicts this will be the more routined task to handle.