AS-Stats v1.32 (2010-08-02)
a simple tool to generate per-AS traffic graphs from NetFlow/sFlow records
by Manuel Kasper, Monzoon Networks AG <mkasper@monzoon.net>
--------------------------------------------------------------------

Changes
-------
xxx		Fix for multiple entries with the same tag in the
		knownlinks file (e.g. for LACP)
		(contributed by Michal Buchtik)
		
		Added sync_ds.pl script to contrib directory, which can
		synchronize the data sources of RRD files with the tags
		defined in the knownlinks file
		(contributed by Michal Buchtik)

v1.32   Fix add_ds.sh to support new directory structure
		(contributed by Sergei Veltistov)
		
		Fix PHP warnings and move $ntop to config.inc
		(contributed by Michal Buchtik)

v1.31	Set memory_limit to 256 MB in PHP pages (suggested by Steve Colam
		<steve@colam.co.uk>).
		
		Allow NetFlow aggregation version 0 as well as 2 to make NetFlow
		exports from Juniper routers work (suggested by Thomas Mangin
     	<thomas.mangin@exa-networks.co.uk>).

		Updated asinfo.txt with data from whois.cymru.com.

v1.3	Changes by Steve Colam <steve@colam.co.uk>:
			- ...-asstatd.pl now accepts parameters (UDP listen port,
			  sampling rate etc.) on the command line
			
			  ****************************************************
			  * Mind the new command line syntax when upgrading! *
			  * Change your start script accordingly...          *
			  ****************************************************
			
			- hierarchical RRD structure for more efficient storage
			  (one directory per low byte of AS number)
			
			  *********************************************
			  * Use tools/migraterrdfiles.pl to move your *
			  * RRD files when upgrading!                 *
			  *********************************************

			- ...-asstatd.pl now re-reads known links file upon SIGHUP
			
		Added contrib/generate-asinfo.py script to generate AS list from WHOIS
        data (contributed by Thomas Mangin
     	<thomas.mangin@exa-networks.co.uk>).

		Moved site-specific parameters of www frontend to config.inc.
		
		New flag images from famfamfam.com.
		
		Updated asinfo.txt.

v1.2	Support for sFlow (through sflow-asstatd.pl); fix for link names
		with upper-case characters
		Allow inbound/outbound in graphs to be swapped (via option
		in www/config.inc)

v1.1	Fix for a potential race condition surrounding $childrunning
		(reported by Yann Gauteron; experienced on a Linux system)

v1		Initial release


How it works
------------

A Perl script (netflow-asstatd.pl) collects NetFlow v8 AS aggregation records
or sFlow v5 samples from one or more routers. It caches them for about a
minute (to prevent excessive writes to RRD files), identifies the link that
each record refers to (by means of the SNMP in/out interface index), maps it
to a corresponding "known link" and RRD data source, and then runs RRDtool. To
avoid losing new records while the RRD files are updated, the update task is
run in a separate process.

For each AS, a separate RRD file is created as needed. It contains two data
sources for each link - one for inbound and one for outbound traffic.
In generated per-AS traffic graphs, inbound traffic is shown as positive,
while outbound traffic is shown as negative values.

Another Perl script, rrd-extractstats.pl, is meant to run about once per hour.
It sums up per-AS and link traffic during the last 24 hours, sorts the ASes
by total traffic (descending) and writes the results to a text file. This
is then used to display the "top N AS" and other stats by the provided PHP
scripts.


Prerequisites
-------------

- Perl 5.8
- RRDtool 1.2 (with Perl "RRDs" library)
- web server with PHP 5
- one or more routers than can generate NetFlow v8 AS aggregation records
  or sFlow samples


Installation
------------
In the instructions below, "xx-asstatd.pl" refers to either netflow-asstatd.pl
or sflow-asstatd.pl, depending on whether your routers generate NetFlow or
sFlow data.

- Copy the perl scripts xx-asstatd.pl and rrd-extractstats.pl to the
  machine that will collect NetFlow/sFlow records

- Create a "known links" file with the following information about each
  link that you want to appear in your AS stats:
  	
  	- IP address of router (= source IP of NetFlow datagrams)
  	- SNMP interface index of interface (use "show snmp mib ifmib ifindex"
  	  to find out)
  	- a short "tag" (15 chars max., alphanumerics only) that will be used
  	  internally (e.g. for RRD DS names)
  	- a human-readable description (will appear in the generated graphs)
  	- a color code for the graphs (HTML style, 6 hex digits)
  
  See the example file provided (knownlinks) for the format.

- Create a directory to hold per-AS RRD files. For each AS, about 128 KB of
  storage are required, and there could be (in theory) up to 64511 ASes.
  AS-Stats automatically creates 256 subdirectories in this directory for
  more efficient storage of RRD files (one directory per lower byte of
  AS number, in hex).

- Start xx-asstatd.pl in the background (or, better yet, write a
  startup script for your operating system to automatically start
  xx-asstatd.pl on boot):
  
  	nohup xx-asstatd.pl -r /path/to/rrd/dir -k /path/to/knownlinks &

  By default, netflow-asstatd.pl will listen on port 9000 (UDP) for NetFlow
  datagrams, and sflow-asstatd.pl will listen on port 6343 (UDP) for sFlow
  datagrams. Use the -p option if you want to change that.
  If you use sampled NetFlow or sFlow, set the sampling rate with the -s
  option.
  sflow-asstatd.pl also needs you to specify your own AS number with the -a
  option for accurate classification of inbound and outbound traffic.
  It's a good idea to make sure only UDP datagrams from your trusted routers
  will reach the machine running xx-asstatd.pl (firewall etc.).

- NetFlow only:
  Have your router(s) send NetFlow v8 AS aggregation records to your machine.
  This is typically done with commands like the following (Cisco IOS):

		ip flow-cache timeout active 5
		
		int Gi0/x.y
		  ip flow ingress
		  
		ip flow-export source <source interface>
		ip flow-export version 5 origin-as
		ip flow-aggregation cache as
		 cache timeout active 5
		 export destination <IP address of server running AS stats> 9000
		 enabled  

  Note that the version has to be specified as 5, even though the AS
  aggregation records will actually be v8. Also, setting the global flow
  cache timeout to 5 minutes is necessary to get "smooth" traffic graphs
  (default is 30 minutes), as a flow is only counted when it expires from
  the cache. Decreasing the flow-cache timeout may result in a slight
  increase in CPU usage (and NetFlow AS aggregation takes its fair share of
  CPU as well, of course).

- sFlow only:
  Have your router(s) send sFlow samples to your machine. Your routers
  may need a software upgrade to make them include AS path information for
  both inbound and outbound packets (this is a good thing to check if
  your graphs only show traffic on one direction).

- Wait 1-2 minutes. You should then see new RRD files popping up in the
  directory that you defined/created earlier on. If not, make sure that
  xx-asstatd.pl is running, not spewing out any error messages, and that
  the NetFlow/sFlow datagrams are actually reaching your machine (tcpdump...).

- Add a cronjob to run the following command every hour:

	rrd-extractstats.pl /path/to/rrd/dir /path/to/knownlinks \
		/path/to/asstats_day.txt

  That script will go through all RRD files and collect per-link summary
  stats for each AS, sort them by total traffic (descending), and write them
  to a text file. The "top N AS" page uses this to determine which ASes to
  show.
  
- Copy the contents of the "www" directory to somewhere within your web
  server's document root and change file paths in config.inc as necessary.

- Wait a few hours for data to accumulate. :)

- Access the provided PHP scripts via your web server and marvel at the
  (hopefully) beautiful graphs.


Adding a new link
-----------------
Adding a new link involves adding two new data sources to all RRD files.
This is a bit of a PITA since RRDtool itself doesn't provide a command to do
that. A simple (but slow) Perl script that is meant to be used with RRDtool's
XML dump/restore feature is provided (add_ds_proc.pl, add_ds.sh). Note that
netflow-asstatd.pl should be stopped while modifying RRD files, to avoid
breaking them with concurrent modifications.


Changing the RRAs
-----------------
By default, the created RRDs keep data as follows:

	* 48 hours at 5 minute resolution
	* 1 week at 1 hour resolution
	* 1 year at 1 day resolution

If you want to change that, modify the getrrdfile() function in
xx-asstatd.pl and delete any old RRD files.


To do
-----

- rrd-extractstats.pl uses a lot of memory and could probably use some
  optimization.