From 878134a81c9fa87154656ce6e5f7426553e48346 Mon Sep 17 00:00:00 2001 From: Manuel Kasper Date: Tue, 30 Jul 2013 16:43:57 +0200 Subject: [PATCH] Delete README It's redundant now that we have a Markdown version. --- README | 447 --------------------------------------------------------- 1 file changed, 447 deletions(-) delete mode 100644 README diff --git a/README b/README deleted file mode 100644 index 41d80e9..0000000 --- a/README +++ /dev/null @@ -1,447 +0,0 @@ -AS-Stats v1.41 (2013-03-17) -a simple tool to generate per-AS traffic graphs from NetFlow/sFlow records -by Manuel Kasper for Monzoon Networks AG --------------------------------------------------------------------- - -Changes -------- -vx.xx Try harder to determine IP version for sFlow samples. - - Add IPFIX support - (contributed by Daniel Piekacz) - - Add experimental NetFlow v5 support - (contributed by Charlie Allom) - - Fix handling of 64-bit counters for NetFlow v9 - -v1.41 Generate v6 RRD DS in contrib/sync_ds.pl too - (spotted by Peter Hansen) - - sFlow: fix creation of new RRDs when multiple entries with the same - tag are present in the known links file - (spotted by Michal Buchtik) - - Add startup scripts for FreeBSD (contrib/freebsd) - (contributed by Michal Buchtik) - - Add support for setting the sampling rate per link in the - knownlinks file. - -v1.40 Add support for NetFlow v9 to netflow-asstatd.pl - (sponsored by Xplornet Communications Inc.) - - Add support for IPv6 (for NetFlow v9 and sFlow). Note: existing - RRDs need to be upgraded (new data sources added for v6) for - this to work. Enable $showv6 in www/config.inc to see separate - graphs for IPv6. - - Add support for 4-byte ASNs (NetFlow v9 and sFlow). Needs testing. - - Add RRA for 1 month at 4 hour resolution to newly created RRD files. - - Add links to PeeringDB and robtex - (suggested by Steve Glendinning) - - Fix AS-SET lookup on systems where the whois command returns - additional lines. - - -v1.36 Fix creation of new RRDs when multiple entries with the same - tag are present in the known links file - (spotted by Michel Moriniaux) - - Add feature to inspect all ASes in an AS-SET (automatic whois - lookup). - (contributed by Nicolas Debrigode) - - Updated asinfo.txt with data from whois.cymru.com. - - -v1.35 Allow hyphens in link names. - (contributed by Gareth Campling) - - Smooth I/O burstiness and reduce overall IOPS requirements - by flushing only 10% of the cache every 25 seconds (instead - of the entire cache at once every 60 seconds). - (contributed by James A. T. Rice) - - -v1.34 Fix for NaN detection in rrd-extractstats.pl for - 64-bit Perl versions - (contributed by Benjamin Schlageter) - - Skip missing data sources in rrd-extractstats.pl to avoid - abort if new data sources are added but the RRDs are not - updated. - - -v1.33 Fix for multiple entries with the same tag in the - knownlinks file (e.g. for LACP) - (contributed by Michal Buchtik) - - Added sync_ds.pl script to contrib directory, which can - synchronize the data sources of RRD files with the tags - defined in the knownlinks file - (contributed by Michal Buchtik) - - sflow-asstatd.pl now uses the agent IP instead of the - UDP source address - this makes it behave properly when - a proxy is being used - (contributed by Michel Moriniaux) - -v1.32 Fix add_ds.sh to support new directory structure - (contributed by Sergei Veltistov) - - Fix PHP warnings and move $ntop to config.inc - (contributed by Michal Buchtik) - -v1.31 Set memory_limit to 256 MB in PHP pages (suggested by Steve Colam - ). - - Allow NetFlow aggregation version 0 as well as 2 to make NetFlow - exports from Juniper routers work (suggested by Thomas Mangin - ). - - Updated asinfo.txt with data from whois.cymru.com. - -v1.3 Changes by Steve Colam : - - ...-asstatd.pl now accepts parameters (UDP listen port, - sampling rate etc.) on the command line - - **************************************************** - * Mind the new command line syntax when upgrading! * - * Change your start script accordingly... * - **************************************************** - - - hierarchical RRD structure for more efficient storage - (one directory per low byte of AS number) - - ********************************************* - * Use tools/migraterrdfiles.pl to move your * - * RRD files when upgrading! * - ********************************************* - - - ...-asstatd.pl now re-reads known links file upon SIGHUP - - Added contrib/generate-asinfo.py script to generate AS list from WHOIS - data (contributed by Thomas Mangin - ). - - Moved site-specific parameters of www frontend to config.inc. - - New flag images from famfamfam.com. - - Updated asinfo.txt. - -v1.2 Support for sFlow (through sflow-asstatd.pl); fix for link names - with upper-case characters - Allow inbound/outbound in graphs to be swapped (via option - in www/config.inc) - -v1.1 Fix for a potential race condition surrounding $childrunning - (reported by Yann Gauteron; experienced on a Linux system) - -v1 Initial release - - -How it works ------------- - -A Perl script (netflow-asstatd.pl) collects NetFlow v8/v9 AS aggregation records -or sFlow v5 samples from one or more routers. It caches them for about a -minute (to prevent excessive writes to RRD files), identifies the link that -each record refers to (by means of the SNMP in/out interface index), maps it -to a corresponding "known link" and RRD data source, and then runs RRDtool. To -avoid losing new records while the RRD files are updated, the update task is -run in a separate process. - -For each AS, a separate RRD file is created as needed. It contains two data -sources for each link - one for inbound and one for outbound traffic. -In generated per-AS traffic graphs, inbound traffic is shown as positive, -while outbound traffic is shown as negative values. - -Another Perl script, rrd-extractstats.pl, is meant to run about once per hour. -It sums up per-AS and link traffic during the last 24 hours, sorts the ASes -by total traffic (descending) and writes the results to a text file. This -is then used to display the "top N AS" and other stats by the provided PHP -scripts. - - -Prerequisites -------------- - -- Perl 5.8 -- RRDtool 1.2 (with Perl "RRDs" library) -- if using sFlow: the Net::sFlow module (CPAN) -- web server with PHP 5 -- one or more routers than can generate NetFlow v8/v9 AS aggregation records - or sFlow samples - - -Installation ------------- -In the instructions below, "xx-asstatd.pl" refers to either netflow-asstatd.pl -or sflow-asstatd.pl, depending on whether your routers generate NetFlow or -sFlow data. - -- Copy the perl scripts xx-asstatd.pl and rrd-extractstats.pl to the - machine that will collect NetFlow/sFlow records - -- Create a "known links" file with the following information about each - link that you want to appear in your AS stats: - - - IP address of router (= source IP of NetFlow datagrams) - - SNMP interface index of interface (use "show snmp mib ifmib ifindex" - to find out) - - a short "tag" (12 chars max., alphanumerics only) that will be used - internally (e.g. for RRD DS names) - - a human-readable description (will appear in the generated graphs) - - a color code for the graphs (HTML style, 6 hex digits) - - See the example file provided (knownlinks) for the format. - -- Create a directory to hold per-AS RRD files. For each AS, about 128 KB of - storage are required, and there could be (in theory) up to 64511 ASes. - AS-Stats automatically creates 256 subdirectories in this directory for - more efficient storage of RRD files (one directory per lower byte of - AS number, in hex). - -- Start xx-asstatd.pl in the background (or, better yet, write a - startup script for your operating system to automatically start - xx-asstatd.pl on boot): - - nohup xx-asstatd.pl -r /path/to/rrd/dir -k /path/to/knownlinks & - - By default, netflow-asstatd.pl will listen on port 9000 (UDP) for NetFlow - datagrams, and sflow-asstatd.pl will listen on port 6343 (UDP) for sFlow - datagrams. Use the -p option if you want to change that. - If you use sampled NetFlow or sFlow, set the sampling rate with the -s - option. - sflow-asstatd.pl also needs you to specify your own AS number with the -a - option for accurate classification of inbound and outbound traffic. - It's a good idea to make sure only UDP datagrams from your trusted routers - will reach the machine running xx-asstatd.pl (firewall etc.). - -- NetFlow only: - Have your router(s) send NetFlow v8 or v9 AS aggregation records to - your machine. This is typically done with commands like the following - (Cisco IOS): - - ip flow-cache timeout active 5 - - int Gi0/x.y - ip flow ingress - - ip flow-export source - ip flow-export version 5 origin-as - ip flow-aggregation cache as - cache timeout active 5 - cache entries 16384 - export destination 9000 - enabled - - Adjust the number of cache entries if necessary (i.e. if you get messages - like "Netflow as aggregation cache is almost full" in the logs). - - Note that the version has to be specified as 5, even though the AS - aggregation records will actually be v8. Also, setting the global flow - cache timeout to 5 minutes is necessary to get "smooth" traffic graphs - (default is 30 minutes), as a flow is only counted when it expires from - the cache. Decreasing the flow-cache timeout may result in a slight - increase in CPU usage (and NetFlow AS aggregation takes its fair share of - CPU as well, of course). - - Routers with MLS (Multi-Layer Switching, e.g. Cisco 7600 series) require - additional commands like the following in order to enable NetFlow - processing/aggregation for packets processed in hardware: - - mls aging fast time 4 threshold 2 - mls aging long 128 - mls aging normal 64 - mls flow ip interface-full - - For IOS XR, the configuration looks as follows: - - flow exporter-map FEM - version v9 - ! - transport udp 9000 - source - destination vrf default - - flow monitor-map IPV4-FMM - record ipv4 - exporter FEM - cache entries 16384 - cache timeout active 5 - ! - flow monitor-map IPV6-FMM - record ipv6 - exporter FEM - cache entries 16384 - cache timeout active 5 - ! - - sampler-map SM - random 1 out-of 10000 - - router bgp 100 - address-family ipv4 unicast - bgp attribute-download - address-family ipv6 unicast - bgp attribute-download - - For JunOS, the configuration looks as follows: - - forwarding-options { - sampling { - input { - rate 2048; - max-packets-per-second 4096; - } - family inet { - output { - flow-active-timeout 60; - flow-server x.x.x.x { - port 9000; - autonomous-system-type origin; - aggregation { - autonomous-system; - } - version 8; - } - } - } - } - } - - JunOS IPFIX configuration: - chassis { - tfeb { - slot 0 { - sampling-instance flow-ipfix; - } - } - } - interfaces { - ge-1/0/0 { - unit 0 { - family inet { - sampling { - input; - output; - } - } - } - } - } - forwarding-options { - sampling { - instance { - flow-ipfix { - input { - rate 1; - } - family inet { - output { - flow-server 192.0.2.10 { - port 9000; - autonomous-system-type origin; - no-local-dump; - version-ipfix { - template { - ipv4; - } - } - } - inline-jflow { - source-address 192.0.2.1; - } - } - } - } - } - } - } - services { - flow-monitoring { - version-ipfix { - template ipv4 { - flow-active-timeout 60; - flow-inactive-timeout 60; - template-refresh-rate { - packets 1000; - seconds 10; - } - option-refresh-rate { - packets 1000; - seconds 10; - } - ipv4-template; - } - } - } - } - -- sFlow only: - Have your router(s) send sFlow samples to your machine. Your routers - may need a software upgrade to make them include AS path information for - both inbound and outbound packets (this is a good thing to check if - your graphs only show traffic on one direction). - -- Wait 1-2 minutes. You should then see new RRD files popping up in the - directory that you defined/created earlier on. If not, make sure that - xx-asstatd.pl is running, not spewing out any error messages, and that - the NetFlow/sFlow datagrams are actually reaching your machine (tcpdump...). - -- Add a cronjob to run the following command every hour: - - rrd-extractstats.pl /path/to/rrd/dir /path/to/knownlinks \ - /path/to/asstats_day.txt - - That script will go through all RRD files and collect per-link summary - stats for each AS, sort them by total traffic (descending), and write them - to a text file. The "top N AS" page uses this to determine which ASes to - show. - -- Copy the contents of the "www" directory to somewhere within your web - server's document root and change file paths in config.inc as necessary. - -- Make the directory "asset" within www writable by the web server (this - is used to cache AS-SETs and avoid having to query whois for every request). - -- Wait a few hours for data to accumulate. :) - -- Access the provided PHP scripts via your web server and marvel at the - (hopefully) beautiful graphs. - - -Adding a new link ------------------ -Adding a new link involves adding two new data sources to all RRD files. -This is a bit of a PITA since RRDtool itself doesn't provide a command to do -that. A simple (but slow) Perl script that is meant to be used with RRDtool's -XML dump/restore feature is provided (add_ds_proc.pl, add_ds.sh). Note that -netflow-asstatd.pl should be stopped while modifying RRD files, to avoid -breaking them with concurrent modifications. - - -Changing the RRAs ------------------ -By default, the created RRDs keep data as follows: - - * 48 hours at 5 minute resolution - * 1 week at 1 hour resolution - * 1 month at 4 hour resolution - * 1 year at 1 day resolution - -If you want to change that, modify the getrrdfile() function in -xx-asstatd.pl and delete any old RRD files. - - -To do ------ - -- rrd-extractstats.pl uses a lot of memory and could probably use some - optimization.