malhuda/AS-Stats
1
0
Fork 0
mirror of https://github.com/manuelkasper/AS-Stats.git synced 2025-02-20 11:44:12 +08:00
Code Issues Actions Packages Projects Releases Wiki Activity

Merge pull request #12 from nshopik/docs

Browse Source 
Split docs and convert to markdown
This commit is contained in:
Manuel Kasper 2013-07-30 07:43:04 -07:00
commit a8deedce97
2 changed files with 423 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

115
HISTORY.md Normal file
View File

@ -0,0 +1,115 @@
## x.xx
* Try harder to determine IP version for sFlow samples.
* Add IPFIX support (contributed by Daniel Piekacz)
* Add experimental NetFlow v5 support (contributed by Charlie Allom)
* Fix handling of 64-bit counters for NetFlow v9
## v1.41
* Generate v6 RRD DS in contrib/sync_ds.pl too (spotted by Peter Hansen)
* sFlow: fix creation of new RRDs when multiple entries with the same
tag are present in the known links file (spotted by Michal Buchtik)
* Add startup scripts for FreeBSD (contrib/freebsd)
(contributed by Michal Buchtik)
* Add support for setting the sampling rate per link in the knownlinks file.
## v1.40
* Add support for NetFlow v9 to netflow-asstatd.pl
(sponsored by Xplornet Communications Inc.)
* Add support for IPv6 (for NetFlow v9 and sFlow). Note: existing
RRDs need to be upgraded (new data sources added for v6) for
this to work. Enable $showv6 in www/config.inc to see separate
graphs for IPv6.
* Add support for 4-byte ASNs (NetFlow v9 and sFlow). Needs testing.
* Add RRA for 1 month at 4 hour resolution to newly created RRD files.
* Add links to PeeringDB and robtex (suggested by Steve Glendinning)
* Fix AS-SET lookup on systems where the whois command returns additional lines.
## v1.36
* Fix creation of new RRDs when multiple entries with the same
tag are present in the known links file
(spotted by Michel Moriniaux)
* Add feature to inspect all ASes in an AS-SET (automatic whois lookup).
(contributed by Nicolas Debrigode)
* Updated asinfo.txt with data from whois.cymru.com.
## v1.35
* Allow hyphens in link names. (contributed by Gareth Campling)
* Smooth I/O burstiness and reduce overall IOPS requirements
by flushing only 10% of the cache every 25 seconds (instead
of the entire cache at once every 60 seconds).
(contributed by James A. T. Rice)
## v1.34
* Fix for NaN detection in rrd-extractstats.pl for
64-bit Perl versions
(contributed by Benjamin Schlageter)
* Skip missing data sources in rrd-extractstats.pl to avoid
abort if new data sources are added but the RRDs are not
updated.
## v1.33
* Fix for multiple entries with the same tag in the
knownlinks file (e.g. for LACP) (contributed by Michal Buchtik)
* Added sync_ds.pl script to contrib directory, which can
synchronize the data sources of RRD files with the tags
defined in the knownlinks file
(contributed by Michal Buchtik)
* sflow-asstatd.pl now uses the agent IP instead of the
UDP source address - this makes it behave properly when
a proxy is being used (contributed by Michel Moriniaux)
## v1.32
* Fix add_ds.sh to support new directory structure
(contributed by Sergei Veltistov)
* Fix PHP warnings and move $ntop to config.inc
(contributed by Michal Buchtik)
## v1.31
* Set memory_limit to 256 MB in PHP pages (suggested by Steve Colam
<steve@colam.co.uk>).
* Allow NetFlow aggregation version 0 as well as 2 to make NetFlow
exports from Juniper routers work (suggested by Thomas Mangin
<thomas.mangin@exa-networks.co.uk>).
* Updated asinfo.txt with data from whois.cymru.com.
## v1.3
**Change your start script accordingly...**
* Changes by Steve Colam <steve@colam.co.uk>:
- ...-asstatd.pl now accepts parameters (UDP listen port,
sampling rate etc.) on the command line;
**Mind the new command line syntax when upgrading!**
- hierarchical RRD structure for more efficient storage
(one directory per low byte of AS number);
**Use `tools/migraterrdfiles.pl` to move your RRD files when upgrading!**
- ...-asstatd.pl now re-reads known links file upon SIGHUP
* Added contrib/generate-asinfo.py script to generate AS list from WHOIS
data (contributed by Thomas Mangin <thomas.mangin@exa-networks.co.uk>).
* Moved site-specific parameters of www frontend to config.inc.
* New flag images from famfamfam.com.
* Updated asinfo.txt.
## v1.2
* Support for sFlow (through sflow-asstatd.pl); fix for link names
with upper-case characters
* Allow inbound/outbound in graphs to be swapped (via option
in www/config.inc)
## v1.1
* Fix for a potential race condition surrounding $childrunning
(reported by Yann Gauteron; experienced on a Linux system)
## v1
* Initial release

308
README.md Normal file
View File

@ -0,0 +1,308 @@
AS-Stats v1.41 (2013-03-17)
===========================
A simple tool to generate per-AS traffic graphs from NetFlow/sFlow records
by Manuel Kasper <mk@neon1.net> for Monzoon Networks AG
How it works
------------
A Perl script (netflow-asstatd.pl) collects NetFlow v8/v9 AS aggregation records
or sFlow v5 samples from one or more routers. It caches them for about a
minute (to prevent excessive writes to RRD files), identifies the link that
each record refers to (by means of the SNMP in/out interface index), maps it
to a corresponding "known link" and RRD data source, and then runs RRDtool. To
avoid losing new records while the RRD files are updated, the update task is
run in a separate process.
For each AS, a separate RRD file is created as needed. It contains two data
sources for each link - one for inbound and one for outbound traffic.
In generated per-AS traffic graphs, inbound traffic is shown as positive,
while outbound traffic is shown as negative values.
Another Perl script, rrd-extractstats.pl, is meant to run about once per hour.
It sums up per-AS and link traffic during the last 24 hours, sorts the ASes
by total traffic (descending) and writes the results to a text file. This
is then used to display the "top N AS" and other stats by the provided PHP
scripts.
Prerequisites
-------------
- Perl 5.8
- RRDtool 1.2 (with Perl "RRDs" library)
- if using sFlow: the Net::sFlow module (CPAN)
- web server with PHP 5
- one or more routers than can generate NetFlow v8/v9 AS aggregation records
or sFlow samples
Installation
------------
In the instructions below, "xx-asstatd.pl" refers to either netflow-asstatd.pl
or sflow-asstatd.pl, depending on whether your routers generate NetFlow or
sFlow data.
- Copy the perl scripts xx-asstatd.pl and rrd-extractstats.pl to the
machine that will collect NetFlow/sFlow records
- Create a "known links" file with the following information about each
link that you want to appear in your AS stats:
- IP address of router (= source IP of NetFlow datagrams)
- SNMP interface index of interface (use "show snmp mib ifmib ifindex"
to find out)
- a short "tag" (12 chars max., alphanumerics only) that will be used
internally (e.g. for RRD DS names)
- a human-readable description (will appear in the generated graphs)
- a color code for the graphs (HTML style, 6 hex digits)
See the example file provided (knownlinks) for the format.
- Create a directory to hold per-AS RRD files. For each AS, about 128 KB of
storage are required, and there could be (in theory) up to 64511 ASes.
AS-Stats automatically creates 256 subdirectories in this directory for
more efficient storage of RRD files (one directory per lower byte of
AS number, in hex).
- Start xx-asstatd.pl in the background (or, better yet, write a
startup script for your operating system to automatically start
xx-asstatd.pl on boot):
`nohup xx-asstatd.pl -r /path/to/rrd/dir -k /path/to/knownlinks &`
By default, netflow-asstatd.pl will listen on port 9000 (UDP) for NetFlow
datagrams, and sflow-asstatd.pl will listen on port 6343 (UDP) for sFlow
datagrams. Use the -p option if you want to change that.
If you use sampled NetFlow or sFlow, set the sampling rate with the -s
option.
sflow-asstatd.pl also needs you to specify your own AS number with the -a
option for accurate classification of inbound and outbound traffic.
It's a good idea to make sure only UDP datagrams from your trusted routers
will reach the machine running xx-asstatd.pl (firewall etc.).
- NetFlow only:
Have your router(s) send NetFlow v8 or v9 AS aggregation records to
your machine. This is typically done with commands like the following
(Cisco IOS):
ip flow-cache timeout active 5
int Gi0/x.y
ip flow ingress
ip flow-export source <source interface>
ip flow-export version 5 origin-as
ip flow-aggregation cache as
cache timeout active 5
cache entries 16384
export destination <IP address of server running AS stats> 9000
enabled
Adjust the number of cache entries if necessary (i.e. if you get messages
like "Netflow as aggregation cache is almost full" in the logs).
Note that the version has to be specified as 5, even though the AS
aggregation records will actually be v8. Also, setting the global flow
cache timeout to 5 minutes is necessary to get "smooth" traffic graphs
(default is 30 minutes), as a flow is only counted when it expires from
the cache. Decreasing the flow-cache timeout may result in a slight
increase in CPU usage (and NetFlow AS aggregation takes its fair share of
CPU as well, of course).
Routers with MLS (Multi-Layer Switching, e.g. Cisco 7600 series) require
additional commands like the following in order to enable NetFlow
processing/aggregation for packets processed in hardware:
mls aging fast time 4 threshold 2
mls aging long 128
mls aging normal 64
mls flow ip interface-full
For IOS XR, the configuration looks as follows:
flow exporter-map FEM
version v9
!
transport udp 9000
source <source interface>
destination <IP address of server running AS stats> vrf default
flow monitor-map IPV4-FMM
record ipv4
exporter FEM
cache entries 16384
cache timeout active 5
!
flow monitor-map IPV6-FMM
record ipv6
exporter FEM
cache entries 16384
cache timeout active 5
!
sampler-map SM
random 1 out-of 10000
router bgp 100
address-family ipv4 unicast
bgp attribute-download
address-family ipv6 unicast
bgp attribute-download
For JunOS, the configuration looks as follows:
forwarding-options {
sampling {
input {
rate 2048;
max-packets-per-second 4096;
}
family inet {
output {
flow-active-timeout 60;
flow-server x.x.x.x {
port 9000;
autonomous-system-type origin;
aggregation {
autonomous-system;
}
version 8;
}
}
}
}
}
JunOS IPFIX configuration:
chassis {
tfeb {
slot 0 {
sampling-instance flow-ipfix;
}
}
}
interfaces {
ge-1/0/0 {
unit 0 {
family inet {
sampling {
input;
output;
}
}
}
}
}
forwarding-options {
sampling {
instance {
flow-ipfix {
input {
rate 1;
}
family inet {
output {
flow-server 192.0.2.10 {
port 9000;
autonomous-system-type origin;
no-local-dump;
version-ipfix {
template {
ipv4;
}
}
}
inline-jflow {
source-address 192.0.2.1;
}
}
}
}
}
}
}
services {
flow-monitoring {
version-ipfix {
template ipv4 {
flow-active-timeout 60;
flow-inactive-timeout 60;
template-refresh-rate {
packets 1000;
seconds 10;
}
option-refresh-rate {
packets 1000;
seconds 10;
}
ipv4-template;
}
}
}
}
- sFlow only:
Have your router(s) send sFlow samples to your machine. Your routers
may need a software upgrade to make them include AS path information for
both inbound and outbound packets (this is a good thing to check if
your graphs only show traffic on one direction).
- Wait 1-2 minutes. You should then see new RRD files popping up in the
directory that you defined/created earlier on. If not, make sure that
xx-asstatd.pl is running, not spewing out any error messages, and that
the NetFlow/sFlow datagrams are actually reaching your machine (tcpdump...).
- Add a cronjob to run the following command every hour:
`rrd-extractstats.pl /path/to/rrd/dir /path/to/knownlinks \
/path/to/asstats_day.txt`
That script will go through all RRD files and collect per-link summary
stats for each AS, sort them by total traffic (descending), and write them
to a text file. The "top N AS" page uses this to determine which ASes to
show.
- Copy the contents of the "www" directory to somewhere within your web
server's document root and change file paths in config.inc as necessary.
- Make the directory "asset" within www writable by the web server (this
is used to cache AS-SETs and avoid having to query whois for every request).
- Wait a few hours for data to accumulate. :)
- Access the provided PHP scripts via your web server and marvel at the
(hopefully) beautiful graphs.
Adding a new link
-----------------
Adding a new link involves adding two new data sources to all RRD files.
This is a bit of a PITA since RRDtool itself doesn't provide a command to do
that. A simple (but slow) Perl script that is meant to be used with RRDtool's
XML dump/restore feature is provided (add_ds_proc.pl, add_ds.sh). Note that
netflow-asstatd.pl should be stopped while modifying RRD files, to avoid
breaking them with concurrent modifications.
Changing the RRAs
-----------------
By default, the created RRDs keep data as follows:
* 48 hours at 5 minute resolution
* 1 week at 1 hour resolution
* 1 month at 4 hour resolution
* 1 year at 1 day resolution
If you want to change that, modify the getrrdfile() function in
xx-asstatd.pl and delete any old RRD files.
To do
-----
- rrd-extractstats.pl uses a lot of memory and could probably use some
optimization.