AWStats (XI)

Installation

_$: apt-get install libfcgi-perl
_$: wget http://nginxlibrary.com/downloads/perl-fcgi/fastcgi-wrapper -O /usr/bin/fastcgi-wrapper.pl
_$: wget http://nginxlibrary.com/downloads/perl-fcgi/perl-fcgi -O /etc/init.d/perl-fcgi
_$: chmod +x /usr/bin/fastcgi-wrapper.pl
_$: chmod +x /etc/init.d/perl-fcgi
_$: update-rc.d perl-fcgi defaults 99 00
_$: service perl-fcgi start
_$: netstat -naptu | grep perl
tcp        0      0 127.0.0.1:8999          0.0.0.0:*               LISTEN      2569/perl

Update to a more recent version

Installation directories in Ubuntu 12.04:

/usr/local/awstats
/usr/share/awstats

_$: mkdir -p apps
_$: cd apps
_$: wget https://prdownloads.sourceforge.net/awstats/awstats-7.5.tar.gz
_$: tar xvfz awstats-7.5.tar.gz
_$: cd awstats-7.5
_$: find . -name "awstats.pl"
./wwwroot/cgi-bin/awstats.pl
_$: find . -name "awstats_buildstaticpages.pl"
./tools/awstats_buildstaticpages.pl
_$: find . -name "maillogconvert.pl"
./tools/maillogconvert.pl
_$: find . -name "logresolvemerge.pl"
./tools/logresolvemerge.pl

Make a back up of the files currently installed:

_$: mv /usr/lib/cgi-bin/awstats{.pl,.pl.bak}
_$: mv /usr/lib/cgi-bin/awstats_buildstaticpages{.pl,.pl.bak}
_$: for file in "awstats_configure.pl awstats_exportlib.pl awstats_updateall.pl maillogconvert.pl"
do
    mv /usr/local/awstats/${file} /usr/local/awstats/${file}.bak
done

Recursively copy the files in cgi-bin to /usr/lib/cgi-bin:

_$: cp /root/apps/awstats-7.5/wwwroot/cgi-bin/* --recursive /usr/lib/cgi-bin

Recursively copy the files in icon to /usr/share/awstats:

_$: cp /root/apps/awstats-7.5/wwwroot/{classes,css,icon,js} --recursive /usr/share/awstats

Copy the files in tools to /usr/lib/cgi-bin:

_$: cp /root/apps/awstats-7.5/tools/*.pl /usr/lib/cgi-bin

Tunnel

The AWStats web site will be accesed through port 8443. This port is hidden behind a firewall for good reasons, so in order to see the web site we will need to create a tunnel:

(user@local)_$: ssh -L 8443:localhost:8443 user@<DevOps server>

Once created, all things going to the 8443 port in our machine will be forwarded over the secure channel to the 8443 port in the DevOps server and we will be able to see the web site.

Configuration

The awstats.pl script is located at /usr/lib/cgi-bin.
The awstats_buildstaticpages.pl script is located at /usr/share/awstats/tools.
The maillogconvert.pl script is located at /usr/local/awstats.

_$: mkdir /usr/local/awstats
_$: cp /usr/share/doc/awstats/examples/awstats_*.pl /usr/local/awstats/
_$: cp /usr/share/doc/awstats/examples/maillogconvert.pl /usr/local/awstats/
_$: cp /usr/share/awstats/tools/logresolvemerge.pl /usr/local/awstats/
_$: cp /etc/awstats/awstats.conf /etc/awstats/awstats.nginx.web.conf

/etc/nginx/sites-available/awstats:
-----------------------------------
server {
    listen 8443;
    root /var/www/awstats;

    location ^~ /awstats-icon {
        alias /usr/share/awstats/icon/;
        access_log off;
    }

    location ^~ /awstatscss {
        alias /usr/share/doc/awstats/examples/css/;
        access_log off;
    }

    location ^~ /awstatsclasses {
        alias /usr/share/doc/awstats/examples/classes/;
        access_log off;
    }

    # Perl execution
    location ~ \.pl$ {
        include /etc/nginx/fastcgi_params;
        fastcgi_index index.pl;
        fastcgi_pass  127.0.0.1:8999;
        fastcgi_param SCRIPT_FILENAME /usr/lib$fastcgi_script_name;
    }
}

Note: The rest of the fastcgi parameters are in /etc/nginx/fastcgi_params.

_$: ln -s /etc/nginx/sites-available/awstats /etc/nginx/sites-enabled/
_$: service nginx restart

/usr/lib/index.pl:
------------------
#!/usr/bin/perl

print "Content-type:text/html\n\n";
print <<EndOfHTML;
<html>
<head><title>Perl CGI Test</title></head>
<body><h1>Perl CGI is working</h1></body>
</html>
EndOfHTML

_$: chmod +x /usr/lib/index.pl

If we go to http://127.0.0.1:8443/index.pl we will see the message “Perl CGI is working”.

Analyzing stats from other servers

We must place the configuration files in the /etc/awstats directory. Those files must conform to the following naming standard: awstats.<name>.conf. We will use the <name> part to do the database updates.

Example

Filename: awstats.mail.01.conf
Command needed to update the database: awstats.pl -config=mail.01 -update

There cannot be two files with the same name despite being in two different directories. For instance, we cannot have /etc/awstats/web1/site.conf and /etc/awstats/web2/site.conf at the same time.

On those configuration files we configure the LogFile and DataDir sections:

LogFile:
- mail: perl /usr/local/awstats/maillogconvert.pl standard < /srv/backup/logs/mail/mailserver/mail.log |
- web: /srv/backup/logs/<site>/webserver/access.log
DataDir:
- mail: /var/lib/awstats/mailserver
- web: /var/lib/awstats/webserver

_$: mkdir -p /var/lib/awstats/webserver
_$: mkdir -p /var/lib/awstats/mailserver

/etc/awstats/awstats.webserver.site.conf:
-----------------------------------------
...
LogFile="/srv/backup/logs/site/web/access.log"
...
DirData="/var/lib/awstats/webserver"
...

/etc/awstats/awstats.mailserver.conf:
-------------------------------------
...
LogFile="perl /usr/local/awstats/maillogconvert.pl standard < /srv/backup/logs/mail/mailserver/mail.log |"
...
DirData="/var/lib/awstats/mailserver"
...

Now we create a web page to see the stats for those sites. Please take a time to appreciate how fast, responsive and lean this web page is:

/var/www/awstats/index.html:
----------------------------
<html>

<head>
<title>AWStats</title>
</head>

<body>

<h2>mail.example.com</h2>
<a href="/mailserver/mail.awstats.html">mail.example.com</a></p>

<h2>web.example.com</h2>
<a href="/webserver/site.awstats.html">example.com</a></p>
<a href="/webserver/anothersite.awstats.html">example.net</a></p>

</body>

</html>

Create a script to update the stats:

/root/cron/awstats-update.sh:
-----------------------------
awstats="/usr/lib/cgi-bin/awstats.pl"
WWWPATH="/var/www/awstats"

# mail
$awstats -config=mailserver              -update -output > ${WWWPATH}/mailserver/mail.awstats.html

# web
$awstats -config=webserver.site          -update -output > ${WWWPATH}/webserver/site.awstats.html
$awstats -config=webserver.anothersite   -update -output > ${WWWPATH}/webserver/anothersite.awstats.html

# Fix permissions
/bin/chown www-data:www-data --recursive ${WWWPATH}

and another one to backup the stats already created:

/root/cron/awstats-backup.sh:
-----------------------------
for host in mailserver webserver
do
    # Configuration files
    cp /etc/awstats/awstats.$host*  /srv/backup/awstats/$host/conf

    # History files
    cp /var/lib/awstats/$host/*     /srv/backup/awstats/$host/txt

    # HTML Reports
    cp /var/www/awstats/$host/*     /srv/backup/awstats/$host/www
done

Once created, we can add those scripts to our crontab entries:

_$: crontab -e
...
### AWStats
10  *  *   *   *     /root/cron/awstats-update.sh
0   0  2   *   *     /root/cron/awstats-backup.sh
...

Basic authentication

/etc/nginx/sites-available/awstats:
-----------------------------------
server {
    listen 8443;
    root /var/www/awstats;

    # Authentication
    location / {
       auth_basic            "System user name and password required";
       auth_basic_user_file  /var/www/awstats/admin/.htpasswd;
    }

    location ^~ /admin {
        deny  all;
    }

    # Icons, images and CSS
    location ^~ /awstats-icon {
        alias /usr/share/awstats/icon/;
        access_log off;
    }

    location ^~ /awstatscss {
        alias /usr/share/doc/awstats/examples/css/;
        access_log off;
    }

    location ^~ /awstatsclasses {
        alias /usr/share/doc/awstats/examples/classes/;
        access_log off;
    }

    # Perl execution
    location ~ \.pl$ {
        include /etc/nginx/fastcgi_params;
        fastcgi_index index.pl;
        fastcgi_pass  127.0.0.1:8999;
        fastcgi_param SCRIPT_FILENAME /usr/lib$fastcgi_script_name;
        #fastcgi_intercept_errors on;
    }
}

_$: mkdir /var/awstats/admin

Now we will generate passwords for two ficticious users. You can do this using htpasswd from the apache2-utils or there is also a PIP package with the same name.

_$: htpasswd -bc user1 "pass1" /var/www/awstats/admin/.htpasswd
_$: htpasswd -b  user2 "pass2" /var/www/awstats/admin/.htpasswd
_$: nginx -t && service nginx reload

Processing logs from Windows’ IIS

We will split this process in two parts: the first one is the general use case and the second one the checkout step of our web application. In order to create a monthly log for the checkout step, which is extracted from the general use case, we will use the checkoutm.sh script.

The directory structure is as follows:

/srv/backup/logs/site
├── app
│   ├── fr000001.xml
│   ├── fr000002.xml
│   ...
│   └── freb.xsl
└── web
    ├── 201311
    │   ├── u_ex131119.zip
    │   ├── u_ex131120.zip
    │   ...
    │   ├── u_ex131128.zip
    │   ├── u_ex131129.zip
    │   └── u_ex131130.zip
    ├── 201312
    │   ├── u_ex131201.zip
    │   ├── u_ex131202.zip
    │   ...
    │   ├── u_ex131228.zip
    │   ├── u_ex131229.zip
    │   ├── u_ex131230.zip
    │   └── u_ex131231.zip
    ├── unzipped
    │   ├── checkout
    │   │   ├── checkout.log
    │   │   └── checkout.sh
    │   ├── u_ex131119.log
    │   ├── u_ex131120.log
    │   ...
    │   ├── u_ex131228.log
    │   ├── u_ex131229.log
    │   ├── u_ex131230.log
    │   └── u_ex131231.log
    └── unzip.sh

AWStats configuration

/etc/awstats/awstats.iis.site.conf:
-----------------------------------
...
LogFile="cat /srv/backup/logs/site/web/unzipped/u_ex1312*.log |"
LogFormat=2
SiteDomain="www.example.com"
DNSLookup=0
DirData="/var/lib/awstats/iis"
...

/etc/awstats/awstats.iis.site-checkout.conf:
--------------------------------------------
...
LogFile="/srv/backup/logs/site/web/unzipped/checkout/checkout.log"
LogFormat=2
SiteDomain="www.example.com"
DNSLookup=0
DirData="/var/lib/awstats/iis"
...

Helper shell scripts

/srv/backup/logs/site/web/unzip.sh:
-----------------------------------
# Unzip all files in "unzipped" folder

FOLDER="unzipped"
for f in ./${FOLDER}/*.zip
do
    7z x $f -o${FOLDER} && rm $f
done

/srv/backup/logs/site/web/unzipped/checkout/checkoutm.sh:
---------------------------------------------------------
# Creates checkout logs for every month
# Arguments:
#  - year:  00, 01, ... , 99
#  - month: 01, 02, ... , 12
# Example:
# $ checkoutm.sh 14 03    # March 2014
# $ checkoutm.sh 15 02    # February 2015

if [ $# -eq 2 ]
then
    year=${1}
    month=${2}
else
    year=`date +"%y"`
    month=`date +"%m"`
fi

echo "Using year: $year"
echo "Using month: $month"

file="checkout-${year}${month}.log"

# Create file with header
touch ${file}
echo "#Software: Microsoft Internet Information Services 7.5
#Version: 1.0
#Date: 20${year}-${month}-01 00:00:00
#Fields: date time s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) sc
-status sc-substatus sc-win32-status time-taken" > $file

# Get all entries from this month that are checkout
# grep -h "${year}-${month}" ../u_ex${year}${month}*.log | grep "/intranet/checkout" >> ${file}
grep -h "/intranet/checkout" ../u_ex${year}${month}*.log >> ${file}

# Change windows CRLF to LF (remove ^M)
sed -i 's/\r//g' ${file}

Process regular logs:

_$: cd /srv/backup/logs/site/web
_$: cp ./201312/*.zip ./unzipped
_$: ./unzip.sh

Process checkout logs. In this case we will process all checkouts that happened during January of 2014:

_$: cd /srv/backup/logs/site/web/unzipped/checkout
_$: ./checkoutm.sh 14 01    # 14 --> 2014 ; 01 --> January

We can either wait until awstats is automatically updated or we can manually force an update. To force an update run the following command:

_$: /root/cron/awstats-update.sh

Analyzing old log files

Let’s suppose that in March 2015 some log files are missing and that we realized they were missing in May 2015.

Take a backup of the months after March that are fine:

_$: cd /var/lib/awstats/iis
_$: mkdir ./tmp
_$: mv awstats0[45]2015.iis.site.txt ./tmp

Edit the configuration file so that March is analyzed again, but this time completely ;)

/etc/awstats/awstats.iis.site.conf:
-----------------------------------
...
LogFile="cat /srv/backup/logs/site/web/unzipped/u_ex1503*.log |"

Run awstats:

_$: awstats="/usr/lib/cgi-bin/awstats.pl"
_$: WWWPATH="/var/www/awstats"
_$: $awstats -config=iis.site -update -output > ${WWWPATH}/iis/site.awstats.html

It might happen that running awstats does not create the file for March. If that happens, do it manually:

_$: touch /var/lib/awstats/iis/awstats032015.iis.site.txt

and run again:

_$: $awstats -config=iis.site -update -output > ${WWWPATH}/iis/site.awstats.html

Once the awstats032015.iis.site.txt has data, copy back the files for April and May:

_$: cd /var/lib/awstats/iis
_$: mv ./tmp/awstats0[45]2015.iis.site.txt .
_$: rmdir ./tmp

Create PDF reports

Check that the awstats_buildstaticpages.pl is located in /usr/lib/cgi-bin:

_$: [[ -f /usr/lib/cgi-bin/awstats_buildstaticpages.pl ]] || echo "Missing"

_$: perl ./awstats_buildstaticpages.pl -month=05 -year=2016 \
         -config=webserver.site \
         -dir=/tmp -buildpdf=/usr/bin/htmldoc \
         -diricon=/usr/share/awstats/icon \
         -awstatsprog=/usr/lib/cgi-bin/awstats.pl

Comprobar que los archivos temporales tienen cierto tamaño (han sido rellenados): _$: find /tmp -name “awstats.*.html” -size +500c | sort

Troubleshooting

While really big files are being processed, the following warning may appear: “Flush history file on disk (unique hosts reach flush limit of 20000)”. It is not an error, just information.