To improve responsiveness, tried to make awstats generate static pages instead of dynamic report generation with the cgi-bin perl script.
Most probably the unresponsive nature was due to the DNS settings in the /etc/awstats/*.conf files for the domains we were reporting on. As the conf file says,
# If you want/need to set DNSLookup to 1, don't forget that this will
# dramatically reduce AWStats's update process speed. Do not use on large web
# sites.
Then, there was also the dynamic DNS lookup option, which also needed to be set to 0 -
# For very large sites, setting DNSLookup to 0 (or 2) might be the only
# reasonable choice. DynamicDNSLookup allows to resolve host names for
# items shown in html tables only, when data is output on reports instead
# of resolving once during log analysis step.
# Possible values:
# 0 - No dynamic DNS lookup
Then two more settings were changed, changed to 0
# Possible values:
# 0 - Report is not shown at all
# 1 - Report is shown in main page with an entry in menu and default columns
# XYZ - Report shows column informations defined by code X,Y,Z...
# X,Y,Z... are code letters among the following:
# U = Unique visitors
# V = Visits
# P = Number of pages
# H = Number of hits (or mails)
# B = Bandwidth (or total mail size for mail logs)
# L = Last access date
# E = Entry pages
# X = Exit pages
# C = Web compression (mod_gzip,mod_deflate)
# M = Average mail size (mail logs)
# Show domains/country chart
# Context: Web, Streaming, Mail, Ftp
# Default: PHB, Possible column codes: UVPHB
ShowDomainsStats=0
# Show hosts chart
# Context: Web, Streaming, Mail, Ftp
# Default: PHBL, Possible column codes: PHBL
ShowHostsStats=0
Then, with much trial and error, the following lines were added to the root crontab for creating the static pages, and an index file was created to easily go to the relevant server/month page.
sudo su -
crontab -e
58 23 * * * /usr/share/awstats/tools/awstats_buildstaticpages.pl -update -dir=/var/www/pathtostatsfromawstats/ -config=server1.org -builddate=$(date '+\%Y\%m') > /dev/null 2>&1
56 23 * * * /usr/share/awstats/tools/awstats_buildstaticpages.pl -update
-dir=/var/www/pathtostatsfromawstats/ -config=server2.org
-builddate=$(date '+\%Y\%m') > /dev/null 2>&1
and so on.
There were lots of emails to root being generated every 10 minutes from cron, saying awstats.pl could not read the log files in /var/log/apache2/ourlogfile.log - it turned out that these were being generated by another cron which would probably have been created by the awstats installation via apt, which was located in /etc/cron.d directory, the file was /etc/cron.d/awstats which was being run as an unprivileged user, hence unable to read log files etc. So I just commented out those awstat update lines in that file. To note:
# /etc/crontab: system-wide crontab
# Unlike any other crontab you don't have to run the `crontab'
# command to install the new version when you edit this file
# and files in /etc/cron.d. These files also have username fields,
# that none of the other crontabs do.
Initial setup -
followed the post
to get Ubuntu install documentation,
and followed that for the setup, with separate access.log files for each domain set in their respective conf files.
To set up static pages.
Found the path to awstats executables (perl scripts) - /usr/share/awstats/tools -
And if we need to password protect it,
https://www.digitalocean.com/community/tutorials/how-to-set-up-password-authentication-with-apache-on-ubuntu-20-04
Running the perl script without any arguments gave the listing of possible arguments -
perl /usr/share/awstats/tools/awstats_buildstaticpages.pl
Usage:
awstats_buildstaticpages.pl (awstats_options) [awstatsbuildstaticpages_options]
where awstats_options are any option known by AWStats
-config=configvalue is value for -config parameter (REQUIRED)
-update option used to update statistics before to generate pages
-lang=LL to output a HTML report in language LL (en,de,es,fr,...)
-month=MM to output a HTML report for an old month=MM
-year=YYYY to output a HTML report for an old year=YYYY
and awstatsbuildstaticpages_options can be
-awstatsprog=pathtoawstatspl AWStats software (awstats.pl) path
-dir=outputdir Output directory for generated pages
-diricons=icondir Relative path to use as icon dir in <img> links
-builddate=%YY%MM%DD Used to add build date in built pages filenames
-staticlinksext=xxx Build pages with .xxx extension (default .html)
-buildpdf[=pathtohtmldoc] Build a PDF file after building HTML pages.
Output directory must contains icon directory
when this option is used (need 'htmldoc')
At first, I thought I would need to manually create directories and chown them to www-data
# mkdir 202502
# chown www-data:www-data 202502
but then I could use the builddate parameter instead. Note that in the builddate parameter, when the $(date) is called in a cron script, the % characters have to be escaped.
Reports for older year/month combinations could be created like
perl /usr/share/awstats/tools/awstats_buildstaticpages.pl -dir=/var/www/pathtostatsfromawstats/ -config=ourserver.org -year=2024 -month=12 -builddate=202412
And finally, asked Github Copilot to generate a form with dummy servernames and paths, which I manually edited to get the correct paths, as below -
<script>
function redirectToUrl() {
var server = document.getElementById("server").value;
var year = document.getElementById("year").value;
var month = document.getElementById("month").value;
var url = "https://ourstatsurlforstatsfromawstats/awstats." + server + year + month + ".html";
window.location.href = url;
}
</script>
</HEAD>
<body>
<form onsubmit="event.preventDefault(); redirectToUrl();">
<label for="server">Select Server:</label>
<select id="server" name="server">
<option value="1.org.">CMS</option>
<option value="2.org.">devel2</option>
</select>
<br /><br />
<label for="year">Select Year:</label>
<select id="year" name="year">
<option value="2024">2024</option>
<option value="2025">2025</option>
<option value="2026">2026</option>
</select>
<br /><br />
<label for="month">Select Month:</label>
<select id="month" name="month">
<option value="01">01</option>
<option value="02">02</option>
<option value="03">03</option>
<option value="04">04</option>
<option value="05">05</option>
<option value="06">06</option>
<option value="07">07</option>
<option value="08">08</option>
<option value="09">09</option>
<option value="10">10</option>
<option value="11">11</option>
<option value="12">12</option>
</select>
<br /><br />
<input type="submit" value="Submit" />
</form>
</body>
