This is a generic mechanism for evaluating DaCHS' server logs.
Operators can run

svn checkout http://svn.ari.uni-heidelberg.de/svn/gavo/hdinputs/accesslogs

in their /var/gavo/inputs, add

logFormat: combined
preloadRDs: accesslogs/q

in the [web] section of their /etc/gavo.rc and then look at their access
logs at <serverurl>accesslogs/q/inspect.

The origin field of the grammar will be empty unless you install the
python3-geoip package.


This tries to have a guess at what are robots using two different
mechanisms:

(1) there are ignored ips; after te first import, see robots-hosts.txt
in the resource directory.  These are matched literally and dynamically
extended if a host requests robots.txt

(b) there are fragments of ignored user agents; after the first import,
see robots-sigs.txt in the resource directory.  These are matched as
fragments ("frag in userAgent"), so you will get a lot of false
positives if you make these strings too short.

Feel free to add to these as you go.