[pkgsrc/trunk]: pkgsrc/www/e2guardian e2guardian: update to 5.1.1

[sborrill <sborrill%pkgsrc.org@localhost>]

details:   https://anonhg.NetBSD.org/pkgsrc/rev/1f1246136dea
branches:  trunk
changeset: 382861:1f1246136dea
user:      sborrill <sborrill%pkgsrc.org@localhost>
date:      Mon Jul 16 14:12:34 2018 +0000

description:
e2guardian: update to 5.1.1

Note that large sections of the code has been re-written and there are
significant changes to the configuration files in this release.

The v5 is written in c++11 and so to compile it you will need gcc v5.4
or later. (or another complier that supports the full c++11).
Note that the target systems may also need an c++11 library update.

REVISED LIST and STORYBOARDING MODEL

Version 5 has a completely revised model for defining and using lists.

List definition is now separated from list application. Lists are no longer
hard-coded, but mapped to a function using a storyboard.

Filtering logic flow is simplified and made more consistent.  Requests are
analysed first and flags set (exception, grey, blocked etc) and once this
checking is complete actions are taken.  Large sections of duplicate logic
has been removed from ConnectionHandler and large sections are now
separate functions.

Storyboarding is a simple scripting language that maps lists to functions
and allows flags to be set.

This means that new lists can be added without changing the code, by adding
a new list definition and then applying it in a revised storyboard.

A different storyboard can be applied to each filtergroup, so if required,
each filtergroup can have a different logic flow.

Please read notes/V5_list_definition.pdf & notes/V5_Storyboard.pdf for
details.

TRANSPARENT HTTPS

    Detects SNI and flags whether traffic is TLS.
    Currently limited to port 443 traffic.

ICAP SERVER

    REQMOD and RESPMOD mode supported.
    See notes/icap.

DIRECT UPSTREAM ACCESS I.e. not via proxy.

    To implement globaly comment out 'proxyip =' in e2guardian.conf.

    The storyboard action setgodirect can be used within checkrequest functions
    to enable selected protocols/site/urls to godirect.
    e.g. to send all connect requests directly add
                if(connect) setgodirect
        to a requestchecks function.
    This can be also useful to by-pass squid for some requests (e.g. os update
    sites) when squid authentication methods are being used.

STORYBOARD TRACING

    New option 'storyboardtrace' to enable tracing output -
    for storyboard bug-fixing

READABLE THREAD_ID FOR LOGS & DEBUG

    Most debug and syslog messages are now prefixed with a thread ID as
    follows:-

    master: for master thread

    listen1_proxy:  normal proxy listener
    listen1_thttps: tranparent https listener
    listen1_icap: icap listener

    where '1' is index

    hw10: for http_worker threads where '10' is the thread number

    log: for logging thread

REVISED DEBUG STAGE 1

    The following low level debugs are no longer enabled by DGDEBUG:

    Network sockets - use NETDEBUG instead

    Regular expressions - use REDEBUG instead

    This reduces the volume of information and makes the debug
    log easier to read.

REVISED DEBUG STAGE 2

    New debuglevel option in e2guardian.conf.  Allows some debuging on
    production systems.   Currently just for ICAP and CLAMAV.  Will be
    extended to other sections of code in future releases.

HTTP/1.1

    Support for HTTP v1.1 completed - including Chunked encoding

ANTI-VIRUS PLUGINS

    Anti-virus plugins implimented for proxy, trans and ICAP

INCLUDE FILES IN e2guardianf1.conf

    Filtergroup configuration files may now include other files, enabling
    a more DRY approach to configuration. So configuration common to several
    filtergroups can be placed in a file which is included in the filtergroup
    config file.

    Syntax is same as list includes  -
    .include<full_path_to_file>

    Where single options and list defines with the same name are
    repeated only the last one read will be actioned.  This differs
    from pre-v5 versions where the first single option was actioned and any
    repeats ignored.  This allows the overwriting of single options and
    re-definion of lists in a structured way.

LIST INPUT VIA STDIN

This replaces the totalblocklist in previous versions allowing multiple
lists to be loaded via stdin.  See notes/lists_via_stdin.

OPENSSL v1.1 SUPPORT

    Will now support OpenSSL v1.1 as well as v1.0.2 or above

------------------------------
New in v4 (v4.1).

The v4 is written in c++11 and so to compile it you will need gcc v5.4
or later. (or another complier that supports the c++11 std::thread library).
Note that the target systems may also need an c++11 library update.

REVISED PROCESS MODEL

The parent children process model (which does not scale for very large numbers
of connections) is replaced with a queue/threads based model.

The main thread now only deals with set-up of the logging, listener, and worker
threads, the input (and reinput) of the lists, signals and statistics.

The treads communicate via fi-fo queues within memory and so there is no need
for ipc pipes.

A listener thread is set up for each ip/port combination. They listen for a
connection, accept it and then push the new connection socket on the
appropriate worker queue.

The worker threads pop connections from the worker queue and deal with the
connection.

When a worker wants to log a request it pushes the logging data onto the log
queue. The logging thread will pop the data from the queue, format it and
write it to the log.

Most of the above logic is in FatControler.cpp.  The logic is now much simpler
and has reduced the amount of code in FatControler by over 50%.

Socket classes have been extensively modified to remove interrupt handling (for
list reload etc) and all select calls are removed.  So there is no longer a
need to modify FD_SETSIZE.

New LOptionContainer class has been written to hold list and filter group
setings.   On gentle restart a new LOptionContainer object is created and
loaded with filter group and list settings.   Once fully read in a global
shared pointer is switched from the old list to the new, making actioning
list changes immediate an with no interruption to service.

NOTES FOR PREVIOUS VERSION  - v4.0.1

All pics support has been removed
Mail option not yet implemented.
Url cache not implimented
IP cache not implimented
Auth plugins - tested and working
Scan plugins - some tested
New e2guardian.conf options
   httpworkers
   enablessl

diffstat:

 www/e2guardian/DESCR             |   5 ++++-
 www/e2guardian/Makefile          |   9 +++++----
 www/e2guardian/PLIST             |  38 ++++++++++++++++++++++++++++++--------
 www/e2guardian/distinfo          |  10 +++++-----
 www/e2guardian/files/configfiles |   5 -----
 www/e2guardian/options.mk        |  14 ++++++++++++--
 6 files changed, 56 insertions(+), 25 deletions(-)

diffs (248 lines):

diff -r b110fe427528 -r 1f1246136dea www/e2guardian/DESCR
--- a/www/e2guardian/DESCR      Mon Jul 16 12:57:02 2018 +0000
+++ b/www/e2guardian/DESCR      Mon Jul 16 14:12:34 2018 +0000
@@ -5,4 +5,7 @@
 
 e2guardian is a fork of DansGuardian and the maintainers fully acknowledge
 the work carried out by and the copyright of Daniel Baron and other
-contributors to the Dansguardian project.
+contributors to the DansGuardian project.
+
+Please note that version 5 is not backward compatible with v4 (and earlier)
+configuration files.
diff -r b110fe427528 -r 1f1246136dea www/e2guardian/Makefile
--- a/www/e2guardian/Makefile   Mon Jul 16 12:57:02 2018 +0000
+++ b/www/e2guardian/Makefile   Mon Jul 16 14:12:34 2018 +0000
@@ -1,11 +1,10 @@
-# $NetBSD: Makefile,v 1.5 2017/05/12 11:42:33 jperkin Exp $
+# $NetBSD: Makefile,v 1.6 2018/07/16 14:12:34 sborrill Exp $
 
-DISTNAME=      e2guardian-3.5.1
-PKGREVISION=   1
+DISTNAME=      e2guardian-5.1.1
 CATEGORIES=    www
 MASTER_SITES=  ${MASTER_SITE_GITHUB:=e2guardian/}
 GITHUB_PROJECT=        e2guardian
-GITHUB_TAG=    v${PKGVERSION_NOREV}
+GITHUB_TAG=    V${PKGVERSION_NOREV}
 
 MAINTAINER=    prlw1%cam.ac.uk@localhost
 HOMEPAGE=      http://www.e2guardian.org/
@@ -51,6 +50,8 @@
 
 LIBS.SunOS+=           -liconv
 
+WRKSRC=                ${WRKDIR}/${DISTNAME}
+
 .include "options.mk"
 
 CHECK_WRKREF_SKIP=     sbin/e2guardian
diff -r b110fe427528 -r 1f1246136dea www/e2guardian/PLIST
--- a/www/e2guardian/PLIST      Mon Jul 16 12:57:02 2018 +0000
+++ b/www/e2guardian/PLIST      Mon Jul 16 14:12:34 2018 +0000
@@ -1,4 +1,4 @@
-@comment $NetBSD: PLIST,v 1.4 2018/01/01 22:30:02 rillig Exp $
+@comment $NetBSD: PLIST,v 1.5 2018/07/16 14:12:34 sborrill Exp $
 man/man8/e2guardian.8
 sbin/e2guardian
 share/doc/e2guardian/AuthPlugins
@@ -65,7 +65,6 @@
 share/e2guardian/languages/polish/template.html
 share/e2guardian/languages/portuguese/fancydmtemplate.html
 share/e2guardian/languages/portuguese/messages
-share/e2guardian/languages/portuguese/messagesaccents
 share/e2guardian/languages/portuguese/template.html
 share/e2guardian/languages/ptbrazilian/fancydmtemplate.html
 share/e2guardian/languages/ptbrazilian/messages
@@ -90,7 +89,6 @@
 share/e2guardian/languages/turkish/template.html
 share/e2guardian/languages/ukenglish/fancydmtemplate.html
 share/e2guardian/languages/ukenglish/messages
-share/e2guardian/languages/ukenglish/messages.alt
 share/e2guardian/languages/ukenglish/template.html
 share/e2guardian/scripts/bsd-init
 share/e2guardian/scripts/e2guardian
@@ -104,28 +102,37 @@
 share/examples/e2guardian/authplugins/port.conf
 share/examples/e2guardian/authplugins/proxy-basic.conf
 share/examples/e2guardian/authplugins/proxy-digest.conf
-share/examples/e2guardian/authplugins/proxy-header.conf
 share/examples/e2guardian/authplugins/proxy-ntlm.conf
 ${PLIST.clamd}share/examples/e2guardian/contentscanners/clamdscan.conf
 ${PLIST.cli}share/examples/e2guardian/contentscanners/commandlinescan.conf
+${PLIST.icap}share/examples/e2guardian/contentscanners/icapscan.conf
+share/examples/e2guardian/common.story
 share/examples/e2guardian/downloadmanagers/default.conf
-share/examples/e2guardian/downloadmanagers/fancy.conf
 share/examples/e2guardian/e2guardian.conf
 share/examples/e2guardian/e2guardianf1.conf
+share/examples/e2guardian/examplef1.story
 share/examples/e2guardian/lists/addheaderregexplist
+share/examples/e2guardian/lists/authexceptioniplist
+share/examples/e2guardian/lists/authexceptionsiteiplist
 share/examples/e2guardian/lists/authexceptionsitelist
 share/examples/e2guardian/lists/authexceptionurllist
 share/examples/e2guardian/lists/authplugins/ipgroups
 share/examples/e2guardian/lists/authplugins/portgroups
+share/examples/e2guardian/lists/bannedclientlist
 share/examples/e2guardian/lists/bannedextensionlist
 share/examples/e2guardian/lists/bannediplist
 share/examples/e2guardian/lists/bannedmimetypelist
 share/examples/e2guardian/lists/bannedphraselist
 share/examples/e2guardian/lists/bannedregexpheaderlist
 share/examples/e2guardian/lists/bannedregexpurllist
+share/examples/e2guardian/lists/bannedregexpuseragentlist
 share/examples/e2guardian/lists/bannedrooms/default
 share/examples/e2guardian/lists/bannedsearchlist
+share/examples/e2guardian/lists/bannedsearchoveridelist
+share/examples/e2guardian/lists/bannedsiteiplist
 share/examples/e2guardian/lists/bannedsitelist
+share/examples/e2guardian/lists/bannedsitelistwithbypass
+share/examples/e2guardian/lists/bannedsslsiteiplist
 share/examples/e2guardian/lists/bannedsslsitelist
 share/examples/e2guardian/lists/bannedurllist
 share/examples/e2guardian/lists/contentregexplist
@@ -133,36 +140,49 @@
 ${PLIST.avscan}share/examples/e2guardian/lists/contentscanners/exceptionvirusmimetypelist
 ${PLIST.avscan}share/examples/e2guardian/lists/contentscanners/exceptionvirussitelist
 ${PLIST.avscan}share/examples/e2guardian/lists/contentscanners/exceptionvirusurllist
-share/examples/e2guardian/lists/downloadmanagers/managedextensionlist
-share/examples/e2guardian/lists/downloadmanagers/managedmimetypelist
 share/examples/e2guardian/lists/embededreferersitelist
+share/examples/e2guardian/lists/embededreferersiteiplist
 share/examples/e2guardian/lists/embededrefererurllist
+share/examples/e2guardian/lists/exceptionclientlist
 share/examples/e2guardian/lists/exceptionextensionlist
+share/examples/e2guardian/lists/exceptionfilesiteiplist
 share/examples/e2guardian/lists/exceptionfilesitelist
 share/examples/e2guardian/lists/exceptionfileurllist
 share/examples/e2guardian/lists/exceptioniplist
 share/examples/e2guardian/lists/exceptionmimetypelist
 share/examples/e2guardian/lists/exceptionphraselist
+share/examples/e2guardian/lists/exceptionregexpheaderlist
 share/examples/e2guardian/lists/exceptionregexpurllist
+share/examples/e2guardian/lists/exceptionregexpuseragentlist
 share/examples/e2guardian/lists/exceptionsitelist
+share/examples/e2guardian/lists/exceptionsiteiplist
 share/examples/e2guardian/lists/exceptionurllist
 share/examples/e2guardian/lists/filtergroupslist
 share/examples/e2guardian/lists/greysitelist
+share/examples/e2guardian/lists/greysiteiplist
+share/examples/e2guardian/lists/greysslsiteiplist
 share/examples/e2guardian/lists/greysslsitelist
 share/examples/e2guardian/lists/greyurllist
 share/examples/e2guardian/lists/headerregexplist
 share/examples/e2guardian/lists/localbannedsearchlist
+share/examples/e2guardian/lists/localbannedsiteiplist
 share/examples/e2guardian/lists/localbannedsitelist
+share/examples/e2guardian/lists/localbannedsslsiteiplist
 share/examples/e2guardian/lists/localbannedsslsitelist
 share/examples/e2guardian/lists/localbannedurllist
+share/examples/e2guardian/lists/localexceptionsiteiplist
 share/examples/e2guardian/lists/localexceptionsitelist
 share/examples/e2guardian/lists/localexceptionurllist
+share/examples/e2guardian/lists/localgreysiteiplist
 share/examples/e2guardian/lists/localgreysitelist
+share/examples/e2guardian/lists/localgreysslsiteiplist
 share/examples/e2guardian/lists/localgreysslsitelist
 share/examples/e2guardian/lists/localgreyurllist
 share/examples/e2guardian/lists/logregexpurllist
+share/examples/e2guardian/lists/logsiteiplist
 share/examples/e2guardian/lists/logsitelist
 share/examples/e2guardian/lists/logurllist
+share/examples/e2guardian/lists/nocheckcertsiteiplist
 share/examples/e2guardian/lists/nocheckcertsitelist
 share/examples/e2guardian/lists/phraselists/badwords/weighted_dutch
 share/examples/e2guardian/lists/phraselists/badwords/weighted_french
@@ -240,7 +260,7 @@
 share/examples/e2guardian/lists/phraselists/weapons/weighted
 share/examples/e2guardian/lists/phraselists/weapons/weighted_portuguese
 share/examples/e2guardian/lists/phraselists/webmail/weighted
-share/examples/e2guardian/lists/pics
+share/examples/e2guardian/lists/refererexceptionsiteiplist
 share/examples/e2guardian/lists/refererexceptionsitelist
 share/examples/e2guardian/lists/refererexceptionurllist
 share/examples/e2guardian/lists/searchregexplist
@@ -248,5 +268,7 @@
 share/examples/e2guardian/lists/urlredirectregexplist
 share/examples/e2guardian/lists/urlregexplist
 share/examples/e2guardian/lists/weightedphraselist
+share/examples/e2guardian/preauth.story
+share/examples/e2guardian/site.story
 @pkgdir share/examples/e2guardian/lists/contentscanners
 @pkgdir share/examples/e2guardian/contentscanners
diff -r b110fe427528 -r 1f1246136dea www/e2guardian/distinfo
--- a/www/e2guardian/distinfo   Mon Jul 16 12:57:02 2018 +0000
+++ b/www/e2guardian/distinfo   Mon Jul 16 14:12:34 2018 +0000
@@ -1,6 +1,6 @@
-$NetBSD: distinfo,v 1.3 2017/03/14 15:38:01 sborrill Exp $
+$NetBSD: distinfo,v 1.4 2018/07/16 14:12:34 sborrill Exp $
 
-SHA1 (e2guardian-3.5.1.tar.gz) = ba9190e35c937e934273d1f64aa2f08dbcc489ad
-RMD160 (e2guardian-3.5.1.tar.gz) = 50b1dd13f59d3c3159ebe112560d145ecb016bfe
-SHA512 (e2guardian-3.5.1.tar.gz) = deef797c7ed8ef36c122ab232581c7b1aee91ec7286a26c6b392465a2523dfaf74e0c5ed969f5eef04a10cede3c64b6cb3bcebd8969bce032298bec96ae37b30
-Size (e2guardian-3.5.1.tar.gz) = 529677 bytes
+SHA1 (e2guardian-5.1.1.tar.gz) = 9946688a6c6884807b82f050c5b3dfa48226fc29
+RMD160 (e2guardian-5.1.1.tar.gz) = 26f35ebd3a7addcf62f07b31616aa4c719c6f733
+SHA512 (e2guardian-5.1.1.tar.gz) = a678c6d5b5672e40985cfbb9c718b61a7aee0f67092db6e20706cc525ad58c5b87d296575987a0c3e8ec4fd61b802cfa9d9f8b8d0b3a46cdee0f1f637c9bac37
+Size (e2guardian-5.1.1.tar.gz) = 850980 bytes
diff -r b110fe427528 -r 1f1246136dea www/e2guardian/files/configfiles
--- a/www/e2guardian/files/configfiles  Mon Jul 16 12:57:02 2018 +0000
+++ b/www/e2guardian/files/configfiles  Mon Jul 16 14:12:34 2018 +0000
@@ -3,9 +3,7 @@
 authplugins/port.conf
 authplugins/proxy-basic.conf
 authplugins/proxy-digest.conf
-authplugins/proxy-header.conf
 downloadmanagers/default.conf
-downloadmanagers/fancy.conf
 e2guardian.conf
 e2guardianf1.conf
 lists/addheaderregexplist
@@ -25,8 +23,6 @@
 lists/bannedsslsitelist
 lists/bannedurllist
 lists/contentregexplist
-lists/downloadmanagers/managedextensionlist
-lists/downloadmanagers/managedmimetypelist
 lists/embededreferersitelist
 lists/embededrefererurllist
 lists/exceptionextensionlist
@@ -132,7 +128,6 @@
 lists/phraselists/weapons/weighted
 lists/phraselists/weapons/weighted_portuguese
 lists/phraselists/webmail/weighted
-lists/pics
 lists/refererexceptionsitelist
 lists/refererexceptionurllist
 lists/searchregexplist
diff -r b110fe427528 -r 1f1246136dea www/e2guardian/options.mk
--- a/www/e2guardian/options.mk Mon Jul 16 12:57:02 2018 +0000
+++ b/www/e2guardian/options.mk Mon Jul 16 14:12:34 2018 +0000
@@ -1,7 +1,7 @@
-# $NetBSD: options.mk,v 1.2 2017/03/15 13:42:03 sborrill Exp $
+# $NetBSD: options.mk,v 1.3 2018/07/16 14:12:34 sborrill Exp $
 
 PKG_OPTIONS_VAR=       PKG_OPTIONS.e2guardian
-PKG_SUPPORTED_OPTIONS= debug email commandline clamd ssl
+PKG_SUPPORTED_OPTIONS= debug email commandline clamd ssl icap
 PKG_SUGGESTED_OPTIONS= email ssl
 
 .include "../../mk/bsd.options.mk"
@@ -18,6 +18,16 @@
 .endif
 
 ##
+## Enable support for ICAP AV server content scanner
+##
+PLIST_VARS+=           icap
+.if !empty(PKG_OPTIONS:Micap)
+CONFIGURE_ARGS+=       --enable-icap
+PLIST.icap=            yes
+EGFILES_EXTRA+=                contentscanners/icap.conf
+.endif
+
+##
 ## Enables ssl MITM
 ##
 .if !empty(PKG_OPTIONS:Mssl)