[pkgsrc/trunk]: pkgsrc/sysutils/beats beats: Update to 6.8.8. Changes since ...

[jperkin <jperkin%pkgsrc.org@localhost>]

details:   https://anonhg.NetBSD.org/pkgsrc/rev/1c892d671920
branches:  trunk
changeset: 429048:1c892d671920
user:      jperkin <jperkin%pkgsrc.org@localhost>
date:      Wed Apr 15 13:00:55 2020 +0000

description:
beats: Update to 6.8.8.  Changes since 6.6.2:

Beats version 6.8.8

Add support for Cisco syslog format used by their switch. 10760

Beats version 6.8.7

Fixed bug with elasticsearch/cluster_stats metricset not recording license expiration date correctly. 14541 14591
Make kibana module more resilient to Kibana unavailability. 15258 15270

Beats version 6.8.6

Fix recording of SSL cert metadata for Expired/Unvalidated x509 certs. 13687
Fix marshaling of ms-since-epoch values in elasticsearch/cluster_stats metricset. 14378
Fixed bug with elasticsearch/cluster_stats metricset not recording license ID in the correct field. 14592

Beats version 6.8.5

Convert indexed ms-since-epoch timestamp fields in elasticsearch/ml_job metricset to ints from float64s. 14220 14222

Beats version 6.8.4

Fix delay in enforcing close_renamed and close_removed options. 13488 13907
Fix merging of fields specified in global scope with fields specified under an input’s scope. 3628 13909
Fixed early expiration of templates (Netflow v9 and IPFIX). 13821
Fixed bad handling of sequence numbers when multiple observation domains were exported by a single device (Netflow V9 and IPFIX). 13821
Fixed increased memory usage with large files when multiline pattern does not match. 14068
Mark Kibana usage stats as collected only if API call succeeds. 13881

Beats version 6.8.3

Iterate over journal correctly, so no duplicate entries are sent. 12716
Fix panic in Redis Key metricset when collecting information from a removed key. 13426
Remove _nodes field from under cluster_stats as it’s not being used. 13010
Collect license expiry date fields as well. 11652

Beats version 6.8.2

Process dataset: Do not show non-root warning on Windows. 12740
Host dataset: Export Host fields to gob encoder. 12940
Skipping unparsable log entries from docker json reader 12268
Limit memory usage of Redis replication sessions. https://github.com/elastic/beats/issues/12657[12657

Beats version 6.8.1

Fixed a memory leak when using the add_process_metadata processor under Windows. 12100
Package dataset: Log error when Homebrew is not installed. 11667
Process dataset: Fixed a memory leak under Windows. 12100
Login dataset: Fix re-read of utmp files. 12028
Package dataset: Fixed a crash inside librpm after Auditbeat has been running for a while. 12147 12168
Fix direction of incoming IPv6 sockets. 12248
Package dataset: Auto-detect package directories. 12289
System module: Start system module without host ID. 12373
Host dataset: Fix reboot detection logic. 12591
Fix goroutine leak happening when harvesters are dynamically stopped. 11263
Fix initialization of the TCP input logger. 11605
Fix goroutine leak caused on initialization failures of log input. 12125
Fix memory leak in Filebeat pipeline acker. 12063
Fix goroutine leak on non-explicit finalization of log input. 12164
When TLS is configured for the TCP input and a certificate_authorities is configured we now default to required for the client_authentication. 12584
Avoid generating hints-based configuration with empty hosts when no exposed port is suitable for the hosts hint. 8264 12086
Fix direction of incoming IPv6 sockets. 12248
Validate that kibana/status metricset cannot be used when xpack is enabled. 12264
In the kibana/stats metricset, only log error (don’t also index it) if xpack is enabled. 12353
The elasticsearch/index_summary metricset gracefully handles an empty Elasticsearch cluster when xpack.enabled: true is set. 12489 12487
When TLS is configured for the http metricset and a certificate_authorities is configured we now default to required for the client_authentication. 12584
Fixed a memory leak when using process monitoring under Windows. 12100
Improved debug logging efficiency in PGQSL module. 12150
Add support to the system package dataset for the SUSE OS family. 11634
Add validation for elasticsearch and kibana modules' metricsets when xpack.enabled is set to true. 12386

Beats version 6.8.0

Updates to support changes to licensing of security features.
Some Elastic Stack security features, such as encrypted communications, file and native authentication, and role-based access control, are now available in more subscription levels. For details, see 
https://www.elastic.co/subscriptions.

Beats version 6.7.2

Relax validation of the X-Pack license UID value. 11640
Fix a parsing error with the X-Pack license check on 32-bit system. 11650
Fix OS family classification in add_host_metadata for Amazon Linux, Raspbian, and RedHat Linux. 9134 11494
Fix false positives reported in the host.containerized field added by add_host_metadata. 11494
Fix the add_host_metadata’s host.id field on older Linux versions. 11494
Package dataset: dlopen versioned librpm shared objects. 11565
Package dataset: Nullify Librpm’s rpmsqEnable. 11628
Don’t apply multiline rules in Logstash json logs. 11346
Prevent the docker/memory metricset from processing invalid events before container start 11676
Add support to the system package dataset for the SUSE OS family. 11634

Beats version 6.7.1

Initialize the Paths before the keystore and save the keystore into data/{beatname}.keystore. 10706
Remove IP fields from default_field in Elasticsearch template. 11399

Beats version 6.7.0

Port settings have been deprecated in redis/logstash output and will be removed in 7.0. 9915
Update the code of Central Management to align with the new returned format. 10019
Allow Central Management to send events back to kibana. 9382
Fix panic if fields settting is used to configure hosts.x fields. 10824 10935
Introduce query.default_field as part of the template. 11205
Beats Xpack now checks for Basic license on connect. 11296
Filesets with multiple ingest pipelines added in 8914 only work with Elasticsearch >= 6.5.0 10001
Add grok pattern to support redis 5.0.3 log timestamp. 9819 10033
Ingesting Elasticsearch audit logs is only supported with Elasticsearch 6.5.0 and above 8852
Remove ecs option from user_agent processors when loading pipelines with Filebeat 6.7.x into Elasticsearch < 6.7.0. 10655 11362
Remove monitor generator script that was rarely used. 9648
Fix TLS certificate DoS vulnerability. 10303
Fix panic and file unlock in spool on atomic operation (arm, x86-32). File lock was not released when panic occurs, leading to the beat deadlocking on startup. 10289
Adding logging traces at debug level when the pipeline client receives the following events: onFilteredOut, onDroppedOnPublish. 9016
Do not panic when no tokenizer string is configured for a dissect processor. 8895
Fix a issue when remote and local configuration didn’t match when fetching configuration from Central Management. 10587
Add ECS-like selectors and dedotting to docker autodiscover. 10757 10862
Fix encoding of timestamps when using disk spool. 10099
Include ip and boolean type when generating index pattern. 10995
Using an environment variable for the password when enrolling a beat will now raise an error if the variable doesn’t exist. 10936
Cancelling enrollment of a beat will not enroll the beat. 10150
Remove IP fields from default_field in Elasticsearch template. 11399
Package: Disable librpm signal handlers. 10694
Login: Handle different bad login UTMP types. 10865
Fix hostname references in System module dashbords. 11064
User dataset: Numerous fixes to error handling. 10942
Support IPv6 addresses with zone id in IIS ingest pipeline. 9836 error log: 9869 access log: 10029
Fix bad bytes count in docker input when filtering by stream. 10211
Fixed data types for roles and indices fields in elasticsearch/audit fileset 10307
Cover empty request data, url and version in Apache2 modulehttps://github.com/elastic/beats/pull/10846[10846]
Fix a bug with the convert_timezone option using the incorrect timezone field. 11055 11164
Change URLPATH grok pattern to support brackets. 11135 11252
Add support for iis log with different address format. 11255 11256
Add fix to parse syslog message with priority value 0. 11010
Host header can now be overridden for HTTP requests sent by Heartbeat monitors. 9516
Fix checks for TCP send/receive data 10777
Do not stop collecting events when journal entries change. 9994
Fix MongoDB dashboard that had some incorrect field names from status Metricset 9795 9715
Fix issue that would prevent collection of processes without command line on Windows. 10196
Fixed data type for tags field in docker/container metricset 10307
Fixed data type for tags field in docker/image metricset 10307
Fixed data type for isr field in kafka/partition metricset 10307
Fixed data types for various hosts fields in mongodb/replstatus metricset 10307
Added function to close sql database connection. 10355
Fix parsing error using GET in Jolokia module. 11075 11071
Fix Winlogbeat escaping CR, LF and TAB characters. 11328 11357
Correctly extract Kinesis Data field from the Kinesis Record. 11141
Add the required permissions to the role when deployment SQS functions. 9152
Affecting all Beats
Add ip fields to default_field in Elasticsearch template. 11035
Add cleanup_timeout option to docker autodiscover, to wait some time before removing configurations after a container is stopped. 10374 10905
System module process dataset: Add user information to processes. 9963
Add system package dataset. 10225
Add system module login dataset. 9327
Add entity_id fields. 10500
Add seven dashboards for the system module. 10511
Add field log.source.address and log.file.path to replace source. 9435
Support mysql 5.7.22 slowlog starting with time information. 7892 9647
Add support for ssl_request_log in apache2 module. 8088 9833
Add support for iis 7.5 log format. 9753 9967
Add support for MariaDB in the slowlog fileset of mysql module. 9731
Add convert_timezone to nginx module. 9839 10148
Add support for Percona in the slowlog fileset of mysql module. 6665 10227
Added support for ingesting structured Elasticsearch audit logs 8852
New iptables module that receives iptables/ip6tables logs over syslog or file. Supports Ubiquiti Firewall extensions. 8781 10176
Populate more ECS fields in the Suricata module. 10006
Made monitors.d configuration part of the default config. 9004
Autodiscover metadata is now included in events by default. So, if you are using the docker provider for instance, you’ll see the correct fields under the docker key. 10258
Add field event.dataset which is {module}.{metricset}.
Add more TCP statuses to socket_summary metricset. 9430
Remove experimental tag from ceph metricsets. 9708
Add key metricset to the Redis module. 9582 9657
Add DeDot for kubernetes labels and annotations. 9860 9939
Add docker event metricset. 9856
Release Ceph module as GA. 10202
Release windows Metricbeat module as GA. 10163
Release traefik Metricbeat module as GA. 10166
List filesystems on Windows that have an access path but not an assigned letter 8916 10196
Release uswgi Metricbeat module GA. 10164
Release php_fpm module as GA. 10198
Release Memcached module as GA. 10199
Release etcd module as GA. 10200
Release kubernetes apiserver and event metricsets as GA 10212
Release Couchbase module as GA. 10201
Release aerospike module as GA. 10203
Release envoyproxy module GA. 10223
Release mongodb.metrics and mongodb.replstatus as GA. 10242
Release mysql.galera_status as Beta. 10242
Release postgresql.statement as GA. 10242
Release RabbitMQ Metricbeat module GA. 10165
Release Dropwizard module as GA. 10240
Release Graphite module as GA. 10240
Release http.server metricset as GA. 10240
Add support for MySQL 8.0 and tests also for Percona and MariaDB. 10261
Release use of xpack.enabled: true flag in Elasticsearch and Kibana modules as GA. 10222
Release Elastic stack modules (Elasticsearch, Logstash, and Kibana) as GA. 10094
Add remaining memory metrics of pods in Kubernetes metricbeat module 10157
Added 'server' Metricset to Zookeeper Metricbeat module 8938 10341
Add overview dashboard to Zookeeper Metricbeat module 10379
Mark Functionbeat as GA. 10564
Functionbeat can now deploy a function for Kinesis. 10116
Allow functionbeat to use the keystore. 9009
Deprecate field source. Will be replaced by log.source.address and log.file.path in 7.0. 9435
Deprecate field metricset.rtt. Replaced by event.duration which is in nano instead of micro seconds.
Support new TLS version negotiation introduced in TLS 1.3. 8647.
Journalbeat requires at least systemd v233 in order to follow entries after journal changes (rotation, vacuum).

diffstat:

 sysutils/beats/Makefile                                                  |  13 +-
 sysutils/beats/distinfo                                                  |  13 +-
 sysutils/beats/patches/patch-libbeat_common_file_stderr__other.go        |  20 +++
 sysutils/beats/patches/patch-vendor_github.com_tsg_gopacket_pcap_pcap.go |  63 +++++++++-
 4 files changed, 95 insertions(+), 14 deletions(-)

diffs (157 lines):

diff -r 127d77464e0a -r 1c892d671920 sysutils/beats/Makefile
--- a/sysutils/beats/Makefile   Wed Apr 15 11:57:35 2020 +0000
+++ b/sysutils/beats/Makefile   Wed Apr 15 13:00:55 2020 +0000
@@ -1,13 +1,12 @@
-# $NetBSD: Makefile,v 1.51 2020/04/12 11:01:46 bsiegert Exp $
+# $NetBSD: Makefile,v 1.52 2020/04/15 13:00:55 jperkin Exp $
 
-DISTNAME=              beats-6.6.2
-PKGREVISION=           11
+DISTNAME=              beats-6.8.8
 CATEGORIES=            sysutils
 MASTER_SITES=          ${MASTER_SITE_GITHUB:=elastic/}
 GITHUB_PROJECT=                beats
 GITHUB_TAG=            v${PKGVERSION_NOREV}
 
-MAINTAINER=            fhajny%NetBSD.org@localhost
+MAINTAINER=            pkgsrc-users%NetBSD.org@localhost
 HOMEPAGE=              https://www.elastic.co/products/beats
 COMMENT=               Data Shippers for Elasticsearch
 LICENSE=               apache-2.0
@@ -34,10 +33,12 @@
 
 .for beat in ${BEATS}
 CHECK_RELRO_SKIP+=     bin/${beat}
-CONF_FILES+=           share/examples/beats/${beat}.yml                        \
-                               ${PKG_SYSCONFDIR}/${beat}.yml
+CONF_FILES+=           share/examples/beats/${beat}.yml \
+                       ${PKG_SYSCONFDIR}/${beat}.yml
 .endfor
 
+BUILD_DEFS+=           VARBASE
+
 BEATS_DIR?=            ${VARBASE}/db/beats
 BEATS_LOGDIR?=         ${VARBASE}/log/beats
 
diff -r 127d77464e0a -r 1c892d671920 sysutils/beats/distinfo
--- a/sysutils/beats/distinfo   Wed Apr 15 11:57:35 2020 +0000
+++ b/sysutils/beats/distinfo   Wed Apr 15 13:00:55 2020 +0000
@@ -1,10 +1,11 @@
-$NetBSD: distinfo,v 1.26 2019/03/18 12:36:42 he Exp $
+$NetBSD: distinfo,v 1.27 2020/04/15 13:00:55 jperkin Exp $
 
-SHA1 (beats-6.6.2.tar.gz) = ca8b05fa23d383e90b5d8357224491b8d3f38c10
-RMD160 (beats-6.6.2.tar.gz) = cdb95337644c09a40a2eb556b76e9cff6e93bde7
-SHA512 (beats-6.6.2.tar.gz) = 1bc16fdd4525a197e4deeedb29fbc77ec44418c5eb371be95fa09555c1344a2e423aab83207c8313ac6cd33ce7fe91fe7ca06837302917465b5e1803dbf7f66d
-Size (beats-6.6.2.tar.gz) = 29119533 bytes
+SHA1 (beats-6.8.8.tar.gz) = 2619cff0df437439b2a800dde11fdfc8935dca5f
+RMD160 (beats-6.8.8.tar.gz) = b4916e3a5f7afcfc35e9fc860cccf7ff44ca93c0
+SHA512 (beats-6.8.8.tar.gz) = 6a86948000e77d32587c523d018e4f1561ec7f44f68644971f44f8fd23351b3aad48f3682563467020204de1447a9ed34bd9366d2b44c13bdcb2a28af41ab6d6
+Size (beats-6.8.8.tar.gz) = 35564363 bytes
+SHA1 (patch-libbeat_common_file_stderr__other.go) = 9e1f28a9721c996ba75a7712d39c70186f549067
 SHA1 (patch-vendor_github.com_insomniacslk_dhcp_dhcpv4_client.go) = 4588ee4feba431bdc1c8250d58fa9ac66f733bb3
-SHA1 (patch-vendor_github.com_tsg_gopacket_pcap_pcap.go) = 2ef13f84cbf2e1da428e7b08625ffa6111ebcdf5
+SHA1 (patch-vendor_github.com_tsg_gopacket_pcap_pcap.go) = 887b009bb2613ed5241fbf9a9ff3e8e75df98cae
 SHA1 (patch-vendor_golang.org_x_crypto_blake2b_blake2bAVX2__amd64.go) = 2ee9c516c6f0da5789b97b498f5db7c7eae106cd
 SHA1 (patch-vendor_golang.org_x_crypto_blake2b_blake2bAVX2__amd64.s) = b6aabeda2abad29d9f5d4032f57ea66eb4231688
diff -r 127d77464e0a -r 1c892d671920 sysutils/beats/patches/patch-libbeat_common_file_stderr__other.go
--- /dev/null   Thu Jan 01 00:00:00 1970 +0000
+++ b/sysutils/beats/patches/patch-libbeat_common_file_stderr__other.go Wed Apr 15 13:00:55 2020 +0000
@@ -0,0 +1,20 @@
+$NetBSD: patch-libbeat_common_file_stderr__other.go,v 1.1 2020/04/15 13:00:55 jperkin Exp $
+
+Use more portable unix.Dup2().
+
+--- libbeat/common/file/stderr_other.go.orig   2020-03-05 13:27:52.000000000 +0000
++++ libbeat/common/file/stderr_other.go
+@@ -21,11 +21,11 @@ package file
+ 
+ import (
+       "os"
+-      "syscall"
++      "golang.org/x/sys/unix"
+ )
+ 
+ // RedirectStandardError causes all standard error output to be directed to the
+ // given file.
+ func RedirectStandardError(toFile *os.File) error {
+-      return syscall.Dup2(int(toFile.Fd()), 2)
++      return unix.Dup2(int(toFile.Fd()), 2)
+ }
diff -r 127d77464e0a -r 1c892d671920 sysutils/beats/patches/patch-vendor_github.com_tsg_gopacket_pcap_pcap.go
--- a/sysutils/beats/patches/patch-vendor_github.com_tsg_gopacket_pcap_pcap.go  Wed Apr 15 11:57:35 2020 +0000
+++ b/sysutils/beats/patches/patch-vendor_github.com_tsg_gopacket_pcap_pcap.go  Wed Apr 15 13:00:55 2020 +0000
@@ -1,8 +1,9 @@
-$NetBSD: patch-vendor_github.com_tsg_gopacket_pcap_pcap.go,v 1.1 2017/01/05 16:43:09 fhajny Exp $
+$NetBSD: patch-vendor_github.com_tsg_gopacket_pcap_pcap.go,v 1.2 2020/04/15 13:00:55 jperkin Exp $
 
 Just require pcap, fixes NetBSD.
+Pull in upstream patch to fix newer cgo syntax.
 
---- vendor/github.com/tsg/gopacket/pcap/pcap.go.orig   2016-12-06 07:41:08.000000000 +0000
+--- vendor/github.com/tsg/gopacket/pcap/pcap.go.orig   2020-03-05 13:27:52.000000000 +0000
 +++ vendor/github.com/tsg/gopacket/pcap/pcap.go
 @@ -8,14 +8,7 @@
  package pcap
@@ -20,3 +21,61 @@
  #include <stdlib.h>
  #include <pcap.h>
  
+@@ -170,7 +163,7 @@ type InterfaceAddress struct {
+ // BPF is a compiled filter program, useful for offline packet matching.
+ type BPF struct {
+       orig string
+-      bpf  _Ctype_struct_bpf_program // takes a finalizer, not overriden by outsiders
++      bpf  C.struct_bpf_program // takes a finalizer, not overriden by outsiders
+ }
+ 
+ // BlockForever, when passed into OpenLive/SetTimeout, causes it to block forever
+@@ -382,7 +375,7 @@ func (p *Handle) Error() error {
+ 
+ // Stats returns statistics on the underlying pcap handle.
+ func (p *Handle) Stats() (stat *Stats, err error) {
+-      var cstats _Ctype_struct_pcap_stat
++      var cstats C.struct_pcap_stat
+       if -1 == C.pcap_stats(p.cptr, &cstats) {
+               return nil, p.Error()
+       }
+@@ -443,7 +436,7 @@ func (p *Handle) SetBPFFilter(expr strin
+               }
+       }
+ 
+-      var bpf _Ctype_struct_bpf_program
++      var bpf C.struct_bpf_program
+       cexpr := C.CString(expr)
+       defer C.free(unsafe.Pointer(cexpr))
+ 
+@@ -486,7 +479,7 @@ func (b *BPF) String() string {
+ }
+ 
+ // BPF returns the compiled BPF program.
+-func (b *BPF) BPF() _Ctype_struct_bpf_program {
++func (b *BPF) BPF() C.struct_bpf_program {
+       return b.bpf
+ }
+ 
+@@ -549,10 +542,10 @@ func FindAllDevs() (ifs []Interface, err
+       return
+ }
+ 
+-func findalladdresses(addresses *_Ctype_struct_pcap_addr) (retval []InterfaceAddress) {
++func findalladdresses(addresses *C.struct_pcap_addr) (retval []InterfaceAddress) {
+       // TODO - make it support more than IPv4 and IPv6?
+       retval = make([]InterfaceAddress, 0, 1)
+-      for curaddr := addresses; curaddr != nil; curaddr = (*_Ctype_struct_pcap_addr)(curaddr.next) {
++      for curaddr := addresses; curaddr != nil; curaddr = (*C.struct_pcap_addr)(curaddr.next) {
+               var a InterfaceAddress
+               var err error
+               // In case of a tun device on Linux the link layer has no curaddr.addr.
+@@ -818,7 +811,7 @@ func (h *Handle) NewDumper(file string)
+ // Writes a packet to the file. The return values of ReadPacketData
+ // can be passed to this function as arguments.
+ func (d *Dumper) WritePacketData(data []byte, ci gopacket.CaptureInfo) (err error) {
+-      var pkthdr _Ctype_struct_pcap_pkthdr
++      var pkthdr C.struct_pcap_pkthdr
+       pkthdr.caplen = C.bpf_u_int32(ci.CaptureLength)
+       pkthdr.len = C.bpf_u_int32(ci.Length)
+