CVS commit: pkgsrc/sysutils/beats

["Filip Hajny" <fhajny%netbsd.org@localhost>]

Module Name:    pkgsrc
Committed By:   fhajny
Date:           Mon Nov 20 15:55:16 UTC 2017

Modified Files:
        pkgsrc/sysutils/beats: Makefile PLIST distinfo

Log Message:
Update sysutils/beats to 6.0.0.

=== Beats version 6.0.0

==== Breaking changes

Affecting all Beats

- The log directory (`path.log`) for Windows services is now set to
  `C:\ProgramData\[beatname]\logs`.
- The _all field is disabled in Elasticsearch 6.0.
- Fail if removed setting output.X.flush_interval is explicitly
  configured.
- Rename the `/usr/bin/beatname.sh` script (e.g. `metricbeat.sh`) to
  `/usr/bin/beatname`.
- Beat does not start if elasticsearch index pattern was modified but
  not the template name and pattern.
- Fail if removed setting output.X.flush_interval is explicitly
  configured.
- Rename `kubernetes` processor to `add_kubernetes_metadata`.
- Rename `.full.yml` config files to `*.reference.yml`.
- The `scripts/import_dashboards` is removed from packages. Use the
  `setup` command instead.
- Change format of the saved kibana dashboards to have a single JSON
  file for each dashboard
- Rename `configtest` command to `test config`.
- Remove setting `queue_size` and `bulk_queue_size`.
- Remove setting `dashboard.snapshot` and `dashboard.snapshot_url`.
- Beats can no longer be launched from Windows Explorer (GUI), command
  line is required.

Filebeat

- Rename `input_type` field to `prospector.type`
- The `@metadata.type` field, added by the Logstash output, is now
  hardcoded to `doc` and will be removed in future versions.

Metricbeat

- Change all `system.cpu..pct` metrics to be scaled by the number of
  CPU cores.
- Remove filters setting from metricbeat modules.
- Added `type` field to filesystem metrics.

Heartbeat

- Renamed the heartbeat RPM/DEB name to `heartbeat-elastic`.

Packetbeat

- Remove not-working `runoptions.uid` and `runoptions.gid` options in
  Packetbeat.
- Remove the already unsupported `pf_ring` sniffer option.

Auditbeat

- Changed file metricset config to make `file.paths` a list instead of
  a dictionary.

==== Bugfixes

Affecting all Beats

- Fix data race accessing watched containers.
- Do not require template if index change and template disabled
- Fix missing ACK in redis output.
- Fix the `/usr/bin/beatname` script to accept `-d ""` as a parameter.
- Combine `fields.yml` properties when they are defined in different
  sources.
- Keep Docker & Kubernetes pod metadata after container dies while
  they are needed by processors.
- Fix `fields.yml` lookup when using `export template` with a custom
  `path.config` param.
- Remove runner creation from every reload check
- Fix add_kubernetes_metadata matcher registry lookup.
- Register kubernetes `field_format` matcher and remove logger in
  `Encode` API
- Fix go plugins not loaded when beat starts
- Add support for `initContainers` in `add_kubernetes_metadata`
  processor.
- Eliminate deprecated _default_ mapping in 6.x
- Fix pod name indexer to use both namespace, pod name to frame index
  key
- Don't stop with error loading the ES template if the ES output is
  not enabled.
- Fix race condition in internal logging rotator.
- Normalize all times to UTC to ensure proper index naming.
- Fix issue with loading dashboards to ES 6.0 when .kibana index did
  not already exist.
- Fix importing the dashboards when the limit for max open files is
  too low.
- Fix configuration documentation for kubernetes processor
- Fix misspelling in `add_locale` configuration option for
  abbreviation.

Filebeat

- Fix machine learning jobs setup for dynamic modules.
- Fix default paths for redis 4.0.1 logs on macOS
- Fix Filebeat not starting if command line and modules configs are
  used together.
- Fix double `@timestamp` field when JSON decoding was used.
- Fix issue where the `fileset.module` could have the wrong value.
- Fix race condition on harvester stopping with reloading enabled.
- Fix recursive glob config parsing and resolution across restarts.
- Allow string characters in user agent patch version (NGINX and
  Apache)
- Fix grok pattern in filebeat module system/auth without hostname.

Winlogbeat

- Removed validation of top-level config keys.

Metricbeat

- Use `beat.name` instead of `beat.hostname` in the Host Overview
  dashboard.
- Fix the loading of 5.x dashboards.
- Fix a memory allocation issue where more memory was allocated than
  needed in the windows-perfmon metricset.
- Don't start metricbeat if external modules config is wrong and
  reload is disabled
- The MongoDB module now connects on each fetch, to avoid stopping the
  whole Metricbeat instance if MongoDB is not up when starting.
- Fix kubernetes events module to be able to index time fields
  properly.
- Fixed `cmd_set` and `cmd_get` being mixed in the Memcache module.
- Added missing mongodb configuration file to the `modules.d` folder.
- Fix wrong MySQL CRUD queries timelion visualization
- Add new metrics to CPU metricsset
- Fix issue affecting Windows services timing out at startup.
- Fix incorrect docker.diskio.total metric calculation.
- Vsphere module: used memory field corrected.
- Set correct format for percent fields in memory module.
- Fix a debug statement that said a module wrapper had stopped when it
  hadn't.
- Use MemAvailable value from /proc/meminfo on Linux 3.14.
- Fix panic when events were dropped by filters.
- Add filtering to system filesystem metricset to remove relative
  mountpoints like those from Linux network namespaces.
- Remove unnecessary print statement in schema apis.
- Fix type of field `haproxy.stat.check.health.last`.

Heartbeat

- Fix monitor.name being empty by default.
- Fix wrong event timestamps.

Packetbeat

- Fix missing length check in the PostgreSQL module.
- Fix panic in ACK handler if event is dropped on blocked queue
- Update flow timestamp on each packet being received.
- Enabled /proc/net/tcp6 scanning and fixed ip v6 parsing.
- Enable memcache filtering only if a port is specified in the config
  file.

Auditbeat

- Fix `file.max_file_size` config option for the audit file metricset.

==== Added

Affecting all Beats

- Enable flush timeout by default.
- Add @metadata.version to events send to Logstash.
- Add setting to enable/disable the slow start in logstash output.
- Update init scripts to use the `test config` subcommand instead of
  the deprecated `-configtest` flag.
- Get by default the credentials for connecting to Kibana from the
  Elasticsearch output configuration.
- Added `cloud.id` and `cloud.auth` settings, for simplifying using
  Beats with the Elastic Cloud.
- Add lz4 compression support to kafka output.
- Add newer kafka versions to kafka output.
- Configure the index name when loading the dashboards and the index
  pattern.
- New cli subcommands interface.
- Allow source path matching in `add_docker_metadata` processor.
- Add support for analyzers and multifields in fields.yml.
- Add support for JSON logging.
- Add `test output` command, to test Elasticsearch and Logstash output
  settings.
- Introduce configurable event queue settings: queue.mem.events,
  queue.mem.flush.min_events and queue.mem.flush.timeout.
- Enable pipelining in Logstash output by default.
- Added 'result' field to Elasticsearch QueryResult struct for
  compatibility with 6.x Index and Delete API responses.
- The sample dashboards are now included in the Beats packages.
- Add `pattern` option to be used in the fields.yml to specify the
  pattern for a number field.
- Upgraded to Golang 1.8.3.
- Added the possibility to set Elasticsearch mapping template settings
  from the Beat configuration file.
- Add a variable to the SysV init scripts to make it easier to change
  the user.
- Add the option to write the generated Elasticsearch mapping template
  into a file.
- Add `instance_name` in GCE add_cloud_metadata processor.
- Add `add_docker_metadata` processor.
- Add `logging.files` `permissions` option.

Filebeat

- Add Kubernetes manifests to deploy Filebeat.
- Changed the number of shards in the default configuration to 3.
- Don't start filebeat if external modules/prospectors config is wrong
  and reload is disabled
- Add `filebeat.registry_flush` setting, to delay the registry
  updates.
- Add experimental Redis module.
- Nginx module: use the first not-private IP address as the remote_ip.
- Load Ingest Node pipelines when the Elasticsearch connection is
  established, instead of only once at startup.
- Add support for loading Xpack Machine Learning configurations from
  the modules, and added sample configurations for the Nginx module.
- Add udp prospector type.
- Enabled Cgo which means libc is dynamically compiled.
- Add Beta module config reloading mechanism
- Remove spooler and publisher components and settings.
- Added ability to sort harvested files.
- Add experimental Redis slow log prospector type.

Winlogbeat

- Changed the number of shards in the default configuration to 3.
- Add the ability to use LevelRaw if Level isn't populated in the
  event XML.

Metricbeat

- Add Kubernetes manifests to deploy Metricbeat.
- Auto-select a hostname (based on the host on which the Beat is
  running) in the Host Overview dashboard.
- Add `filesystem.ignore_types` to system module for ignoring
  filesystem types.
- Add support to exclude labels from kubernetes pod metadata.
- Add random startup delay to each metricset to avoid the thundering
  herd problem.
- Add the ability to configure audit rules to the kernel module.
- Add the ability to configure kernel's audit failure mode.
- Add experimental Aerospike module.
- Vsphere module: collect custom fields from virtual machines.
- Add `test modules` command, to test modules expected output.
- Add `processors` setting to metricbeat modules.
- Support `npipe` protocol (Windows) in Docker module.
- Add macOS implementation of the system diskio metricset.
- Add process_summary metricset that records high level metrics about
  processes.
- Add `kube-state-metrics` based metrics to `kubernetes` module
- Add debug logging to Jolokia JMX metricset.
- Add events metricset for kubernetes metricbeat module
- Change Metricbeat default configuration file to be better optimized
  for most users.
- Add experimental RabbitMQ module.
- Add Kibana dashboard for the Kubernetes modules.

Heartbeat

- Changed the number of shards in the default configuration to 1.
- Enabled Cgo which means libc is dynamically compiled.

Packetbeat

- Changed the number of shards in the default configuration to 3.

Auditbeat

- Changed the number of shards in the default configuration to 3.
- Add support for receiving audit events using a multicast socket.
- Added `file.hash_types` config option for controlling the hash
  types.
- Added the ability to specify byte unit suffixes to
  `file.max_file_size`.
- Add file integrity metricset to the audit module.

==== Deprecated

Affecting all Beats

- The `@metadata.type` field, added by the Logstash output, is
  deprecated, hardcoded to `doc` and will be removed in future
versions.

Filebeat

- The `filebeat.config_dir` option is deprecated. Use
  `filebeat.config.prospector` options instead.
- Deprecate `input_type` prospector config. Use `type` config option
  instead.

=== Beats version 5.6.4

- Fix race condition in internal logging rotator.
- Add support for enabling TLS renegotiation.
- Add setting to enable/disable the slow start in logstash output.
- Packetbeat: Fix missing length check in the PostgreSQL module.


To generate a diff of this commit:
cvs rdiff -u -r1.19 -r1.20 pkgsrc/sysutils/beats/Makefile
cvs rdiff -u -r1.1 -r1.2 pkgsrc/sysutils/beats/PLIST
cvs rdiff -u -r1.16 -r1.17 pkgsrc/sysutils/beats/distinfo

Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.

Modified files:

Index: pkgsrc/sysutils/beats/Makefile
diff -u pkgsrc/sysutils/beats/Makefile:1.19 pkgsrc/sysutils/beats/Makefile:1.20
--- pkgsrc/sysutils/beats/Makefile:1.19 Thu Nov  2 15:30:50 2017
+++ pkgsrc/sysutils/beats/Makefile      Mon Nov 20 15:55:16 2017
@@ -1,6 +1,6 @@
-# $NetBSD: Makefile,v 1.19 2017/11/02 15:30:50 fhajny Exp $
+# $NetBSD: Makefile,v 1.20 2017/11/20 15:55:16 fhajny Exp $
 
-DISTNAME=              beats-5.6.3
+DISTNAME=              beats-6.0.0
 CATEGORIES=            sysutils
 MASTER_SITES=          ${MASTER_SITE_GITHUB:=elastic/}
 GITHUB_PROJECT=                beats
@@ -28,10 +28,6 @@ CHECK_RELRO_SKIP+=   bin/packetbeat
 .for beat in ${BEATS}
 CONF_FILES+=           share/examples/beats/${beat}.yml                        \
                                ${PKG_SYSCONFDIR}/${beat}.yml
-CONF_FILES+=           share/examples/beats/${beat}.template.json              \
-                               ${PKG_SYSCONFDIR}/${beat}.template.json
-CONF_FILES+=           share/examples/beats/${beat}.template-es2x.json         \
-                               ${PKG_SYSCONFDIR}/${beat}.template-es2x.json
 .endfor
 
 BEATS_DIR?=            ${VARBASE}/db/beats
@@ -60,11 +56,7 @@ do-build:
 do-install:
 .for beat in ${BEATS}
        ${INSTALL_PROGRAM} ${WRKSRC}/${beat}/${beat} ${DESTDIR}${PREFIX}/bin
-       ${INSTALL_DATA} ${WRKSRC}/${beat}/${beat}.full.yml                      \
-               ${DESTDIR}${PREFIX}/share/examples/beats
-       ${INSTALL_DATA} ${WRKSRC}/${beat}/${beat}.template-es2x.json            \
-               ${DESTDIR}${PREFIX}/share/examples/beats
-       ${INSTALL_DATA} ${WRKSRC}/${beat}/${beat}.template.json                 \
+       ${INSTALL_DATA} ${WRKSRC}/${beat}/${beat}.reference.yml                 \
                ${DESTDIR}${PREFIX}/share/examples/beats
        ${INSTALL_DATA} ${WRKSRC}/${beat}/${beat}.yml                           \
                ${DESTDIR}${PREFIX}/share/examples/beats

Index: pkgsrc/sysutils/beats/PLIST
diff -u pkgsrc/sysutils/beats/PLIST:1.1 pkgsrc/sysutils/beats/PLIST:1.2
--- pkgsrc/sysutils/beats/PLIST:1.1     Thu Jan  5 16:43:09 2017
+++ pkgsrc/sysutils/beats/PLIST Mon Nov 20 15:55:16 2017
@@ -1,21 +1,13 @@
-@comment $NetBSD: PLIST,v 1.1 2017/01/05 16:43:09 fhajny Exp $
+@comment $NetBSD: PLIST,v 1.2 2017/11/20 15:55:16 fhajny Exp $
 bin/filebeat
 bin/heartbeat
 bin/metricbeat
 bin/packetbeat
-share/examples/beats/filebeat.full.yml
-share/examples/beats/filebeat.template-es2x.json
-share/examples/beats/filebeat.template.json
+share/examples/beats/filebeat.reference.yml
 share/examples/beats/filebeat.yml
-share/examples/beats/heartbeat.full.yml
-share/examples/beats/heartbeat.template-es2x.json
-share/examples/beats/heartbeat.template.json
+share/examples/beats/heartbeat.reference.yml
 share/examples/beats/heartbeat.yml
-share/examples/beats/metricbeat.full.yml
-share/examples/beats/metricbeat.template-es2x.json
-share/examples/beats/metricbeat.template.json
+share/examples/beats/metricbeat.reference.yml
 share/examples/beats/metricbeat.yml
-share/examples/beats/packetbeat.full.yml
-share/examples/beats/packetbeat.template-es2x.json
-share/examples/beats/packetbeat.template.json
+share/examples/beats/packetbeat.reference.yml
 share/examples/beats/packetbeat.yml

Index: pkgsrc/sysutils/beats/distinfo
diff -u pkgsrc/sysutils/beats/distinfo:1.16 pkgsrc/sysutils/beats/distinfo:1.17
--- pkgsrc/sysutils/beats/distinfo:1.16 Tue Oct 17 11:31:36 2017
+++ pkgsrc/sysutils/beats/distinfo      Mon Nov 20 15:55:16 2017
@@ -1,7 +1,7 @@
-$NetBSD: distinfo,v 1.16 2017/10/17 11:31:36 fhajny Exp $
+$NetBSD: distinfo,v 1.17 2017/11/20 15:55:16 fhajny Exp $
 
-SHA1 (beats-5.6.3.tar.gz) = 1c246928966cd2d6f22bd227104b2134cc8421ba
-RMD160 (beats-5.6.3.tar.gz) = 6b874330830f43831d13547bf069d81eb6e80ffd
-SHA512 (beats-5.6.3.tar.gz) = 6b3d61f18b1bd45db4ba5d08e92b94f1fb48610658237c91f4404a4074f82badbac2ab9bc236c1f64567a0dd1f62022c3d092a9017073069b4497c01f1b1d94b
-Size (beats-5.6.3.tar.gz) = 18140948 bytes
+SHA1 (beats-6.0.0.tar.gz) = b9e80f854c7042b6f5110393d9a09733026f4ad5
+RMD160 (beats-6.0.0.tar.gz) = 0eaee5bf8e8ff764e1e09bb3be8c52ecc06ef8f3
+SHA512 (beats-6.0.0.tar.gz) = 6dbeecb640fe2e39b81be90fdd05d03c1b3ffddd20ef06ab7c51805eede82035a8d0a00de3132edb77594383095bd5e7a79609e3bdf3d62377140cde7ed3fcd7
+Size (beats-6.0.0.tar.gz) = 15118739 bytes
 SHA1 (patch-vendor_github.com_tsg_gopacket_pcap_pcap.go) = 2ef13f84cbf2e1da428e7b08625ffa6111ebcdf5