CVS commit: pkgsrc/devel/librdkafka

["Adam Ciarcinski" <adam%netbsd.org@localhost>]

Module Name:    pkgsrc
Committed By:   adam
Date:           Sat Feb  1 20:49:43 UTC 2020

Modified Files:
        pkgsrc/devel/librdkafka: Makefile PLIST distinfo

Log Message:
librdkafka: updated to 1.3.0

librdkafka v1.3.0 release
This is a feature release adding support for KIP-392 Fetch from follower, allowing a consumer to fetch messages from the closest replica to increase throughput and reduce cost.

Features
KIP-392 - Fetch messages from closest replica / follower (by @mhowlett)
Added experimental (subject to change or removal) mock broker to make application and librdkafka development testing easier.

Fixes
Fix consumer_lag in stats when consuming from broker versions <0.11.0.0 (regression in librdkafka v1.2.0).

librdkafka v1.2.2 release
v1.2.2 fixes the producer performance regression introduced in v1.2.1 which may affect high-throughput producer applications.

Fixes
Fix producer insert msgq regression in v1.2.1.
Upgrade builtin lz4 to 1.9.2.
Don't trigger error when broker hostname changes.
Less strict message.max.bytes check for individual messages.
Don't call timespec_get() on OSX (since it was removed in recent XCode) by @maparent .
configure: add --runstatedir for compatibility with autoconf.
LZ4 is available from ProduceRequest 0, not 3.
Address 12 code issues identified by Coverity static code analysis.

Enhancements
Add warnings for inconsistent security configuration.
Optimizations to hdr histogram (stats) rollover.
Reorganized examples and added a cleaner consumer example, added minimal C++ producer example.
Print compression type per message-set when debug=msg

librdkafka v1.2.1 release
Warning: v1.2.1 has a producer performance regression which may affect high-throughput producer applications. We recommend such users to stay on v1.2.0.

v1.2.1 is a maintenance release:
Properly handle new Kafka-framed SASL GSSAPI frame semantics on Windows.
This bug was introduced in v1.2.0 and broke GSSAPI authentication on Windows.
Fix msgq (re)insertion code to avoid O(N^2) insert sort operations on retry
The msgq insert code now properly handles interleaved and overlapping
message range inserts, which may occur during Producer retries for
high-throughput applications.
configure: added --disable-c11threads to avoid using libc-provided C11 threads.
configure: added more autoconf compatibility options to ignore

librdkafka v1.2.0 release

WARNING: There is an issue with SASL GSSAPI authentication on Windows with this release. Upgrade directly to v1.2.1 which fixes the issue.

v1.2.0 is a feature release making the consumer transaction aware.

Transaction aware consumer (isolation.level=read_committed) implemented by @mhowlett.
Sub-millisecond buffering (linger.ms) on the producer.
Improved authentication errors (KIP-152)

Consumer-side transaction support
This release adds consumer-side support for transactions.
In previous releases, the consumer always delivered all messages to the application, even those in aborted or not yet committed transactions. In this release, the consumer will by default skip 
messages in aborted transactions.
This is controlled through the new isolation.level configuration property which
defaults to read_committed (only read committed messages, filter out aborted and not-yet committed transactions), to consume all messages, including for aborted transactions, you may set this 
property to read_uncommitted to get the behaviour of previous releases.
For consumers in read_committed mode, the end of a partition is now defined to be the offset of the last message of a successfully committed transaction (referred to as the 'Last Stable Offset').
For non-transactional messages there is no change from previous releases, they will always be read, but a consumer will not advance into a not yet committed transaction on the partition.

Upgrade considerations
linger.ms default was changed from 0 to 0.5 ms to promote some level of batching even with default settings.

New configuration properties
Consumer property isolation.level=read_committed ensures the consumer will only read messages from successfully committed producer transactions. Default is read_committed. To get the previous 
behaviour, set the property to read_uncommitted, which will read all messages produced to a topic, regardless if the message was part of an aborted or not yet committed transaction.
Enhancements
Offset commit metadata (arbitrary application-specified data) is now returned by rd_kafka_committed() and rd_kafka_offsets_for_times()
C++: Added Conf::c_ptr*() to retrieve the underlying C config object.
Added on_thread_start() and on_thread_exit() interceptors.
Increase queue.buffering.max.kbytes max to INT_MAX.
Optimize varint decoding, increasing consume performance by ~15%.

Fixes
General:
Rate limit IO-based queue wakeups to linger.ms, this reduces CPU load and lock contention for high throughput producer applications.
Reduce memory allocations done by rd_kafka_topic_partition_list_new().
Fix socket recv error handling on MSVC (by Jinsu Lee).
Avoid 1s stalls in some scenarios when broker wakeup-fd is triggered.
SSL: Use only hostname (not port) when valid broker hostname (by Hunter Jacksson)
SSL: Ignore OpenSSL cert verification results if enable.ssl.certificate.verification=false
rdkafka_example_cpp: fix metadata listing mode (@njzcx)
SASL Kerberos/GSSAPI: don't treat kinit ECHILD errors as errors
Fix compare overflows
configure: Add option to disable automagic dependency on zstd (by Thomas Deutschmann)
Documentation updates and fixes by Cedric Cellier and @ngrandem
Set thread name on MacOS X (by Nikhil Benesch)
C++: Fix memory leak in Headers (by Vladimir Sakharuk)
Fix UBSan (undefined behaviour errors)
CONFIGURATION.md: escape || inside markdown table
Refresh broker list metadata even if no topics to refresh

Consumer:
Make rd_kafka_pause|resume_partitions() synchronous, making sure that a subsequent consumer_poll() will not return messages for the paused partitions.
Fix incorrect toppar destroy in OffsetRequest
Fix message version 1 offset calculation
Defer commit in transport error to avoid consumer_close hang.

Producer:
Messages were not timed out for leader-less partitions.
Improve message timeout granularity to millisecond precision (the smallest ffective message timeout will still be 1000ms).
message.timeout.ms=0 is now accepted even if linger.ms > 0 (by Jeff Snyder)
Don't track max.poll.interval.ms unless in Consumer mode, this saves quite a few memory barries for high-performance Producers.
Optimization: avoid atomic fatal error code check when idempotence is disabled.


To generate a diff of this commit:
cvs rdiff -u -r1.10 -r1.11 pkgsrc/devel/librdkafka/Makefile
cvs rdiff -u -r1.2 -r1.3 pkgsrc/devel/librdkafka/PLIST
cvs rdiff -u -r1.7 -r1.8 pkgsrc/devel/librdkafka/distinfo

Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.

Modified files:

Index: pkgsrc/devel/librdkafka/Makefile
diff -u pkgsrc/devel/librdkafka/Makefile:1.10 pkgsrc/devel/librdkafka/Makefile:1.11
--- pkgsrc/devel/librdkafka/Makefile:1.10       Sat Jan 18 21:48:07 2020
+++ pkgsrc/devel/librdkafka/Makefile    Sat Feb  1 20:49:43 2020
@@ -1,17 +1,15 @@
-# $NetBSD: Makefile,v 1.10 2020/01/18 21:48:07 jperkin Exp $
+# $NetBSD: Makefile,v 1.11 2020/02/01 20:49:43 adam Exp $
 
-DISTNAME=      librdkafka-1.0.1
-PKGREVISION=   1
+DISTNAME=      librdkafka-1.3.0
 CATEGORIES=    devel
 MASTER_SITES=  ${MASTER_SITE_GITHUB:=edenhill/}
+GITHUB_TAG=    v${PKGVERSION_NOREV}
 
 MAINTAINER=    fhajny%NetBSD.org@localhost
-HOMEPAGE=      https://github.com/edenhill/librdkafka/
+HOMEPAGE=      https://github.com/edenhill/librdkafka
 COMMENT=       Apache Kafka C/C++ library
 LICENSE=       2-clause-bsd
 
-GITHUB_TAG=    v${PKGVERSION_NOREV}
-
 HAS_CONFIGURE= yes
 USE_LANGUAGES= c c99 c++
 USE_TOOLS+=    bash gmake pkg-config
@@ -19,9 +17,11 @@ USE_TOOLS+=  bash gmake pkg-config
 CONFIG_SHELL=  bash
 
 CONFIGURE_ARGS+=       --prefix=${PREFIX}
+CONFIGURE_ARGS+=       --disable-gssapi
+CONFIGURE_ARGS+=       --disable-optimization
 CONFIGURE_ARGS+=       --enable-lz4
 CONFIGURE_ARGS+=       --enable-ssl
-CONFIGURE_ARGS+=       --disable-sasl
+CONFIGURE_ARGS+=       --enable-zstd
 
 TEST_TARGET=   check
 
@@ -33,6 +33,7 @@ PKGCONFIG_OVERRIDE+=  src/rdkafka.pc
 PYTHON_FOR_BUILD_ONLY= yes
 
 .include "../../archivers/lz4/buildlink3.mk"
+.include "../../archivers/zstd/buildlink3.mk"
 .include "../../devel/zlib/buildlink3.mk"
 .include "../../lang/python/tool.mk"
 .include "../../security/openssl/buildlink3.mk"

Index: pkgsrc/devel/librdkafka/PLIST
diff -u pkgsrc/devel/librdkafka/PLIST:1.2 pkgsrc/devel/librdkafka/PLIST:1.3
--- pkgsrc/devel/librdkafka/PLIST:1.2   Thu Apr  5 09:01:16 2018
+++ pkgsrc/devel/librdkafka/PLIST       Sat Feb  1 20:49:43 2020
@@ -1,5 +1,6 @@
-@comment $NetBSD: PLIST,v 1.2 2018/04/05 09:01:16 fhajny Exp $
+@comment $NetBSD: PLIST,v 1.3 2020/02/01 20:49:43 adam Exp $
 include/librdkafka/rdkafka.h
+include/librdkafka/rdkafka_mock.h
 include/librdkafka/rdkafkacpp.h
 lib/librdkafka++.a
 lib/librdkafka++.so

Index: pkgsrc/devel/librdkafka/distinfo
diff -u pkgsrc/devel/librdkafka/distinfo:1.7 pkgsrc/devel/librdkafka/distinfo:1.8
--- pkgsrc/devel/librdkafka/distinfo:1.7        Wed Jul 10 01:31:21 2019
+++ pkgsrc/devel/librdkafka/distinfo    Sat Feb  1 20:49:43 2020
@@ -1,9 +1,9 @@
-$NetBSD: distinfo,v 1.7 2019/07/10 01:31:21 minskim Exp $
+$NetBSD: distinfo,v 1.8 2020/02/01 20:49:43 adam Exp $
 
-SHA1 (librdkafka-1.0.1.tar.gz) = 95b78a6607addc36b1cd00ff034478a9502c7217
-RMD160 (librdkafka-1.0.1.tar.gz) = 66af0e1f29b67074e1f07fa72bc248e862a58c85
-SHA512 (librdkafka-1.0.1.tar.gz) = 094599e974236814c03e688eaa5e39d817cca110ff691501826268c45653fcd6b21b60e741140a1f7ffccd49902247f08fcec82475be2e9f85feaa4340b32d4c
-Size (librdkafka-1.0.1.tar.gz) = 2466259 bytes
+SHA1 (librdkafka-1.3.0.tar.gz) = 20c4ddb2437fc875ba92777a3906fb3a375d7e7f
+RMD160 (librdkafka-1.3.0.tar.gz) = bbb790083dbeb87dcb757d1d9f88e14cf2e28244
+SHA512 (librdkafka-1.3.0.tar.gz) = 94b6202a83d54dfa181466a7ea4e0b492cd695e2bc08e7e46e60efb9c1dacc912fc891cb2bb50d189dc6d0fd6c7553266196dded460957ba0f569d3e08090669
+Size (librdkafka-1.3.0.tar.gz) = 2627028 bytes
 SHA1 (patch-Makefile) = 31731833723acada1c53d01a632f8ba726788a5a
 SHA1 (patch-src_rd.h) = 698000dcad93ccaa07e2462e0932967272b2a915
 SHA1 (patch-src_rdendian.h) = 75cb69e2cbdbcad6a82a943539d5707ff1df7d6d