Private symbols in libpcap, libmagic, libexpat

[Taylor R Campbell <riastradh%NetBSD.org@localhost>]

Currently we build libpcap, libmagic, and libexpat with various
private symbols exported accidentally because our build system was
missing -fvisibility=hidden and associated CPPFLAGS.

The attached patch fixes this (and as a side effect should unbreak the
MKINET6=no build).

Now, fixing this means deleting symbols from shared libraries.

On the one hand, that normally requires a major bump.

On the other hand, all of the symbols are clearly private: none of
them have ever appeared in public header files, so they can only have
been used either:

(a) by applications reaching into places they shouldn't; or

(b) by accidental namespace collisions with names like `der_cmp',
    `file_names', `sock_open', `sappend' -- which may result in
    obscure bugs at _run-time_ not detected at build-time.

So it might be worthwhile to pull up the change _without_ a major bump
even though that technically violates the rules.

On the third hand, I put this off until after 10.1, so maybe it's a
moot point and I should just bump the major (and recursively bump the
major of all dependent libraries).

Thoughts?

# HG changeset patch
# User Taylor R Campbell <riastradh%NetBSD.org@localhost>
# Date 1732297730 0
#      Fri Nov 22 17:48:50 2024 +0000
# Branch trunk
# Node ID bd54919cfdb74d3d8f6eddfe9d15be3101f2c21f
# Parent  fdd2a8b62ce7ef83fb5433dbfd053c271728d8a2
# EXP-Topic riastradh-pr58839-pr58842-pr58840-hideprivsyms
external/bsd/libpcap: Hide private symbols.

1. Build with -fvisibility=hidden -DBUILDING_PCAP -Dpcap_EXPORT like
   upstream does.

   => Verified that this only suppresses symbols which are _not_
      declared in libpcap's public header files, and never have been
      since shlib major 5 which we moved past back in 2017 before all
      released versions of NetBSD.

      NOTE: There are some symbols (eproto_db) that are still
      exported even though they aren't listed in any public .h file.
      This appears to be intentional.

2. Update pcap.expsym to reflect this.

XXX This deletes symbols, so the rules technically demand major bump.

PR lib/58839: libpcap leaks internal symbols

PR lib/58838: shared libraries in base should all have expsym lists

diff -r fdd2a8b62ce7 -r bd54919cfdb7 external/bsd/libpcap/lib/Makefile
--- a/external/bsd/libpcap/lib/Makefile	Tue Dec 17 12:44:33 2024 +0000
+++ b/external/bsd/libpcap/lib/Makefile	Fri Nov 22 17:48:50 2024 +0000
@@ -4,6 +4,8 @@ USE_SHLIBDIR=yes
 .include <bsd.init.mk>
 USE_FORT?= yes	# network protocol library
 
+COPTS+= -fvisibility=hidden
+CPPFLAGS+= -DBUILDING_PCAP -Dpcap_EXPORTS
 CPPFLAGS+= -DPCAP_DONT_INCLUDE_PCAP_BPF_H -DHAVE_CONFIG_H -DENABLE_REMOTE
 CPPFLAGS+= -DPCAP_SUPPORT_RPCAP
 CPPFLAGS+= -I${.OBJDIR}
diff -r fdd2a8b62ce7 -r bd54919cfdb7 external/bsd/libpcap/lib/pcap.expsym
--- a/external/bsd/libpcap/lib/pcap.expsym	Tue Dec 17 12:44:33 2024 +0000
+++ b/external/bsd/libpcap/lib/pcap.expsym	Fri Nov 22 17:48:50 2024 +0000
@@ -1,80 +1,12 @@
-__pcap_atodn
-__pcap_atoin
-__pcap_nametodnaddr
 bpf_dump
 bpf_filter
 bpf_image
-bpf_optimize
-bpf_set_error
 bpf_validate
-dlt_to_linktype
 eproto_db
-finish_parse
-gen_acode
-gen_and
-gen_arth
-gen_atmfield_code
-gen_atmmulti_abbrev
-gen_atmtype_abbrev
-gen_broadcast
-gen_byteop
-gen_ecode
-gen_geneve
-gen_greater
-gen_ifindex
-gen_inbound
-gen_less
-gen_llc
-gen_llc_i
-gen_llc_s
-gen_llc_s_subtype
-gen_llc_u
-gen_llc_u_subtype
-gen_load
-gen_loadi
-gen_loadlen
-gen_mcode
-gen_mcode6
-gen_mpls
-gen_mtp2type_abbrev
-gen_mtp3field_code
-gen_multicast
-gen_ncode
-gen_neg
-gen_not
-gen_or
-gen_p80211_fcdir
-gen_p80211_type
-gen_pf_action
-gen_pf_ifname
-gen_pf_reason
-gen_pf_rnr
-gen_pf_ruleset
-gen_pf_srnr
-gen_portop6
-gen_pppoed
-gen_pppoes
-gen_proto_abbrev
-gen_relation
-gen_scode
-gen_vlan
-icode_to_fcode
-linktype_to_dlt
-max_snaplen_for_dlt
-pcap__create_buffer
-pcap__delete_buffer
-pcap__flush_buffer
-pcap__scan_buffer
-pcap__scan_bytes
-pcap__scan_string
-pcap__switch_to_buffer
 pcap_activate
-pcap_add_any_dev
-pcap_alloc
 pcap_breakloop
 pcap_bufsize
 pcap_can_set_rfmon
-pcap_check_header
 pcap_close
 pcap_compile
 pcap_compile_nopcap
@@ -86,7 +18,6 @@ pcap_datalink_name_to_val
 pcap_datalink_val_to_description
 pcap_datalink_val_to_description_or_dlt
 pcap_datalink_val_to_name
-pcap_debug
 pcap_dispatch
 pcap_dump
 pcap_dump_close
@@ -103,35 +34,20 @@ pcap_file
 pcap_fileno
 pcap_findalldevs
 pcap_findalldevs_ex
-pcap_findalldevs_ex_remote
 pcap_fopen_offline
 pcap_fopen_offline_with_tstamp_precision
-pcap_free
 pcap_free_datalinks
 pcap_free_tstamp_types
 pcap_freealldevs
 pcap_freecode
-pcap_get_column
-pcap_get_debug
-pcap_get_extra
-pcap_get_in
-pcap_get_leng
-pcap_get_lineno
-pcap_get_lval
-pcap_get_out
 pcap_get_required_select_timeout
 pcap_get_selectable_fd
-pcap_get_text
 pcap_get_tstamp_precision
 pcap_geterr
 pcap_getnonblock
 pcap_init
 pcap_inject
 pcap_is_swapped
-pcap_lex
-pcap_lex_destroy
-pcap_lex_init
-pcap_lex_init_extra
 pcap_lib_version
 pcap_list_datalinks
 pcap_list_tstamp_types
@@ -151,7 +67,6 @@ pcap_nametoproto
 pcap_next
 pcap_next_etherent
 pcap_next_ex
-pcap_ng_check_header
 pcap_offline_filter
 pcap_open
 pcap_open_dead
@@ -159,30 +74,17 @@ pcap_open_dead_with_tstamp_precision
 pcap_open_live
 pcap_open_offline
 pcap_open_offline_with_tstamp_precision
-pcap_open_rpcap
-pcap_parse
 pcap_parsesrcstr
 pcap_perror
-pcap_pop_buffer_state
-pcap_push_buffer_state
-pcap_realloc
 pcap_remoteact_accept
 pcap_remoteact_accept_ex
 pcap_remoteact_cleanup
 pcap_remoteact_close
 pcap_remoteact_list
-pcap_restart
 pcap_sendpacket
 pcap_set_buffer_size
-pcap_set_column
 pcap_set_datalink
-pcap_set_debug
-pcap_set_extra
 pcap_set_immediate_mode
-pcap_set_in
-pcap_set_lineno
-pcap_set_lval
-pcap_set_out
 pcap_set_parser_debug
 pcap_set_promisc
 pcap_set_rfmon
@@ -202,66 +104,3 @@ pcap_tstamp_type_name_to_val
 pcap_tstamp_type_val_to_description
 pcap_tstamp_type_val_to_name
 pcap_version
-pcapint_add_addr_to_dev
-pcapint_add_addr_to_if
-pcapint_add_dev
-pcapint_add_to_pcaps_to_close
-pcapint_adjust_snapshot
-pcapint_breakloop_common
-pcapint_check_activated
-pcapint_cleanup_live_common
-pcapint_create_common
-pcapint_create_interface
-pcapint_createsrcstr_ex
-pcapint_do_addexit
-pcapint_filter
-pcapint_find_dev
-pcapint_find_or_add_dev
-pcapint_find_or_add_if
-pcapint_findalldevs_interfaces
-pcapint_fmt_errmsg_for_errno
-pcapint_fmt_set_encoding
-pcapint_getnonblock_fd
-pcapint_install_bpf_program
-pcapint_new_api
-pcapint_offline_read
-pcapint_oneshot
-pcapint_open_offline_common
-pcapint_parsesrcstr_ex
-pcapint_platform_finddevs
-pcapint_post_process
-pcapint_remove_from_pcaps_to_close
-pcapint_setnonblock_fd
-pcapint_sf_cleanup
-pcapint_strcasecmp
-pcapint_utf_8_mode
-pcapint_validate_filter
-pcapint_vfmt_errmsg_for_errno
-rpcap_create
-rpcap_createhdr
-rpcap_msg_type_string
-rpcap_senderror
-sappend
-sdup
-sock_bufferize
-sock_check_hostlist
-sock_cleanup
-sock_close
-sock_cmpaddr
-sock_discard
-sock_fmterrmsg
-sock_getascii_addrport
-sock_geterrcode
-sock_geterrmsg
-sock_getmyinfo
-sock_init
-sock_initaddress
-sock_open
-sock_present2network
-sock_recv
-sock_recv_dgram
-sock_send
-sock_vfmterrmsg
-stoulen
-usb_create
-usb_findalldevs
diff -r fdd2a8b62ce7 -r bd54919cfdb7 sys/net/bpf_filter.c
--- a/sys/net/bpf_filter.c	Tue Dec 17 12:44:33 2024 +0000
+++ b/sys/net/bpf_filter.c	Fri Nov 22 17:48:50 2024 +0000
@@ -212,6 +212,7 @@ u_int
 bpf_filter_ext(const bpf_ctx_t *bc, const struct bpf_insn *pc, bpf_args_t *args)
 #else
 __strong_alias(pcapint_filter, bpf_filter)
+__asm(".hidden	pcapint_filter"); /* XXX not for libpcap export */
 u_int
 bpf_filter(const struct bpf_insn *pc, const u_char *p, u_int wirelen,
     u_int buflen)
@@ -611,6 +612,7 @@ int
 bpf_validate_ext(const bpf_ctx_t *bc, const struct bpf_insn *f, int signed_len)
 #else
 __strong_alias(pcapint_validate_filter, bpf_validate)
+__asm(".hidden	pcapint_validate_filter"); /* XXX not for libpcap export */
 int
 bpf_validate(const struct bpf_insn *f, int signed_len)
 #endif
# HG changeset patch
# User Taylor R Campbell <riastradh%NetBSD.org@localhost>
# Date 1732300670 0
#      Fri Nov 22 18:37:50 2024 +0000
# Branch trunk
# Node ID 348764c3d427b8638537671d1e26f4ce02c137f8
# Parent  bd54919cfdb74d3d8f6eddfe9d15be3101f2c21f
# EXP-Topic riastradh-pr58839-pr58842-pr58840-hideprivsyms
external/bsd/file libmagic: Hide private symbols.

Build with -fvisibility=hidden so only symbols explicitly marked for
export are exported.

NOTE: This deletes some previously exported symbols, but these
previously exported symbols have never been declared in magic.h, so
applications could use them only either (a) by reaching into places
they shouldn't, or (b) by colliding with other uses because it
exported names like `der_cmp' and `file_names'.

XXX This deletes symbols, so the rules technically demand major bump.

PR lib/58842: libmagic leaks private and zstd symbols

diff -r bd54919cfdb7 -r 348764c3d427 external/bsd/file/lib/Makefile
--- a/external/bsd/file/lib/Makefile	Fri Nov 22 17:48:50 2024 +0000
+++ b/external/bsd/file/lib/Makefile	Fri Nov 22 18:37:50 2024 +0000
@@ -11,6 +11,8 @@ LIB=		magic
 LIBDPLIBS+=	${DPLIBS}
 CPPFLAGS+=-I.
 
+COPTS+=	-fvisibility=hidden
+
 # XXX Avoid exporting symbols of statically linked deps like zstd.
 # XXX Should do this in bsd.lib.mk, perhaps.
 .for _lib _dir in ${LIBDPLIBS}
diff -r bd54919cfdb7 -r 348764c3d427 external/bsd/file/lib/magic.expsym
--- a/external/bsd/file/lib/magic.expsym	Fri Nov 22 17:48:50 2024 +0000
+++ b/external/bsd/file/lib/magic.expsym	Fri Nov 22 18:37:50 2024 +0000
@@ -1,42 +1,3 @@
-accept_ranges
-cdf_count_chain
-cdf_ctime
-cdf_find_stream
-cdf_print_classid
-cdf_print_elapsed_time
-cdf_print_property_name
-cdf_read_dir
-cdf_read_doc_summary_info
-cdf_read_header
-cdf_read_long_sector_chain
-cdf_read_property_info
-cdf_read_sat
-cdf_read_sector
-cdf_read_sector_chain
-cdf_read_short_sector
-cdf_read_short_sector_chain
-cdf_read_short_stream
-cdf_read_ssat
-cdf_read_summary_info
-cdf_read_user_stream
-cdf_swap_class
-cdf_swap_dir
-cdf_swap_header
-cdf_timespec_to_timestamp
-cdf_timestamp_to_timespec
-cdf_tole2
-cdf_tole4
-cdf_tole8
-cdf_u16tos8
-cdf_unpack_catalog
-cdf_unpack_dir
-cdf_unpack_header
-cdf_unpack_summary_info
-cdf_zero_stream
-der_cmp
-der_offs
-file_names
-file_nnames
 magic_buffer
 magic_check
 magic_close
# HG changeset patch
# User Taylor R Campbell <riastradh%NetBSD.org@localhost>
# Date 1732301187 0
#      Fri Nov 22 18:46:27 2024 +0000
# Branch trunk
# Node ID 75b7036e5cdd97556f358743fcddf61ae4dca28a
# Parent  348764c3d427b8638537671d1e26f4ce02c137f8
# EXP-Topic riastradh-pr58839-pr58842-pr58840-hideprivsyms
external/mit/expat: Hide private symbols.

Build with -fvisibility=hidden -DXML_ENABLE_VISIBILITY like upstream
does.

NOTE: This deletes some symbols that were previously exported, but I
verified that none of them have ever appeared in expat.h or
expat_external.h, so they can only have been used either (a) by
applications reaching into places they shouldn't, or (b) by
accidental namespace collisions.

XXX This deletes symbols, so the rules technically demand major bump.

PR lib/58840: expat leaks internal symbols

diff -r 348764c3d427 -r 75b7036e5cdd external/mit/expat/lib/libexpat/Makefile
--- a/external/mit/expat/lib/libexpat/Makefile	Fri Nov 22 18:37:50 2024 +0000
+++ b/external/mit/expat/lib/libexpat/Makefile	Fri Nov 22 18:46:27 2024 +0000
@@ -12,6 +12,8 @@ INCSDIR=	/usr/include
 
 CPPFLAGS+=		-I${EXPATSRCDIR} -I${.CURDIR}
 CPPFLAGS+=		-DHAVE_EXPAT_CONFIG_H
+CPPFLAGS+=		-DXML_ENABLE_VISIBILITY
+COPTS+=			-fvisibility=hidden
 COPTS.xmlparse.c=	-Wno-error # macppc
 COPTS.xmlrole.c=	-Wno-error # macppc
 COPTS.xmltok.c=		-Wno-error # macppc
diff -r 348764c3d427 -r 75b7036e5cdd external/mit/expat/lib/libexpat/expat.expsym
--- a/external/mit/expat/lib/libexpat/expat.expsym	Fri Nov 22 18:37:50 2024 +0000
+++ b/external/mit/expat/lib/libexpat/expat.expsym	Fri Nov 22 18:46:27 2024 +0000
@@ -66,22 +66,3 @@ XML_SetXmlDeclHandler
 XML_StopParser
 XML_UseForeignDTD
 XML_UseParserAsHandlerArg
-XmlGetUtf16InternalEncoding
-XmlGetUtf16InternalEncodingNS
-XmlGetUtf8InternalEncoding
-XmlGetUtf8InternalEncodingNS
-XmlInitEncoding
-XmlInitEncodingNS
-XmlInitUnknownEncoding
-XmlInitUnknownEncodingNS
-XmlParseXmlDecl
-XmlParseXmlDeclNS
-XmlPrologStateInit
-XmlPrologStateInitExternalEntity
-XmlSizeOfUnknownEncoding
-XmlUtf16Encode
-XmlUtf8Encode
-_INTERNAL_trim_to_complete_utf8_characters
-testingAccountingGetCountBytesDirect
-testingAccountingGetCountBytesIndirect
-unsignedCharToPrintable