CVS commit: pkgsrc/www/py-w3lib

["Adam Ciarcinski" <adam%netbsd.org@localhost>]

Module Name:    pkgsrc
Committed By:   adam
Date:           Mon Oct  5 05:12:02 UTC 2026

Modified Files:
        pkgsrc/www/py-w3lib: Makefile PLIST distinfo

Log Message:
py-w3lib: updated to 2.5.0

2.5.0 (2026-09-30)

New features:

- Added support for Python 3.15

- Added :func:`~w3lib.url.add_http_if_no_scheme`, ported from Scrapy, which
  adds ``http`` as the default scheme to a URL that has none

- ``w3lib.url.parse_qsl_to_bytes``, :func:`~w3lib.url.url_query_parameter`,
  :func:`~w3lib.url.add_or_replace_parameter`,
  :func:`~w3lib.url.add_or_replace_parameters` and
  :func:`~w3lib.url.canonicalize_url` now accept a *separator* (or
  *query_separator* for :func:`~w3lib.url.canonicalize_url`) keyword argument,
  to support query strings that use a separator other than ``&``

- :func:`~w3lib.html.get_base_url` and :func:`~w3lib.html.get_meta_refresh`
  now accept a *max_scan* keyword argument, an upper bound on how much of the
  document they look at

- Improved the performance of most :mod:`w3lib.url` functions,
  :func:`~w3lib.url.safe_url_string` in particular
  :mod:`w3lib.html` functions

- :func:`~w3lib.html.get_base_url` and :func:`~w3lib.html.get_meta_refresh`
  are now several times faster on typical pages, which have no ``<base>`` tag
  and no ``<meta>`` refresh tag

Deprecations and removals:

- The ``w3lib.util`` module is deprecated, and now emits a
  :exc:`DeprecationWarning` on import

- The undocumented ``w3lib_replace`` codec error handler is no longer
  registered

Security and correctness fixes:

- :func:`~w3lib.url.safe_url_string`, :func:`~w3lib.url.canonicalize_url` and
  ``w3lib.url.parse_url`` no longer disagree with how browsers and
  :mod:`urllib.parse` read a URL in the following cases, which could let a URL
  resolve to a different host or path than the one these functions reported:

    - ``\`` is now treated like ``/`` in the authority and path of
      special-scheme URLs

    - An NFKC-normalized backslash in the host is now rejected, like other
      normalized authority delimiters already were

    - ASCII tab, carriage return and line feed characters are now stripped
      from the whole URL, including the host

  Additionally, :func:`~w3lib.url.safe_url_string` now adds a ``/`` path to a
  URL that has a query but no path, since some HTTP clients otherwise send
  the query alone as the request target
  :exc:`UnicodeDecodeError` for a host that cannot be IDNA-encoded, such as
  one with an empty label, when *encoding* is not UTF-8

- :func:`~w3lib.url.canonicalize_url` now resolves dot segments (``.`` and
  ``..``) in the path and normalizes IPv6 addresses in the host, so that
  equivalent URLs canonicalize the same
  characters whose decoding would change their meaning, such as a semicolon
  (``%3B``, a path parameter delimiter) or a percent sign (``%25``,
  which would otherwise decode to a bare ``%``), in the path encoded.

- :func:`~w3lib.url.canonicalize_url` no longer IDNA-encodes the userinfo of a
  URL together with its host, which changed both, e.g.
  ``http://user@éxample.com`` became ``http://xn--user%xample-fbb.com@localhost``

- :func:`~w3lib.url.safe_download_url` now resolves percent-encoded dot
  segments, such as ``%2e`` or ``%2e%2e``, like ``.`` and ``..``
  decides whether the path keeps its trailing ``/`` based on the path alone,
  rather than on whether the whole URL ends with ``/``

- :func:`~w3lib.url.url_query_parameter` now accepts a ``bytes`` URL, instead
  of parsing its ``b'…'`` representation
  :func:`~w3lib.url.url_query_cleaner` accepts one as well, instead of
  raising :exc:`TypeError`, while the type hint of its *parameterlist*
  argument no longer claims to accept ``bytes`` items, which never matched
  anything

- :func:`~w3lib.url.path_to_file_uri` and :func:`~w3lib.url.file_uri_to_path`
  now round-trip Windows UNC paths, ``\\server\share\file`` becoming
  ``file://server/share/file``
  no longer raises for a path that starts with an extra pair of slashes, such
  as ``////foo/bar``

- Fixed excessive backtracking on malformed input, which could be used for
  denial of service, in :func:`~w3lib.html.get_base_url`
  :func:`~w3lib.html.get_meta_refresh`
  :func:`~w3lib.html.remove_tags_with_content`
  :func:`~w3lib.html.replace_tags` and :func:`~w3lib.html.remove_tags`
  ), and :func:`~w3lib.html.unquote_markup` on unclosed CDATA sections

- :func:`~w3lib.html.get_base_url` no longer picks up a ``<base>`` tag
  written as text inside ``<script>`` or ``<noscript>``, where a browser
  would ignore it, or one broken by a comment, such as
  ``<base h<!--c-->ref="…">``, which a browser does not read as a tag at all

- :func:`~w3lib.html.get_base_url` now reads only the first ``href`` attribute
  of the ``<base>`` tag, as browsers do, instead of the last attribute whose
  name ends in ``href``, such as ``data-href``, and now also reads an unquoted
  ``href`` value

- :func:`~w3lib.html.get_meta_refresh` and
  :func:`~w3lib.html.remove_tags_with_content` no longer treat a ``<script>``
  or ``<noscript>`` element as closed only by a bare ``</script>``, now also
  recognizing a closing tag followed by whitespace, ``/`` or another token

- :func:`~w3lib.url.parse_data_uri` no longer raises on an empty quoted
  parameter value, such as ``charset=""``

- :func:`~w3lib.html.replace_entities` and :func:`~w3lib.html.get_meta_refresh`
  now only treat ASCII digits as numeric character references and refresh
  intervals, matching browsers

- :func:`~w3lib.html.replace_entities` now resolves a null or surrogate
  numeric character reference, such as ``&#0;`` or ``&#xD800;``, to U+FFFD,
  as browsers do, instead of producing a NUL or a lone surrogate that cannot
  be encoded

- :func:`~w3lib.encoding.resolve_encoding`, and with it every function that
  detects the encoding of a response, no longer resolves UTF-7, which
  browsers do not support and which lets ASCII-only bytes smuggle markup
  past filters

- :func:`~w3lib.encoding.http_content_type_encoding` now recognizes a quoted
  ``charset`` parameter value, and no longer stops matching a valid
  ``charset`` because of what follows it or of a quoted-string parameter
  value that itself contains ``charset``

- :func:`~w3lib.encoding.html_body_declared_encoding` now skips HTML
  comments, like :func:`~w3lib.html.get_base_url` and
  :func:`~w3lib.html.get_meta_refresh` already did
  charset declaration regardless of the order of the attributes in the tag,
  of how the ``http-equiv`` pragma is spelled, e.g.
  ``httpequiv="ContentType"``, and of what surrounds ``charset=`` within a
  ``content`` attribute value

- :func:`~w3lib.encoding.html_to_unicode` now decodes a lead ``0x80`` byte as
  the euro sign when the encoding is GB18030, matching the WHATWG Encoding
  Standard
  declared in the body or auto-detected as big-endian, as it already did for
  one declared in the ``Content-Type`` header

- :func:`~w3lib.html.replace_tags`, :func:`~w3lib.html.remove_tags` and
  :func:`~w3lib.html.remove_tags_with_content` no longer end a tag at an
  angle bracket inside a quoted attribute value, such as the ``>`` of
  ``<img alt="a>b" src=x>``

- :func:`~w3lib.html.replace_entities` and :func:`~w3lib.html.unquote_markup`
  no longer consume *keep* when it is an iterator, such as a generator, which
  left some entities unkept
  no longer raises :exc:`ValueError` when *which_ones* or *keep* is an empty
  iterator and the other one is not

- :func:`~w3lib.html.get_meta_refresh` no longer reports a redirect for a
  refresh ``content`` value that browsers do not follow because it uses
  non-ASCII whitespace, such as U+00A0

- :func:`~w3lib.html.get_meta_refresh` no longer prints the HTML content it
  was given to stdout when a :exc:`UnicodeDecodeError` is raised

- Tests, benchmarking and CI improvements


To generate a diff of this commit:
cvs rdiff -u -r1.16 -r1.17 pkgsrc/www/py-w3lib/Makefile
cvs rdiff -u -r1.4 -r1.5 pkgsrc/www/py-w3lib/PLIST
cvs rdiff -u -r1.14 -r1.15 pkgsrc/www/py-w3lib/distinfo

Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.

Modified files:

Index: pkgsrc/www/py-w3lib/Makefile
diff -u pkgsrc/www/py-w3lib/Makefile:1.16 pkgsrc/www/py-w3lib/Makefile:1.17
--- pkgsrc/www/py-w3lib/Makefile:1.16   Fri Mar 27 09:01:16 2026
+++ pkgsrc/www/py-w3lib/Makefile        Mon Oct  5 05:12:02 2026
@@ -1,6 +1,6 @@
-# $NetBSD: Makefile,v 1.16 2026/03/27 09:01:16 adam Exp $
+# $NetBSD: Makefile,v 1.17 2026/10/05 05:12:02 adam Exp $
 
-DISTNAME=      w3lib-2.4.1
+DISTNAME=      w3lib-2.5.0
 PKGNAME=       ${PYPKGPREFIX}-${DISTNAME}
 CATEGORIES=    www python
 MASTER_SITES=  ${MASTER_SITE_PYPI:=w/w3lib/}

Index: pkgsrc/www/py-w3lib/PLIST
diff -u pkgsrc/www/py-w3lib/PLIST:1.4 pkgsrc/www/py-w3lib/PLIST:1.5
--- pkgsrc/www/py-w3lib/PLIST:1.4       Mon Feb  9 13:38:35 2026
+++ pkgsrc/www/py-w3lib/PLIST   Mon Oct  5 05:12:02 2026
@@ -1,4 +1,4 @@
-@comment $NetBSD: PLIST,v 1.4 2026/02/09 13:38:35 adam Exp $
+@comment $NetBSD: PLIST,v 1.5 2026/10/05 05:12:02 adam Exp $
 ${PYSITELIB}/${WHEEL_INFODIR}/METADATA
 ${PYSITELIB}/${WHEEL_INFODIR}/RECORD
 ${PYSITELIB}/${WHEEL_INFODIR}/WHEEL
@@ -15,6 +15,9 @@ ${PYSITELIB}/w3lib/_types.pyo
 ${PYSITELIB}/w3lib/_url.py
 ${PYSITELIB}/w3lib/_url.pyc
 ${PYSITELIB}/w3lib/_url.pyo
+${PYSITELIB}/w3lib/_util.py
+${PYSITELIB}/w3lib/_util.pyc
+${PYSITELIB}/w3lib/_util.pyo
 ${PYSITELIB}/w3lib/encoding.py
 ${PYSITELIB}/w3lib/encoding.pyc
 ${PYSITELIB}/w3lib/encoding.pyo

Index: pkgsrc/www/py-w3lib/distinfo
diff -u pkgsrc/www/py-w3lib/distinfo:1.14 pkgsrc/www/py-w3lib/distinfo:1.15
--- pkgsrc/www/py-w3lib/distinfo:1.14   Fri Mar 27 09:01:16 2026
+++ pkgsrc/www/py-w3lib/distinfo        Mon Oct  5 05:12:02 2026
@@ -1,5 +1,5 @@
-$NetBSD: distinfo,v 1.14 2026/03/27 09:01:16 adam Exp $
+$NetBSD: distinfo,v 1.15 2026/10/05 05:12:02 adam Exp $
 
-BLAKE2s (w3lib-2.4.1.tar.gz) = 7dce8ff91ce45b9711874091c05c349ccef31e9def3aef550428be76c6829f8e
-SHA512 (w3lib-2.4.1.tar.gz) = 493a6889efcda86cb8d467fb7bc4b90bb5ad8d41f39045401c829fcf9bc2462fb8d077da39a22e18f70b66eb90c92c73cf35582af8180bee06958ac8c3fb5821
-Size (w3lib-2.4.1.tar.gz) = 48494 bytes
+BLAKE2s (w3lib-2.5.0.tar.gz) = 7770ac46c786e2b546e18acbd424e168309ce5fc9e27b32d8c702334e597e2e7
+SHA512 (w3lib-2.5.0.tar.gz) = 100e2bcc690cdf40ca64e0c31e1e56f382b41cb75431aac850394df82197b5c7f087ac7b3bad98d3ddc09ad14daa5806964f94f91eeed296b2cee062fce62987
+Size (w3lib-2.5.0.tar.gz) = 385035 bytes