CVS commit: pkgsrc/www/py-scrapy

To: pkgsrc-changes%NetBSD.org@localhost
Subject: CVS commit: pkgsrc/www/py-scrapy
From: "Adam Ciarcinski" <adam%netbsd.org@localhost>
Date: Wed, 20 May 2026 12:47:09 +0000

Module Name:    pkgsrc
Committed By:   adam
Date:           Wed May 20 12:47:09 UTC 2026

Modified Files:
        pkgsrc/www/py-scrapy: Makefile PLIST distinfo

Log Message:
py-scrapy: updated to 2.16.0

Scrapy 2.16.0 (2026-05-19)

Highlights:

-   Official support for Python 3.14
-   Support for Twisted 26.4.0+

Modified requirements

-   Increased the minimum versions of the following dependencies:

    - service_identity_: 18.1.0 → 23.1.0

-   Added support for Twisted 26.4.0+.
-   Added support for Python 3.14.

Backward-incompatible changes

-   The following classes and functions, intended for internal use by
    :class:`~scrapy.core.downloader.handlers.http11.HTTP11DownloadHandler`
    and :class:`~scrapy.core.downloader.handlers.http2.H2DownloadHandler`, have
    been made private:

    - ``scrapy.core.downloader.handlers.http11.ScrapyAgent``
    - ``scrapy.core.downloader.handlers.http11.ScrapyProxyAgent``
    - ``scrapy.core.downloader.handlers.http11.TunnelingAgent``
    - ``scrapy.core.downloader.handlers.http11.TunnelingTCP4ClientEndpoint``
    - ``scrapy.core.downloader.handlers.http11.tunnel_request_data()``
    - ``scrapy.core.downloader.handlers.http2.ScrapyH2Agent``

Deprecations

-   ``scrapy.FormRequest`` is deprecated. You can use the :doc:`form2request
    <form2request:index>` library instead, see :ref:`form`.
-   ``scrapy.utils.python.MutableChain`` is deprecated.

Deprecation removals

-   The ``start_requests()`` method of :class:`~scrapy.Spider`, deprecated in
    2.13.0, is removed and no longer called. Use :meth:`~scrapy.Spider.start`
    instead, or both to maintain support for lower Scrapy versions.
-   Support for ``process_start_requests()`` methods of :ref:`spider middlewares
    <topics-spider-middleware>`, deprecated in 2.13.0, is removed. Use
    :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_start` instead,
    or both to maintain support for lower Scrapy versions.
-   Support for synchronous ``process_spider_output()`` methods of spider
    middlewares, deprecated in Scrapy 2.13.0, is removed. You should upgrade
    the affected middlewares to have asynchronous ``process_spider_output()``
    methods.
-   The ``spider`` arguments of the following methods of
    :class:`~scrapy.core.scraper.Scraper`, deprecated in Scrapy 2.13.0, are
    removed:

    - ``close_spider()``
    - ``enqueue_scrape()``
    - ``handle_spider_error()``
    - ``handle_spider_output()``

-   HTTP/1.0 support code, deprecated in Scrapy 2.13.0, is removed. This
    includes:

    - ``scrapy.core.downloader.handlers.http10.HTTP10DownloadHandler``
    - The ``scrapy.core.downloader.webclient`` module.
    - The ``DOWNLOADER_HTTPCLIENTFACTORY`` setting.

-   The following functions, deprecated in Scrapy 2.13.0, are removed, you
    should import them from :mod:`w3lib.url` directly instead:

    - ``scrapy.utils.url.add_or_replace_parameter()``
    - ``scrapy.utils.url.add_or_replace_parameters()``
    - ``scrapy.utils.url.any_to_uri()``
    - ``scrapy.utils.url.canonicalize_url()``
    - ``scrapy.utils.url.file_uri_to_path()``
    - ``scrapy.utils.url.is_url()``
    - ``scrapy.utils.url.parse_data_uri()``
    - ``scrapy.utils.url.parse_url()``
    - ``scrapy.utils.url.path_to_file_uri()``
    - ``scrapy.utils.url.safe_download_url()``
    - ``scrapy.utils.url.safe_url_string()``
    - ``scrapy.utils.url.url_query_cleaner()``
    - ``scrapy.utils.url.url_query_parameter()``

-   The following test-related code, deprecated in Scrapy 2.13.0, is removed:

    - the ``scrapy.utils.testproc`` module
    - the ``scrapy.utils.testsite`` module
    - ``scrapy.utils.test.assert_gcs_environ()``
    - ``scrapy.utils.test.get_ftp_content_and_delete()``
    - ``scrapy.utils.test.get_gcs_content_and_delete()``
    - ``scrapy.utils.test.mock_google_cloud_storage()``
    - ``scrapy.utils.test.skip_if_no_boto()``
    - ``scrapy.utils.test.TestSpider``

-   ``scrapy.utils.versions.scrapy_components_versions()``, deprecated in
    Scrapy 2.13.0, is removed, you can use
    :func:`scrapy.utils.versions.get_versions` instead.
-   ``scrapy.downloadermiddlewares.ajaxcrawl.AjaxCrawlMiddleware`` and
    ``scrapy.utils.url.escape_ajax()``, deprecated in Scrapy 2.13.0, are
    removed.
-   The ``__init__()`` method of priority queue classes (see
    :setting:`SCHEDULER_PRIORITY_QUEUE`) now needs to support a keyword-only
    ``start_queue_cls`` parameter, not supporting it was deprecated in Scrapy
    2.13.0.

-   ``scrapy.spiders.init.InitSpider``, deprecated in Scrapy 2.13.0, is
    removed.

New features

-   New features and improvements for
    :class:`~scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler`:

    - Support for proxies.
    - Support for the :reqmeta:`download_latency` meta key.
    - Support for :attr:`Response.certificate
      <scrapy.http.Response.certificate>`.
    - Default headers set by the ``httpx`` library are no longer added to
      requests.

-   :class:`~scrapy.core.downloader.handlers.http11.HTTP11DownloadHandler` now
    skips HTTPS proxy certificate verification when the
    :setting:`DOWNLOAD_VERIFY_CERTIFICATES` setting is set to ``False``.

Improvements

-   :func:`time.monotonic` is used instead of :func:`time.time` to calculate
    elapsed time in various places.
-   Improved extraction of the file extension from the URL in
    :class:`~scrapy.pipelines.files.FilesPipeline`.
-   Other code refactoring and improvements.

Bug fixes

-   :class:`~scrapy.core.downloader.handlers.http11.HTTP11DownloadHandler` now
    raises an exception when a request has an ``https://`` destination and an
    ``https://`` proxy, which is not supported by this handler. Previously it
    tried to connect to the proxy via HTTP in this case.
-   :class:`~scrapy.core.downloader.handlers.http2.H2DownloadHandler` now
    raises an exception for requests with ``http://`` URLs instead of trying to
    connect, which is not supported by this handler.
-   :class:`~scrapy.core.downloader.handlers.http2.H2DownloadHandler` no longer
    adds the ``:status`` pseudo-header to :attr:`Response.headers
    <scrapy.http.Response.headers>`.
-   Fixed :func:`scrapy.utils.response.open_in_browser` removing the ``<head>``
    tag when adding the ``<base>`` tag.

Documentation

-   Documented that
    :class:`~scrapy.core.downloader.handlers.http11.HTTP11DownloadHandler`
    doesn't support HTTPS proxies for HTTPS destinations and that
    :class:`~scrapy.core.downloader.handlers.http2.H2DownloadHandler` doesn't
    support proxies at all.
-   Added an example of using
    :class:`logging.handlers.TimedRotatingFileHandler` to rotate Scrapy logs.
-   Added a ``CITATION.cff`` file.
-   Mentioned :setting:`DOWNLOADER_CLIENT_TLS_METHOD` in :ref:`bans`.
-   Other documentation improvements and fixes.


To generate a diff of this commit:
cvs rdiff -u -r1.39 -r1.40 pkgsrc/www/py-scrapy/Makefile
cvs rdiff -u -r1.17 -r1.18 pkgsrc/www/py-scrapy/PLIST
cvs rdiff -u -r1.29 -r1.30 pkgsrc/www/py-scrapy/distinfo

Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.

Modified files:

Index: pkgsrc/www/py-scrapy/Makefile
diff -u pkgsrc/www/py-scrapy/Makefile:1.39 pkgsrc/www/py-scrapy/Makefile:1.40
--- pkgsrc/www/py-scrapy/Makefile:1.39  Thu Apr 30 12:27:26 2026
+++ pkgsrc/www/py-scrapy/Makefile       Wed May 20 12:47:09 2026
@@ -1,6 +1,6 @@
-# $NetBSD: Makefile,v 1.39 2026/04/30 12:27:26 adam Exp $
+# $NetBSD: Makefile,v 1.40 2026/05/20 12:47:09 adam Exp $
 
-DISTNAME=      scrapy-2.15.2
+DISTNAME=      scrapy-2.16.0
 PKGNAME=       ${PYPKGPREFIX}-${DISTNAME}
 CATEGORIES=    www python
 MASTER_SITES=  ${MASTER_SITE_PYPI:=S/Scrapy/}
@@ -21,7 +21,7 @@ DEPENDS+=     ${PYPKGPREFIX}-parsel>=1.5.0:.
 DEPENDS+=      ${PYPKGPREFIX}-protego>=0.1.15:../../www/py-protego
 DEPENDS+=      ${PYPKGPREFIX}-pydispatcher>=2.0.5:../../devel/py-pydispatcher
 DEPENDS+=      ${PYPKGPREFIX}-queuelib>=1.4.2:../../devel/py-queuelib
-DEPENDS+=      ${PYPKGPREFIX}-service_identity>=18.1.0:../../security/py-service_identity
+DEPENDS+=      ${PYPKGPREFIX}-service_identity>=23.1.0:../../security/py-service_identity
 DEPENDS+=      ${PYPKGPREFIX}-tldextract-[0-9]*:../../net/py-tldextract
 DEPENDS+=      ${PYPKGPREFIX}-twisted>=21.7.0:../../net/py-twisted
 DEPENDS+=      ${PYPKGPREFIX}-w3lib>=1.17.0:../../www/py-w3lib

Index: pkgsrc/www/py-scrapy/PLIST
diff -u pkgsrc/www/py-scrapy/PLIST:1.17 pkgsrc/www/py-scrapy/PLIST:1.18
--- pkgsrc/www/py-scrapy/PLIST:1.17     Wed Apr 22 11:13:56 2026
+++ pkgsrc/www/py-scrapy/PLIST  Wed May 20 12:47:09 2026
@@ -1,4 +1,4 @@
-@comment $NetBSD: PLIST,v 1.17 2026/04/22 11:13:56 adam Exp $
+@comment $NetBSD: PLIST,v 1.18 2026/05/20 12:47:09 adam Exp $
 bin/scrapy-${PYVERSSUFFIX}
 ${PYSITELIB}/${WHEEL_INFODIR}/METADATA
 ${PYSITELIB}/${WHEEL_INFODIR}/RECORD
@@ -82,6 +82,12 @@ ${PYSITELIB}/scrapy/core/downloader/cont
 ${PYSITELIB}/scrapy/core/downloader/handlers/__init__.py
 ${PYSITELIB}/scrapy/core/downloader/handlers/__init__.pyc
 ${PYSITELIB}/scrapy/core/downloader/handlers/__init__.pyo
+${PYSITELIB}/scrapy/core/downloader/handlers/_base_http.py
+${PYSITELIB}/scrapy/core/downloader/handlers/_base_http.pyc
+${PYSITELIB}/scrapy/core/downloader/handlers/_base_http.pyo
+${PYSITELIB}/scrapy/core/downloader/handlers/_base_streaming.py
+${PYSITELIB}/scrapy/core/downloader/handlers/_base_streaming.pyc
+${PYSITELIB}/scrapy/core/downloader/handlers/_base_streaming.pyo
 ${PYSITELIB}/scrapy/core/downloader/handlers/_httpx.py
 ${PYSITELIB}/scrapy/core/downloader/handlers/_httpx.pyc
 ${PYSITELIB}/scrapy/core/downloader/handlers/_httpx.pyo
@@ -100,9 +106,6 @@ ${PYSITELIB}/scrapy/core/downloader/hand
 ${PYSITELIB}/scrapy/core/downloader/handlers/http.py
 ${PYSITELIB}/scrapy/core/downloader/handlers/http.pyc
 ${PYSITELIB}/scrapy/core/downloader/handlers/http.pyo
-${PYSITELIB}/scrapy/core/downloader/handlers/http10.py
-${PYSITELIB}/scrapy/core/downloader/handlers/http10.pyc
-${PYSITELIB}/scrapy/core/downloader/handlers/http10.pyo
 ${PYSITELIB}/scrapy/core/downloader/handlers/http11.py
 ${PYSITELIB}/scrapy/core/downloader/handlers/http11.pyc
 ${PYSITELIB}/scrapy/core/downloader/handlers/http11.pyo
@@ -118,9 +121,6 @@ ${PYSITELIB}/scrapy/core/downloader/midd
 ${PYSITELIB}/scrapy/core/downloader/tls.py
 ${PYSITELIB}/scrapy/core/downloader/tls.pyc
 ${PYSITELIB}/scrapy/core/downloader/tls.pyo
-${PYSITELIB}/scrapy/core/downloader/webclient.py
-${PYSITELIB}/scrapy/core/downloader/webclient.pyc
-${PYSITELIB}/scrapy/core/downloader/webclient.pyo
 ${PYSITELIB}/scrapy/core/engine.py
 ${PYSITELIB}/scrapy/core/engine.pyc
 ${PYSITELIB}/scrapy/core/engine.pyo
@@ -151,9 +151,6 @@ ${PYSITELIB}/scrapy/crawler.pyo
 ${PYSITELIB}/scrapy/downloadermiddlewares/__init__.py
 ${PYSITELIB}/scrapy/downloadermiddlewares/__init__.pyc
 ${PYSITELIB}/scrapy/downloadermiddlewares/__init__.pyo
-${PYSITELIB}/scrapy/downloadermiddlewares/ajaxcrawl.py
-${PYSITELIB}/scrapy/downloadermiddlewares/ajaxcrawl.pyc
-${PYSITELIB}/scrapy/downloadermiddlewares/ajaxcrawl.pyo
 ${PYSITELIB}/scrapy/downloadermiddlewares/cookies.py
 ${PYSITELIB}/scrapy/downloadermiddlewares/cookies.pyc
 ${PYSITELIB}/scrapy/downloadermiddlewares/cookies.pyo
@@ -396,9 +393,6 @@ ${PYSITELIB}/scrapy/spiders/crawl.pyo
 ${PYSITELIB}/scrapy/spiders/feed.py
 ${PYSITELIB}/scrapy/spiders/feed.pyc
 ${PYSITELIB}/scrapy/spiders/feed.pyo
-${PYSITELIB}/scrapy/spiders/init.py
-${PYSITELIB}/scrapy/spiders/init.pyc
-${PYSITELIB}/scrapy/spiders/init.pyo
 ${PYSITELIB}/scrapy/spiders/sitemap.py
 ${PYSITELIB}/scrapy/spiders/sitemap.pyc
 ${PYSITELIB}/scrapy/spiders/sitemap.pyo
@@ -537,12 +531,6 @@ ${PYSITELIB}/scrapy/utils/template.pyo
 ${PYSITELIB}/scrapy/utils/test.py
 ${PYSITELIB}/scrapy/utils/test.pyc
 ${PYSITELIB}/scrapy/utils/test.pyo
-${PYSITELIB}/scrapy/utils/testproc.py
-${PYSITELIB}/scrapy/utils/testproc.pyc
-${PYSITELIB}/scrapy/utils/testproc.pyo
-${PYSITELIB}/scrapy/utils/testsite.py
-${PYSITELIB}/scrapy/utils/testsite.pyc
-${PYSITELIB}/scrapy/utils/testsite.pyo
 ${PYSITELIB}/scrapy/utils/trackref.py
 ${PYSITELIB}/scrapy/utils/trackref.pyc
 ${PYSITELIB}/scrapy/utils/trackref.pyo

Index: pkgsrc/www/py-scrapy/distinfo
diff -u pkgsrc/www/py-scrapy/distinfo:1.29 pkgsrc/www/py-scrapy/distinfo:1.30
--- pkgsrc/www/py-scrapy/distinfo:1.29  Thu Apr 30 12:27:26 2026
+++ pkgsrc/www/py-scrapy/distinfo       Wed May 20 12:47:09 2026
@@ -1,5 +1,5 @@
-$NetBSD: distinfo,v 1.29 2026/04/30 12:27:26 adam Exp $
+$NetBSD: distinfo,v 1.30 2026/05/20 12:47:09 adam Exp $
 
-BLAKE2s (scrapy-2.15.2.tar.gz) = a6c6a3644debc09caf5a1dffa096a6a2e21051c52dd69826212fd899dbfd3f0b
-SHA512 (scrapy-2.15.2.tar.gz) = 619cb00abefb48657c055eeb1e90d79c5fdaf8468edf44770dafcf0f9d58d117f955aefd3d3cd284f917d2b55b4e86dc08bfd15198717ea12c7051a376dd3761
-Size (scrapy-2.15.2.tar.gz) = 1291748 bytes
+BLAKE2s (scrapy-2.16.0.tar.gz) = 56089af78fecd84bc9d1d89a94d78b8fa18591fadcbad1136179556a1e32e32c
+SHA512 (scrapy-2.16.0.tar.gz) = b6cb3b99c0b49313a1fd2770fdabd2b2cab71cbdbe52216eb5b965bbb27034d7419ae9d3b32b0db4340898045163c2675087f3f2993f57a5fc4ad0c2586b72e8
+Size (scrapy-2.16.0.tar.gz) = 1280834 bytes

Prev by Date: CVS commit: pkgsrc/doc
Next by Date: CVS commit: pkgsrc/lang/py-findpython
Previous by Thread: CVS commit: pkgsrc/doc
Next by Thread: CVS commit: pkgsrc/lang/py-findpython
Indexes:

Home | Main Index | Thread Index | Old Index