CVS commit: pkgsrc/www/py-scrapy

["Adam Ciarcinski" <adam%netbsd.org@localhost>]

Module Name:    pkgsrc
Committed By:   adam
Date:           Mon Oct  5 07:26:41 UTC 2026

Modified Files:
        pkgsrc/www/py-scrapy: Makefile PLIST distinfo

Log Message:
py-scrapy: updated to 2.19.0

Scrapy 2.19.0 (2026-09-10)

Highlights:

-   New ``RemoteControl`` extension which allows inspecting and controlling a
    running crawl over HTTP, used by the :ref:`Scrapy MCP server
    <using-mcp-server>`

-   Experimental ``aiohttp``-based download handler (now the default when
    running without a reactor)

Modified requirements

-   Added support for Python 3.15.
-   New dependencies:

    - aiohttp_ >= 3.13.3
    - charset-normalizer_ >= 3.4.0
    - platformdirs_ >= 2.0.0

Backward-incompatible changes

-   When running :ref:`without a Twisted reactor <asyncio-without-reactor>`,
    i.e. with :setting:`TWISTED_REACTOR_ENABLED` set to ``False``, the default
    download handler for ``http`` and ``https`` is now
    :class:`~scrapy.core.downloader.handlers._aiohttp.AiohttpDownloadHandler`
    instead of
    :class:`~scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler`. You
    can configure
    :class:`~scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler` in
    the :setting:`DOWNLOAD_HANDLERS` setting if you want.

-   ``scrapy.http.cookies.WrappedRequest.is_unverifiable()`` now always returns
    ``False``, and the undocumented ``is_unverifiable`` request meta key that
    it used to read is ignored.

Deprecations

-   The ``RANDOMIZE_DOWNLOAD_DELAY`` setting is deprecated. Use the new
    :setting:`DOWNLOAD_DELAY_JITTER` setting instead. Similarly, the
    ``randomize_delay`` key of the :setting:`DOWNLOAD_SLOTS` setting is
    deprecated in favor of a new ``jitter`` key, the ``randomize_delay``
    attribute of :class:`scrapy.core.downloader.Slot` is deprecated in favor of
    its new ``jitter`` attribute, and the ``randomize_delay`` attribute
    of :class:`scrapy.core.downloader.Downloader` is deprecated in favor of the
    :setting:`DOWNLOAD_DELAY_JITTER` setting.

-   The ``install_root_handler`` parameter of
    :func:`~scrapy.utils.log.configure_logging`,
    :class:`~scrapy.crawler.CrawlerProcess` and
    :class:`~scrapy.crawler.AsyncCrawlerProcess` is deprecated. Use the new
    :setting:`LOG_INSTALL_ROOT_HANDLER` setting instead.

-   The ``scrapy.dupefilters.RFPDupeFilter.fingerprints`` attribute is
    deprecated. Overriding
    :meth:`scrapy.dupefilters.RFPDupeFilter.request_fingerprint` is deprecated
    as well; set the :setting:`REQUEST_FINGERPRINTER_CLASS` setting instead.

-   Overriding the ``add_pre_hook()`` or ``add_post_hook()`` methods of
    :class:`~scrapy.contracts.Contract` is deprecated. Define
    ``pre_process()`` or ``post_process()`` instead.

New features

-   Added a :class:`~scrapy.extensions.remote_control.RemoteControl`
    extension, enabled by default, which allows connecting to crawl processes
    via HTTP and running code inside them.

-   Added an experimental HTTP download handler based on aiohttp_,
    :class:`~scrapy.core.downloader.handlers._aiohttp.AiohttpDownloadHandler`.

-   Added SQLite-backed scheduler queues:
    :class:`~scrapy.squeues.PickleFifoSQLiteQueue`,
    :class:`~scrapy.squeues.PickleLifoSQLiteQueue`,
    :class:`~scrapy.squeues.MarshalFifoSQLiteQueue` and
    :class:`~scrapy.squeues.MarshalLifoSQLiteQueue`. They write each request
    within its own transaction, so that an unclean shutdown cannot corrupt the
    on-disk queue, at the cost of slower scheduling.

-   Added a :setting:`DOWNLOAD_DELAY_JITTER` setting, and a ``jitter`` key for
    :setting:`DOWNLOAD_SLOTS`, which set the magnitude of the random variation
    applied to :setting:`DOWNLOAD_DELAY`. They replace the
    ``RANDOMIZE_DOWNLOAD_DELAY`` setting and the ``randomize_delay`` key, which
    could only toggle a fixed ±50%.

-   Added a :setting:`LOG_COLOR` setting, which colorizes log output by log
    level when logging to a terminal. It needs the new :ref:`color <extras>`
    extra.

-   The encoding of a response that doesn't declare one is now detected with
    charset-normalizer_ when the response body is neither ASCII nor UTF-8,
    instead of always falling back to ``cp1252``.

-   Added a :setting:`LOG_INSTALL_ROOT_HANDLER` setting, which replaces the
    deprecated ``install_root_handler`` parameter and, unlike it, can also be
    set from a :file:`settings.py` module or from the command line.

-   :ref:`Contracts <topics-contracts>` now support callbacks defined with
    ``async def``, including asynchronous generators.

-   Added the :class:`~scrapy.contracts.default.MethodContract` (``@method``),
    :class:`~scrapy.contracts.default.BodyContract` (``@body``),
    :class:`~scrapy.contracts.default.HeaderContract` (``@header``) and
    :class:`~scrapy.contracts.default.CookieContract` (``@cookie``)
    contracts, which set the corresponding attributes of the sample request,
    and an ``-a`` option for the :command:`check` command, to set spider
    arguments as in the :command:`crawl` command.

-   Added :meth:`Response.to_dict() <scrapy.http.Response.to_dict>`,
    :meth:`Response.from_dict() <scrapy.http.Response.from_dict>` and
    :func:`~scrapy.utils.response.response_from_dict`, and used them in the
    built-in :ref:`HTTP cache storages <httpcache-storage-fs>`, which now
    restore cached responses of any response class, including those of
    third-party plugins, with all their attributes.

-   Cached responses now indicate when they were stored, through the new
    :reqmeta:`cache_timestamp` request meta key.

-   The ``format`` key of :setting:`FEEDS` is now inferred from the file
    extension of the feed URI when not set, e.g. ``json`` for a URI ending in
    :file:`.json`.

-   Added :func:`~scrapy.utils.project.find_projects`, which yields the root
    directory of every Scrapy project in a directory tree.

-   Added a public :attr:`~scrapy.core.engine.ExecutionEngine.scheduler`
    attribute to :class:`~scrapy.core.engine.ExecutionEngine`.

-   :class:`~scrapy.statscollectors.StatsCollector` objects now implement
    ``__str__()``, which returns the pretty-printed stats.

-   :func:`~scrapy.utils.response.open_in_browser` now also supports
    responses that are neither HTML nor plain text, picking a file extension
    based on the ``Content-Type`` header.

-   The ``-t``/``--template`` option of the :command:`genspider` command now
    also takes a path to a :file:`.tmpl` file, so that a custom template can
    be used without setting :setting:`TEMPLATES_DIR`.

-   The ``--pdb`` command-line option now uses ipdb_ instead of :mod:`pdb`
    when ipdb_ is installed.

-   Log records about item processing, i.e. about items scraped, dropped or
    raising an exception, now carry the item in their ``extra`` dict, under
    the ``item`` key, so that custom logging handlers can read it.

Improvements

-   :class:`~scrapy.dupefilters.RFPDupeFilter` now keeps request fingerprints
    as :class:`bytes`, to reduce memory usage. The :file:`requests.seen` file
    that it writes in the :ref:`job directory <job-dir>` is now a binary file.

-   The ``--pdb`` command-line option now starts a post-mortem debugging
    session on every logged error that comes with a traceback, instead of on
    every :class:`~twisted.python.failure.Failure` object built anywhere in
    the process.

-   The Scrapy :command:`shell` no longer prints the ``DEBUG`` messages of
    parso_, an indirect dependency of IPython, when :setting:`LOG_LEVEL` is
    ``DEBUG``.

-   Other code refactoring and improvements.

Bug fixes

-   :class:`~scrapy.core.downloader.handlers.http11.HTTP11DownloadHandler`
    now sets :attr:`~scrapy.http.Response.certificate` and
    :attr:`~scrapy.http.Response.ip_address` attributes even for responses
    without a body.

-   HTTP connection pool keys in
    :class:`~scrapy.core.downloader.handlers.http11.HTTP11DownloadHandler` and
    :class:`~scrapy.core.downloader.handlers.http2.H2DownloadHandler` now
    include the request bind address (see :setting:`DOWNLOAD_BIND_ADDRESS`), so
    that requests bound to different addresses no longer reuse each other's
    pooled connections.

-   :class:`~scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware` now
    updates the headers of a cached response, and stores it again, when a
    revalidation request gets a 304 response.

-   An exception raised in a spider callback that no
    :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_spider_exception`
    method handles is now offered to each of those methods only once.

-   Defining :attr:`~scrapy.Spider.allowed_domains` as a property no longer
    breaks the commands that match a URL to a spider, i.e. :command:`shell`,
    :command:`fetch` and :command:`parse`. A property cannot be evaluated on
    a spider class, so it is now ignored, with a warning, instead of raising
    :exc:`TypeError`.

-   :func:`~scrapy.utils.project.get_project_settings` now adds the project
    directory to :data:`sys.path` also when the ``SCRAPY_SETTINGS_MODULE``
    environment variable is set, so that the settings module that the variable
    points to can be imported.

-   :class:`scrapy.utils.datatypes.LocalCache` no longer evicts the oldest item
    when updating an existing one.

Documentation

-   Added a page :ref:`about using Scrapy with coding agents <agents>`,
    covering the official agent plugin and the Scrapy MCP server.

-   Documented the request metadata keys that were missing from the
    :ref:`list of special keys <topics-request-meta>`, and documented that keys
    whose name starts with an underscore are internal.

-   Documented :ref:`how to write custom Scrapy commands <custom-commands>`,
    :ref:`how to write custom spider templates <spider-templates>`, :ref:`how
    to access the response of a failed media download
    <media-pipeline-failed-downloads>` and :ref:`how to set request headers
    for media pipeline requests <media-request-headers>`.

-   Documented how to subclass
    :class:`~scrapy.downloadermiddlewares.redirect.RedirectMiddleware` to
    allow or deny redirects based on the target URL, and how to set the
    priority of the requests that :class:`~scrapy.spiders.CrawlSpider`
    generates from its rules.

-   Documented that :func:`~scrapy.utils.project.get_project_settings`
    returns project settings only, that spiders :ref:`running concurrently in
    the same process <run-multiple-spiders>` get independent crawlers,
    middlewares and settings, and that Scrapy sets the ``request`` attribute
    of the :class:`~twisted.python.failure.Failure` objects that it passes to
    errbacks.

-   Documented :class:`~scrapy.commands.ScrapyCommand` and
    :class:`~scrapy.statscollectors.StatsCollector`.

-   Improved the contents of ``llms-full.txt`` and Markdown versions of
    documentation pages.


To generate a diff of this commit:
cvs rdiff -u -r1.43 -r1.44 pkgsrc/www/py-scrapy/Makefile
cvs rdiff -u -r1.19 -r1.20 pkgsrc/www/py-scrapy/PLIST
cvs rdiff -u -r1.32 -r1.33 pkgsrc/www/py-scrapy/distinfo

Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.

Modified files:

Index: pkgsrc/www/py-scrapy/Makefile
diff -u pkgsrc/www/py-scrapy/Makefile:1.43 pkgsrc/www/py-scrapy/Makefile:1.44
--- pkgsrc/www/py-scrapy/Makefile:1.43  Thu Sep  3 08:35:37 2026
+++ pkgsrc/www/py-scrapy/Makefile       Mon Oct  5 07:26:41 2026
@@ -1,6 +1,6 @@
-# $NetBSD: Makefile,v 1.43 2026/09/03 08:35:37 adam Exp $
+# $NetBSD: Makefile,v 1.44 2026/10/05 07:26:41 adam Exp $
 
-DISTNAME=      scrapy-2.18.0
+DISTNAME=      scrapy-2.19.0
 PKGNAME=       ${PYPKGPREFIX}-${DISTNAME}
 CATEGORIES=    www python
 MASTER_SITES=  ${MASTER_SITE_PYPI:=S/Scrapy/}
@@ -12,7 +12,9 @@ LICENSE=      modified-bsd
 
 TOOL_DEPENDS+= ${PYPKGPREFIX}-hatchling>=1.27.0:../../devel/py-hatchling
 DEPENDS+=      ${PYPKGPREFIX}-OpenSSL>=24.3.0:../../security/py-OpenSSL
+DEPENDS+=      ${PYPKGPREFIX}-aiohttp>=3.13.3:../../www/py-aiohttp
 DEPENDS+=      ${PYPKGPREFIX}-brotli>=1.2.0:../../archivers/py-brotli
+DEPENDS+=      ${PYPKGPREFIX}-charset-normalizer>=3.4.0:../../converters/py-charset-normalizer
 DEPENDS+=      ${PYPKGPREFIX}-cryptography>=41.0.5:../../security/py-cryptography
 DEPENDS+=      ${PYPKGPREFIX}-cssselect>=0.9.1:../../textproc/py-cssselect
 DEPENDS+=      ${PYPKGPREFIX}-defusedxml>=0.7.1:../../textproc/py-defusedxml
@@ -20,6 +22,7 @@ DEPENDS+=     ${PYPKGPREFIX}-itemadapter>=0.
 DEPENDS+=      ${PYPKGPREFIX}-itemloaders>=1.0.1:../../textproc/py-itemloaders
 DEPENDS+=      ${PYPKGPREFIX}-lxml>=4.6.4:../../textproc/py-lxml
 DEPENDS+=      ${PYPKGPREFIX}-packaging>=0:../../devel/py-packaging
+DEPENDS+=      ${PYPKGPREFIX}-platformdirs>=2.0.0:../../misc/py-platformdirs
 DEPENDS+=      ${PYPKGPREFIX}-parsel>=1.5.0:../../www/py-parsel
 DEPENDS+=      ${PYPKGPREFIX}-protego>=0.1.15:../../www/py-protego
 DEPENDS+=      ${PYPKGPREFIX}-pydispatcher>=2.0.5:../../devel/py-pydispatcher

Index: pkgsrc/www/py-scrapy/PLIST
diff -u pkgsrc/www/py-scrapy/PLIST:1.19 pkgsrc/www/py-scrapy/PLIST:1.20
--- pkgsrc/www/py-scrapy/PLIST:1.19     Thu Sep  3 08:35:37 2026
+++ pkgsrc/www/py-scrapy/PLIST  Mon Oct  5 07:26:41 2026
@@ -1,4 +1,4 @@
-@comment $NetBSD: PLIST,v 1.19 2026/09/03 08:35:37 adam Exp $
+@comment $NetBSD: PLIST,v 1.20 2026/10/05 07:26:41 adam Exp $
 bin/scrapy-${PYVERSSUFFIX}
 ${PYSITELIB}/${WHEEL_INFODIR}/METADATA
 ${PYSITELIB}/${WHEEL_INFODIR}/RECORD
@@ -97,6 +97,9 @@ ${PYSITELIB}/scrapy/core/downloader/cont
 ${PYSITELIB}/scrapy/core/downloader/handlers/__init__.py
 ${PYSITELIB}/scrapy/core/downloader/handlers/__init__.pyc
 ${PYSITELIB}/scrapy/core/downloader/handlers/__init__.pyo
+${PYSITELIB}/scrapy/core/downloader/handlers/_aiohttp.py
+${PYSITELIB}/scrapy/core/downloader/handlers/_aiohttp.pyc
+${PYSITELIB}/scrapy/core/downloader/handlers/_aiohttp.pyo
 ${PYSITELIB}/scrapy/core/downloader/handlers/_base_http.py
 ${PYSITELIB}/scrapy/core/downloader/handlers/_base_http.pyc
 ${PYSITELIB}/scrapy/core/downloader/handlers/_base_http.pyo
@@ -241,6 +244,9 @@ ${PYSITELIB}/scrapy/extensions/periodic_
 ${PYSITELIB}/scrapy/extensions/postprocessing.py
 ${PYSITELIB}/scrapy/extensions/postprocessing.pyc
 ${PYSITELIB}/scrapy/extensions/postprocessing.pyo
+${PYSITELIB}/scrapy/extensions/remote_control.py
+${PYSITELIB}/scrapy/extensions/remote_control.pyc
+${PYSITELIB}/scrapy/extensions/remote_control.pyo
 ${PYSITELIB}/scrapy/extensions/spiderstate.py
 ${PYSITELIB}/scrapy/extensions/spiderstate.pyc
 ${PYSITELIB}/scrapy/extensions/spiderstate.pyo
@@ -435,6 +441,9 @@ ${PYSITELIB}/scrapy/utils/_deps_compat.p
 ${PYSITELIB}/scrapy/utils/_download_handlers.py
 ${PYSITELIB}/scrapy/utils/_download_handlers.pyc
 ${PYSITELIB}/scrapy/utils/_download_handlers.pyo
+${PYSITELIB}/scrapy/utils/_remote_control.py
+${PYSITELIB}/scrapy/utils/_remote_control.pyc
+${PYSITELIB}/scrapy/utils/_remote_control.pyo
 ${PYSITELIB}/scrapy/utils/asyncgen.py
 ${PYSITELIB}/scrapy/utils/asyncgen.pyc
 ${PYSITELIB}/scrapy/utils/asyncgen.pyo

Index: pkgsrc/www/py-scrapy/distinfo
diff -u pkgsrc/www/py-scrapy/distinfo:1.32 pkgsrc/www/py-scrapy/distinfo:1.33
--- pkgsrc/www/py-scrapy/distinfo:1.32  Thu Sep  3 08:35:37 2026
+++ pkgsrc/www/py-scrapy/distinfo       Mon Oct  5 07:26:41 2026
@@ -1,5 +1,5 @@
-$NetBSD: distinfo,v 1.32 2026/09/03 08:35:37 adam Exp $
+$NetBSD: distinfo,v 1.33 2026/10/05 07:26:41 adam Exp $
 
-BLAKE2s (scrapy-2.18.0.tar.gz) = c8cab4d08ccf4fe404cc787a780887c2833a8d89e32fd4a47d1967476dc9ba8c
-SHA512 (scrapy-2.18.0.tar.gz) = 237a0ebc0937f871d4d0ec6e29622993f211337b3ff16e2087139b39f9753697526f9145c8c88feee7760516bf8249f7b9393e3df80c6d792e098210424c7e34
-Size (scrapy-2.18.0.tar.gz) = 1407695 bytes
+BLAKE2s (scrapy-2.19.0.tar.gz) = c8285f32018df238cb9ad0e7fd84444f1a014d0b7efed6c28e2b716cfeee493e
+SHA512 (scrapy-2.19.0.tar.gz) = c8c21c4157ae542b96fcf1cde0594485af65d5fd1822e1933dfabc00de64b5599b13f331af13db34604ae7aced4d7f65a88d6d3d3fb634e7d8e877534900137a
+Size (scrapy-2.19.0.tar.gz) = 1457780 bytes