CVS commit: pkgsrc/www/py-scrapy

["Adam Ciarcinski" <adam%netbsd.org@localhost>]

Module Name:    pkgsrc
Committed By:   adam
Date:           Thu Jul 23 07:40:43 UTC 2026

Modified Files:
        pkgsrc/www/py-scrapy: Makefile distinfo

Log Message:
py-scrapy: updated to 2.17.0

Scrapy 2.17.0 (2026-07-07)

Highlights:

-   Security bug fixes
-   HTTP/2 and SOCKS proxy support for ``HttpxDownloadHandler``
-   Improved settings for changing allowed TLS versions

Security bug fixes

-   ``s3://`` requests now use HTTPS by default, instead of plaintext HTTP.

    Previously, :class:`~scrapy.core.downloader.handlers.s3.S3DownloadHandler`
    sent signed S3 requests over plaintext HTTP unless
    ``request.meta["is_secure"]`` was set to a true value, exposing the request
    path, the AWS ``Authorization`` header, the ``X-Amz-Security-Token`` header
    (when using temporary credentials), and the response contents to network
    attackers, who could also tamper with responses. See the `76g3-c3x4-crvx`_
    security advisory for details.

    To restore the previous behavior for a given request, set
    ``request.meta["is_secure"]`` to ``False``.

    .. _76g3-c3x4-crvx: https://github.com/scrapy/scrapy/security/advisories/GHSA-76g3-c3x4-crvx

Deprecations

-   The ``DOWNLOADER_CLIENT_TLS_METHOD`` setting is deprecated. You should use
    the :setting:`DOWNLOAD_TLS_MIN_VERSION` and/or
    :setting:`DOWNLOAD_TLS_MAX_VERSION` settings instead if you want to change
    the TLS method selection.

-   The following spider attributes are deprecated in favor of settings:

    - ``http_user`` (use :setting:`HTTPAUTH_USER`)
    - ``http_pass`` (use :setting:`HTTPAUTH_PASS`)
    - ``http_auth_domain`` (use :setting:`HTTPAUTH_DOMAIN`)

-   The ``scrapy.commands.ScrapyCommand.help()`` method is deprecated. It was
    never called by Scrapy.

-   The following TLS-related functions and constants, intended for internal
    use, are deprecated:

    - ``scrapy.core.downloader.tls.METHOD_TLS``
    - ``scrapy.core.downloader.tls.METHOD_TLSv10``
    - ``scrapy.core.downloader.tls.METHOD_TLSv11``
    - ``scrapy.core.downloader.tls.METHOD_TLSv12``
    - ``scrapy.core.downloader.tls.openssl_methods``
    - ``scrapy.core.downloader.tls.DEFAULT_CIPHERS``
    - ``scrapy.utils.ssl.ffi_buf_to_string()``
    - ``scrapy.utils.ssl.get_temp_key_info()``
    - ``scrapy.utils.ssl.x509name_to_string()``

-   The ``CRAWLSPIDER_FOLLOW_LINKS`` setting is deprecated. You can set
    ``follow=False`` in your rules to achieve the same effect.

-   Instantiating
    :class:`~scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware`
    without a ``crawler`` argument is deprecated.

-   Instantiating
    :class:`~scrapy.spidermiddlewares.referer.RefererMiddleware` without a
    ``settings`` argument is deprecated.

New features

-   Added support for HTTP/2 requests to
    :class:`~scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler`. It
    requires setting the new :setting:`HTTPX_HTTP2_ENABLED` setting to
    ``True``.

-   Added support for SOCKS proxies to
    :class:`~scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler`.

-   Added :setting:`DOWNLOAD_TLS_MIN_VERSION` and
    :setting:`DOWNLOAD_TLS_MAX_VERSION` settings as replacements for the
    ``DOWNLOADER_CLIENT_TLS_METHOD`` setting (which is now deprecated).
    Compared to the old setting, they support specifying a range of allowed
    versions and support newer TLS versions.

-   Added :setting:`HTTPAUTH_USER`, :setting:`HTTPAUTH_PASS` and
    :setting:`HTTPAUTH_DOMAIN` settings and :reqmeta:`http_user`,
    :reqmeta:`http_pass` and :reqmeta:`http_auth_domain` meta keys as more
    flexible ways to set HTTP authentication data.

-   Added a :reqmeta:`verbatim_url` meta key that can be set to ``True`` to
    skip request URL canonicalization.

-   Added ``deny_tags`` and ``deny_attrs`` arguments to :class:`LinkExtractor
    <scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor>`.

-   :attr:`scrapy.Item.fields` now returns the fields in the definition order
    instead of the alphabetical one.

-   Added a :setting:`RETRY_GIVE_UP_LOG_LEVEL` setting, a
    :reqmeta:`give_up_log_level` meta key and a ``give_up_log_level`` argument
    of the
    :func:`~scrapy.downloadermiddlewares.retry.get_retry_request` function that
    allow changing the log level of the message logged when the retry limit has
    been reached.

-   It's now possible to set :setting:`DOWNLOADER_CLIENT_TLS_CIPHERS` to
    ``None`` to use the default ciphers of the underlying TLS implementation.

Improvements

-   :class:`~scrapy.FormRequest` is no longer deprecated, only its
    ``from_response()`` method is still deprecated.

-   Switched the item definition in the default project template from a
    :class:`scrapy.item.Item` to a dataclass.

-   Fixed deprecation warnings with pyOpenSSL 26.3.0.

-   Removed the runtime warnings for :attr:`Spider.allowed_domains
    <scrapy.Spider.allowed_domains>` containing URLs or domains with ports
    instead of just domains and for spider classes having a ``start_url``
    attribute instead of :class:`~scrapy.spiders.Spider.start_urls`. Please use
    :doc:`scrapy-lint <scrapy-lint:index>` to find mistakes in your spider code
    instead.

-   :func:`scrapy.utils.test.get_crawler` now disables
    :setting:`TELNETCONSOLE_ENABLED` by default.

-   Other code refactoring and improvements.

Bug fixes

-   :class:`~scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler` no
    longer ignores proxy credentials for redirected or retried requests.

-   :class:`~scrapy.extensions.feedexport.GCSFeedStorage` now closes the
    temporary file after the upload.

-   Fixed ``scrapy shell <URL>`` running a full spider crawl when there is a
    spider for the requested URL. This bug was introduced in Scrapy 2.13.0.

-   The :setting:`IMAGES_STORE_S3_ACL` and :setting:`IMAGES_STORE_GCS_ACL`
    settings are no longer ignored. This bug was introduced in Scrapy 2.12.0.

-   :class:`~scrapy.core.downloader.handlers.ftp.FTPDownloadHandler` now closes
    the connection after making the request.

-   Removed the deprecated ``spider`` argument from the pipeline defined in the
    default project template.

-   Fixed ``scrapy genspider --edit`` not working.

-   When a :class:`~scrapy.crawler.Crawler` instance is passed to
    :meth:`AsyncCrawlerRunner.create_crawler()
    <scrapy.crawler.AsyncCrawlerRunner.create_crawler>` or
    :meth:`CrawlerRunner.create_crawler()
    <scrapy.crawler.CrawlerRunner.create_crawler>`, settings from both classes
    are now merged, previously only the settings from the
    :class:`~scrapy.crawler.Crawler` instance were used.

-   Fixed several issues with cookie handling in
    :func:`scrapy.utils.request.request_to_curl`.

-   Fixed :class:`scrapy.resolver.CachingThreadedResolver` not disabling the
    cache when :setting:`DNSCACHE_ENABLED` is set to ``False``.

-   Fixed :func:`scrapy.utils.response.open_in_browser` not removing comments
    when looking for the ``<base>`` tag.

-   Fixed checking for deprecated methods in custom :setting:`ITEM_PROCESSOR`
    implementations.

-   Fixed :func:`scrapy.utils.url.strip_url` corrupting some URLs with
    credentials.

-   :func:`scrapy.utils.misc.rel_has_nofollow` now ignores the case when
    looking for "nofollow" strings.

-   Fixed an exception in :class:`scrapy.utils.sitemap.Sitemap` when parsing
    some malformed sitemaps.


To generate a diff of this commit:
cvs rdiff -u -r1.41 -r1.42 pkgsrc/www/py-scrapy/Makefile
cvs rdiff -u -r1.30 -r1.31 pkgsrc/www/py-scrapy/distinfo

Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.

Modified files:

Index: pkgsrc/www/py-scrapy/Makefile
diff -u pkgsrc/www/py-scrapy/Makefile:1.41 pkgsrc/www/py-scrapy/Makefile:1.42
--- pkgsrc/www/py-scrapy/Makefile:1.41  Sun Jun 28 15:41:49 2026
+++ pkgsrc/www/py-scrapy/Makefile       Thu Jul 23 07:40:43 2026
@@ -1,6 +1,6 @@
-# $NetBSD: Makefile,v 1.41 2026/06/28 15:41:49 wiz Exp $
+# $NetBSD: Makefile,v 1.42 2026/07/23 07:40:43 adam Exp $
 
-DISTNAME=      scrapy-2.16.0
+DISTNAME=      scrapy-2.17.0
 PKGNAME=       ${PYPKGPREFIX}-${DISTNAME}
 CATEGORIES=    www python
 MASTER_SITES=  ${MASTER_SITE_PYPI:=S/Scrapy/}

Index: pkgsrc/www/py-scrapy/distinfo
diff -u pkgsrc/www/py-scrapy/distinfo:1.30 pkgsrc/www/py-scrapy/distinfo:1.31
--- pkgsrc/www/py-scrapy/distinfo:1.30  Wed May 20 12:47:09 2026
+++ pkgsrc/www/py-scrapy/distinfo       Thu Jul 23 07:40:43 2026
@@ -1,5 +1,5 @@
-$NetBSD: distinfo,v 1.30 2026/05/20 12:47:09 adam Exp $
+$NetBSD: distinfo,v 1.31 2026/07/23 07:40:43 adam Exp $
 
-BLAKE2s (scrapy-2.16.0.tar.gz) = 56089af78fecd84bc9d1d89a94d78b8fa18591fadcbad1136179556a1e32e32c
-SHA512 (scrapy-2.16.0.tar.gz) = b6cb3b99c0b49313a1fd2770fdabd2b2cab71cbdbe52216eb5b965bbb27034d7419ae9d3b32b0db4340898045163c2675087f3f2993f57a5fc4ad0c2586b72e8
-Size (scrapy-2.16.0.tar.gz) = 1280834 bytes
+BLAKE2s (scrapy-2.17.0.tar.gz) = 3866c1c08618b848614800f313d516fab563bb5a15913f9f74485cd18442a2f2
+SHA512 (scrapy-2.17.0.tar.gz) = 97482910f75bdf2812fa8a13b9d31dea579454d2235d761fd655e154d4096cc86483ca88150153d7cb0a14449fcb550515a4a23493c55e4ec0a435adaf0be80b
+Size (scrapy-2.17.0.tar.gz) = 1312077 bytes