pkgsrc-Changes archive
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]
CVS commit: pkgsrc/www/py-scrapy
Module Name: pkgsrc
Committed By: adam
Date: Thu Apr 27 09:33:44 UTC 2023
Modified Files:
pkgsrc/www/py-scrapy: Makefile PLIST distinfo
Log Message:
py-scrapy: updated to 2.8.0
Scrapy 2.8.0 (2023-02-02)
-------------------------
This is a maintenance release, with minor features, bug fixes, and cleanups.
Deprecation removals
~~~~~~~~~~~~~~~~~~~~
- The ``scrapy.utils.gz.read1`` function, deprecated in Scrapy 2.0, has now
been removed. Use the :meth:`~io.BufferedIOBase.read1` method of
:class:`~gzip.GzipFile` instead.
- The ``scrapy.utils.python.to_native_str`` function, deprecated in Scrapy
2.0, has now been removed. Use :func:`scrapy.utils.python.to_unicode`
instead.
- The ``scrapy.utils.python.MutableChain.next`` method, deprecated in Scrapy
2.0, has now been removed. Use
:meth:`~scrapy.utils.python.MutableChain.__next__` instead.
- The ``scrapy.linkextractors.FilteringLinkExtractor`` class, deprecated
in Scrapy 2.0, has now been removed. Use
:class:`LinkExtractor <scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor>`
instead.
- Support for using environment variables prefixed with ``SCRAPY_`` to
override settings, deprecated in Scrapy 2.0, has now been removed.
- Support for the ``noconnect`` query string argument in proxy URLs,
deprecated in Scrapy 2.0, has now been removed. We expect proxies that used
to need it to work fine without it.
- The ``scrapy.utils.python.retry_on_eintr`` function, deprecated in Scrapy
2.3, has now been removed.
- The ``scrapy.utils.python.WeakKeyCache`` class, deprecated in Scrapy 2.4,
has now been removed.
Deprecations
~~~~~~~~~~~~
- :exc:`scrapy.pipelines.images.NoimagesDrop` is now deprecated.
- :meth:`ImagesPipeline.convert_image
<scrapy.pipelines.images.ImagesPipeline.convert_image>` must now accept a
``response_body`` parameter.
New features
~~~~~~~~~~~~
- Applied black_ coding style to files generated with the
:command:`genspider` and :command:`startproject` commands.
.. _black: https://black.readthedocs.io/en/stable/
- :setting:`FEED_EXPORT_ENCODING` is now set to ``"utf-8"`` in the
``settings.py`` file that the :command:`startproject` command generates.
With this value, JSON exports won’t force the use of escape sequences for
non-ASCII characters.
- The :class:`~scrapy.extensions.memusage.MemoryUsage` extension now logs the
peak memory usage during checks, and the binary unit MiB is now used to
avoid confusion.
- The ``callback`` parameter of :class:`~scrapy.http.Request` can now be set
to :func:`scrapy.http.request.NO_CALLBACK`, to distinguish it from
``None``, as the latter indicates that the default spider callback
(:meth:`~scrapy.Spider.parse`) is to be used.
Bug fixes
~~~~~~~~~
- Enabled unsafe legacy SSL renegotiation to fix access to some outdated
websites.
- Fixed STARTTLS-based email delivery not working with Twisted 21.2.0 and
better.
- Fixed the :meth:`finish_exporting` method of :ref:`item exporters
<topics-exporters>` not being called for empty files.
- Fixed HTTP/2 responses getting only the last value for a header when
multiple headers with the same name are received.
- Fixed an exception raised by the :command:`shell` command on some cases
when :ref:`using asyncio <using-asyncio>`.
- When using :class:`~scrapy.spiders.CrawlSpider`, callback keyword arguments
(``cb_kwargs``) added to a request in the ``process_request`` callback of a
:class:`~scrapy.spiders.Rule` will no longer be ignored.
- The :ref:`images pipeline <images-pipeline>` no longer re-encodes JPEG
files.
- Fixed the handling of transparent WebP images by the :ref:`images pipeline
<images-pipeline>`.
- :func:`scrapy.shell.inspect_response` no longer inhibits ``SIGINT``
(Ctrl+C).
- :class:`LinkExtractor <scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor>`
with ``unique=False`` no longer filters out links that have identical URL
*and* text.
- :class:`~scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware` now
ignores URL protocols that do not support ``robots.txt`` (``data://``,
``file://``).
- Silenced the ``filelock`` debug log messages introduced in Scrapy 2.6.
- Fixed the output of ``scrapy -h`` showing an unintended ``**commands**``
line.
- Made the active project indication in the output of :ref:`commands
<topics-commands>` more clear.
Documentation
~~~~~~~~~~~~~
- Documented how to :ref:`debug spiders from Visual Studio Code
<debug-vscode>`.
- Documented how :setting:`DOWNLOAD_DELAY` affects per-domain concurrency.
- Improved consistency.
- Fixed typos.
Quality assurance
~~~~~~~~~~~~~~~~~
- Applied :ref:`black coding style <coding-style>`, sorted import statements,
and introduced :ref:`pre-commit <scrapy-pre-commit>`.
- Switched from :mod:`os.path` to :mod:`pathlib`.
- Addressed many issues reported by Pylint.
- Improved code readability.
- Improved package metadata.
- Removed direct invocations of ``setup.py``.
- Removed unnecessary :class:`~collections.OrderedDict` usages.
- Removed unnecessary ``__str__`` definitions.
- Removed obsolete code and comments.
- Fixed test and CI issues.
To generate a diff of this commit:
cvs rdiff -u -r1.16 -r1.17 pkgsrc/www/py-scrapy/Makefile
cvs rdiff -u -r1.8 -r1.9 pkgsrc/www/py-scrapy/PLIST
cvs rdiff -u -r1.12 -r1.13 pkgsrc/www/py-scrapy/distinfo
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
Modified files:
Index: pkgsrc/www/py-scrapy/Makefile
diff -u pkgsrc/www/py-scrapy/Makefile:1.16 pkgsrc/www/py-scrapy/Makefile:1.17
--- pkgsrc/www/py-scrapy/Makefile:1.16 Wed Oct 19 14:25:20 2022
+++ pkgsrc/www/py-scrapy/Makefile Thu Apr 27 09:33:44 2023
@@ -1,8 +1,7 @@
-# $NetBSD: Makefile,v 1.16 2022/10/19 14:25:20 nia Exp $
+# $NetBSD: Makefile,v 1.17 2023/04/27 09:33:44 adam Exp $
-DISTNAME= Scrapy-2.4.1
+DISTNAME= Scrapy-2.8.0
PKGNAME= ${PYPKGPREFIX}-${DISTNAME:tl}
-PKGREVISION= 2
CATEGORIES= www python
MASTER_SITES= ${MASTER_SITE_PYPI:=S/Scrapy/}
@@ -11,25 +10,27 @@ HOMEPAGE= https://scrapy.org/
COMMENT= High-level Web Crawling and Web Scraping framework
LICENSE= modified-bsd
-DEPENDS+= ${PYPKGPREFIX}-ZopeInterface>=4.1.3:../../devel/py-ZopeInterface
+DEPENDS+= ${PYPKGPREFIX}-ZopeInterface>=5.1.0:../../devel/py-ZopeInterface
DEPENDS+= ${PYPKGPREFIX}-cssselect>=0.9.1:../../textproc/py-cssselect
+DEPENDS+= ${PYPKGPREFIX}-itemadapter>=0.1.0:../../textproc/py-itemadapter
DEPENDS+= ${PYPKGPREFIX}-itemloaders>=1.0.1:../../textproc/py-itemloaders
-DEPENDS+= ${PYPKGPREFIX}-lxml>=3.5.0:../../textproc/py-lxml
+DEPENDS+= ${PYPKGPREFIX}-lxml>=4.3.0:../../textproc/py-lxml
+DEPENDS+= ${PYPKGPREFIX}-packaging-[0-9]*:../../devel/py-packaging
DEPENDS+= ${PYPKGPREFIX}-parsel>=1.5.0:../../www/py-parsel
DEPENDS+= ${PYPKGPREFIX}-protego>=0.1.15:../../www/py-protego
DEPENDS+= ${PYPKGPREFIX}-pydispatcher>=2.0.5:../../devel/py-pydispatcher
DEPENDS+= ${PYPKGPREFIX}-queuelib>=1.4.2:../../devel/py-queuelib
-DEPENDS+= ${PYPKGPREFIX}-service_identity>=16.0.0:../../security/py-service_identity
-DEPENDS+= ${PYPKGPREFIX}-six>=1.10.0:../../lang/py-six
-DEPENDS+= ${PYPKGPREFIX}-twisted>=17.9.0:../../net/py-twisted
+DEPENDS+= ${PYPKGPREFIX}-service_identity>=18.1.0:../../security/py-service_identity
+DEPENDS+= ${PYPKGPREFIX}-tldextract-[0-9]*:../../net/py-tldextract
+DEPENDS+= ${PYPKGPREFIX}-twisted>=18.9.0:../../net/py-twisted
DEPENDS+= ${PYPKGPREFIX}-w3lib>=1.17.0:../../www/py-w3lib
USE_LANGUAGES= # none
-PYTHON_VERSIONS_INCOMPATIBLE= 27
-
USE_PKG_RESOURCES= yes
+PYTHON_VERSIONS_INCOMPATIBLE= 27
+
post-install:
cd ${DESTDIR}${PREFIX}/bin && \
${MV} scrapy scrapy-${PYVERSSUFFIX} || ${TRUE}
Index: pkgsrc/www/py-scrapy/PLIST
diff -u pkgsrc/www/py-scrapy/PLIST:1.8 pkgsrc/www/py-scrapy/PLIST:1.9
--- pkgsrc/www/py-scrapy/PLIST:1.8 Mon Mar 22 08:56:56 2021
+++ pkgsrc/www/py-scrapy/PLIST Thu Apr 27 09:33:44 2023
@@ -1,4 +1,4 @@
-@comment $NetBSD: PLIST,v 1.8 2021/03/22 08:56:56 triaxx Exp $
+@comment $NetBSD: PLIST,v 1.9 2023/04/27 09:33:44 adam Exp $
bin/scrapy-${PYVERSSUFFIX}
${PYSITELIB}/${EGG_INFODIR}/PKG-INFO
${PYSITELIB}/${EGG_INFODIR}/SOURCES.txt
@@ -98,6 +98,9 @@ ${PYSITELIB}/scrapy/core/downloader/hand
${PYSITELIB}/scrapy/core/downloader/handlers/http11.py
${PYSITELIB}/scrapy/core/downloader/handlers/http11.pyc
${PYSITELIB}/scrapy/core/downloader/handlers/http11.pyo
+${PYSITELIB}/scrapy/core/downloader/handlers/http2.py
+${PYSITELIB}/scrapy/core/downloader/handlers/http2.pyc
+${PYSITELIB}/scrapy/core/downloader/handlers/http2.pyo
${PYSITELIB}/scrapy/core/downloader/handlers/s3.py
${PYSITELIB}/scrapy/core/downloader/handlers/s3.pyc
${PYSITELIB}/scrapy/core/downloader/handlers/s3.pyo
@@ -113,6 +116,18 @@ ${PYSITELIB}/scrapy/core/downloader/webc
${PYSITELIB}/scrapy/core/engine.py
${PYSITELIB}/scrapy/core/engine.pyc
${PYSITELIB}/scrapy/core/engine.pyo
+${PYSITELIB}/scrapy/core/http2/__init__.py
+${PYSITELIB}/scrapy/core/http2/__init__.pyc
+${PYSITELIB}/scrapy/core/http2/__init__.pyo
+${PYSITELIB}/scrapy/core/http2/agent.py
+${PYSITELIB}/scrapy/core/http2/agent.pyc
+${PYSITELIB}/scrapy/core/http2/agent.pyo
+${PYSITELIB}/scrapy/core/http2/protocol.py
+${PYSITELIB}/scrapy/core/http2/protocol.pyc
+${PYSITELIB}/scrapy/core/http2/protocol.pyo
+${PYSITELIB}/scrapy/core/http2/stream.py
+${PYSITELIB}/scrapy/core/http2/stream.pyc
+${PYSITELIB}/scrapy/core/http2/stream.pyo
${PYSITELIB}/scrapy/core/scheduler.py
${PYSITELIB}/scrapy/core/scheduler.pyc
${PYSITELIB}/scrapy/core/scheduler.pyo
@@ -209,6 +224,9 @@ ${PYSITELIB}/scrapy/extensions/memdebug.
${PYSITELIB}/scrapy/extensions/memusage.py
${PYSITELIB}/scrapy/extensions/memusage.pyc
${PYSITELIB}/scrapy/extensions/memusage.pyo
+${PYSITELIB}/scrapy/extensions/postprocessing.py
+${PYSITELIB}/scrapy/extensions/postprocessing.pyc
+${PYSITELIB}/scrapy/extensions/postprocessing.pyo
${PYSITELIB}/scrapy/extensions/spiderstate.py
${PYSITELIB}/scrapy/extensions/spiderstate.pyc
${PYSITELIB}/scrapy/extensions/spiderstate.pyo
@@ -396,6 +414,9 @@ ${PYSITELIB}/scrapy/templates/spiders/xm
${PYSITELIB}/scrapy/utils/__init__.py
${PYSITELIB}/scrapy/utils/__init__.pyc
${PYSITELIB}/scrapy/utils/__init__.pyo
+${PYSITELIB}/scrapy/utils/asyncgen.py
+${PYSITELIB}/scrapy/utils/asyncgen.pyc
+${PYSITELIB}/scrapy/utils/asyncgen.pyo
${PYSITELIB}/scrapy/utils/benchserver.py
${PYSITELIB}/scrapy/utils/benchserver.pyc
${PYSITELIB}/scrapy/utils/benchserver.pyo
@@ -435,9 +456,6 @@ ${PYSITELIB}/scrapy/utils/ftp.pyo
${PYSITELIB}/scrapy/utils/gz.py
${PYSITELIB}/scrapy/utils/gz.pyc
${PYSITELIB}/scrapy/utils/gz.pyo
-${PYSITELIB}/scrapy/utils/http.py
-${PYSITELIB}/scrapy/utils/http.pyc
-${PYSITELIB}/scrapy/utils/http.pyo
${PYSITELIB}/scrapy/utils/httpobj.py
${PYSITELIB}/scrapy/utils/httpobj.pyc
${PYSITELIB}/scrapy/utils/httpobj.pyo
@@ -450,24 +468,15 @@ ${PYSITELIB}/scrapy/utils/job.pyo
${PYSITELIB}/scrapy/utils/log.py
${PYSITELIB}/scrapy/utils/log.pyc
${PYSITELIB}/scrapy/utils/log.pyo
-${PYSITELIB}/scrapy/utils/markup.py
-${PYSITELIB}/scrapy/utils/markup.pyc
-${PYSITELIB}/scrapy/utils/markup.pyo
${PYSITELIB}/scrapy/utils/misc.py
${PYSITELIB}/scrapy/utils/misc.pyc
${PYSITELIB}/scrapy/utils/misc.pyo
-${PYSITELIB}/scrapy/utils/multipart.py
-${PYSITELIB}/scrapy/utils/multipart.pyc
-${PYSITELIB}/scrapy/utils/multipart.pyo
${PYSITELIB}/scrapy/utils/ossignal.py
${PYSITELIB}/scrapy/utils/ossignal.pyc
${PYSITELIB}/scrapy/utils/ossignal.pyo
${PYSITELIB}/scrapy/utils/project.py
${PYSITELIB}/scrapy/utils/project.pyc
${PYSITELIB}/scrapy/utils/project.pyo
-${PYSITELIB}/scrapy/utils/py36.py
-${PYSITELIB}/scrapy/utils/py36.pyc
-${PYSITELIB}/scrapy/utils/py36.pyo
${PYSITELIB}/scrapy/utils/python.py
${PYSITELIB}/scrapy/utils/python.pyc
${PYSITELIB}/scrapy/utils/python.pyo
Index: pkgsrc/www/py-scrapy/distinfo
diff -u pkgsrc/www/py-scrapy/distinfo:1.12 pkgsrc/www/py-scrapy/distinfo:1.13
--- pkgsrc/www/py-scrapy/distinfo:1.12 Tue Oct 26 11:30:48 2021
+++ pkgsrc/www/py-scrapy/distinfo Thu Apr 27 09:33:44 2023
@@ -1,5 +1,5 @@
-$NetBSD: distinfo,v 1.12 2021/10/26 11:30:48 nia Exp $
+$NetBSD: distinfo,v 1.13 2023/04/27 09:33:44 adam Exp $
-BLAKE2s (Scrapy-2.4.1.tar.gz) = 38d33dd75b56a710b0624558094c326b280669ad81fc29d5df2bf6037e93df8e
-SHA512 (Scrapy-2.4.1.tar.gz) = 65e1f6b92a7ca1b46b3edbe3e668e11cc5140fbf983ac7fce38c31282009a848c02883bda8d56ea3019c84658839ee10e7237c9290cfe9a8d6b6abee07566b2a
-Size (Scrapy-2.4.1.tar.gz) = 1044246 bytes
+BLAKE2s (Scrapy-2.8.0.tar.gz) = e3bd0d640992aab05d1a5c9108b5b741cea4ad9dc902afc5d385eb9ba14c6a61
+SHA512 (Scrapy-2.8.0.tar.gz) = 6e631f84e27aeab5aeae672047869deb783b3c2f6de66c9bd6df768598a638d7e76a3f38945bfdf82f5ca0eb69491c262960f1645fe2f4947f72c0829a0eefaf
+Size (Scrapy-2.8.0.tar.gz) = 1140185 bytes
Home |
Main Index |
Thread Index |
Old Index