pkgsrc-WIP-changes archive
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]
py-distributed: Updated to 2022.6.2 2022.6.1 Released on June 24, 2022 Highlights This release includes the Worker State Machine refactor. The expectation should be that the worker state is its own synchronous subclass. Pulling all the state out into its own class allows us to write targeted unit tests without invoking any concurrent or asynchronous code. See GH#5736 for more information. Enhancements     Make worker state machine methods private (GH#6564) crusaderky     Yank state machine out of Worker class (GH#6566) crusaderky     Track worker_state_machine.TaskState instances (GH#6525) Hendrik Makait     Trivial tweaks to the Worker State Machine (GH#6586) crusaderky     Replace loop.call_later and loop.add_callback with background tasks added to Server. (GH#6603) Thomas Grainger     Support for neater WorkerState tests (GH#6609) crusaderky     Limit TCP writes with Tornado to 2GB (GH#6557) hhuuggoo     Enable no_implicit_optional for scheduler (GH#6622) Thomas Grainger Bug Fixes     Partial revert of compute-task message format (GH#6626) Florian Jetter     Restore log message about received signals in CLI (GH#6618) Florian Jetter     Handle empty memoryviews of bytearrays when (de)serializing (GH#6576) Benjamin Zaitlen     Ensure steal requests from same-IP but distinct workers are rejected (GH#6585) Florian Jetter     Fix tls_(min|max)_ version having no effect on openssl 1.1.0g or lower (GH#6562) Thomas Grainger     Fix idle_timeout and unxfail test (GH#6563) Matthew Rocklin     Fix crashing debug statement in _purge_state (GH#6589) crusaderky     Abort connections on CancelledError (GH#6574) Thomas Grainger     Fix Active Memory Manager ignores nbytes thresholds (GH#6583) crusaderky Deprecations     Deprecate WorkerState accessors (GH#6579) crusaderky Maintenance     Remove ipython hack (GH#6599) crusaderky     Mypy enforce --no-implicit-optional (GH#6606) crusaderky     Update versioneer: change from using SafeConfigParser to ConfigParser (GH#6605) Thomas A Caswell     Warn unreachable for scheduler.py (GH#6611) Florian Jetter     Refactor wait_for_state() (GH#6581) crusaderky     Hardcode wait_for_signals signal list (GH#6619) Thomas Grainger     Always pick an open port when running tests (GH#6591) Florian Jetter     Log popen stdout/err when subprocess times out (GH#6567) Gabe Joseph     Fix test_error_during_startup (GH#6608) Florian Jetter     Make test_idle_timeout_no_workers more robust (GH#6602) Florian Jetter     Mypy enforce --disallow-incomplete-defs (GH#6601) crusaderky     Do not log during signal handler (GH#6590) Florian Jetter     Don’t initialize mp_context on import (GH#6580) Lawrence Mitchell     Test retire workers deadlock (GH#6240) Gabe Joseph     Rework some tests related to gather_dep (GH#6472) crusaderky     Minor cosmetic review of scheduler_story and worker_story (GH#6442) crusaderky     Force __future__.annotations with isort (GH#6621) Thomas Grainger 2022.6.0 Released on June 10, 2022 Enhancements     Make disk access in system monitor configurable (GH#6537) Matthew Rocklin     Log and except errors on preload start (GH#6553) Matthew Rocklin     Fix Scheduler.restart logic (GH#6504) Gabe Joseph     Don’t heartbeat while Worker is closing (GH#6543) Gabe Joseph     No longer retry LocalCluster on errno.EADDRINUSE (GH#6369) Thomas Grainger     Don’t invoke log_event from state machine (GH#6512) crusaderky     Add config option to disable profiling and disable it in many tests per default (GH#6490) Hendrik Makait     Encapsulate Worker.batched_stream.send() (GH#6475) crusaderky Bug Fixes     refresh-who-has can break the worker state machine (GH#6529) crusaderky     Restart worker if it’s unrecognized by scheduler (GH#6505) Gabe Joseph     Fix import error when distributed.rmm.pool-size is set (GH#6482) KoyamaSohei Deprecations     Restore signature compatibility for dask-gateway (GH#6561) Tom Augspurger     Deprecate the io_loop and loop kwarg to Server, Worker, and Nanny (GH#6473) Thomas Grainger     Deprecate the loop kwarg to Scheduler (GH#6443) Thomas Grainger Documentation     Fix typo in .nthreads() docstring example (GH#6545) Pavithra Eswaramoorthy     Update docs theme for rebranding (GH#6495) Sarah Charlotte Johnson Maintenance     Refactor gather_dep (GH#6388) crusaderky     Fix flaky test_gather_dep_one_worker_always_busy (GH#6554) crusaderky     Remove missing-data message (GH#6546) crusaderky     Port test_local.LocalTest to pytest tests to allow use of loop fixture (GH#6523) Thomas Grainger     Fix test_quiet_client_close (GH#6541) Gabe Joseph     Use default_initializer in WorkerProcess (GH#6534) jakirkham     Avoid deadlocks in tests that use popen (GH#6483) Gabe Joseph     Revert “Fix CLI Scheduler Tests (GH#6502)” (GH#6547) Gabe Joseph     Update test report URL in summary message (GH#6532) Gabe Joseph     Update test report url (GH#6531) Ian Rose     Assert AsyncProcess.set_exit_callback is not called with a coroutine function (GH#6526) Thomas Grainger     Typing and docstring for Worker.close (GH#6518) Hendrik Makait     Fix CLI Scheduler Tests (GH#6502) Benjamin Zaitlen     Collect assertions in test_as_current_is_thread_local (GH#6520) Thomas Grainger     Link test report from test results comment (GH#6524) Hendrik Makait     Ignore the return value of signal.signal (GH#6519) Thomas Grainger     Refactor all event handlers (GH#6410) crusaderky     Fix dashboard favicon background (GH#6514) Jacob Tomlinson     Update dashboard logo (GH#6513) Jacob Tomlinson     Fix test_stress_scatter_death (GH#6404) Florian Jetter     Remove CrossFilter widget (GH#6484) crusaderky     data_needed exclusively contains tasks in fetch state (GH#6481) crusaderky     Assert possible previous states (GH#6488) Florian Jetter     @fail_hard can kill the whole test suite; hide errors (GH#6474) crusaderky     Assert that a fetch->cancelled->resumed->fetch cycle is impossible (GH#6460) crusaderky     Refactor busy workers reinsertion (GH#6379) crusaderky     Refactor find_missing and refresh_who_has (GH#6348) crusaderky     Rename test_collections.py to test_dask_collections.py (GH#6486) crusaderky     update_who_has can remove workers (GH#6342) crusaderky     Restructure test_watch_requires_lock_to_run to avoid flakes (GH#6469) Hendrik Makait     Fix intermittent test_profile_plot failure (GH#6456) Matthew Rocklin     Use asyncio.run to run gen_cluster, gen_test and cluster (GH#6231) Thomas Grainger     Improve tests that watch for subprocess logs (GH#6461) Gabe Joseph 2022.05.2 Released on May 26, 2022 Enhancements     Add a lock to distributed.profile for better concurrency control (GH#6421) Hendrik Makait     Send SIGKILL after SIGTERM when passing 95% memory (GH#6419) crusaderky Bug Fixes     Log rather than raise exceptions in preload.teardown() (GH#6458) Matthew Rocklin     Handle failing plugin.close() calls during scheduler shutdown (GH#6450) Matthew Rocklin     Fix slicing bug in ensure_memoryview (GH#6449) jakirkham     Generalize UCX errors on connect() and correct pytest fixtures (GH#6434) Peter Andreas Entschev     Run cluster widget periodic callbacks on the correct event loop (GH#6444) Thomas Grainger Maintenance     Disable pytest-asyncio if installed (GH#6436) Jacob Tomlinson     Close client in sync test_actor tests (GH#6459) Thomas Grainger     Ignore ServerSession.with_document_locked unawaited (GH#6447) Thomas Grainger     Remove coverage pin from Python 3.10 environment (GH#6439) Thomas Grainger     Annotate remove_worker (GH#6441) crusaderky     Update gpuCI RAPIDS_VER to 22.08 (GH#6428) 2022.05.1 Released on May 24, 2022 New Features     Add HTTP API to scheduler (GH#6270) Matthew Murray     Shuffle Service with Scheduler Logic (GH#6007) Matthew Rocklin Enhancements     Follow-up on removing report and safe from Worker.close (GH#6423) Gabe Joseph     Server close faster (GH#6415) Florian Jetter     Disable HTTP API by default (GH#6420) Jacob Tomlinson     Remove report and safe from Worker.close (GH#6363) Florian Jetter     Allow deserialized plugins in register_scheduler_plugin (GH#6401) Matthew Rocklin     WorkerState are different for different addresses (GH#6398) Florian Jetter     Do not filter tasks before gathering data (GH#6371) crusaderky     Remove worker reconnect (GH#6361) Gabe Joseph     Add SchedulerPlugin.log_event handler (GH#6381) Matthew Rocklin     Ensure occupancy tracking works as expected for long running tasks (GH#6351) Florian Jetter     stimulus_id for all Instructions (GH#6347) crusaderky     Refactor missing-data command (GH#6332) crusaderky     Add idempotent to register_scheduler_plugin client (GH#6328) Alex Ford     Add option to specify a scheduler address for workers to use (GH#5944) Enric Tejedor Bug Fixes     Remove stray breakpoint (GH#6417) Thomas Grainger     Fix API JSON MIME type (GH#6397) Jacob Tomlinson     Remove wrong assert in handle compute (GH#6370) Florian Jetter     Ensure multiple clients can cancel their key without interference (GH#6016) Florian Jetter     Fix Nanny shutdown assertion (GH#6357) Gabe Joseph     Fix fail_hard for sync functions (GH#6269) Gabe Joseph     Prevent infinite transition loops; more aggressive validate_state() (GH#6318) crusaderky     Ensure cleanup of many GBs of spilled data on terminate (GH#6280) crusaderky     Fix WORKER_ANY_RUNNING regression (GH#6297) Florian Jetter     Race conditions from fetch to compute while AMM requests replica (GH#6248) Florian Jetter     Ensure resumed tasks are not accidentally forgotten (GH#6217) Florian Jetter     Do not allow closing workers to be awaited again (GH#5910) Florian Jetter Deprecations     Move wait_for_signals to private module and deprecate distributed.cli.utils (GH#6367) Hendrik Makait Documentation     Fix typos and whitespace in worker.py (GH#6326) Hendrik Makait     Fix link to memory trimming documentation (GH#6317) Marco Wolsza Maintenance     Make gen_test show up in VSCode test discovery (GH#6424) Gabe Joseph     WSMR / deserialize_task (GH#6411) crusaderky     Restore signal handlers after wait for signals is done (GH#6400) Thomas Grainger     fail_hard should reraise (GH#6399) crusaderky     Revisit tests mocking gather_dep (GH#6385) crusaderky     Fix flaky test_in_flight_lost_after_resumed (GH#6372) Florian Jetter     Restore install_signal_handlers due to downstream dependencies (GH#6366) Hendrik Makait     Improve catch_unhandled_exceptions (GH#6358) Gabe Joseph     Remove all invocations of IOLoop.run_sync from CLI (GH#6205) Hendrik Makait     Remove transition-counter-max from config (GH#6349) crusaderky     Use list comprehension in pickle_loads (GH#6343) jakirkham     Improve ensure_memoryview test coverage & make minor fixes (GH#6333) jakirkham     Remove leaking reference to workers from gen_cluster (GH#6337) Hendrik Makait     Partial annotations for stealing.py (GH#6338) crusaderky     Validate and debug state machine on handle_compute_task (GH#6327) crusaderky     Bump pyupgrade and clean up # type: ignore (GH#6293) crusaderky     gen_cluster to write to /tmp (GH#6335) crusaderky     Transition table as a ClassVar (GH#6331) crusaderky     Simplify ensure_memoryview test with array (GH#6322) jakirkham     Refactor ensure_communicating (GH#6165) crusaderky     Review scheduler annotations, part 2 (GH#6253) crusaderky     Use w for writeable branch in pickle_loads (GH#6314) jakirkham     Simplify frame handling in ws (GH#6294) jakirkham     Use ensure_bytes from dask.utils (GH#6295) jakirkham     Use ensure_memoryview in array deserialization (GH#6300) jakirkham     Escape < > when generating Junit report (GH#6306) crusaderky     Use codecs.decode to deserialize errors (GH#6274) jakirkham     Minimize copying in maybe_compress & byte_sample (GH#6273) jakirkham     Skip test_release_evloop_while_spilling on OSX (GH#6291) Florian Jetter     Simplify logic in get_default_compression (GH#6260) jakirkham     Cleanup old compression workarounds (GH#6259) jakirkham     Re-enable NVML monitoring for WSL (GH#6119) Charles Blackmon-Luca
- To: pkgsrc-wip-changes%NetBSD.org@localhost
 
- Subject: py-distributed: Updated to 2022.6.2 2022.6.1 Released on June 24, 2022 Highlights This release includes the Worker State Machine refactor. The expectation should be that the worker state is its own synchronous subclass. Pulling all the state out into its own class allows us to write targeted unit tests without invoking any concurrent or asynchronous code. See GH#5736 for more information. Enhancements     Make worker state machine methods private (GH#6564) crusaderky     Yank state machine out of Worker class (GH#6566) crusaderky     Track worker_state_machine.TaskState instances (GH#6525) Hendrik Makait     Trivial tweaks to the Worker State Machine (GH#6586) crusaderky     Replace loop.call_later and loop.add_callback with background tasks added to Server. (GH#6603) Thomas Grainger     Support for neater WorkerState tests (GH#6609) crusaderky     Limit TCP writes with Tornado to 2GB (GH#6557) hhuuggoo     Enable no_implicit_optional for scheduler (GH#6622) Thomas Grainger Bug Fixes     Partial revert of compute-task message format (GH#6626) Florian Jetter     Restore log message about received signals in CLI (GH#6618) Florian Jetter     Handle empty memoryviews of bytearrays when (de)serializing (GH#6576) Benjamin Zaitlen     Ensure steal requests from same-IP but distinct workers are rejected (GH#6585) Florian Jetter     Fix tls_(min|max)_ version having no effect on openssl 1.1.0g or lower (GH#6562) Thomas Grainger     Fix idle_timeout and unxfail test (GH#6563) Matthew Rocklin     Fix crashing debug statement in _purge_state (GH#6589) crusaderky     Abort connections on CancelledError (GH#6574) Thomas Grainger     Fix Active Memory Manager ignores nbytes thresholds (GH#6583) crusaderky Deprecations     Deprecate WorkerState accessors (GH#6579) crusaderky Maintenance     Remove ipython hack (GH#6599) crusaderky     Mypy enforce --no-implicit-optional (GH#6606) crusaderky     Update versioneer: change from using SafeConfigParser to ConfigParser (GH#6605) Thomas A Caswell     Warn unreachable for scheduler.py (GH#6611) Florian Jetter     Refactor wait_for_state() (GH#6581) crusaderky     Hardcode wait_for_signals signal list (GH#6619) Thomas Grainger     Always pick an open port when running tests (GH#6591) Florian Jetter     Log popen stdout/err when subprocess times out (GH#6567) Gabe Joseph     Fix test_error_during_startup (GH#6608) Florian Jetter     Make test_idle_timeout_no_workers more robust (GH#6602) Florian Jetter     Mypy enforce --disallow-incomplete-defs (GH#6601) crusaderky     Do not log during signal handler (GH#6590) Florian Jetter     Don’t initialize mp_context on import (GH#6580) Lawrence Mitchell     Test retire workers deadlock (GH#6240) Gabe Joseph     Rework some tests related to gather_dep (GH#6472) crusaderky     Minor cosmetic review of scheduler_story and worker_story (GH#6442) crusaderky     Force __future__.annotations with isort (GH#6621) Thomas Grainger 2022.6.0 Released on June 10, 2022 Enhancements     Make disk access in system monitor configurable (GH#6537) Matthew Rocklin     Log and except errors on preload start (GH#6553) Matthew Rocklin     Fix Scheduler.restart logic (GH#6504) Gabe Joseph     Don’t heartbeat while Worker is closing (GH#6543) Gabe Joseph     No longer retry LocalCluster on errno.EADDRINUSE (GH#6369) Thomas Grainger     Don’t invoke log_event from state machine (GH#6512) crusaderky     Add config option to disable profiling and disable it in many tests per default (GH#6490) Hendrik Makait     Encapsulate Worker.batched_stream.send() (GH#6475) crusaderky Bug Fixes     refresh-who-has can break the worker state machine (GH#6529) crusaderky     Restart worker if it’s unrecognized by scheduler (GH#6505) Gabe Joseph     Fix import error when distributed.rmm.pool-size is set (GH#6482) KoyamaSohei Deprecations     Restore signature compatibility for dask-gateway (GH#6561) Tom Augspurger     Deprecate the io_loop and loop kwarg to Server, Worker, and Nanny (GH#6473) Thomas Grainger     Deprecate the loop kwarg to Scheduler (GH#6443) Thomas Grainger Documentation     Fix typo in .nthreads() docstring example (GH#6545) Pavithra Eswaramoorthy     Update docs theme for rebranding (GH#6495) Sarah Charlotte Johnson Maintenance     Refactor gather_dep (GH#6388) crusaderky     Fix flaky test_gather_dep_one_worker_always_busy (GH#6554) crusaderky     Remove missing-data message (GH#6546) crusaderky     Port test_local.LocalTest to pytest tests to allow use of loop fixture (GH#6523) Thomas Grainger     Fix test_quiet_client_close (GH#6541) Gabe Joseph     Use default_initializer in WorkerProcess (GH#6534) jakirkham     Avoid deadlocks in tests that use popen (GH#6483) Gabe Joseph     Revert “Fix CLI Scheduler Tests (GH#6502)” (GH#6547) Gabe Joseph     Update test report URL in summary message (GH#6532) Gabe Joseph     Update test report url (GH#6531) Ian Rose     Assert AsyncProcess.set_exit_callback is not called with a coroutine function (GH#6526) Thomas Grainger     Typing and docstring for Worker.close (GH#6518) Hendrik Makait     Fix CLI Scheduler Tests (GH#6502) Benjamin Zaitlen     Collect assertions in test_as_current_is_thread_local (GH#6520) Thomas Grainger     Link test report from test results comment (GH#6524) Hendrik Makait     Ignore the return value of signal.signal (GH#6519) Thomas Grainger     Refactor all event handlers (GH#6410) crusaderky     Fix dashboard favicon background (GH#6514) Jacob Tomlinson     Update dashboard logo (GH#6513) Jacob Tomlinson     Fix test_stress_scatter_death (GH#6404) Florian Jetter     Remove CrossFilter widget (GH#6484) crusaderky     data_needed exclusively contains tasks in fetch state (GH#6481) crusaderky     Assert possible previous states (GH#6488) Florian Jetter     @fail_hard can kill the whole test suite; hide errors (GH#6474) crusaderky     Assert that a fetch->cancelled->resumed->fetch cycle is impossible (GH#6460) crusaderky     Refactor busy workers reinsertion (GH#6379) crusaderky     Refactor find_missing and refresh_who_has (GH#6348) crusaderky     Rename test_collections.py to test_dask_collections.py (GH#6486) crusaderky     update_who_has can remove workers (GH#6342) crusaderky     Restructure test_watch_requires_lock_to_run to avoid flakes (GH#6469) Hendrik Makait     Fix intermittent test_profile_plot failure (GH#6456) Matthew Rocklin     Use asyncio.run to run gen_cluster, gen_test and cluster (GH#6231) Thomas Grainger     Improve tests that watch for subprocess logs (GH#6461) Gabe Joseph 2022.05.2 Released on May 26, 2022 Enhancements     Add a lock to distributed.profile for better concurrency control (GH#6421) Hendrik Makait     Send SIGKILL after SIGTERM when passing 95% memory (GH#6419) crusaderky Bug Fixes     Log rather than raise exceptions in preload.teardown() (GH#6458) Matthew Rocklin     Handle failing plugin.close() calls during scheduler shutdown (GH#6450) Matthew Rocklin     Fix slicing bug in ensure_memoryview (GH#6449) jakirkham     Generalize UCX errors on connect() and correct pytest fixtures (GH#6434) Peter Andreas Entschev     Run cluster widget periodic callbacks on the correct event loop (GH#6444) Thomas Grainger Maintenance     Disable pytest-asyncio if installed (GH#6436) Jacob Tomlinson     Close client in sync test_actor tests (GH#6459) Thomas Grainger     Ignore ServerSession.with_document_locked unawaited (GH#6447) Thomas Grainger     Remove coverage pin from Python 3.10 environment (GH#6439) Thomas Grainger     Annotate remove_worker (GH#6441) crusaderky     Update gpuCI RAPIDS_VER to 22.08 (GH#6428) 2022.05.1 Released on May 24, 2022 New Features     Add HTTP API to scheduler (GH#6270) Matthew Murray     Shuffle Service with Scheduler Logic (GH#6007) Matthew Rocklin Enhancements     Follow-up on removing report and safe from Worker.close (GH#6423) Gabe Joseph     Server close faster (GH#6415) Florian Jetter     Disable HTTP API by default (GH#6420) Jacob Tomlinson     Remove report and safe from Worker.close (GH#6363) Florian Jetter     Allow deserialized plugins in register_scheduler_plugin (GH#6401) Matthew Rocklin     WorkerState are different for different addresses (GH#6398) Florian Jetter     Do not filter tasks before gathering data (GH#6371) crusaderky     Remove worker reconnect (GH#6361) Gabe Joseph     Add SchedulerPlugin.log_event handler (GH#6381) Matthew Rocklin     Ensure occupancy tracking works as expected for long running tasks (GH#6351) Florian Jetter     stimulus_id for all Instructions (GH#6347) crusaderky     Refactor missing-data command (GH#6332) crusaderky     Add idempotent to register_scheduler_plugin client (GH#6328) Alex Ford     Add option to specify a scheduler address for workers to use (GH#5944) Enric Tejedor Bug Fixes     Remove stray breakpoint (GH#6417) Thomas Grainger     Fix API JSON MIME type (GH#6397) Jacob Tomlinson     Remove wrong assert in handle compute (GH#6370) Florian Jetter     Ensure multiple clients can cancel their key without interference (GH#6016) Florian Jetter     Fix Nanny shutdown assertion (GH#6357) Gabe Joseph     Fix fail_hard for sync functions (GH#6269) Gabe Joseph     Prevent infinite transition loops; more aggressive validate_state() (GH#6318) crusaderky     Ensure cleanup of many GBs of spilled data on terminate (GH#6280) crusaderky     Fix WORKER_ANY_RUNNING regression (GH#6297) Florian Jetter     Race conditions from fetch to compute while AMM requests replica (GH#6248) Florian Jetter     Ensure resumed tasks are not accidentally forgotten (GH#6217) Florian Jetter     Do not allow closing workers to be awaited again (GH#5910) Florian Jetter Deprecations     Move wait_for_signals to private module and deprecate distributed.cli.utils (GH#6367) Hendrik Makait Documentation     Fix typos and whitespace in worker.py (GH#6326) Hendrik Makait     Fix link to memory trimming documentation (GH#6317) Marco Wolsza Maintenance     Make gen_test show up in VSCode test discovery (GH#6424) Gabe Joseph     WSMR / deserialize_task (GH#6411) crusaderky     Restore signal handlers after wait for signals is done (GH#6400) Thomas Grainger     fail_hard should reraise (GH#6399) crusaderky     Revisit tests mocking gather_dep (GH#6385) crusaderky     Fix flaky test_in_flight_lost_after_resumed (GH#6372) Florian Jetter     Restore install_signal_handlers due to downstream dependencies (GH#6366) Hendrik Makait     Improve catch_unhandled_exceptions (GH#6358) Gabe Joseph     Remove all invocations of IOLoop.run_sync from CLI (GH#6205) Hendrik Makait     Remove transition-counter-max from config (GH#6349) crusaderky     Use list comprehension in pickle_loads (GH#6343) jakirkham     Improve ensure_memoryview test coverage & make minor fixes (GH#6333) jakirkham     Remove leaking reference to workers from gen_cluster (GH#6337) Hendrik Makait     Partial annotations for stealing.py (GH#6338) crusaderky     Validate and debug state machine on handle_compute_task (GH#6327) crusaderky     Bump pyupgrade and clean up # type: ignore (GH#6293) crusaderky     gen_cluster to write to /tmp (GH#6335) crusaderky     Transition table as a ClassVar (GH#6331) crusaderky     Simplify ensure_memoryview test with array (GH#6322) jakirkham     Refactor ensure_communicating (GH#6165) crusaderky     Review scheduler annotations, part 2 (GH#6253) crusaderky     Use w for writeable branch in pickle_loads (GH#6314) jakirkham     Simplify frame handling in ws (GH#6294) jakirkham     Use ensure_bytes from dask.utils (GH#6295) jakirkham     Use ensure_memoryview in array deserialization (GH#6300) jakirkham     Escape < > when generating Junit report (GH#6306) crusaderky     Use codecs.decode to deserialize errors (GH#6274) jakirkham     Minimize copying in maybe_compress & byte_sample (GH#6273) jakirkham     Skip test_release_evloop_while_spilling on OSX (GH#6291) Florian Jetter     Simplify logic in get_default_compression (GH#6260) jakirkham     Cleanup old compression workarounds (GH#6259) jakirkham     Re-enable NVML monitoring for WSL (GH#6119) Charles Blackmon-Luca
 
- From: Matthew Danielson <matthewd%fastmail.us@localhost>
 
- Date: Mon, 04 Jul 2022 02:21:43 +0000
 
Module Name:	pkgsrc-wip
Committed By:	Matthew Danielson <matthewd%fastmail.us@localhost>
Pushed By:	matthewd
Date:		Sun Jul 3 19:21:43 2022 -0700
Changeset:	0b55b7f25b132db3f0980f475ab98cf28a497b13
Modified Files:
	py-distributed/Makefile
	py-distributed/PLIST
	py-distributed/distinfo
Log Message:
py-distributed: Updated to 2022.6.2
2022.6.1
Released on June 24, 2022
Highlights
This release includes the Worker State Machine refactor. The expectation should be that the worker state is its own synchronous subclass. Pulling all the state out into its own class allows us to write targeted unit tests without invoking any concurrent or asynchronous code.
See GH#5736 for more information.
Enhancements
    Make worker state machine methods private (GH#6564) crusaderky
    Yank state machine out of Worker class (GH#6566) crusaderky
    Track worker_state_machine.TaskState instances (GH#6525) Hendrik Makait
    Trivial tweaks to the Worker State Machine (GH#6586) crusaderky
    Replace loop.call_later and loop.add_callback with background tasks added to Server. (GH#6603) Thomas Grainger
    Support for neater WorkerState tests (GH#6609) crusaderky
    Limit TCP writes with Tornado to 2GB (GH#6557) hhuuggoo
    Enable no_implicit_optional for scheduler (GH#6622) Thomas Grainger
Bug Fixes
    Partial revert of compute-task message format (GH#6626) Florian Jetter
    Restore log message about received signals in CLI (GH#6618) Florian Jetter
    Handle empty memoryviews of bytearrays when (de)serializing (GH#6576) Benjamin Zaitlen
    Ensure steal requests from same-IP but distinct workers are rejected (GH#6585) Florian Jetter
    Fix tls_(min|max)_ version having no effect on openssl 1.1.0g or lower (GH#6562) Thomas Grainger
    Fix idle_timeout and unxfail test (GH#6563) Matthew Rocklin
    Fix crashing debug statement in _purge_state (GH#6589) crusaderky
    Abort connections on CancelledError (GH#6574) Thomas Grainger
    Fix Active Memory Manager ignores nbytes thresholds (GH#6583) crusaderky
Deprecations
    Deprecate WorkerState accessors (GH#6579) crusaderky
Maintenance
    Remove ipython hack (GH#6599) crusaderky
    Mypy enforce --no-implicit-optional (GH#6606) crusaderky
    Update versioneer: change from using SafeConfigParser to ConfigParser (GH#6605) Thomas A Caswell
    Warn unreachable for scheduler.py (GH#6611) Florian Jetter
    Refactor wait_for_state() (GH#6581) crusaderky
    Hardcode wait_for_signals signal list (GH#6619) Thomas Grainger
    Always pick an open port when running tests (GH#6591) Florian Jetter
    Log popen stdout/err when subprocess times out (GH#6567) Gabe Joseph
    Fix test_error_during_startup (GH#6608) Florian Jetter
    Make test_idle_timeout_no_workers more robust (GH#6602) Florian Jetter
    Mypy enforce --disallow-incomplete-defs (GH#6601) crusaderky
    Do not log during signal handler (GH#6590) Florian Jetter
    Don’t initialize mp_context on import (GH#6580) Lawrence Mitchell
    Test retire workers deadlock (GH#6240) Gabe Joseph
    Rework some tests related to gather_dep (GH#6472) crusaderky
    Minor cosmetic review of scheduler_story and worker_story (GH#6442) crusaderky
    Force __future__.annotations with isort (GH#6621) Thomas Grainger
2022.6.0
Released on June 10, 2022
Enhancements
    Make disk access in system monitor configurable (GH#6537) Matthew Rocklin
    Log and except errors on preload start (GH#6553) Matthew Rocklin
    Fix Scheduler.restart logic (GH#6504) Gabe Joseph
    Don’t heartbeat while Worker is closing (GH#6543) Gabe Joseph
    No longer retry LocalCluster on errno.EADDRINUSE (GH#6369) Thomas Grainger
    Don’t invoke log_event from state machine (GH#6512) crusaderky
    Add config option to disable profiling and disable it in many tests per default (GH#6490) Hendrik Makait
    Encapsulate Worker.batched_stream.send() (GH#6475) crusaderky
Bug Fixes
    refresh-who-has can break the worker state machine (GH#6529) crusaderky
    Restart worker if it’s unrecognized by scheduler (GH#6505) Gabe Joseph
    Fix import error when distributed.rmm.pool-size is set (GH#6482) KoyamaSohei
Deprecations
    Restore signature compatibility for dask-gateway (GH#6561) Tom Augspurger
    Deprecate the io_loop and loop kwarg to Server, Worker, and Nanny (GH#6473) Thomas Grainger
    Deprecate the loop kwarg to Scheduler (GH#6443) Thomas Grainger
Documentation
    Fix typo in .nthreads() docstring example (GH#6545) Pavithra Eswaramoorthy
    Update docs theme for rebranding (GH#6495) Sarah Charlotte Johnson
Maintenance
    Refactor gather_dep (GH#6388) crusaderky
    Fix flaky test_gather_dep_one_worker_always_busy (GH#6554) crusaderky
    Remove missing-data message (GH#6546) crusaderky
    Port test_local.LocalTest to pytest tests to allow use of loop fixture (GH#6523) Thomas Grainger
    Fix test_quiet_client_close (GH#6541) Gabe Joseph
    Use default_initializer in WorkerProcess (GH#6534) jakirkham
    Avoid deadlocks in tests that use popen (GH#6483) Gabe Joseph
    Revert “Fix CLI Scheduler Tests (GH#6502)” (GH#6547) Gabe Joseph
    Update test report URL in summary message (GH#6532) Gabe Joseph
    Update test report url (GH#6531) Ian Rose
    Assert AsyncProcess.set_exit_callback is not called with a coroutine function (GH#6526) Thomas Grainger
    Typing and docstring for Worker.close (GH#6518) Hendrik Makait
    Fix CLI Scheduler Tests (GH#6502) Benjamin Zaitlen
    Collect assertions in test_as_current_is_thread_local (GH#6520) Thomas Grainger
    Link test report from test results comment (GH#6524) Hendrik Makait
    Ignore the return value of signal.signal (GH#6519) Thomas Grainger
    Refactor all event handlers (GH#6410) crusaderky
    Fix dashboard favicon background (GH#6514) Jacob Tomlinson
    Update dashboard logo (GH#6513) Jacob Tomlinson
    Fix test_stress_scatter_death (GH#6404) Florian Jetter
    Remove CrossFilter widget (GH#6484) crusaderky
    data_needed exclusively contains tasks in fetch state (GH#6481) crusaderky
    Assert possible previous states (GH#6488) Florian Jetter
    @fail_hard can kill the whole test suite; hide errors (GH#6474) crusaderky
    Assert that a fetch->cancelled->resumed->fetch cycle is impossible (GH#6460) crusaderky
    Refactor busy workers reinsertion (GH#6379) crusaderky
    Refactor find_missing and refresh_who_has (GH#6348) crusaderky
    Rename test_collections.py to test_dask_collections.py (GH#6486) crusaderky
    update_who_has can remove workers (GH#6342) crusaderky
    Restructure test_watch_requires_lock_to_run to avoid flakes (GH#6469) Hendrik Makait
    Fix intermittent test_profile_plot failure (GH#6456) Matthew Rocklin
    Use asyncio.run to run gen_cluster, gen_test and cluster (GH#6231) Thomas Grainger
    Improve tests that watch for subprocess logs (GH#6461) Gabe Joseph
2022.05.2
Released on May 26, 2022
Enhancements
    Add a lock to distributed.profile for better concurrency control (GH#6421) Hendrik Makait
    Send SIGKILL after SIGTERM when passing 95% memory (GH#6419) crusaderky
Bug Fixes
    Log rather than raise exceptions in preload.teardown() (GH#6458) Matthew Rocklin
    Handle failing plugin.close() calls during scheduler shutdown (GH#6450) Matthew Rocklin
    Fix slicing bug in ensure_memoryview (GH#6449) jakirkham
    Generalize UCX errors on connect() and correct pytest fixtures (GH#6434) Peter Andreas Entschev
    Run cluster widget periodic callbacks on the correct event loop (GH#6444) Thomas Grainger
Maintenance
    Disable pytest-asyncio if installed (GH#6436) Jacob Tomlinson
    Close client in sync test_actor tests (GH#6459) Thomas Grainger
    Ignore ServerSession.with_document_locked unawaited (GH#6447) Thomas Grainger
    Remove coverage pin from Python 3.10 environment (GH#6439) Thomas Grainger
    Annotate remove_worker (GH#6441) crusaderky
    Update gpuCI RAPIDS_VER to 22.08 (GH#6428)
2022.05.1
Released on May 24, 2022
New Features
    Add HTTP API to scheduler (GH#6270) Matthew Murray
    Shuffle Service with Scheduler Logic (GH#6007) Matthew Rocklin
Enhancements
    Follow-up on removing report and safe from Worker.close (GH#6423) Gabe Joseph
    Server close faster (GH#6415) Florian Jetter
    Disable HTTP API by default (GH#6420) Jacob Tomlinson
    Remove report and safe from Worker.close (GH#6363) Florian Jetter
    Allow deserialized plugins in register_scheduler_plugin (GH#6401) Matthew Rocklin
    WorkerState are different for different addresses (GH#6398) Florian Jetter
    Do not filter tasks before gathering data (GH#6371) crusaderky
    Remove worker reconnect (GH#6361) Gabe Joseph
    Add SchedulerPlugin.log_event handler (GH#6381) Matthew Rocklin
    Ensure occupancy tracking works as expected for long running tasks (GH#6351) Florian Jetter
    stimulus_id for all Instructions (GH#6347) crusaderky
    Refactor missing-data command (GH#6332) crusaderky
    Add idempotent to register_scheduler_plugin client (GH#6328) Alex Ford
    Add option to specify a scheduler address for workers to use (GH#5944) Enric Tejedor
Bug Fixes
    Remove stray breakpoint (GH#6417) Thomas Grainger
    Fix API JSON MIME type (GH#6397) Jacob Tomlinson
    Remove wrong assert in handle compute (GH#6370) Florian Jetter
    Ensure multiple clients can cancel their key without interference (GH#6016) Florian Jetter
    Fix Nanny shutdown assertion (GH#6357) Gabe Joseph
    Fix fail_hard for sync functions (GH#6269) Gabe Joseph
    Prevent infinite transition loops; more aggressive validate_state() (GH#6318) crusaderky
    Ensure cleanup of many GBs of spilled data on terminate (GH#6280) crusaderky
    Fix WORKER_ANY_RUNNING regression (GH#6297) Florian Jetter
    Race conditions from fetch to compute while AMM requests replica (GH#6248) Florian Jetter
    Ensure resumed tasks are not accidentally forgotten (GH#6217) Florian Jetter
    Do not allow closing workers to be awaited again (GH#5910) Florian Jetter
Deprecations
    Move wait_for_signals to private module and deprecate distributed.cli.utils (GH#6367) Hendrik Makait
Documentation
    Fix typos and whitespace in worker.py (GH#6326) Hendrik Makait
    Fix link to memory trimming documentation (GH#6317) Marco Wolsza
Maintenance
    Make gen_test show up in VSCode test discovery (GH#6424) Gabe Joseph
    WSMR / deserialize_task (GH#6411) crusaderky
    Restore signal handlers after wait for signals is done (GH#6400) Thomas Grainger
    fail_hard should reraise (GH#6399) crusaderky
    Revisit tests mocking gather_dep (GH#6385) crusaderky
    Fix flaky test_in_flight_lost_after_resumed (GH#6372) Florian Jetter
    Restore install_signal_handlers due to downstream dependencies (GH#6366) Hendrik Makait
    Improve catch_unhandled_exceptions (GH#6358) Gabe Joseph
    Remove all invocations of IOLoop.run_sync from CLI (GH#6205) Hendrik Makait
    Remove transition-counter-max from config (GH#6349) crusaderky
    Use list comprehension in pickle_loads (GH#6343) jakirkham
    Improve ensure_memoryview test coverage & make minor fixes (GH#6333) jakirkham
    Remove leaking reference to workers from gen_cluster (GH#6337) Hendrik Makait
    Partial annotations for stealing.py (GH#6338) crusaderky
    Validate and debug state machine on handle_compute_task (GH#6327) crusaderky
    Bump pyupgrade and clean up # type: ignore (GH#6293) crusaderky
    gen_cluster to write to /tmp (GH#6335) crusaderky
    Transition table as a ClassVar (GH#6331) crusaderky
    Simplify ensure_memoryview test with array (GH#6322) jakirkham
    Refactor ensure_communicating (GH#6165) crusaderky
    Review scheduler annotations, part 2 (GH#6253) crusaderky
    Use w for writeable branch in pickle_loads (GH#6314) jakirkham
    Simplify frame handling in ws (GH#6294) jakirkham
    Use ensure_bytes from dask.utils (GH#6295) jakirkham
    Use ensure_memoryview in array deserialization (GH#6300) jakirkham
    Escape < > when generating Junit report (GH#6306) crusaderky
    Use codecs.decode to deserialize errors (GH#6274) jakirkham
    Minimize copying in maybe_compress & byte_sample (GH#6273) jakirkham
    Skip test_release_evloop_while_spilling on OSX (GH#6291) Florian Jetter
    Simplify logic in get_default_compression (GH#6260) jakirkham
    Cleanup old compression workarounds (GH#6259) jakirkham
    Re-enable NVML monitoring for WSL (GH#6119) Charles Blackmon-Luca
To see a diff of this commit:
https://wip.pkgsrc.org/cgi-bin/gitweb.cgi?p=pkgsrc-wip.git;a=commitdiff;h=0b55b7f25b132db3f0980f475ab98cf28a497b13
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
diffstat:
 py-distributed/Makefile |  2 +-
 py-distributed/PLIST    | 21 ++++++++++++++++++---
 py-distributed/distinfo |  6 +++---
 3 files changed, 22 insertions(+), 7 deletions(-)
diffs:
diff --git a/py-distributed/Makefile b/py-distributed/Makefile
index 2b10469d56..3453ee1459 100644
--- a/py-distributed/Makefile
+++ b/py-distributed/Makefile
@@ -1,6 +1,6 @@
 # $NetBSD$
 
-DISTNAME=	distributed-2022.5.0
+DISTNAME=	distributed-2022.6.1
 PKGNAME=	${PYPKGPREFIX}-${DISTNAME}
 CATEGORIES=	devel net
 MASTER_SITES=	${MASTER_SITE_PYPI:=d/distributed/}
diff --git a/py-distributed/PLIST b/py-distributed/PLIST
index b491f76eab..88123b8007 100644
--- a/py-distributed/PLIST
+++ b/py-distributed/PLIST
@@ -15,6 +15,9 @@ ${PYSITELIB}/distributed/__init__.pyo
 ${PYSITELIB}/distributed/_concurrent_futures_thread.py
 ${PYSITELIB}/distributed/_concurrent_futures_thread.pyc
 ${PYSITELIB}/distributed/_concurrent_futures_thread.pyo
+${PYSITELIB}/distributed/_signals.py
+${PYSITELIB}/distributed/_signals.pyc
+${PYSITELIB}/distributed/_signals.pyo
 ${PYSITELIB}/distributed/_stories.py
 ${PYSITELIB}/distributed/_stories.pyc
 ${PYSITELIB}/distributed/_stories.pyo
@@ -63,6 +66,9 @@ ${PYSITELIB}/distributed/client.pyo
 ${PYSITELIB}/distributed/cluster_dump.py
 ${PYSITELIB}/distributed/cluster_dump.pyc
 ${PYSITELIB}/distributed/cluster_dump.pyo
+${PYSITELIB}/distributed/collections.py
+${PYSITELIB}/distributed/collections.pyc
+${PYSITELIB}/distributed/collections.pyo
 ${PYSITELIB}/distributed/comm/__init__.py
 ${PYSITELIB}/distributed/comm/__init__.pyc
 ${PYSITELIB}/distributed/comm/__init__.pyo
@@ -168,9 +174,6 @@ ${PYSITELIB}/distributed/deploy/ssh.pyo
 ${PYSITELIB}/distributed/deploy/utils.py
 ${PYSITELIB}/distributed/deploy/utils.pyc
 ${PYSITELIB}/distributed/deploy/utils.pyo
-${PYSITELIB}/distributed/deploy/utils_test.py
-${PYSITELIB}/distributed/deploy/utils_test.pyc
-${PYSITELIB}/distributed/deploy/utils_test.pyo
 ${PYSITELIB}/distributed/diagnostics/__init__.py
 ${PYSITELIB}/distributed/diagnostics/__init__.pyc
 ${PYSITELIB}/distributed/diagnostics/__init__.pyo
@@ -233,6 +236,9 @@ ${PYSITELIB}/distributed/http/routing.pyo
 ${PYSITELIB}/distributed/http/scheduler/__init__.py
 ${PYSITELIB}/distributed/http/scheduler/__init__.pyc
 ${PYSITELIB}/distributed/http/scheduler/__init__.pyo
+${PYSITELIB}/distributed/http/scheduler/api.py
+${PYSITELIB}/distributed/http/scheduler/api.pyc
+${PYSITELIB}/distributed/http/scheduler/api.pyo
 ${PYSITELIB}/distributed/http/scheduler/info.py
 ${PYSITELIB}/distributed/http/scheduler/info.pyc
 ${PYSITELIB}/distributed/http/scheduler/info.pyo
@@ -401,6 +407,15 @@ ${PYSITELIB}/distributed/semaphore.pyo
 ${PYSITELIB}/distributed/shuffle/__init__.py
 ${PYSITELIB}/distributed/shuffle/__init__.pyc
 ${PYSITELIB}/distributed/shuffle/__init__.pyo
+${PYSITELIB}/distributed/shuffle/arrow.py
+${PYSITELIB}/distributed/shuffle/arrow.pyc
+${PYSITELIB}/distributed/shuffle/arrow.pyo
+${PYSITELIB}/distributed/shuffle/multi_comm.py
+${PYSITELIB}/distributed/shuffle/multi_comm.pyc
+${PYSITELIB}/distributed/shuffle/multi_comm.pyo
+${PYSITELIB}/distributed/shuffle/multi_file.py
+${PYSITELIB}/distributed/shuffle/multi_file.pyc
+${PYSITELIB}/distributed/shuffle/multi_file.pyo
 ${PYSITELIB}/distributed/shuffle/shuffle.py
 ${PYSITELIB}/distributed/shuffle/shuffle.pyc
 ${PYSITELIB}/distributed/shuffle/shuffle.pyo
diff --git a/py-distributed/distinfo b/py-distributed/distinfo
index 9d3435650d..fac0ca7f23 100644
--- a/py-distributed/distinfo
+++ b/py-distributed/distinfo
@@ -1,5 +1,5 @@
 $NetBSD$
 
-BLAKE2s (distributed-2022.5.0.tar.gz) = fe5a05b8d7674b5edf4ea6f51b46b8169783ecf65efd8b7f44b0b70a1770698f
-SHA512 (distributed-2022.5.0.tar.gz) = b7138be0a3548bec586e7a460e583613b55866d104da006607141415d14a5070909524341df77f963fbaafdf7d5ede7c89912d913c101928f59c572931df8307
-Size (distributed-2022.5.0.tar.gz) = 934072 bytes
+BLAKE2s (distributed-2022.6.1.tar.gz) = cc1c32d90fb17186076beb9e33f6002f167559b452255a287cbab0bb1d2b7c50
+SHA512 (distributed-2022.6.1.tar.gz) = 84e3c8136caccde71639f64405663a7a44632fbd76a63ff0372f192b4b23cbe3e21b3ab7fa87e17918eb6542b96c24c047dc8849f74f16c2da805cd8e3aeb233
+Size (distributed-2022.6.1.tar.gz) = 959007 bytes
Home |
Main Index |
Thread Index |
Old Index