pkgsrc-WIP-changes archive
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]
py-distributed: Updated to 2022.6.2 2022.6.1 Released on June 24, 2022 Highlights This release includes the Worker State Machine refactor. The expectation should be that the worker state is its own synchronous subclass. Pulling all the state out into its own class allows us to write targeted unit tests without invoking any concurrent or asynchronous code. See GH#5736 for more information. Enhancements Make worker state machine methods private (GH#6564) crusaderky Yank state machine out of Worker class (GH#6566) crusaderky Track worker_state_machine.TaskState instances (GH#6525) Hendrik Makait Trivial tweaks to the Worker State Machine (GH#6586) crusaderky Replace loop.call_later and loop.add_callback with background tasks added to Server. (GH#6603) Thomas Grainger Support for neater WorkerState tests (GH#6609) crusaderky Limit TCP writes with Tornado to 2GB (GH#6557) hhuuggoo Enable no_implicit_optional for scheduler (GH#6622) Thomas Grainger Bug Fixes Partial revert of compute-task message format (GH#6626) Florian Jetter Restore log message about received signals in CLI (GH#6618) Florian Jetter Handle empty memoryviews of bytearrays when (de)serializing (GH#6576) Benjamin Zaitlen Ensure steal requests from same-IP but distinct workers are rejected (GH#6585) Florian Jetter Fix tls_(min|max)_ version having no effect on openssl 1.1.0g or lower (GH#6562) Thomas Grainger Fix idle_timeout and unxfail test (GH#6563) Matthew Rocklin Fix crashing debug statement in _purge_state (GH#6589) crusaderky Abort connections on CancelledError (GH#6574) Thomas Grainger Fix Active Memory Manager ignores nbytes thresholds (GH#6583) crusaderky Deprecations Deprecate WorkerState accessors (GH#6579) crusaderky Maintenance Remove ipython hack (GH#6599) crusaderky Mypy enforce --no-implicit-optional (GH#6606) crusaderky Update versioneer: change from using SafeConfigParser to ConfigParser (GH#6605) Thomas A Caswell Warn unreachable for scheduler.py (GH#6611) Florian Jetter Refactor wait_for_state() (GH#6581) crusaderky Hardcode wait_for_signals signal list (GH#6619) Thomas Grainger Always pick an open port when running tests (GH#6591) Florian Jetter Log popen stdout/err when subprocess times out (GH#6567) Gabe Joseph Fix test_error_during_startup (GH#6608) Florian Jetter Make test_idle_timeout_no_workers more robust (GH#6602) Florian Jetter Mypy enforce --disallow-incomplete-defs (GH#6601) crusaderky Do not log during signal handler (GH#6590) Florian Jetter Don’t initialize mp_context on import (GH#6580) Lawrence Mitchell Test retire workers deadlock (GH#6240) Gabe Joseph Rework some tests related to gather_dep (GH#6472) crusaderky Minor cosmetic review of scheduler_story and worker_story (GH#6442) crusaderky Force __future__.annotations with isort (GH#6621) Thomas Grainger 2022.6.0 Released on June 10, 2022 Enhancements Make disk access in system monitor configurable (GH#6537) Matthew Rocklin Log and except errors on preload start (GH#6553) Matthew Rocklin Fix Scheduler.restart logic (GH#6504) Gabe Joseph Don’t heartbeat while Worker is closing (GH#6543) Gabe Joseph No longer retry LocalCluster on errno.EADDRINUSE (GH#6369) Thomas Grainger Don’t invoke log_event from state machine (GH#6512) crusaderky Add config option to disable profiling and disable it in many tests per default (GH#6490) Hendrik Makait Encapsulate Worker.batched_stream.send() (GH#6475) crusaderky Bug Fixes refresh-who-has can break the worker state machine (GH#6529) crusaderky Restart worker if it’s unrecognized by scheduler (GH#6505) Gabe Joseph Fix import error when distributed.rmm.pool-size is set (GH#6482) KoyamaSohei Deprecations Restore signature compatibility for dask-gateway (GH#6561) Tom Augspurger Deprecate the io_loop and loop kwarg to Server, Worker, and Nanny (GH#6473) Thomas Grainger Deprecate the loop kwarg to Scheduler (GH#6443) Thomas Grainger Documentation Fix typo in .nthreads() docstring example (GH#6545) Pavithra Eswaramoorthy Update docs theme for rebranding (GH#6495) Sarah Charlotte Johnson Maintenance Refactor gather_dep (GH#6388) crusaderky Fix flaky test_gather_dep_one_worker_always_busy (GH#6554) crusaderky Remove missing-data message (GH#6546) crusaderky Port test_local.LocalTest to pytest tests to allow use of loop fixture (GH#6523) Thomas Grainger Fix test_quiet_client_close (GH#6541) Gabe Joseph Use default_initializer in WorkerProcess (GH#6534) jakirkham Avoid deadlocks in tests that use popen (GH#6483) Gabe Joseph Revert “Fix CLI Scheduler Tests (GH#6502)” (GH#6547) Gabe Joseph Update test report URL in summary message (GH#6532) Gabe Joseph Update test report url (GH#6531) Ian Rose Assert AsyncProcess.set_exit_callback is not called with a coroutine function (GH#6526) Thomas Grainger Typing and docstring for Worker.close (GH#6518) Hendrik Makait Fix CLI Scheduler Tests (GH#6502) Benjamin Zaitlen Collect assertions in test_as_current_is_thread_local (GH#6520) Thomas Grainger Link test report from test results comment (GH#6524) Hendrik Makait Ignore the return value of signal.signal (GH#6519) Thomas Grainger Refactor all event handlers (GH#6410) crusaderky Fix dashboard favicon background (GH#6514) Jacob Tomlinson Update dashboard logo (GH#6513) Jacob Tomlinson Fix test_stress_scatter_death (GH#6404) Florian Jetter Remove CrossFilter widget (GH#6484) crusaderky data_needed exclusively contains tasks in fetch state (GH#6481) crusaderky Assert possible previous states (GH#6488) Florian Jetter @fail_hard can kill the whole test suite; hide errors (GH#6474) crusaderky Assert that a fetch->cancelled->resumed->fetch cycle is impossible (GH#6460) crusaderky Refactor busy workers reinsertion (GH#6379) crusaderky Refactor find_missing and refresh_who_has (GH#6348) crusaderky Rename test_collections.py to test_dask_collections.py (GH#6486) crusaderky update_who_has can remove workers (GH#6342) crusaderky Restructure test_watch_requires_lock_to_run to avoid flakes (GH#6469) Hendrik Makait Fix intermittent test_profile_plot failure (GH#6456) Matthew Rocklin Use asyncio.run to run gen_cluster, gen_test and cluster (GH#6231) Thomas Grainger Improve tests that watch for subprocess logs (GH#6461) Gabe Joseph 2022.05.2 Released on May 26, 2022 Enhancements Add a lock to distributed.profile for better concurrency control (GH#6421) Hendrik Makait Send SIGKILL after SIGTERM when passing 95% memory (GH#6419) crusaderky Bug Fixes Log rather than raise exceptions in preload.teardown() (GH#6458) Matthew Rocklin Handle failing plugin.close() calls during scheduler shutdown (GH#6450) Matthew Rocklin Fix slicing bug in ensure_memoryview (GH#6449) jakirkham Generalize UCX errors on connect() and correct pytest fixtures (GH#6434) Peter Andreas Entschev Run cluster widget periodic callbacks on the correct event loop (GH#6444) Thomas Grainger Maintenance Disable pytest-asyncio if installed (GH#6436) Jacob Tomlinson Close client in sync test_actor tests (GH#6459) Thomas Grainger Ignore ServerSession.with_document_locked unawaited (GH#6447) Thomas Grainger Remove coverage pin from Python 3.10 environment (GH#6439) Thomas Grainger Annotate remove_worker (GH#6441) crusaderky Update gpuCI RAPIDS_VER to 22.08 (GH#6428) 2022.05.1 Released on May 24, 2022 New Features Add HTTP API to scheduler (GH#6270) Matthew Murray Shuffle Service with Scheduler Logic (GH#6007) Matthew Rocklin Enhancements Follow-up on removing report and safe from Worker.close (GH#6423) Gabe Joseph Server close faster (GH#6415) Florian Jetter Disable HTTP API by default (GH#6420) Jacob Tomlinson Remove report and safe from Worker.close (GH#6363) Florian Jetter Allow deserialized plugins in register_scheduler_plugin (GH#6401) Matthew Rocklin WorkerState are different for different addresses (GH#6398) Florian Jetter Do not filter tasks before gathering data (GH#6371) crusaderky Remove worker reconnect (GH#6361) Gabe Joseph Add SchedulerPlugin.log_event handler (GH#6381) Matthew Rocklin Ensure occupancy tracking works as expected for long running tasks (GH#6351) Florian Jetter stimulus_id for all Instructions (GH#6347) crusaderky Refactor missing-data command (GH#6332) crusaderky Add idempotent to register_scheduler_plugin client (GH#6328) Alex Ford Add option to specify a scheduler address for workers to use (GH#5944) Enric Tejedor Bug Fixes Remove stray breakpoint (GH#6417) Thomas Grainger Fix API JSON MIME type (GH#6397) Jacob Tomlinson Remove wrong assert in handle compute (GH#6370) Florian Jetter Ensure multiple clients can cancel their key without interference (GH#6016) Florian Jetter Fix Nanny shutdown assertion (GH#6357) Gabe Joseph Fix fail_hard for sync functions (GH#6269) Gabe Joseph Prevent infinite transition loops; more aggressive validate_state() (GH#6318) crusaderky Ensure cleanup of many GBs of spilled data on terminate (GH#6280) crusaderky Fix WORKER_ANY_RUNNING regression (GH#6297) Florian Jetter Race conditions from fetch to compute while AMM requests replica (GH#6248) Florian Jetter Ensure resumed tasks are not accidentally forgotten (GH#6217) Florian Jetter Do not allow closing workers to be awaited again (GH#5910) Florian Jetter Deprecations Move wait_for_signals to private module and deprecate distributed.cli.utils (GH#6367) Hendrik Makait Documentation Fix typos and whitespace in worker.py (GH#6326) Hendrik Makait Fix link to memory trimming documentation (GH#6317) Marco Wolsza Maintenance Make gen_test show up in VSCode test discovery (GH#6424) Gabe Joseph WSMR / deserialize_task (GH#6411) crusaderky Restore signal handlers after wait for signals is done (GH#6400) Thomas Grainger fail_hard should reraise (GH#6399) crusaderky Revisit tests mocking gather_dep (GH#6385) crusaderky Fix flaky test_in_flight_lost_after_resumed (GH#6372) Florian Jetter Restore install_signal_handlers due to downstream dependencies (GH#6366) Hendrik Makait Improve catch_unhandled_exceptions (GH#6358) Gabe Joseph Remove all invocations of IOLoop.run_sync from CLI (GH#6205) Hendrik Makait Remove transition-counter-max from config (GH#6349) crusaderky Use list comprehension in pickle_loads (GH#6343) jakirkham Improve ensure_memoryview test coverage & make minor fixes (GH#6333) jakirkham Remove leaking reference to workers from gen_cluster (GH#6337) Hendrik Makait Partial annotations for stealing.py (GH#6338) crusaderky Validate and debug state machine on handle_compute_task (GH#6327) crusaderky Bump pyupgrade and clean up # type: ignore (GH#6293) crusaderky gen_cluster to write to /tmp (GH#6335) crusaderky Transition table as a ClassVar (GH#6331) crusaderky Simplify ensure_memoryview test with array (GH#6322) jakirkham Refactor ensure_communicating (GH#6165) crusaderky Review scheduler annotations, part 2 (GH#6253) crusaderky Use w for writeable branch in pickle_loads (GH#6314) jakirkham Simplify frame handling in ws (GH#6294) jakirkham Use ensure_bytes from dask.utils (GH#6295) jakirkham Use ensure_memoryview in array deserialization (GH#6300) jakirkham Escape < > when generating Junit report (GH#6306) crusaderky Use codecs.decode to deserialize errors (GH#6274) jakirkham Minimize copying in maybe_compress & byte_sample (GH#6273) jakirkham Skip test_release_evloop_while_spilling on OSX (GH#6291) Florian Jetter Simplify logic in get_default_compression (GH#6260) jakirkham Cleanup old compression workarounds (GH#6259) jakirkham Re-enable NVML monitoring for WSL (GH#6119) Charles Blackmon-Luca
- To: pkgsrc-wip-changes%NetBSD.org@localhost
- Subject: py-distributed: Updated to 2022.6.2 2022.6.1 Released on June 24, 2022 Highlights This release includes the Worker State Machine refactor. The expectation should be that the worker state is its own synchronous subclass. Pulling all the state out into its own class allows us to write targeted unit tests without invoking any concurrent or asynchronous code. See GH#5736 for more information. Enhancements Make worker state machine methods private (GH#6564) crusaderky Yank state machine out of Worker class (GH#6566) crusaderky Track worker_state_machine.TaskState instances (GH#6525) Hendrik Makait Trivial tweaks to the Worker State Machine (GH#6586) crusaderky Replace loop.call_later and loop.add_callback with background tasks added to Server. (GH#6603) Thomas Grainger Support for neater WorkerState tests (GH#6609) crusaderky Limit TCP writes with Tornado to 2GB (GH#6557) hhuuggoo Enable no_implicit_optional for scheduler (GH#6622) Thomas Grainger Bug Fixes Partial revert of compute-task message format (GH#6626) Florian Jetter Restore log message about received signals in CLI (GH#6618) Florian Jetter Handle empty memoryviews of bytearrays when (de)serializing (GH#6576) Benjamin Zaitlen Ensure steal requests from same-IP but distinct workers are rejected (GH#6585) Florian Jetter Fix tls_(min|max)_ version having no effect on openssl 1.1.0g or lower (GH#6562) Thomas Grainger Fix idle_timeout and unxfail test (GH#6563) Matthew Rocklin Fix crashing debug statement in _purge_state (GH#6589) crusaderky Abort connections on CancelledError (GH#6574) Thomas Grainger Fix Active Memory Manager ignores nbytes thresholds (GH#6583) crusaderky Deprecations Deprecate WorkerState accessors (GH#6579) crusaderky Maintenance Remove ipython hack (GH#6599) crusaderky Mypy enforce --no-implicit-optional (GH#6606) crusaderky Update versioneer: change from using SafeConfigParser to ConfigParser (GH#6605) Thomas A Caswell Warn unreachable for scheduler.py (GH#6611) Florian Jetter Refactor wait_for_state() (GH#6581) crusaderky Hardcode wait_for_signals signal list (GH#6619) Thomas Grainger Always pick an open port when running tests (GH#6591) Florian Jetter Log popen stdout/err when subprocess times out (GH#6567) Gabe Joseph Fix test_error_during_startup (GH#6608) Florian Jetter Make test_idle_timeout_no_workers more robust (GH#6602) Florian Jetter Mypy enforce --disallow-incomplete-defs (GH#6601) crusaderky Do not log during signal handler (GH#6590) Florian Jetter Don’t initialize mp_context on import (GH#6580) Lawrence Mitchell Test retire workers deadlock (GH#6240) Gabe Joseph Rework some tests related to gather_dep (GH#6472) crusaderky Minor cosmetic review of scheduler_story and worker_story (GH#6442) crusaderky Force __future__.annotations with isort (GH#6621) Thomas Grainger 2022.6.0 Released on June 10, 2022 Enhancements Make disk access in system monitor configurable (GH#6537) Matthew Rocklin Log and except errors on preload start (GH#6553) Matthew Rocklin Fix Scheduler.restart logic (GH#6504) Gabe Joseph Don’t heartbeat while Worker is closing (GH#6543) Gabe Joseph No longer retry LocalCluster on errno.EADDRINUSE (GH#6369) Thomas Grainger Don’t invoke log_event from state machine (GH#6512) crusaderky Add config option to disable profiling and disable it in many tests per default (GH#6490) Hendrik Makait Encapsulate Worker.batched_stream.send() (GH#6475) crusaderky Bug Fixes refresh-who-has can break the worker state machine (GH#6529) crusaderky Restart worker if it’s unrecognized by scheduler (GH#6505) Gabe Joseph Fix import error when distributed.rmm.pool-size is set (GH#6482) KoyamaSohei Deprecations Restore signature compatibility for dask-gateway (GH#6561) Tom Augspurger Deprecate the io_loop and loop kwarg to Server, Worker, and Nanny (GH#6473) Thomas Grainger Deprecate the loop kwarg to Scheduler (GH#6443) Thomas Grainger Documentation Fix typo in .nthreads() docstring example (GH#6545) Pavithra Eswaramoorthy Update docs theme for rebranding (GH#6495) Sarah Charlotte Johnson Maintenance Refactor gather_dep (GH#6388) crusaderky Fix flaky test_gather_dep_one_worker_always_busy (GH#6554) crusaderky Remove missing-data message (GH#6546) crusaderky Port test_local.LocalTest to pytest tests to allow use of loop fixture (GH#6523) Thomas Grainger Fix test_quiet_client_close (GH#6541) Gabe Joseph Use default_initializer in WorkerProcess (GH#6534) jakirkham Avoid deadlocks in tests that use popen (GH#6483) Gabe Joseph Revert “Fix CLI Scheduler Tests (GH#6502)” (GH#6547) Gabe Joseph Update test report URL in summary message (GH#6532) Gabe Joseph Update test report url (GH#6531) Ian Rose Assert AsyncProcess.set_exit_callback is not called with a coroutine function (GH#6526) Thomas Grainger Typing and docstring for Worker.close (GH#6518) Hendrik Makait Fix CLI Scheduler Tests (GH#6502) Benjamin Zaitlen Collect assertions in test_as_current_is_thread_local (GH#6520) Thomas Grainger Link test report from test results comment (GH#6524) Hendrik Makait Ignore the return value of signal.signal (GH#6519) Thomas Grainger Refactor all event handlers (GH#6410) crusaderky Fix dashboard favicon background (GH#6514) Jacob Tomlinson Update dashboard logo (GH#6513) Jacob Tomlinson Fix test_stress_scatter_death (GH#6404) Florian Jetter Remove CrossFilter widget (GH#6484) crusaderky data_needed exclusively contains tasks in fetch state (GH#6481) crusaderky Assert possible previous states (GH#6488) Florian Jetter @fail_hard can kill the whole test suite; hide errors (GH#6474) crusaderky Assert that a fetch->cancelled->resumed->fetch cycle is impossible (GH#6460) crusaderky Refactor busy workers reinsertion (GH#6379) crusaderky Refactor find_missing and refresh_who_has (GH#6348) crusaderky Rename test_collections.py to test_dask_collections.py (GH#6486) crusaderky update_who_has can remove workers (GH#6342) crusaderky Restructure test_watch_requires_lock_to_run to avoid flakes (GH#6469) Hendrik Makait Fix intermittent test_profile_plot failure (GH#6456) Matthew Rocklin Use asyncio.run to run gen_cluster, gen_test and cluster (GH#6231) Thomas Grainger Improve tests that watch for subprocess logs (GH#6461) Gabe Joseph 2022.05.2 Released on May 26, 2022 Enhancements Add a lock to distributed.profile for better concurrency control (GH#6421) Hendrik Makait Send SIGKILL after SIGTERM when passing 95% memory (GH#6419) crusaderky Bug Fixes Log rather than raise exceptions in preload.teardown() (GH#6458) Matthew Rocklin Handle failing plugin.close() calls during scheduler shutdown (GH#6450) Matthew Rocklin Fix slicing bug in ensure_memoryview (GH#6449) jakirkham Generalize UCX errors on connect() and correct pytest fixtures (GH#6434) Peter Andreas Entschev Run cluster widget periodic callbacks on the correct event loop (GH#6444) Thomas Grainger Maintenance Disable pytest-asyncio if installed (GH#6436) Jacob Tomlinson Close client in sync test_actor tests (GH#6459) Thomas Grainger Ignore ServerSession.with_document_locked unawaited (GH#6447) Thomas Grainger Remove coverage pin from Python 3.10 environment (GH#6439) Thomas Grainger Annotate remove_worker (GH#6441) crusaderky Update gpuCI RAPIDS_VER to 22.08 (GH#6428) 2022.05.1 Released on May 24, 2022 New Features Add HTTP API to scheduler (GH#6270) Matthew Murray Shuffle Service with Scheduler Logic (GH#6007) Matthew Rocklin Enhancements Follow-up on removing report and safe from Worker.close (GH#6423) Gabe Joseph Server close faster (GH#6415) Florian Jetter Disable HTTP API by default (GH#6420) Jacob Tomlinson Remove report and safe from Worker.close (GH#6363) Florian Jetter Allow deserialized plugins in register_scheduler_plugin (GH#6401) Matthew Rocklin WorkerState are different for different addresses (GH#6398) Florian Jetter Do not filter tasks before gathering data (GH#6371) crusaderky Remove worker reconnect (GH#6361) Gabe Joseph Add SchedulerPlugin.log_event handler (GH#6381) Matthew Rocklin Ensure occupancy tracking works as expected for long running tasks (GH#6351) Florian Jetter stimulus_id for all Instructions (GH#6347) crusaderky Refactor missing-data command (GH#6332) crusaderky Add idempotent to register_scheduler_plugin client (GH#6328) Alex Ford Add option to specify a scheduler address for workers to use (GH#5944) Enric Tejedor Bug Fixes Remove stray breakpoint (GH#6417) Thomas Grainger Fix API JSON MIME type (GH#6397) Jacob Tomlinson Remove wrong assert in handle compute (GH#6370) Florian Jetter Ensure multiple clients can cancel their key without interference (GH#6016) Florian Jetter Fix Nanny shutdown assertion (GH#6357) Gabe Joseph Fix fail_hard for sync functions (GH#6269) Gabe Joseph Prevent infinite transition loops; more aggressive validate_state() (GH#6318) crusaderky Ensure cleanup of many GBs of spilled data on terminate (GH#6280) crusaderky Fix WORKER_ANY_RUNNING regression (GH#6297) Florian Jetter Race conditions from fetch to compute while AMM requests replica (GH#6248) Florian Jetter Ensure resumed tasks are not accidentally forgotten (GH#6217) Florian Jetter Do not allow closing workers to be awaited again (GH#5910) Florian Jetter Deprecations Move wait_for_signals to private module and deprecate distributed.cli.utils (GH#6367) Hendrik Makait Documentation Fix typos and whitespace in worker.py (GH#6326) Hendrik Makait Fix link to memory trimming documentation (GH#6317) Marco Wolsza Maintenance Make gen_test show up in VSCode test discovery (GH#6424) Gabe Joseph WSMR / deserialize_task (GH#6411) crusaderky Restore signal handlers after wait for signals is done (GH#6400) Thomas Grainger fail_hard should reraise (GH#6399) crusaderky Revisit tests mocking gather_dep (GH#6385) crusaderky Fix flaky test_in_flight_lost_after_resumed (GH#6372) Florian Jetter Restore install_signal_handlers due to downstream dependencies (GH#6366) Hendrik Makait Improve catch_unhandled_exceptions (GH#6358) Gabe Joseph Remove all invocations of IOLoop.run_sync from CLI (GH#6205) Hendrik Makait Remove transition-counter-max from config (GH#6349) crusaderky Use list comprehension in pickle_loads (GH#6343) jakirkham Improve ensure_memoryview test coverage & make minor fixes (GH#6333) jakirkham Remove leaking reference to workers from gen_cluster (GH#6337) Hendrik Makait Partial annotations for stealing.py (GH#6338) crusaderky Validate and debug state machine on handle_compute_task (GH#6327) crusaderky Bump pyupgrade and clean up # type: ignore (GH#6293) crusaderky gen_cluster to write to /tmp (GH#6335) crusaderky Transition table as a ClassVar (GH#6331) crusaderky Simplify ensure_memoryview test with array (GH#6322) jakirkham Refactor ensure_communicating (GH#6165) crusaderky Review scheduler annotations, part 2 (GH#6253) crusaderky Use w for writeable branch in pickle_loads (GH#6314) jakirkham Simplify frame handling in ws (GH#6294) jakirkham Use ensure_bytes from dask.utils (GH#6295) jakirkham Use ensure_memoryview in array deserialization (GH#6300) jakirkham Escape < > when generating Junit report (GH#6306) crusaderky Use codecs.decode to deserialize errors (GH#6274) jakirkham Minimize copying in maybe_compress & byte_sample (GH#6273) jakirkham Skip test_release_evloop_while_spilling on OSX (GH#6291) Florian Jetter Simplify logic in get_default_compression (GH#6260) jakirkham Cleanup old compression workarounds (GH#6259) jakirkham Re-enable NVML monitoring for WSL (GH#6119) Charles Blackmon-Luca
- From: Matthew Danielson <matthewd%fastmail.us@localhost>
- Date: Mon, 04 Jul 2022 02:21:43 +0000
Module Name: pkgsrc-wip
Committed By: Matthew Danielson <matthewd%fastmail.us@localhost>
Pushed By: matthewd
Date: Sun Jul 3 19:21:43 2022 -0700
Changeset: 0b55b7f25b132db3f0980f475ab98cf28a497b13
Modified Files:
py-distributed/Makefile
py-distributed/PLIST
py-distributed/distinfo
Log Message:
py-distributed: Updated to 2022.6.2
2022.6.1
Released on June 24, 2022
Highlights
This release includes the Worker State Machine refactor. The expectation should be that the worker state is its own synchronous subclass. Pulling all the state out into its own class allows us to write targeted unit tests without invoking any concurrent or asynchronous code.
See GH#5736 for more information.
Enhancements
Make worker state machine methods private (GH#6564) crusaderky
Yank state machine out of Worker class (GH#6566) crusaderky
Track worker_state_machine.TaskState instances (GH#6525) Hendrik Makait
Trivial tweaks to the Worker State Machine (GH#6586) crusaderky
Replace loop.call_later and loop.add_callback with background tasks added to Server. (GH#6603) Thomas Grainger
Support for neater WorkerState tests (GH#6609) crusaderky
Limit TCP writes with Tornado to 2GB (GH#6557) hhuuggoo
Enable no_implicit_optional for scheduler (GH#6622) Thomas Grainger
Bug Fixes
Partial revert of compute-task message format (GH#6626) Florian Jetter
Restore log message about received signals in CLI (GH#6618) Florian Jetter
Handle empty memoryviews of bytearrays when (de)serializing (GH#6576) Benjamin Zaitlen
Ensure steal requests from same-IP but distinct workers are rejected (GH#6585) Florian Jetter
Fix tls_(min|max)_ version having no effect on openssl 1.1.0g or lower (GH#6562) Thomas Grainger
Fix idle_timeout and unxfail test (GH#6563) Matthew Rocklin
Fix crashing debug statement in _purge_state (GH#6589) crusaderky
Abort connections on CancelledError (GH#6574) Thomas Grainger
Fix Active Memory Manager ignores nbytes thresholds (GH#6583) crusaderky
Deprecations
Deprecate WorkerState accessors (GH#6579) crusaderky
Maintenance
Remove ipython hack (GH#6599) crusaderky
Mypy enforce --no-implicit-optional (GH#6606) crusaderky
Update versioneer: change from using SafeConfigParser to ConfigParser (GH#6605) Thomas A Caswell
Warn unreachable for scheduler.py (GH#6611) Florian Jetter
Refactor wait_for_state() (GH#6581) crusaderky
Hardcode wait_for_signals signal list (GH#6619) Thomas Grainger
Always pick an open port when running tests (GH#6591) Florian Jetter
Log popen stdout/err when subprocess times out (GH#6567) Gabe Joseph
Fix test_error_during_startup (GH#6608) Florian Jetter
Make test_idle_timeout_no_workers more robust (GH#6602) Florian Jetter
Mypy enforce --disallow-incomplete-defs (GH#6601) crusaderky
Do not log during signal handler (GH#6590) Florian Jetter
Don’t initialize mp_context on import (GH#6580) Lawrence Mitchell
Test retire workers deadlock (GH#6240) Gabe Joseph
Rework some tests related to gather_dep (GH#6472) crusaderky
Minor cosmetic review of scheduler_story and worker_story (GH#6442) crusaderky
Force __future__.annotations with isort (GH#6621) Thomas Grainger
2022.6.0
Released on June 10, 2022
Enhancements
Make disk access in system monitor configurable (GH#6537) Matthew Rocklin
Log and except errors on preload start (GH#6553) Matthew Rocklin
Fix Scheduler.restart logic (GH#6504) Gabe Joseph
Don’t heartbeat while Worker is closing (GH#6543) Gabe Joseph
No longer retry LocalCluster on errno.EADDRINUSE (GH#6369) Thomas Grainger
Don’t invoke log_event from state machine (GH#6512) crusaderky
Add config option to disable profiling and disable it in many tests per default (GH#6490) Hendrik Makait
Encapsulate Worker.batched_stream.send() (GH#6475) crusaderky
Bug Fixes
refresh-who-has can break the worker state machine (GH#6529) crusaderky
Restart worker if it’s unrecognized by scheduler (GH#6505) Gabe Joseph
Fix import error when distributed.rmm.pool-size is set (GH#6482) KoyamaSohei
Deprecations
Restore signature compatibility for dask-gateway (GH#6561) Tom Augspurger
Deprecate the io_loop and loop kwarg to Server, Worker, and Nanny (GH#6473) Thomas Grainger
Deprecate the loop kwarg to Scheduler (GH#6443) Thomas Grainger
Documentation
Fix typo in .nthreads() docstring example (GH#6545) Pavithra Eswaramoorthy
Update docs theme for rebranding (GH#6495) Sarah Charlotte Johnson
Maintenance
Refactor gather_dep (GH#6388) crusaderky
Fix flaky test_gather_dep_one_worker_always_busy (GH#6554) crusaderky
Remove missing-data message (GH#6546) crusaderky
Port test_local.LocalTest to pytest tests to allow use of loop fixture (GH#6523) Thomas Grainger
Fix test_quiet_client_close (GH#6541) Gabe Joseph
Use default_initializer in WorkerProcess (GH#6534) jakirkham
Avoid deadlocks in tests that use popen (GH#6483) Gabe Joseph
Revert “Fix CLI Scheduler Tests (GH#6502)” (GH#6547) Gabe Joseph
Update test report URL in summary message (GH#6532) Gabe Joseph
Update test report url (GH#6531) Ian Rose
Assert AsyncProcess.set_exit_callback is not called with a coroutine function (GH#6526) Thomas Grainger
Typing and docstring for Worker.close (GH#6518) Hendrik Makait
Fix CLI Scheduler Tests (GH#6502) Benjamin Zaitlen
Collect assertions in test_as_current_is_thread_local (GH#6520) Thomas Grainger
Link test report from test results comment (GH#6524) Hendrik Makait
Ignore the return value of signal.signal (GH#6519) Thomas Grainger
Refactor all event handlers (GH#6410) crusaderky
Fix dashboard favicon background (GH#6514) Jacob Tomlinson
Update dashboard logo (GH#6513) Jacob Tomlinson
Fix test_stress_scatter_death (GH#6404) Florian Jetter
Remove CrossFilter widget (GH#6484) crusaderky
data_needed exclusively contains tasks in fetch state (GH#6481) crusaderky
Assert possible previous states (GH#6488) Florian Jetter
@fail_hard can kill the whole test suite; hide errors (GH#6474) crusaderky
Assert that a fetch->cancelled->resumed->fetch cycle is impossible (GH#6460) crusaderky
Refactor busy workers reinsertion (GH#6379) crusaderky
Refactor find_missing and refresh_who_has (GH#6348) crusaderky
Rename test_collections.py to test_dask_collections.py (GH#6486) crusaderky
update_who_has can remove workers (GH#6342) crusaderky
Restructure test_watch_requires_lock_to_run to avoid flakes (GH#6469) Hendrik Makait
Fix intermittent test_profile_plot failure (GH#6456) Matthew Rocklin
Use asyncio.run to run gen_cluster, gen_test and cluster (GH#6231) Thomas Grainger
Improve tests that watch for subprocess logs (GH#6461) Gabe Joseph
2022.05.2
Released on May 26, 2022
Enhancements
Add a lock to distributed.profile for better concurrency control (GH#6421) Hendrik Makait
Send SIGKILL after SIGTERM when passing 95% memory (GH#6419) crusaderky
Bug Fixes
Log rather than raise exceptions in preload.teardown() (GH#6458) Matthew Rocklin
Handle failing plugin.close() calls during scheduler shutdown (GH#6450) Matthew Rocklin
Fix slicing bug in ensure_memoryview (GH#6449) jakirkham
Generalize UCX errors on connect() and correct pytest fixtures (GH#6434) Peter Andreas Entschev
Run cluster widget periodic callbacks on the correct event loop (GH#6444) Thomas Grainger
Maintenance
Disable pytest-asyncio if installed (GH#6436) Jacob Tomlinson
Close client in sync test_actor tests (GH#6459) Thomas Grainger
Ignore ServerSession.with_document_locked unawaited (GH#6447) Thomas Grainger
Remove coverage pin from Python 3.10 environment (GH#6439) Thomas Grainger
Annotate remove_worker (GH#6441) crusaderky
Update gpuCI RAPIDS_VER to 22.08 (GH#6428)
2022.05.1
Released on May 24, 2022
New Features
Add HTTP API to scheduler (GH#6270) Matthew Murray
Shuffle Service with Scheduler Logic (GH#6007) Matthew Rocklin
Enhancements
Follow-up on removing report and safe from Worker.close (GH#6423) Gabe Joseph
Server close faster (GH#6415) Florian Jetter
Disable HTTP API by default (GH#6420) Jacob Tomlinson
Remove report and safe from Worker.close (GH#6363) Florian Jetter
Allow deserialized plugins in register_scheduler_plugin (GH#6401) Matthew Rocklin
WorkerState are different for different addresses (GH#6398) Florian Jetter
Do not filter tasks before gathering data (GH#6371) crusaderky
Remove worker reconnect (GH#6361) Gabe Joseph
Add SchedulerPlugin.log_event handler (GH#6381) Matthew Rocklin
Ensure occupancy tracking works as expected for long running tasks (GH#6351) Florian Jetter
stimulus_id for all Instructions (GH#6347) crusaderky
Refactor missing-data command (GH#6332) crusaderky
Add idempotent to register_scheduler_plugin client (GH#6328) Alex Ford
Add option to specify a scheduler address for workers to use (GH#5944) Enric Tejedor
Bug Fixes
Remove stray breakpoint (GH#6417) Thomas Grainger
Fix API JSON MIME type (GH#6397) Jacob Tomlinson
Remove wrong assert in handle compute (GH#6370) Florian Jetter
Ensure multiple clients can cancel their key without interference (GH#6016) Florian Jetter
Fix Nanny shutdown assertion (GH#6357) Gabe Joseph
Fix fail_hard for sync functions (GH#6269) Gabe Joseph
Prevent infinite transition loops; more aggressive validate_state() (GH#6318) crusaderky
Ensure cleanup of many GBs of spilled data on terminate (GH#6280) crusaderky
Fix WORKER_ANY_RUNNING regression (GH#6297) Florian Jetter
Race conditions from fetch to compute while AMM requests replica (GH#6248) Florian Jetter
Ensure resumed tasks are not accidentally forgotten (GH#6217) Florian Jetter
Do not allow closing workers to be awaited again (GH#5910) Florian Jetter
Deprecations
Move wait_for_signals to private module and deprecate distributed.cli.utils (GH#6367) Hendrik Makait
Documentation
Fix typos and whitespace in worker.py (GH#6326) Hendrik Makait
Fix link to memory trimming documentation (GH#6317) Marco Wolsza
Maintenance
Make gen_test show up in VSCode test discovery (GH#6424) Gabe Joseph
WSMR / deserialize_task (GH#6411) crusaderky
Restore signal handlers after wait for signals is done (GH#6400) Thomas Grainger
fail_hard should reraise (GH#6399) crusaderky
Revisit tests mocking gather_dep (GH#6385) crusaderky
Fix flaky test_in_flight_lost_after_resumed (GH#6372) Florian Jetter
Restore install_signal_handlers due to downstream dependencies (GH#6366) Hendrik Makait
Improve catch_unhandled_exceptions (GH#6358) Gabe Joseph
Remove all invocations of IOLoop.run_sync from CLI (GH#6205) Hendrik Makait
Remove transition-counter-max from config (GH#6349) crusaderky
Use list comprehension in pickle_loads (GH#6343) jakirkham
Improve ensure_memoryview test coverage & make minor fixes (GH#6333) jakirkham
Remove leaking reference to workers from gen_cluster (GH#6337) Hendrik Makait
Partial annotations for stealing.py (GH#6338) crusaderky
Validate and debug state machine on handle_compute_task (GH#6327) crusaderky
Bump pyupgrade and clean up # type: ignore (GH#6293) crusaderky
gen_cluster to write to /tmp (GH#6335) crusaderky
Transition table as a ClassVar (GH#6331) crusaderky
Simplify ensure_memoryview test with array (GH#6322) jakirkham
Refactor ensure_communicating (GH#6165) crusaderky
Review scheduler annotations, part 2 (GH#6253) crusaderky
Use w for writeable branch in pickle_loads (GH#6314) jakirkham
Simplify frame handling in ws (GH#6294) jakirkham
Use ensure_bytes from dask.utils (GH#6295) jakirkham
Use ensure_memoryview in array deserialization (GH#6300) jakirkham
Escape < > when generating Junit report (GH#6306) crusaderky
Use codecs.decode to deserialize errors (GH#6274) jakirkham
Minimize copying in maybe_compress & byte_sample (GH#6273) jakirkham
Skip test_release_evloop_while_spilling on OSX (GH#6291) Florian Jetter
Simplify logic in get_default_compression (GH#6260) jakirkham
Cleanup old compression workarounds (GH#6259) jakirkham
Re-enable NVML monitoring for WSL (GH#6119) Charles Blackmon-Luca
To see a diff of this commit:
https://wip.pkgsrc.org/cgi-bin/gitweb.cgi?p=pkgsrc-wip.git;a=commitdiff;h=0b55b7f25b132db3f0980f475ab98cf28a497b13
Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.
diffstat:
py-distributed/Makefile | 2 +-
py-distributed/PLIST | 21 ++++++++++++++++++---
py-distributed/distinfo | 6 +++---
3 files changed, 22 insertions(+), 7 deletions(-)
diffs:
diff --git a/py-distributed/Makefile b/py-distributed/Makefile
index 2b10469d56..3453ee1459 100644
--- a/py-distributed/Makefile
+++ b/py-distributed/Makefile
@@ -1,6 +1,6 @@
# $NetBSD$
-DISTNAME= distributed-2022.5.0
+DISTNAME= distributed-2022.6.1
PKGNAME= ${PYPKGPREFIX}-${DISTNAME}
CATEGORIES= devel net
MASTER_SITES= ${MASTER_SITE_PYPI:=d/distributed/}
diff --git a/py-distributed/PLIST b/py-distributed/PLIST
index b491f76eab..88123b8007 100644
--- a/py-distributed/PLIST
+++ b/py-distributed/PLIST
@@ -15,6 +15,9 @@ ${PYSITELIB}/distributed/__init__.pyo
${PYSITELIB}/distributed/_concurrent_futures_thread.py
${PYSITELIB}/distributed/_concurrent_futures_thread.pyc
${PYSITELIB}/distributed/_concurrent_futures_thread.pyo
+${PYSITELIB}/distributed/_signals.py
+${PYSITELIB}/distributed/_signals.pyc
+${PYSITELIB}/distributed/_signals.pyo
${PYSITELIB}/distributed/_stories.py
${PYSITELIB}/distributed/_stories.pyc
${PYSITELIB}/distributed/_stories.pyo
@@ -63,6 +66,9 @@ ${PYSITELIB}/distributed/client.pyo
${PYSITELIB}/distributed/cluster_dump.py
${PYSITELIB}/distributed/cluster_dump.pyc
${PYSITELIB}/distributed/cluster_dump.pyo
+${PYSITELIB}/distributed/collections.py
+${PYSITELIB}/distributed/collections.pyc
+${PYSITELIB}/distributed/collections.pyo
${PYSITELIB}/distributed/comm/__init__.py
${PYSITELIB}/distributed/comm/__init__.pyc
${PYSITELIB}/distributed/comm/__init__.pyo
@@ -168,9 +174,6 @@ ${PYSITELIB}/distributed/deploy/ssh.pyo
${PYSITELIB}/distributed/deploy/utils.py
${PYSITELIB}/distributed/deploy/utils.pyc
${PYSITELIB}/distributed/deploy/utils.pyo
-${PYSITELIB}/distributed/deploy/utils_test.py
-${PYSITELIB}/distributed/deploy/utils_test.pyc
-${PYSITELIB}/distributed/deploy/utils_test.pyo
${PYSITELIB}/distributed/diagnostics/__init__.py
${PYSITELIB}/distributed/diagnostics/__init__.pyc
${PYSITELIB}/distributed/diagnostics/__init__.pyo
@@ -233,6 +236,9 @@ ${PYSITELIB}/distributed/http/routing.pyo
${PYSITELIB}/distributed/http/scheduler/__init__.py
${PYSITELIB}/distributed/http/scheduler/__init__.pyc
${PYSITELIB}/distributed/http/scheduler/__init__.pyo
+${PYSITELIB}/distributed/http/scheduler/api.py
+${PYSITELIB}/distributed/http/scheduler/api.pyc
+${PYSITELIB}/distributed/http/scheduler/api.pyo
${PYSITELIB}/distributed/http/scheduler/info.py
${PYSITELIB}/distributed/http/scheduler/info.pyc
${PYSITELIB}/distributed/http/scheduler/info.pyo
@@ -401,6 +407,15 @@ ${PYSITELIB}/distributed/semaphore.pyo
${PYSITELIB}/distributed/shuffle/__init__.py
${PYSITELIB}/distributed/shuffle/__init__.pyc
${PYSITELIB}/distributed/shuffle/__init__.pyo
+${PYSITELIB}/distributed/shuffle/arrow.py
+${PYSITELIB}/distributed/shuffle/arrow.pyc
+${PYSITELIB}/distributed/shuffle/arrow.pyo
+${PYSITELIB}/distributed/shuffle/multi_comm.py
+${PYSITELIB}/distributed/shuffle/multi_comm.pyc
+${PYSITELIB}/distributed/shuffle/multi_comm.pyo
+${PYSITELIB}/distributed/shuffle/multi_file.py
+${PYSITELIB}/distributed/shuffle/multi_file.pyc
+${PYSITELIB}/distributed/shuffle/multi_file.pyo
${PYSITELIB}/distributed/shuffle/shuffle.py
${PYSITELIB}/distributed/shuffle/shuffle.pyc
${PYSITELIB}/distributed/shuffle/shuffle.pyo
diff --git a/py-distributed/distinfo b/py-distributed/distinfo
index 9d3435650d..fac0ca7f23 100644
--- a/py-distributed/distinfo
+++ b/py-distributed/distinfo
@@ -1,5 +1,5 @@
$NetBSD$
-BLAKE2s (distributed-2022.5.0.tar.gz) = fe5a05b8d7674b5edf4ea6f51b46b8169783ecf65efd8b7f44b0b70a1770698f
-SHA512 (distributed-2022.5.0.tar.gz) = b7138be0a3548bec586e7a460e583613b55866d104da006607141415d14a5070909524341df77f963fbaafdf7d5ede7c89912d913c101928f59c572931df8307
-Size (distributed-2022.5.0.tar.gz) = 934072 bytes
+BLAKE2s (distributed-2022.6.1.tar.gz) = cc1c32d90fb17186076beb9e33f6002f167559b452255a287cbab0bb1d2b7c50
+SHA512 (distributed-2022.6.1.tar.gz) = 84e3c8136caccde71639f64405663a7a44632fbd76a63ff0372f192b4b23cbe3e21b3ab7fa87e17918eb6542b96c24c047dc8849f74f16c2da805cd8e3aeb233
+Size (distributed-2022.6.1.tar.gz) = 959007 bytes
Home |
Main Index |
Thread Index |
Old Index