Changelog¶
2.10.2¶
(2024-05-07)
2.10.1¶
(2024-03-19)
2.10.0¶
(2024-02-19)
Added support for macOS (#27, #79, #91)
Fixes memory leaks on macOS
Reduced the amount of semaphores used
Issues a warning when
cpu_ids
is used on macOS
Added
mpire.dashboard.set_stacklevel()
to set the stack level in the dashboard. This influences what line to display in the ‘Invoked on line’ section. (#118)Use function details from the __call__ method on the dashboard in case the callable being executed is a class instance (#117)
Use (global) average rate for the estimate on the dashboard when smoothing=0 (#117)
Make it possible to reuse the same progress_bar_options without raising warnings (#117)
Removed deprecated progress_bar_position parameter from the map functions. Use progress_bar_options[‘position’] instead (added since v2.6.0)
2.9.0¶
(2024-01-08)
Added support for the
rich
progress bar style (#96)Added the option to only show progress on the dashboard. (#107)
Progress bars are now supported on Windows when using threading as start method.
Insights now also work when using the
forkserver
andspawn
start methods. (#104)When using insights on Windows the arguments of the top 5 longest tasks are now available as well.
Fixed deprecated
escape
import fromflask
by importing directly frommarkupsafe
. (#106)Fixed
mpire.dashboard.start_dashboard()
freeze when there are no two ports available. (#112)Added
mpire.dashboard.shutdown_dashboard()
to shutdown the dashboard.Added
py.typed
file to promptmypy
for type checking. (#108)
2.8.1¶
(2023-11-08)
Excluded the
tests
folder from MPIRE distributions (#89)Added a workaround for semaphore leakage on macOS and fixed a bug when working in a fork context while the system default is spawn (#92)
Fix progressbar percentage on dashboard (#101)
Fixed a bug where starting multiple apply_async tasks with a task timeout didn’t interrupt all tasks when the timeout was reached (#98)
Add testing python 3.12 to workflow and drop 3.6 and 3.7 (#102)
2.8.0¶
(2023-08-16)
Added support for Python 3.11 (#67)
2.7.1¶
(2023-04-14)
Transfered ownership of the project from Slimmer AI to sybrenjansen
2.7.0¶
(2023-03-17)
Added the
mpire.WorkerPool.apply()
andmpire.WorkerPool.apply_async()
functions (#63)When inside a Jupyter notebook, the progress bar will not automatically switch to a widget anymore.
tqdm
cannot always determine with certainty that someone is in a notebook or, e.g., a Jupyter console. Another reason is to avoid the many errors people get when having widgets or javascript disabled. See Progress bar style for changing the progress bar to a widget (#71)The
mpire.dashboard.connect_to_dashboard()
function now raises a ConnectionRefused error when the dashboard isn’t running, instead of silently failing and deadlocking the nextmap
call with a progress bar (#68)Added support for a progress bar without knowing the size of the iterable. It used to disable the progress bar when the size was unknown
Changed how
max_tasks_active
is handled. It now applies to the number of tasks that are currently being processed, instead of the number of chunks of tasks, as you would expect from the name. Previously, when the chunk size was set to anything other than 1, the number of active tasks could be higher thanmax_tasks_active
Updated some exception messages and docs (#69)
Changed how worker results, restarts, timeouts, unexpected deaths, and exceptions are handled. They are now handled by individual threads such that the main thread is more responsive. The API is the same, so no user changes are needed
Mixing multiple
map
calls now raises an error (see Mixing map functions)Fixed a bug where calling a
map
function with a progress bar multiple times in a row didn’t display the progress bar correctlyFixed a bug where the dashboard didn’t show an error when an exit function raised an exception
2.6.0¶
(2022-08-29)
Added Python 3.10 support
The
tqdm
progress bar can now be customized using theprogress_bar_options
parameter in themap
functions (#57)Using
progress_bar_position
from amap
function is now deprecated and will be removed in MPIRE v2.10.0. Useprogress_bar_options['position']
insteadDeprecated
enable_insights
from amap
function, useenable_insights
in the WorkerPool constructor insteadFixed a bug where a worker could exit before an exception was entirely sent over the queue, causing a deadlock (#56)
Fixed a bug where exceptions with init arguments weren’t handled correctly (#58)
Fixed a rare and weird bug in Windows that could cause a deadlock (probably fixes #55)
2.5.0¶
(2022-07-25)
Added the option to fix the order of tasks given to the workers (#46)
Fixed a bug where updated WorkerPool parameters aren’t used in subsequent
map
calls whenkeep_alive
is enabled
2.4.0¶
(2022-05-25)
2.3.5¶
(2022-04-25)
2.3.4¶
(2022-03-29)
2.3.3¶
(2021-11-29)
Changed progress bar handler process to thread, making it more stable (especially in notebooks)
Changed progress bar tasks completed queue to array, to make it more responsive and faster
Disabled the tqdm monitor thread which, in combination with MPIRE’s own tqdm lock, could result in deadlocks
2.3.2¶
(2021-11-19)
Included license file in source distribution (#25)
2.3.1¶
(2021-11-16)
Made connecting to the tqdm manager more robust (#23)
2.3.0¶
(2021-10-15)
Fixed progress bar in a particular setting with iPython and django installed (#13)
keep_alive
now works even when the function to be called or any other parameter passed to themap
function is changed (#15)Moved
enable_insights
to the WorkerPool constructor. Usingenable_insights
from amap
function is now deprecated and will be removed in MPIRE v2.6.0.Restructured docs and updated several sections for Windows users.
2.2.1¶
(2021-08-31)
Fixed compatibility with newer tqdm versions (
>= 4.62.2
) (#11)
2.2.0¶
(2021-08-30)
Added support for Windows (#6, #7). Support has a few caveats:
When using worker insights the arguments of the top 5 longest tasks are not available
Progress bar is not supported when using threading as start method
When using
dill
and an exception occurs, or when the exception occurs in an exit function, it can print additionalOSError
messages in the terminal, but these can be safely ignored.
2.1.1¶
(2021-08-26)
Fixed a bug with newer versions of tqdm. The progress bar would throw an
AttributeError
when connected to a dashboard.README and documentation updated
2.1.0¶
(2021-08-06)
Workers now have their own task queue, which speeds up tasks with bigger payloads
Fixed progress bar showing error information when completed without error
Fixed progress bar and worker insights not displaying properly when using threading
Progress bar handling improved accross several scenarios
Dashboard can now handle progress bars when using
spawn
orforkserver
as start methodAdded closing of
multiprocessing.JoinableQueue
objects, to clean up intermediate junkRemoved
numpy
dependencyMade
dill
optional again. In many cases it slows processing down
2.0.0¶
(2021-07-07)
Worker insights added, providing users insight in multiprocessing efficiency
worker_init
andworker_exit
parameters added to eachmap
functionmax_active_tasks
is now set ton_jobs * 2
whenmax_active_tasks=None
, to speed up most jobsn_splits
is now set ton_jobs * 64
when bothchunk_size
andn_splits
areNone
Dashboard ports can now be configured
Renamed
func_pointer
tofunc
in eachmap
functionFixed a bug with the threading backend not terminating correctly
Fixed a bug with the progress bar not showing correctly in notebooks
Using
multiprocess
is now the defaultAdded some debug logging
Refactored a lot of code
Minor bug fixes, which should make things more stable.
Removed Python 3.5 support
Removed
add_task
,get_result
,insert_poison_pill
,stop_workers
, andjoin
functions frommpire.WorkerPool
. Madestart_workers
private. There wasn’t any reason to use these functions.
1.2.2¶
(2021-04-23)
Updated documentation CSS which fixes bullet lists not showing properly
1.2.1¶
(2021-04-22)
Updated some unittests and fixed some linting issues
Minor improvements in documentation
1.2.0¶
(2021-04-22)
Workers can be kept alive in between consecutive map calls
Setting CPU affinity is no longer restricted to Linux platforms
README updated to use RST format for better compatibility with PyPI
Added classifiers to the setup file
1.1.3¶
(2020-09-03)
First public release on Github and PyPi
1.1.2¶
(2020-08-27)
Added missing typing information
Updated some docstrings
Added license
1.1.1¶
(2020-02-19)
Changed
collections.Iterable
tocollections.abc.Iterable
due to deprecation of the former
1.1.0¶
(2019-10-31)
Removed custom progress bar support to fix Jupyter notebook support
New
progress_bar_position
parameter is now available to set the position of the progress bar when using nested worker poolsScreen resizing is now supported when using a progress bar
1.0.0¶
(2019-10-29)
Added the MPIRE dashboard
Added
threading
as a possible backendProgress bar handling now occurs in a separate process, instead of a thread, to improve responsiveness
Refactoring of code and small bug fixes in error handling
Removed deprecated functionality
0.9.0¶
(2019-03-11)
Added support for using different start methods (‘spawn’ and ‘forkserver’) instead of only the default method ‘fork’
Added optional support for using dill in multiprocessing by utilizing the multiprocess library
The
mpire.Worker
class is no longer directly available
0.8.1¶
(2019-02-06)
Fixed bug when process would hang when progress bar was set to
True
and an empty iterable was provided
0.8.0¶
(2018-11-01)
Added support for worker state
Chunking numpy arrays is now done using numpy slicing
mpire.WorkerPool.map()
now supports automatic concatenation of numpy array output
0.7.2¶
(2018-06-14)
Small bug fix when not passing on a boolean or
tqdm
object for theprogress_bar
parameter
0.7.1¶
(2017-12-20)
You can now pass on a dictionary as an argument which will be unpacked accordingly using the
**
-operator.New function
mpire.utils.make_single_arguments()
added which allows you to create an iterable of single argument tuples out of an iterable of single arguments
0.7.0¶
(2017-12-11)
mpire.utils.chunk_tasks()
is now available as a public functionChunking in above function and map functions now accept a
n_splits
parameteriterable_of_args
in map functions can now contain single values instead of only iterablestqdm
is now available from the MPIRE package which automatically switches to the Jupyter/IPython notebook widget when availableSmall bugfix in cleaning up a worker pool when no map function was called
0.6.2¶
(2017-11-07)
Fixed a second bug where the main process could get unresponsive when an exception was raised
0.6.1¶
(2017-11-06)
Fixed bug where sometimes exceptions fail to pickle
Fixed a bug where the main process could get unresponsive when an exception was raised
Child processes are now cleaned up in parallel when an exception was raised
0.6.0¶
(2017-11-03)
restart_workers
parameter is now deprecated and will be removed from v1.0.0Progress bar functionality added (using tqdm)
Improved error handling in user provided functions
Fixed randomly occurring
BrokenPipeErrors
and deadlocks
0.5.1¶
(2017-10-12)
Child processes can now also be pinned to a range of CPUs, instead of only a single one. You can also specify a single CPU or range of CPUs that have to be shared between all child processes
0.5.0¶
(2017-10-06)
Added CPU pinning.
Default number of processes to spawn when using
n_jobs=None
is now set to the number of CPUs available, instead ofcpu_count() - 1
0.4.0¶
(2017-10-05)
Workers can now be started as normal child processes (non-deamon) such that nested
mpire.WorkerPool
s are possible
0.3.0¶
(2017-09-15)
The worker ID can now be passed on the function to be executed by using the
mpire.WorkerPool.pass_on_worker_id()
functionRemoved the use of
has_return_value_with_shared_objects
when usingmpire.WorkerPool.set_shared_objects()
. MPIRE now handles both cases out of the box
0.2.0¶
(2017-06-27)
Added docs
0.1.0¶
First release