Progress bar¶
Progress bar support is added through the tqdm package (installed by default when installing MPIRE). The most easy way
to include a progress bar is by enabling the progress_bar
flag in any of the map
functions:
with WorkerPool(n_jobs=4) as pool:
pool.map(task, range(100), progress_bar=True)
This will display a basic tqdm
progress bar displaying the time elapsed and remaining, number of tasks completed
(including a percentage value) and the speed (i.e., number of tasks completed per time unit).
Progress bar style¶
You can switch to a different progress bar style by changing the progress_bar_style
parameter. For example, when
you require a notebook widget use 'notebook'
as the style:
with WorkerPool(n_jobs=4) as pool:
pool.map(task, range(100), progress_bar=True, progress_bar_style='notebook')
The available styles are:
None
: use the default style (='std'
, see below)'std'
: use the standardtqdm
progress bar'rich'
: use the rich progress bar (requires therich
package to be installed, see Rich progress bars)'notebook'
: use the Jupyter notebook widget'dashboard'
: use only the progress bar on the dashboard
When in a terminal and using the 'notebook'
style, the progress bar will behave weirdly. This is not recommended.
Note
If you run into problems with getting the progress bar to work in a Jupyter notebook (with 'notebook'
style),
have a look at Progress bar issues with Jupyter notebooks.
Changing the default style¶
You can change the default style by setting the mpire.tqdm_utils.PROGRESS_BAR_DEFAULT_STYLE
variable:
import mpire.tqdm_utils
mpire.tqdm_utils.PROGRESS_BAR_DEFAULT_STYLE = 'notebook'
Progress bar options¶
The tqdm
progress bar can be configured using the progress_bar_options
parameter. This parameter accepts a
dictionary with keyword arguments that will be passed to the tqdm
constructor.
Some options in tqdm
will be overwritten by MPIRE. These include the iterable
, total
and leave
parameters. The iterable
is set to the iterable passed on to the map
function. The total
parameter is set to
the number of tasks to be completed. The leave
parameter is always set to True
. Some other parameters have a
default value assigned to them, but can be overwritten by the user.
Here’s an example where we change the description, the units, and the colour of the progress bar:
with WorkerPool(n_jobs=4) as pool:
pool.map(some_func, some_data, progress_bar=True,
progress_bar_options={'desc': 'Processing', 'unit': 'items', 'colour': 'green'})
For a complete list of available options, check out the tqdm docs.
Progress bar position¶
You can easily print a progress bar on a different position on the terminal using the position
parameter of
tqdm
, which facilitates the use of multiple progress bars. Here’s an example of using multiple progress bars using
nested WorkerPools:
def dispatcher(worker_id, X):
with WorkerPool(n_jobs=4) as nested_pool:
return nested_pool.map(task, X, progress_bar=True,
progress_bar_options={'position': worker_id + 1})
def main():
with WorkerPool(n_jobs=4, daemon=False, pass_worker_id=True) as pool:
pool.map(dispatcher, ((range(x, x + 100),) for x in range(100)), iterable_len=100,
n_splits=4, progress_bar=True)
main()
We use worker_id + 1
here because the worker IDs start at zero and we reserve position 0 for the progress bar of
the main WorkerPool (which is the default).
It goes without saying that you shouldn’t specify the same progress bar position multiple times.
Note
When using the rich
progress bar style, the position
parameter cannot be used. An exception will be raised
when trying to do so.
Note
Most progress bar options are completely ignored when in a Jupyter/IPython notebook session or in the MPIRE dashboard.