Welcome to the MPIRE documentation!

MPIRE, short for MultiProcessing Is Really Easy, is a Python package for multiprocessing, but faster and more user-friendly than the default multiprocessing package. It combines the convenient map like functions of multiprocessing.Pool with the benefits of using copy-on-write shared objects of multiprocessing.Process, together with easy-to-use worker state, worker insights, and progress bar functionality.

Features

  • Faster execution than other multiprocessing libraries. See benchmarks.

  • Intuitive, Pythonic syntax

  • Multiprocessing with map/map_unordered/imap/imap_unordered functions

  • Easy use of copy-on-write shared objects with a pool of workers (copy-on-write is only available for start methods fork and threading)

  • Each worker can have its own state and with convenient worker init and exit functionality this state can be easily manipulated (e.g., to load a memory-intensive model only once for each worker without the need of sending it through a queue)

  • Progress bar support using tqdm

  • Progress dashboard support

  • Worker insights to provide insight into your multiprocessing efficiency

  • Graceful and user-friendly exception handling

  • Automatic task chunking for all available map functions to speed up processing of small task queues (including numpy arrays)

  • Adjustable maximum number of active tasks to avoid memory problems

  • Automatic restarting of workers after a specified number of tasks to reduce memory footprint

  • Nested pool of workers are allowed when setting the daemon option

  • Child processes can be pinned to specific or a range of CPUs

  • Optionally utilizes dill as serialization backend through multiprocess, enabling parallelizing more exotic objects, lambdas, and functions in iPython and Jupyter notebooks.

MPIRE has been tested on both Linux and Windows. There are a few minor known caveats for Windows users, which can be found at Windows.

Contents