Welcome to the MPIRE documentation!

MPIRE, short for MultiProcessing Is Really Easy, is a Python package for multiprocessing, but faster and more user-friendly than the default multiprocessing package. It combines the convenient map like functions of multiprocessing.Pool with the benefits of using copy-on-write shared objects of multiprocessing.Process (not supported for Windows), together with easy-to-use worker state, worker insights, and progress bar functionality.

Features

  • Faster execution than other multiprocessing libraries. See benchmarks.

  • Intuitive, Pythonic syntax

  • Multiprocessing with map/map_unordered/imap/imap_unordered functions

  • Easy use of copy-on-write shared objects with a pool of workers (copy-on-write is only available for start method fork, so it’s not supported on Windows)

  • Each worker can have its own state and with convenient worker init and exit functionality this state can be easily manipulated (e.g., to load a memory-intensive model only once for each worker without the need of sending it through a queue)

  • Progress bar support using tqdm

  • Progress dashboard support

  • Worker insights to provide insight into your multiprocessing efficiency

  • Graceful and user-friendly exception handling

  • Timeouts, including for worker init and exit functions

  • Automatic task chunking for all available map functions to speed up processing of small task queues (including numpy arrays)

  • Adjustable maximum number of active tasks to avoid memory problems

  • Automatic restarting of workers after a specified number of tasks to reduce memory footprint

  • Nested pool of workers are allowed when setting the daemon option

  • Child processes can be pinned to specific or a range of CPUs

  • Optionally utilizes dill as serialization backend through multiprocess, enabling parallelizing more exotic objects, lambdas, and functions in iPython and Jupyter notebooks.

MPIRE has been tested on both Linux and Windows. There are a few minor known caveats for Windows users, which can be found at Windows.

Contents