Welcome to the MPIRE documentation!

MPIRE, short for MultiProcessing Is Really Easy, is a Python package for multiprocessing, but faster and more user-friendly than the default multiprocessing package. It combines the convenient map like functions of multiprocessing.Pool with the benefits of using copy-on-write shared objects of multiprocessing.Process, together with easy-to-use worker state, worker insights, and progress bar functionality.

Features

  • Multiprocessing with map/map_unordered/imap/imap_unordered functions

  • Easy use of copy-on-write shared objects with a pool of workers

  • Each worker can have its own state and with convenient worker init and exit functionality this state can be easily manipulated (e.g., to load a memory-intensive model only once for each worker without the need of sending it through a queue)

  • Progress bar support using tqdm

  • Progress dashboard support

  • Graceful and user-friendly exception handling

  • Worker insights gives you insight in your multiprocessing efficiency

  • Automatic task chunking for all available map functions to speed up processing of small task queues (including numpy arrays)

  • Adjustable maximum number of active tasks to avoid memory problems

  • Automatic restarting of workers after a specified number of tasks to reduce memory footprint

  • Nested pool of workers are allowed when setting the daemon option

  • Child processes can be pinned to specific or a range of CPUs

  • Multiple process start methods available, including: fork (default), forkserver, spawn, and threading

  • Optionally uses dill as serialization backend through multiprocess, enabling parallelizing more exotic functions and objects

Contents