Timeouts¶
Timeouts can be set separately for the target, worker_init
and worker_exit
functions. When a timeout has been
set and reached, it will throw a TimeoutError
:
# Will raise TimeoutError, provided that the target function takes longer
# than half a second to complete
with WorkerPool(n_jobs=5) as pool:
pool.map(time_consuming_function, range(10), task_timeout=0.5)
# Will raise TimeoutError, provided that the worker_init function takes longer
# than 3 seconds to complete or the worker_exit function takes longer than
# 150.5 seconds to complete
with WorkerPool(n_jobs=5) as pool:
pool.map(time_consuming_function, range(10), worker_init=init, worker_exit=exit_,
worker_init_timeout=3.0, worker_exit_timeout=150.5)
Use None
(=default) to disable timeouts.
imap
and imap_unordered
¶
When you’re using one of the lazy map functions (e.g., imap
or imap_unordered
) then an exception will only be
raised when the function is actually running. E.g. when executing:
with WorkerPool(n_jobs=5) as pool:
results = pool.imap(time_consuming_function, range(10), task_timeout=0.5)
this will never raise. This is because imap
and imap_unordered
return a generator object, which stops executing
until it gets the trigger to go beyond the yield
statement. When iterating through the results, it will raise as
expected:
with WorkerPool(n_jobs=5) as pool:
results = pool.imap(time_consuming_function, range(10), task_timeout=0.5)
for result in results:
...
Threading¶
When using threading
as start method MPIRE won’t be able to interrupt certain functions, like time.sleep
.