Dill¶
For some functions or tasks it can be useful to not rely on pickle, but on some more powerful serialization backends
like dill. dill
isn’t installed by default. See Dill for more information on installing the dependencies.
One specific example where dill
shines is when using start method spawn
(the default on Windows) in combination
with iPython or Jupyter notebooks. dill
enables parallelizing more exotic objects like lambdas and functions defined
in iPython and Jupyter notebooks. For all benefits of dill
, please refer to the dill documentation.
Once the dependencies have been installed, you can enable it using the use_dill
flag:
with WorkerPool(n_jobs=4, use_dill=True) as pool:
...
Note
When using dill
it can potentially slow down processing. This is the cost of having a more reliable and
powerful serialization backend.