Dashboard¶
The dashboard allows you to see progress information from a browser. This is convenient when running scripts in a notebook or screen, if you want to share the progress information with others, or if you want to get real-time worker insight information.
The dashboard dependencies are not installed by default. See Dashboard for more information.
Starting the dashboard¶
You can start the dashboard programmatically:
from mpire.dashboard import start_dashboard
# Will return a dictionary with dashboard details
dashboard_details = start_dashboard()
print(dashboard_details)
which will print:
{'dashboard_port_nr': 8080,
'manager_host': 'localhost',
'manager_port_nr': 8081}
This will start a dashboard on your local machine on port 8080. When the port is already in use MPIRE will try the next,
until it finds an unused one. In the rare case that no ports are available up to port 8099 the function will raise an
OSError
. By default, MPIRE tries ports 8080-8100. You can override this range by passing on a custom range object:
dashboard_details = start_dashboard(range(9000, 9100))
The returned dictionary contains the port number that is ultimately chosen. It also contains information on how to connect to this dashboard remotely.
Another way of starting a dashboard is by using the bash script (this doesn’t work on Windows!):
$ mpire-dashboard
This will start a dashboard with the connection details printed on screen. It will say something like:
Starting MPIRE dashboard...
MPIRE dashboard started on http://localhost:8080
Server is listening on localhost:8098
--------------------------------------------------
The server part corresponds to the manager_host
and manager_port_nr
from the dictionary returned by
mpire.dashboard.start_dashboard()
. Similarly to earlier, a custom port range can be provided:
$ mpire-dashboard --port-range 9000-9100
The benefit of starting a dashboard this way is that your dashboard keeps running in case of errors in your script. You will be able to see what the error was, when it occurred and where it occurred in your code.
Connecting to an existing dashboard¶
If you have started a dashboard elsewhere, you can connect to it using:
from mpire.dashboard import connect_to_dashboard
connect_to_dashboard(manager_port_nr=8081, manager_host='localhost')
Make sure you use the manager_port_nr
, not the dashboard_port_nr
in the examples above.
You can connect to an existing dashboard on the same, but also on a remote machine (if the ports are open). If
manager_host
is omitted it will fall back to using 'localhost'
.
Using the dashboard¶
Once connected to a dashboard you don’t need to change anything to your code. When you have enabled the use of
a progress bar in your map
call the progress bar will automatically register itself to the dashboard server and show
up, like here:
from mpire import WorkerPool
from mpire.dashboard import connect_to_dashboard
connect_to_dashboard(8099)
def square(x):
import time
time.sleep(0.01) # To be able to show progress
return x * x
with WorkerPool(4) as pool:
pool.map(square, range(10000), progress_bar=True)
This will show something like:
You can click on a progress bar row to view details about the function that is called (which has already been done in the screenshot above).
It will let you know when a KeyboardInterrupt
signal was send to the running process:
or show the traceback information in case of an exception:
In case you have enabled Worker insights these insights will be shown real-time in the dashboard:
Click on the Insights (click to expand/collapse)
to either expand or collapse the insight details.
The dashboard will refresh automatically every 0.5 seconds.
Stack level¶
By default, the dashboard will show information about the function that is called and where it is called from. However,
in some cases where you have wrapped the function in another function, you might be less interested in the wrapper
function and more interested in the function that is calling this wrapper. In such cases you can use
mpire.dashboard.set_stacklevel()
to set the stack level. This is the number of levels in the stack to go back in
order to find the frame that contains the function that is invoking MPIRE. For example:
from mpire import WorkerPool
from mpire.dashboard import set_stacklevel, start_dashboard
class WorkerPoolWrapper:
def __init__(self, n_jobs, progress_bar=True):
self.n_jobs = n_jobs
self.progress_bar = progress_bar
def __call__(self, func, data):
with WorkerPool(self.n_jobs) as pool:
return pool.map(func, data, progress_bar=self.progress_bar)
def square(x):
return x * x
if __name__ == '__main__':
start_dashboard()
executor = WorkerPoolWrapper(4, progress_bar=True)
set_stacklevel(1) # default
results = executor(square, range(10000))
set_stacklevel(2)
results = executor(square, range(10000))
When you run this code you will see that the dashboard will show two progress bars. In both cases, the dashboard will
show the square
function as the function that is called. However, in the first case, it will show
return pool.map(func, data, progress_bar=self.progress_bar)
as the line where it is called from. In the second case,
it will show the results = executor(square, range(10000))
line.