Reference/API#
- schwimmbad.choose_pool(mpi=False, processes=1, **kwargs)#
Choose between the different pools given options from, e.g., argparse.
- Parameters:
mpi (bool, optional) – Use the MPI processing pool,
MPIPool
. By default,False
, will use theSerialPool
.processes (int, optional) – Use the multiprocessing pool,
MultiPool
, with this number of processes. By default,processes=1
, will use theSerialPool
.**kwargs (
Any
) – Any additional kwargs are passed in to the pool class initializer selected by the arguments.
- Return type:
- class schwimmbad.SerialPool(**_)#
A serial pool that wraps the built-in
map()
function.- Parameters:
None –
- map(func, iterable, callback=None)#
A wrapper around the built-in
map()
function to provide a consistent interface with the otherPool
classes.- Parameters:
worker (callable) – A function or callable object that is executed on each element of the specified
tasks
iterable. This object must be picklable (i.e. it can’t be a function scoped within a function or alambda
function). This should accept a single positional argument and return a single object.tasks (iterable) – A list or iterable of tasks. Each task can be itself an iterable (e.g., tuple) of values or data to pass in to the worker function.
callback (callable, optional) – An optional callback function (or callable) that is called with the result from each worker run and is executed on the master process. This is useful for, e.g., saving results to a file, since the callback is only called on the master thread.
- Returns:
results
- Return type:
generator
- class schwimmbad.MultiPool(processes=None, initializer=None, initargs=(), **kwargs)#
A modified version of
multiprocess.pool.Pool
that has better behavior with regard toKeyboardInterrupts
in themap()
method.NOTE: This is no longer built off of the standard library
multiprocessing.pool.Pool
– this uses the version from multiprocess, which uses dill to pickle objects instead of the standard library pickle.- Parameters:
processes (int, optional) – The number of worker processes to use; defaults to the number of CPUs.
initializer (callable, optional) – If specified, a callable that will be invoked by each worker process when it starts.
initargs (iterable, optional) – Arguments for
initializer
; it will be called asinitializer(*initargs)
.kwargs – Extra arguments passed to the
multiprocess.pool.Pool
superclass.
- class schwimmbad.MPIPool(comm=None, use_dill=False)#
A processing pool that distributes tasks using MPI.
With this pool class, the master process distributes tasks to worker processes using an MPI communicator. This pool therefore supports parallel processing on large compute clusters and in environments with multiple nodes or computers that each have many processor cores.
This implementation is inspired by @juliohm in this module
- Parameters:
comm (
mpi4py.MPI.Comm
, optional) – An MPI communicator to distribute tasks with. IfNone
, this usesMPI.COMM_WORLD
by default.use_dill (Set True to use dill serialization. Default is False.) –
- class schwimmbad.JoblibPool(*args, **kwargs)#
A processing pool that distributes tasks using
joblib.Parallel
.This pool processes parallel tasks using the
Parallel
module of the joblib library. All arguments and keyword arguments are passed directly to the__init__
method of the Parallel object provided by joblib.