Reference/API#

schwimmbad.choose_pool(mpi=False, processes=1, **kwargs)#

Choose between the different pools given options from, e.g., argparse.

Parameters:
  • mpi (bool, optional) – Use the MPI processing pool, MPIPool. By default, False, will use the SerialPool.

  • processes (int, optional) – Use the multiprocessing pool, MultiPool, with this number of processes. By default, processes=1, will use the SerialPool.

  • **kwargs (Any) – Any additional kwargs are passed in to the pool class initializer selected by the arguments.

Return type:

Union[MPIPool, MultiPool, SerialPool]

class schwimmbad.SerialPool(**_)#

A serial pool that wraps the built-in map() function.

Parameters:

None

map(func, iterable, callback=None)#

A wrapper around the built-in map() function to provide a consistent interface with the other Pool classes.

Parameters:
  • worker (callable) – A function or callable object that is executed on each element of the specified tasks iterable. This object must be picklable (i.e. it can’t be a function scoped within a function or a lambda function). This should accept a single positional argument and return a single object.

  • tasks (iterable) – A list or iterable of tasks. Each task can be itself an iterable (e.g., tuple) of values or data to pass in to the worker function.

  • callback (callable, optional) – An optional callback function (or callable) that is called with the result from each worker run and is executed on the master process. This is useful for, e.g., saving results to a file, since the callback is only called on the master thread.

Returns:

results

Return type:

generator

class schwimmbad.MultiPool(processes=None, initializer=None, initargs=(), **kwargs)#

A modified version of multiprocess.pool.Pool that has better behavior with regard to KeyboardInterrupts in the map() method.

NOTE: This is no longer built off of the standard library multiprocessing.pool.Pool – this uses the version from multiprocess, which uses dill to pickle objects instead of the standard library pickle.

Parameters:
  • processes (int, optional) – The number of worker processes to use; defaults to the number of CPUs.

  • initializer (callable, optional) – If specified, a callable that will be invoked by each worker process when it starts.

  • initargs (iterable, optional) – Arguments for initializer; it will be called as initializer(*initargs).

  • kwargs – Extra arguments passed to the multiprocess.pool.Pool superclass.

class schwimmbad.MPIPool(comm=None, use_dill=False)#

A processing pool that distributes tasks using MPI.

With this pool class, the master process distributes tasks to worker processes using an MPI communicator. This pool therefore supports parallel processing on large compute clusters and in environments with multiple nodes or computers that each have many processor cores.

This implementation is inspired by @juliohm in this module

Parameters:
  • comm (mpi4py.MPI.Comm, optional) – An MPI communicator to distribute tasks with. If None, this uses MPI.COMM_WORLD by default.

  • use_dill (Set True to use dill serialization. Default is False.) –

class schwimmbad.JoblibPool(*args, **kwargs)#

A processing pool that distributes tasks using joblib.Parallel.

This pool processes parallel tasks using the Parallel module of the joblib library. All arguments and keyword arguments are passed directly to the __init__ method of the Parallel object provided by joblib.