neurotools.jobs.parallel module
Parallel tools
- neurotools.jobs.parallel.parmap(f, problems, leavefree=1, debug=False, verbose=False, show_progress=True)[source]
Parallel implmenetation of map using multiprocessing
- Parameters:
f (function to apply, takes one argument and returns a tuple) – (i,result) where i is the index into the problems
problems (list of arguments to f to evaulate in parallel)
leavefree (number of cores to leave open)
debug (if True, will run on a single core so exceptions get) – handeled correctly
verbose (set to True to print out detailed logging information)
- Return type:
list of results
- neurotools.jobs.parallel.pararraymap(function, problems, debug=False)[source]
Parmap wrapper for common use-case with Numpy
- neurotools.jobs.parallel.eval_from_cached(fingerprint, args, verbose=False)[source]
Attempts to recompile a function from its source code, and store the compiled result in the global dictionary __indirect_eval_fallback_cache. It then attempts to call the function on the provided arguments.
This is one alternative way to pass newly-created functions to the worker pool in parallel processing, if the usual built-in routines fail.
This can fail if the function closes over additional variables that were not present at the time the worker-pool was initialized.
It can yield undefined results if the function uses mutable variables in the global scope.
This cannot rebuild lambda expressions from source, but can use them if they are stored in the global __indirect_eval_fallback_cache ahead of time.
- neurotools.jobs.parallel.parimap(f, jobs, debug=False, force_cached=False, force_fallback=False, allow_fallback=True, verbose=False, show_progress=True)[source]
Parallel map routine.
In some cases and on some systems, user-defind functions don’t work with the multiprocessing library.
This happens when the system is unable to “pickle” functions which were defined after the worker pool was initiatlized.
There are two workarounds for this:
You can attempt to send the function source-code to the workers, and rebuild from source within the workers. However, this is risky for two reasons. (a) If your funcion closes over globals which were not defined at the time the worker-pool was launched, these globals will be missing and prevent re-compilation of the function. (b) Any mutable variables that the function closes over might have a different state in the worker threads (as I understand it).
You can also ensure that there is a pointer to the defined funcion in the global workspace. Here, we store function pointers in the dictionary ‘__indirect_eval_fallback_cache’. Then, one can re-launch the worker pool, and each worker will gain access to the compiled function via the inhereted global scope (as I understand it).
If normal parallel map fails, this routine will first try (1), and then (2).
- neurotools.jobs.parallel.close_pool(context=None, verbose=False)[source]
Safely halts the worker-pool. If worker threads are stuck, then this function will hang. On the other hand, it avoids doing anything violent to close workers.
- Parameters:
leavefree (int, default 1) – How many cores to “leave free”; The pool size will be the number of system cores minus this value
context (python context, default None) – This context will be used for all workers in the pool
verbose (bool, default False) – Whether to print logging information.
- neurotools.jobs.parallel.reset_pool(leavefree=None, context=None, verbose=False)[source]
Safely halts and restarts the worker-pool. If worker threads are stuck, then this function will hang. On the other hand, it avoids doing anything violent to close workers.
- Parameters:
leavefree (int, default 1) – How many cores to “leave free”; The pool size will be the number of system cores minus this value
context (python context, default None) – This context will be used for all workers in the pool
verbose (bool, default False) – Whether to print logging information.
- neurotools.jobs.parallel.parallel_error_handling(f)[source]
We can’t really use exception handling in parallel calls. This is a wrapper to just catch errors to prevent them from propagating up.