neurotools.jobs.closure module

Code dealing with closures in Python.

The default way that Python handles closures creates problems

1. Late binding can cause suprising errors

2. Closing over mutable state can cause surprising errors

3. Mutable default arguments can causing surprising errors

4. Closing over external state that may change across runs can cause surprising errors, as the same arguments may return different values

5. Closing over extrenal state makes safe caching difficult, as we have to guarantee that the arguments, function source, subroutines, and closed-over state are all IDENTICAL before we can trust a cached value.

This module defines some decorators that improve safety, hopefully automatically detecting functions that are likele to cause trouble.

neurotools.jobs.closure.is_immutable(x)

Checks whether an object is immutable. It must be composed of the built-in types str, int, long, bool, float, and tuple. If it is a tuple, each of its elements must be immutable

neurotools.jobs.closure.is_probably_safe(x)

Objects are probably “safe” (unlikly to change) if they are – immutable – functions – modules Obviously, the latter two may change, but in practice it is likely ok. Still, risky!

neurotools.jobs.closure.csv(x)

neurotools.jobs.closure.inspect_function_closure(f)

Checks if a function is “safe”

Formally this is quite tricky as not all closures can be avoided. Functions MUST close over their namespace in some sense, in order to access subroutines. This cannot be avoided, and changes in subroutines may cause changes in behavior.

neurotools.jobs.closure.verify_function_closure(f)

Checks if a function is “safe” (likely immutable)

Formally this is quite tricky as not all closures can be avoided. Functions MUST close over their namespace in order to access subroutines later, as Python is by design late-binding. This cannot be avoided, and changes in subroutines may cause changes in behavior.

Levels of safety:

Safe:

Function references only other functions. All state is passed via arguments. Hash values of subroutines are incorporated into the hash value for this function.

Risky:

Function may close over state and reference globals, as long as those states / globals are themselves immutable. This is risky as values may still change across invokation. In theory we could snapshot the dependent state and incorporate that into the call signature.

Unsafe:

Function contains mutable defaults. Function references mutable globals or closes over mutable state. Unsafe conditions will cause a ValueError

neurotools.jobs.closure.get_subroutines(f)

Returns things that resemble subroutines of function f. These are all callable objets that are either named or bound via closures. Return value is a set of functions. Non-function callable subroutines will cause an error.