What happened when invoking locals() in python3? - python-3.7

Question

I'm using Python3.7.2
The code which makes me confused looks like this:
def foo(cond):
if cond:
z = 1
return z
else:
loc = locals()
x = 1
locals()['y'] = 2
exec("z = x + y")
print(loc)
print(locals())
foo(False)
Here is the result it prints:
{'cond': False, 'loc': {...}, 'x': 1, 'y': 2, 'z': 3}
{'cond': False, 'loc': {...}, 'x': 1, 'y': 2}
The exec function changes the local namespace, but when I execute locals(), variable z disappeared.
However, if I change the code like this:
def foo(cond):
if cond:
return 1
else:
loc = locals()
x = 1
locals()['y'] = 2
exec("z = x + y")
print(loc)
print(locals())
foo(False)
The result will be:
{'cond': False, 'loc': {...}, 'x': 1, 'y': 2, 'z': 3}
{'cond': False, 'loc': {...}, 'x': 1, 'y': 2, 'z': 3}
I'm really confused. :(

Answer 1

In short: the dictionary returned by locals() is only a copy of the true local variables—which are stored in an array, not in a dictionary. You can manipulate the locals() dictionary, add or remove entries at will. Python does not care since that is merely a copy of the local variables it works with. However, each time you call locals(), Python copies the current values of all local variables into the dictionary, replacing anything else you or exec() put in before. So, if z is a proper local variable (as in the first, but not second version of your code), locals() will restore its current value, which happens to be "not set" or "undefined".
Indeed, conceptually Python stores local variables in a dictionary, which can be retrieved through the function locals(). However, internally, it really is two arrays. The names of the locals are an attribute of the code itself and therefore stored in the code object as code.co_varnames. Their values, however, are stored in the frame as fast (which is inaccessible from Python code). You can find more on co_varnames etc. in inspect and dis modules, respectively.
The built-in function locals() actually calls a method fastToLocals() on the current frame object. If we were to write the frame with its fastToLocals method in Python, it might look like this (I am leaving out a lot of details here):
class frame:
def __init__(self, code):
self.__locals_dict = None
self.f_code = code
self.__fast = [0] * len(code.co_varnames)
def fastToLocals(self):
if self.__locals_dict is None:
self.__locals_dict = {}
for (key, value) in zip(self.f_code.co_varnames, self.__fast):
if value is not null: # Remember: it's actually C-code
self.__locals_dict[key] = value
else:
del self.__locals_dict[key] # <- Here's the magic
return self.__locals_dict
def locals():
frame = sys._getframe(1)
frame.fastToLocals()
In plain English: the dictionary you get when calling locals() is cached in the current frame. Calling locals() repeatedly will give you the same dictionary (__locals_dict in the code above). However, each time, you call locals(), the frame will update the entries in this dictionary with the current values that local variables have at that time. As noted here, when a local variable is not set, the entry in the __locals_dict is deleted. And that's what it is all about.
In your first version of the code, the third line says z = 1, which makes z a local variable. In the else branch, however, that local variable z is not set (i.e. would raise a UnboundLocalError) and is therefore removed from the __locals_dict. In the second version of your code, there is no assignment to z, so it is not a local variable and the locals() function does not care about it.
The set of local variables is actually fixed by the compiler. This means that you cannot add or remove local variables at runtime. This is a problem for exec(), as you are clearly using exec() here to define a local variable z. Python's way out is to say that exec() stores z as a local variable in the _locals_dict dictionary, although it cannot put it into the arrays behind this dictionary.
Conclusion: The values of local variables are actually stored in an array, not in the dictionary returned by locals(). When calling locals(), the dictionary is updated with the true current values taken from the dictionary. exec(), on the other hand, can only store its local variables in the dictionary, not in the array. If z is a proper local variable, it will be overwritten by a call to locals() with its current value (which is "does not exist"). If z is not a proper local variable, it will stay in the dictionary untouched.
Python's specifications say that any variable that you assign to inside a function is a local variable. You can change this default through the use of global or nonlocal, of course. But as soon as you write z = ..., z becomes a local variable. If, on the other hand, you only have x = z in your code, the compiler assumes that z is rather a global variable. That's why the line z = 1 makes all the difference: it marks z as a local variable that receives its place in the fast array.
Concerning exec(): in general, there is no way the compiler could really know what code exec() is going to execute (in your case with a string literal it could, but since this is a are rare and uninteresting case, it never tries, anyway). The compiler can therefore not know what local (or global) variables the code in exec() might access, and cannot include that in its calculation of how large the array of local variables should be.
By the way: that local variables are managed in arrays instead of proper dictionaries is the reason why there might be raised an UnboundLocalError instead of an NameError. In case of local variables, the Python interpreter actually recognises the name and knows exactly where its value is to be found (inside the fast array mentioned above). But if that entry is null, it cannot return something meaningful and therefore raises the UnboundLocalError. For global names, however, Python goes really searched for a variable with the given name in the globals and built-in dictionaries. In that case, a variable of the requested name might truly not exist.

Answer 2

Compare your code with the following:
def foo(cond):
if cond:
return 1
else:
loc = locals().copy()
# ——————————————^
x = 1
locals()['y'] = 2
exec("z = x + y")
print(loc)
print(locals())
foo(False)
It produces
{'cond': False}
{'cond': False, 'loc': {'cond': False}, 'x': 1, 'y': 2, 'z': 3}
as you might expect. The reason is that locals() is a reference, not a value.