You already write Python every day. This is the language underneath the one you use — the object model every feature is built from, read from the bottom up. Each chapter opens on a result that shouldn't happen, then shows the machinery that explains it: why a circular import fails on one line and survives on the next, why super() calls a class you never named, why a threaded counter that "should" race doesn't and the one that shouldn't does. It all runs on CPython 3.11.15, and every number and error message here is real output, not a description of one. The title is meant literally, so the first chapter starts where it has to: everything is an object.
Everything is an object; a name points at one
Ask Python what the type of type is.
>>> type(type)
<class 'type'>type is its own type. That is not a riddle; it is the shape of the whole language. In Python, object is not a category that some of your values fall into and others escape. Every value is one: the integer 3, the function you defined this morning, the class it belongs to, the module it lives in, and type itself.
An object is a thing in memory with three properties — an identity (a fixed address for its lifetime), a type (which decides what it can do), and a value. Functions have all three, so you can treat one as data: store it, pass it, and hang attributes on it.
>>> def greet(): pass
>>> greet.calls = 0
>>> greet.calls, greet.__class__.__name__
(0, 'function')greet is an object of type function, and greet.calls = 0 sets an attribute on it the same way you would on anything else. Classes are objects too. int is an instance of object, and it is also itself a type:
>>> isinstance(int, object), isinstance(int, type)
(True, True)Hold onto this. When a later chapter says that len(x) is a method call on the type, and that a class statement is a function call that builds a class object, they are cashing out this one fact: there is no privileged layer of not-objects underneath. It is objects the whole way down.
So if 3 is an object, what is the x in x = 3?
Not a box holding 3. A name is a key in a namespace, bound to an object that lives elsewhere. A namespace is a mapping from names to objects, and in Python it is a literal dictionary. A module's global namespace is a dict — the module's __dict__:
>>> import sys
>>> globals() is sys.modules['__main__'].__dict__
Trueglobals() does not describe the namespace; it hands you the actual dict the interpreter reads and writes. Assignment is dictionary assignment: x = 3 sets the key 'x' to point at the object 3. This is the seam a later chapter needs — a module is a namespace, importing writes names into its dict, and a half-run import is a half-filled dict.
Two names can point at the same object, and that splits assignment into two different acts. Predict the last line before you read it:
>>> a = [1, 2, 3]
>>> b = a
>>> a.append(4)
>>> b
[1, 2, 3, 4]
>>> a = [9]
>>> b
[1, 2, 3, 4]b = a bound b to the same list object as a. Then a.append(4) mutated that object, and because b points at it, b sees the fourth element. But a = [9] rebound a to a brand-new list; it never touched the old one, so b still holds [1, 2, 3, 4].
Rebinding moves a name to a different object. Mutation changes the object a name already points at. The distinction is invisible while a name has an object to itself, and it starts deciding whether your code is correct the moment two names share one.
is and == ask different questions
Two lists with the same contents. Are they equal? Are they the same?
>>> [1, 2] == [1, 2], [1, 2] is [1, 2]
(True, False)== asks whether two objects have the same value. is asks whether they are the same object — one thing in memory, not two things that happen to match. Equality dispatches to a method, __eq__, which a type defines however it likes; two lists are equal when their elements are. Identity compares the objects themselves and cannot be overridden. Equal value, two objects: == is True, is is False.
They come apart, and people reach for is on numbers anyway, because of a half-remembered rule about caching. The rule is real. Testing it naively gives you nonsense, because a second mechanism sits in front of it. Predict this:
>>> a = 256; b = 256
>>> a is b
True
>>> a = 257; b = 257
>>> a is b
TrueSo the cache runs to at least 257? No. Force Python to build the integers at runtime instead of writing them as literals, and the real boundary appears:
>>> int("256") is int("256")
True
>>> int("257") is int("257")
FalseTwo mechanisms are in play, and they are worth separating because they fail differently.
The first is the small-integer cache. CPython builds the integer objects from −5 to 256 once at startup and hands out the same object every time one of them is needed. int("257") is the first integer past the top of that range, so each call mints a fresh object and the two are not identical. int("-6") falls off the bottom and behaves the same way.
The second — the one that made the literal test lie — is constant folding. When the compiler sees the literal 257 twice in one block of code, it stores the value once and points both uses at it, so a = 257; b = 257 ends up comparing an object with itself. You can see the single shared copy sitting in the code object's constant table:
>>> def two(): return 257, 257
>>> two.__code__.co_consts
(None, (257, 257))The 257 is stored once. Interning is the same trick for short strings: identifier-like literals are stored once, so a string built at runtime is a different object from the literal that spells it.
Both are CPython implementation details, not language guarantees. Another interpreter may draw the boundary elsewhere or not cache at all. Never write logic that leans on two equal values being identical.
The interpreter agrees with you
Modern CPython pushes back on this directly. Write is against a literal and 3.11 warns at compile time:
>>> "name" is "name"
<stdin>:1: SyntaxWarning: "is" with a literal. Did you mean "=="?
TrueThe True is the interning coincidence; the warning is the interpreter telling you the question was wrong.
So when is is the right question? When identity is what you actually mean: x is None, and comparison against a unique sentinel object you made to mean "nothing was passed." None is a singleton — one object for the whole program — so x is None is exact and cheap. For numbers and strings, use ==.
Identity and equality also constrain each other. Define __eq__ on a class and Python silently makes its instances unhashable:
>>> class Money:
... def __init__(self, cents): self.cents = cents
... def __eq__(self, other): return self.cents == other.cents
...
>>> {Money(100)}
TypeError: unhashable type: 'Money'
>>> Money.__hash__ is None
TrueDefining __eq__ set __hash__ to None. Behind it is a contract: two objects that compare equal must hash equal, or they would land in different buckets of a set or dict and the container would quietly lose track of them. Python cannot invent a hash consistent with your new __eq__, so instead of letting you build a broken set it withdraws hashing until you supply __hash__ yourself. A frozen dataclass, later in the guide, writes both together and stays consistent.
One last place identity bites, through the aliasing from the previous chapter. A default argument is evaluated once, when the function is defined — not on each call:
>>> def append_to(item, target=[]):
... target.append(item); return target
...
>>> append_to(1)
[1]
>>> append_to(2)
[1, 2]
>>> append_to(3)
[1, 2, 3]Every call that omits target shares the one list built at definition time. It is the same two-names-one-object aliasing as before, hidden inside a signature. The fix uses the sentinel you just met: default to None, and build a fresh list inside the body when you find it.
def append_to(item, target=None):
if target is None:
target = []
target.append(item)
return targetA closure captures the variable, not its value
Here is a function that prints a module-level counter, then bumps it. Predict what happens when you call it.
count = 10
def bump():
print(count)
count = count + 1>>> bump()
UnboundLocalError: cannot access local variable 'count' where it is not associated with a valueThe print(count) fails, on a name that plainly exists at module level. The cause is one line below it. Because count is assigned somewhere in bump's body, count is local to the entire body — including the print above the assignment. The reference states this as a compile-time decision:
If a name is bound in a block, it is a local variable of that block, unless declared as
nonlocalorglobal.
A block is a stretch of code with its own namespace: a module, a function body, a class body. Python fixes a function's local names by scanning the whole body once before running any of it, and any name it sees assigned becomes local everywhere in that block. So print(count) looks up the local count, which has not been assigned yet, and you get UnboundLocalError rather than the global.
Runtime lookup then walks four scopes in order, LEGB: Local, then any Enclosing function scopes, then the module Global namespace, then Built-ins. bump never reaches the global count, because the compile-time rule already made the name local.
Two statements override that decision. global count binds and reads count in the module namespace; nonlocal count reaches outward to an enclosing function:
The
nonlocalstatement causes corresponding names to refer to previously bound variables in the nearest enclosing function scope.
That enclosing case is where closures live, and where the usual mental model is wrong. Predict the output:
>>> funcs = [lambda: i for i in range(3)]
>>> [f() for f in funcs]
[2, 2, 2]Not [0, 1, 2]. Every lambda returns 2. The common explanation — "each lambda remembers the i it was built with" — is exactly backwards. A closure does not capture the value of i at the moment the lambda is made. It captures the variable i, as a shared cell (a small box that the enclosing scope and every inner function point at), and reads the cell's current contents when the lambda is finally called. By then the loop has run to the end and left i at 2. All three lambdas share the one cell:
>>> funcs[0].__closure__
(<cell at 0x7ffaf7fc3f40: int object at 0xa2c108>,)
>>> funcs[0].__code__.co_freevars
('i',)co_freevars lists the names an inner function reads from an enclosing scope — its free variables — and __closure__ holds the cells they resolve to. Capture the value instead of the variable and the result changes:
>>> fixed = [lambda i=i: i for i in range(3)]
>>> [f() for f in fixed]
[0, 1, 2]A default argument is evaluated once, at definition time — the trap from the previous chapter, turned here into a tool. i=i copies the current value of i into the parameter, so each lambda carries its own number rather than a shared cell.
One scope breaks the LEGB chain, and it catches anyone who assumes nested code always sees outward. A class body is a namespace, but it is not an enclosing function scope, so code nested inside it cannot read the class's own names. A comprehension is exactly such nested code:
class Table:
cols = ["id", "name"]
width = 10
upper = [c.upper() for c in cols] # fine
padded = [c.ljust(width) for c in cols] # NameErrorNameError: name 'width' is not definedThe comprehension's body runs in a hidden function, and the reference is explicit about why:
The scope of names defined in a class block is limited to the class block; it does not extend to the code blocks of methods -- this includes comprehensions and generator expressions since they are implemented using a function scope.
The outermost iterable, cols, is evaluated directly in the class body, so that name resolves; everything inside the comprehension runs one scope removed and cannot see width. Keep the value in view another way — a module constant, an argument, or self inside a real method. The same rule returns when a later chapter builds classes by hand, because a class body running top to bottom is the same kind of namespace a module is.
@d means f = d(f)
Strip the syntax away and a decorator is function application. These two definitions produce the same normalize:
@count_calls
def normalize(s):
return s.strip().lower()def normalize(s):
return s.strip().lower()
normalize = count_calls(normalize)@count_calls runs count_calls(normalize) and rebinds the name normalize to whatever comes back. That is the entire feature. Everything else follows from two facts you already hold: a function is an object you can pass and return, and an inner function closes over the enclosing scope.
Here is count_calls, keeping a running tally in a closure cell:
def count_calls(fn):
calls = 0
def wrapper(*args, **kwargs):
nonlocal calls
calls += 1
wrapper.calls = calls
return fn(*args, **kwargs)
return wrapperwrapper is the object the name ends up bound to; fn and calls are its free variables, living in cells — the previous chapter's machinery, put to work. Calling the decorated function calls wrapper, which counts and then delegates:
>>> normalize(" Hi "); normalize("YES")
>>> normalize.calls
2Because the returned object really is wrapper, it wears wrapper's identity, and that is the first thing that trips people. Decorate a real function, one with a name and a docstring of its own:
@count_calls
def parse(line):
"parse a line"
return line.split()>>> parse.__name__, parse.__doc__
('wrapper', None)The decorated parse now reports its name as wrapper and has lost its docstring, which breaks tracebacks, help(), and anything that inspects functions. functools.wraps repairs it by copying the original's __name__, __doc__, and related attributes onto the wrapper:
def logged(fn):
@functools.wraps(fn)
def wrapper(*a, **k):
return fn(*a, **k)
return wrapperWrap a function in logged and its identity comes through intact:
@logged
def parse2(line):
"parse a line"
return line.split()>>> parse2.__name__, parse2.__doc__
('parse2', 'parse a line')Put @wraps on every wrapper you write.
A decorator that takes arguments needs one more layer, and the layers are easy to miscount. @retry(times=3) is not a decorator; it is a call that returns one. So it takes three nested functions: the outer accepts the arguments, the middle accepts the function, the inner does the work.
def retry(times):
def decorator(fn):
@functools.wraps(fn)
def wrapper(*a, **k):
...
return wrapper
return decorator@retry(times=3) evaluates retry(times=3) first, which returns decorator, and that decorates the function. One argument list buys one extra layer.
The @ applies to a class just as readily, since Cls = d(Cls) is as valid as f = d(f). That is exactly how @dataclass, later in the guide, does its work.
Little of this is code you should be writing often, because the standard library already ships the decorators people keep rebuilding. functools.lru_cache memoises a function by its arguments and reports its own hit rate:
>>> @functools.lru_cache(maxsize=None)
... def fib(n): return n if n < 2 else fib(n-1) + fib(n-2)
>>> fib(30); fib.cache_info()
CacheInfo(hits=28, misses=31, maxsize=None, currsize=31)Thirty-one distinct calls, twenty-eight served from the cache. functools.cached_property turns an expensive method into an attribute computed once per instance; functools.singledispatch turns a function into one that chooses an implementation by the type of its first argument:
>>> describe(7), describe([1, 2, 3]), describe("hi")
('int 7', 'list of 3', "object 'hi'")Reach for these before writing a decorator by hand. When you do write one, it is f = d(f) with @wraps on the wrapper, plus one more layer if it takes arguments of its own.
A circular import is a half-built dictionary
Two modules that need each other. orders.py builds an Order around a Customer, and customers.py places an Order:
# orders.py
print("orders: running")
from customers import Customer # triggers customers.py, which imports back
class Order:
def __init__(self, customer): self.customer = customer
print("orders: done")# customers.py
print("customers: running")
from orders import Order # orders is in sys.modules but half-built
class Customer:
def place(self): return Order(self)
print("customers: done")Import the first one and it detonates:
orders: running
customers: running
Traceback (most recent call last):
File "<string>", line 1, in <module>
File ".../orders.py", line 2, in <module>
from customers import Customer # triggers customers.py, which imports back
File ".../customers.py", line 2, in <module>
from orders import Order # orders is in sys.modules but half-built
ImportError: cannot import name 'Order' from partially initialized module 'orders' (most likely due to a circular import)The message names the mechanism, if you know how to read it: partially initialized module. Not "illegal," not "unsupported." Partially initialized. To see why that is the exact right phrase, you need to know what import does, and it is less than most people assume.
Importing a module for the first time is three steps:
- Python creates a new, empty module object.
- It inserts that object into
sys.modulesunder the module's name — before running a single line of the file. - It runs the file top to bottom, binding each name into the module's
__dict__as the defining line executes.
The reference is explicit that step 2 comes before step 3:
The module will exist in sys.modules before the loader executes the module code. This is crucial because the module code may (directly or indirectly) import itself; adding it to sys.modules beforehand prevents unbounded recursion in the worst case and multiple loading in the best.
That ordering is what stops a cycle looping forever. It is also what lets a cycle hand you a broken, half-built module. A module, from the first chapter, is an object whose namespace is a dict; step 3 fills that dict one line at a time. Interrupt it partway and the dict is real, cached, and incomplete.
This is a runtime problem, and that is the part that surprises people arriving from compiled languages. In C, a header is textually included and the linker resolves every symbol after all files are compiled; the order you include headers never decides whether a symbol exists. Python has no separate link step. import runs the file, and a name exists only after its line has executed. Whether the same two files import cleanly or blow up depends entirely on which line the interpreter is standing on when the cycle closes.
Here is the interpreter standing on that line:
import orders
│
├─ 1 create empty module object for orders
├─ 2 sys.modules['orders'] = <orders> inserted BEFORE any code runs
└─ 3 run orders.py, top to bottom
line 1 print("orders: running") orders namespace so far: { }
line 2 from customers import Customer
│
├─ 4 create empty module object for customers
├─ 5 sys.modules['customers'] = <customers>
└─ 6 run customers.py, top to bottom
line 1 print("customers: running")
line 2 from orders import Order
│
├─ 7 'orders' in sys.modules? yes (from step 2)
├─ 8 read attribute Order on <orders>
└─ 9 orders stopped at line 2 — Order undefined
✗ ImportError: cannot import name 'Order'
from partially initialized module 'orders'By step 8, orders is in sys.modules — inserted at step 2 — but has run only its first line, so its dict does not yet hold Order. from orders import Order demands that attribute now, and it is not there.
You can watch the half-built module directly. Here customers, mid-cycle, reaches into sys.modules and inspects the partial orders:
orders in sys.modules already? True
orders namespace so far: ['sys']
customers sees orders; has 'Order' yet? FalsePresent in the cache, one name in its dict, and the name the other module wants not defined yet.
That last fact is the entire difference between the two import forms. from orders import Order reads an attribute off the module object immediately, so it fails when that attribute has not been defined. import orders only binds the module object — which exists, empty but real — and puts off every orders.Order access to later. Switch both modules to plain import and defer the attribute access into a method, and the same cycle loads:
# orders.py
import customers # bind the module object, not a name inside it
class Order:
def __init__(self, customer): self.customer = customer
def describe(self): return f"order for {self.customer.name}"
# customers.py
import orders
class Customer:
def __init__(self, name): self.name = name
def place(self): return orders.Order(self) # resolved when place() is called>>> customers.Customer("Ada").place().describe()
'order for Ada'Both modules now run to completion, because nothing reads a cross-module attribute until place() is called, long after both dicts are full. That is the first of four reliable fixes, each of which works for a reason you can now state.
Restructure. Pull the shared piece into a third module that both import. No back-edge is left, so there is no cycle. This is the right answer more often than the clever ones.
Import the module, not the name — the fix above. Binding a partial module object is always safe; only the deferred module.attribute access needs the finished module.
Defer the import into the function body. Move the offending from customers import Customer inside the function that uses it, so it runs on call rather than at import time:
# orders.py
class Order:
def __init__(self, customer): self.customer = customer
def make_order(name):
from customers import Customer # runs on call, after both modules load
return Order(Customer(name))>>> orders.make_order("Ada").customer.name
'Ada'Guard a type-only import with TYPE_CHECKING. Sometimes the cycle exists only because two modules name each other's types in annotations. Normally an annotation like service: Service is evaluated when its function is defined, so Service has to be importable by then, which is exactly what forces the cycle. from __future__ import annotations turns every annotation in the module into an unevaluated string, so Service is never looked up at runtime and the import is pure dead weight there. typing.TYPE_CHECKING is False when the program runs and True only under a type checker, so you put the import behind it and it runs only where the annotation is actually read:
# models.py
from __future__ import annotations
from typing import TYPE_CHECKING
if TYPE_CHECKING:
from services import Service # only the type checker executes this
class Model:
def bind(self, service: Service) -> None:
self.service = service[models] loaded; TYPE_CHECKING at runtime = Falsemypy still resolves the annotation and reports Success; at runtime the import never runs, so the cycle never closes.
Two wrong models to retire. Circular imports are not illegal — they are legal and common, and what fails is narrower: reading a name off a module that has not defined it yet. And "move the import to the bottom of the file" is folklore that works only sometimes and for the wrong reason. The four levers above are the ones that hold.
Syntax is a method call on the type
You have a Basket and you want len(basket) to report how many items it holds. So you give the instance a __len__:
>>> b = Basket(["milk", "eggs"])
>>> b.__len__ = lambda: 99
>>> hasattr(b, "__len__")
True
>>> len(b)
TypeError: object of type 'Basket' has no len()The method is right there on the object — hasattr confirms it — and len(b) refuses to call it. Move the identical method onto the class, and it works:
>>> class Basket:
... def __init__(self, items): self.items = items
... def __len__(self): return len(self.items)
...
>>> len(Basket(["milk", "eggs"]))
2The rule underneath: syntax does not look for a method on your object. It looks on your object's type. len(x) runs type(x).__len__(x). The brackets in x[i] are type(x).__getitem__(x, i); x + y is type(x).__add__(x, y); with x is type(x).__enter__ and __exit__; a for loop over x starts at type(x).__iter__(x); x == y is type(x).__eq__. These special methods — the dunder ("double underscore") methods — are the data model, and every one of them is dispatched on the type. A __len__ sitting in an instance's own dict is invisible to len(), which never looks there.
The reference states it plainly:
For custom classes, implicit invocations of special methods are only guaranteed to work correctly if defined on an object's type, not in the object's instance dictionary.
There is a reason it works this way, and it is not arbitrary.
Why the type, not the instance
Dunders like __hash__ and __repr__ are implemented by every object, type objects included. If implicit lookup used the ordinary instance-first path, hash(int) would try to run int.__hash__() with no instance to hash and fail. Routing through the type sidesteps it — hash(int) becomes type(int).__hash__(int):
>>> int.__hash__() == hash(int)
TypeError: descriptor '__hash__' of 'int' object needs an argument
>>> type(int).__hash__(int) == hash(int)
TrueThe practical upshot: to make len(), [], +, in, or with work on your objects, define the dunder on the class. You cannot bolt one onto a single instance.
That leaves ordinary attribute access — basket.owner — which does go through the object, and is governed by two hooks people routinely confuse. Every attribute access runs through __getattribute__, the method that implements normal lookup. __getattr__ is a different thing: a fallback, invoked only when normal lookup fails to find the name.
>>> class Config:
... debug = True
... def __getattr__(self, name):
... return f"<default for {name!r}>"
...
>>> c = Config()
>>> c.debug
True
>>> c.timeout
"<default for 'timeout'>"c.debug exists, so __getattribute__ returns it and __getattr__ is never consulted. c.timeout does not exist, normal lookup fails, and only then does __getattr__ run. Use __getattr__ for defaults and lazy attributes. Override __getattribute__ only with real caution, because it intercepts every access, including the ones your own code depends on.
One consequence ties the two halves of this chapter together. The implicit dunder lookup that skips the instance also skips __getattribute__:
This method may still be bypassed when looking up special methods as the result of implicit invocation via language syntax or built-in functions.
So a __getattribute__ you write to trace attribute reads will never see len(x) reaching for __len__. Syntax takes the type-level path, straight past the per-instance machinery — which is exactly the machinery the next chapter opens up.
Attribute lookup is a cascade, and descriptors sit at the top
Two attributes, two opposite outcomes. First, a property you cannot overwrite from the instance:
>>> class Account:
... @property
... def balance(self): return "computed by property"
...
>>> a = Account()
>>> a.__dict__["balance"] = "smuggled into the instance dict"
>>> a.balance
'computed by property'The value is sitting in the instance's own dict — you put it there by hand — and a.balance ignores it and runs the property. Now the reverse: a method that an instance attribute does overwrite:
>>> class Service:
... def ping(self): return "method ping"
...
>>> s = Service()
>>> s.ping = lambda: "instance ping"
>>> s.ping()
'instance ping'Same shape — a name on the class, a name of the same spelling in the instance dict — and opposite winners. One ordered lookup decides both.
When you write obj.attr, __getattribute__ from the previous chapter walks a fixed cascade:
- Search the type and its bases for
attr. If it is found and it is a data descriptor, use it and stop. - Otherwise, if
attris in the instance's own__dict__, return that. - Otherwise, if the type had
attras a non-data descriptor or a plain class attribute, use that. - Otherwise call
__getattr__if it exists; if not, raiseAttributeError.
The words data descriptor are the hinge, so define them. A descriptor is an object placed as a class attribute that customises what obj.attr does, by implementing __get__ and optionally __set__ or __delete__. The reference splits them in two:
If an object defines
__set__or__delete__, it is considered a data descriptor. Descriptors that only define__get__are called non-data descriptors.If an instance's dictionary has an entry with the same name as a data descriptor, the data descriptor takes precedence. If an instance's dictionary has an entry with the same name as a non-data descriptor, the dictionary entry takes precedence.
Drawn out, that precedence is the cascade:
obj.attr checks, in order — the first match winsobj.attr
│
▼
type + bases have 'attr', and it's a DATA descriptor (__set__/__delete__)?
├─ yes ──▶ call its __get__(obj) property, __slots__
▼ no
obj.__dict__ has 'attr'?
├─ yes ──▶ return obj.__dict__['attr'] the instance's own value
▼ no
type + bases have 'attr' (non-data descriptor or plain class attribute)?
├─ yes ──▶ call __get__(obj), or return the value methods, class variables
▼ no
type has __getattr__?
├─ yes ──▶ return __getattr__(obj, 'attr') the fallback from before
▼ no
raise AttributeErrorNow the two opening surprises collapse into one fact. property is a data descriptor — it defines __set__ — so it sits at rung 1 and beats anything in the instance dict:
>>> hasattr(property, "__set__")
TrueA plain function is a non-data descriptor — it defines only __get__ — so it sits at rung 3, below the instance dict. Assign s.ping and your instance-dict entry at rung 2 wins, shadowing the method. That same function __get__ is what binds self: reading s.ping calls the function's __get__, which returns a bound method with s already supplied. Methods are not a separate mechanism; they are non-data descriptors.
You write a descriptor by writing a class with those methods. The one piece that used to be awkward — a descriptor knowing which attribute name it was assigned to — is handled by __set_name__, called once at class-creation time with the owning class and the name:
class Positive:
def __set_name__(self, owner, name):
self.storage = "_" + name
def __get__(self, obj, objtype=None):
return getattr(obj, self.storage)
def __set__(self, obj, value):
if value <= 0:
raise ValueError("must be positive")
setattr(obj, self.storage, value)
class Account:
balance = Positive()>>> a = Account(); a.balance = 100
>>> a.balance
100
>>> a.balance = -5
ValueError: must be positiveBecause Positive defines __set__, it is a data descriptor: every read and every write of a.balance runs through it, so the validation cannot be dodged by writing to the instance dict. __set_name__ handed it _balance to store the real value under, without you repeating the name.
So property, bound methods, and — a couple of chapters on — __slots__ are not three features to memorise separately. They are the same descriptor protocol read off one cascade, and the only thing to hold is which rung each sits on: data descriptors above the instance dict, everything else below it.
super() doesn't call the parent
A base widget, two mixins that each wrap its rendering, and a panel that combines them. Every mixin does the same thing: prepend its own name, then call super().render().
class Widget:
def render(self): return ["Widget"]
class Bordered(Widget):
def render(self): return ["Bordered"] + super().render()
class Scrollable(Widget):
def render(self): return ["Scrollable"] + super().render()
class Panel(Bordered, Scrollable):
def render(self): return ["Panel"] + super().render()Predict Panel().render(). If super() means "my parent class," then Bordered calls Widget and Scrollable calls Widget, so Widget appears twice and the base render runs twice. Run it:
>>> Panel().render()
['Panel', 'Bordered', 'Scrollable', 'Widget']Widget once. Bordered did not call Widget — it called Scrollable, which is not even one of its base classes. To see why, look at what super() actually consults:
>>> [c.__name__ for c in Panel.__mro__]
['Panel', 'Bordered', 'Scrollable', 'Widget', 'object']super() does not mean "the parent." It means "the next class after this one in the method resolution order" — the MRO, a single flat list of every class in the hierarchy, computed once per class. super() inside Bordered.render starts from where Bordered sits in Panel's MRO and steps one place right, which lands on Scrollable. The chain walks the list, each class exactly once, until Widget — whose render has no super() call — ends it.
inheritance graph method resolution order (the MRO)
Widget Panel ─▶ Bordered ─▶ Scrollable ─▶ Widget ─▶ object
/ \ │
Bordered Scrollable super() in Bordered.render steps here ─┘
\ / = Scrollable, a sibling, not Widget
Panel
Widget sits after BOTH Bordered and Scrollable, so it runs once, last.The MRO is not guesswork. It is produced by an algorithm called C3 linearization, and it guarantees two things: every class comes before its own bases, and the left-to-right order you wrote the bases in is preserved. Panel(Bordered, Scrollable) therefore puts Bordered before Scrollable, and both before Widget. When those constraints cannot all be satisfied at once, C3 does not pick a winner — it refuses, at class-creation time:
>>> class X: pass
>>> class Y: pass
>>> class P(X, Y): pass
>>> class Q(Y, X): pass
>>> class Bad(P, Q): pass
TypeError: Cannot create a consistent method resolution order (MRO) for bases X, YP demands X before Y; Q demands Y before X; no single ordering honours both, so the class never gets built. This is deliberate — a hierarchy that could resolve a method two different ways is refused rather than resolved one way behind your back.
For a super() chain to work, every method in it has to cooperate: call super().method() with compatible arguments, and let object at the end of the MRO terminate the chain. That is the real contract behind "mixins compose." Each mixin advances the cursor with super(), not knowing or caring who comes next; the MRO decides. Write "call my parent" logic instead, and it breaks the first time a diamond puts a sibling between you and the class you assumed was next.
The class machinery underneath all this is itself made of objects. A class statement is sugar for a call to type, whose three arguments are the name, the bases, and the namespace dict the body produced:
>>> Dyn = type("Dyn", (), {"greet": lambda self: "hi from " + type(self).__name__})
>>> Dyn().greet()
'hi from Dyn'type is the default metaclass — the type of a class. type(int) is type, and type(type) is type; the tower stops there. A metaclass is a class whose instances are themselves classes, and subclassing type lets you intercept class creation to register, validate, or rewrite classes as they are defined.
Reach for that last. Two hooks cover the cases people used to write metaclasses for. __set_name__, from the previous chapter, lets a descriptor learn its name. And __init_subclass__ — a method on a base class, run once for each subclass as it is created — handles registration with no metaclass in sight:
registry = {}
class Plugin:
def __init_subclass__(cls, /, key=None, **kw):
super().__init_subclass__(**kw)
if key: registry[key] = cls
class CsvExporter(Plugin, key="csv"): pass
class JsonExporter(Plugin, key="json"): pass>>> {key: cls.__name__ for key, cls in registry.items()}
{'csv': 'CsvExporter', 'json': 'JsonExporter'}A metaclass is still the right tool when you need to touch the class object itself before it exists — but on modern Python that is a narrow need, and __init_subclass__, __set_name__, and a class decorator meet most of what used to require one.
Dataclasses and __slots__ earn their keep
The mutable-default trap from chapter 2 has a sequel: write it in a dataclass and Python stops you before the program runs.
>>> @dataclass
... class Cart:
... items: list = []
...
ValueError: mutable default <class 'list'> for field items is not allowed: use default_factory@dataclass recognises the exact bug — one list shared across every instance — and refuses at class-definition time, naming the fix. field(default_factory=list) calls list() afresh for each instance:
from dataclasses import dataclass, field
@dataclass(frozen=True, slots=True)
class Point:
x: int
y: int
tags: list = field(default_factory=list)
dist: float = field(init=False, default=0.0)
def __post_init__(self):
object.__setattr__(self, "dist", (self.x**2 + self.y**2) ** 0.5)@dataclass reads the annotated fields and writes __init__, __repr__, and __eq__ from them:
>>> Point(3, 4)
Point(x=3, y=4, tags=[], dist=5.0)
>>> a, b = Point(0, 0), Point(1, 1)
>>> a.tags.append("urgent"); (a.tags, b.tags)
(['urgent'], [])Two instances, two separate lists — the default_factory did its job. Three of that decorator's options are worth holding:
frozen=True makes instances read-only. Assigning a field raises FrozenInstanceError, and — from chapter 2 — a frozen dataclass gets a __hash__ written to match its __eq__, so it can go in a set. A computed field like dist then has to be set through the back door, object.__setattr__, because ordinary assignment is blocked. That is what __post_init__ is for: a hook that runs right after the generated __init__.
>>> p = Point(3, 4); p.x = 9
FrozenInstanceError: cannot assign to field 'x'slots=True, from Python 3.10 on, builds the class with __slots__ instead of a per-instance dict — which is the other half of this chapter.
By default every instance carries its own __dict__ to hold attributes. That is flexible and not free. __slots__ replaces the dict with a fixed set of descriptor slots, one per declared name, and the saving is large:
>>> import sys
>>> class Plain: pass
>>> class Slotted: __slots__ = ("x", "y")
>>> sys.getsizeof(Plain()) + sys.getsizeof(Plain().__dict__)
344
>>> sys.getsizeof(Slotted())
48344 bytes against 48. A slotted instance also refuses attributes you did not declare, which is often a feature — a mistyped name becomes an error instead of a silently created field:
>>> s = Slotted(); s.z = 3
AttributeError: 'Slotted' object has no attribute 'z'One trap, and it is quiet. __slots__ is not inherited. A subclass that does not declare its own __slots__ gets a __dict__ back, and every instance of it pays the full cost again:
>>> class Child(Slotted): pass # no __slots__ here
>>> c = Child(); c.anything = 99
>>> hasattr(c, "__dict__")
TrueThe saving you measured on Slotted quietly vanishes in Child. To keep it, every class in the chain has to declare __slots__ — an empty __slots__ = () is enough for a subclass that adds no fields of its own.
Use a dataclass whenever a class is mostly a labelled bundle of values: it removes the __init__/__repr__/__eq__ boilerplate and the mutable-default footgun together. Add slots=True when you will make many of them and memory shows up in a profile. The rest of the time an ordinary class is right, and reaching for either of these is effort spent where the program did not ask for it.
Iteration is a protocol you can implement
for is not a primitive. It is sugar over three method calls, and once you can hand-write the loop it stands for, iteration stops being magic.
colors = ["red", "green", "blue"]
it = iter(colors) # 1. get an iterator
out = []
while True:
try:
out.append(next(it)) # 2. pull the next value
except StopIteration: # 3. stop when it is raised
break>>> out
['red', 'green', 'blue']for x in colors runs exactly this: call iter(colors) once to get an iterator, then call next(...) on it until it raises StopIteration. Anything that supports those two calls can be looped over — that is the whole iteration protocol, and it is why for behaves identically on a list, a file, a dict, and a database cursor.
Two roles hide inside the word "iterable," and separating them removes a category of bugs. An iterable is anything you can call iter() on: it has __iter__, which returns an iterator. An iterator is the object that actually produces values: it has __next__, and its own __iter__ returns itself. A list is iterable but is not an iterator:
>>> hasattr(colors, "__iter__"), hasattr(colors, "__next__")
(True, False)
>>> type(iter(colors)).__name__
'list_iterator'iter(colors) mints a separate list_iterator that holds the cursor position. The list holds the data; the iterator holds the place. That split is why you can nest two loops over the same list without them treading on each other — each iter() call gets its own cursor.
Collapse the two roles into one object and you get a container that works exactly once. Here is an iterator that is its own iterable, with __iter__ returning self:
class Countdown:
def __init__(self, n): self.n = n
def __iter__(self): return self
def __next__(self):
if self.n <= 0: raise StopIteration
self.n -= 1
return self.n + 1>>> c = Countdown(3)
>>> list(c)
[3, 2, 1]
>>> list(c)
[]The second pass is empty. c is the cursor, and the first list(c) ran it to the end; there is no fresh cursor left to hand out. That is not a fault in Countdown — a bare iterator is meant to be one-shot — but it is a fault if you wanted a reusable container. Keep the two roles apart: have __iter__ return a new iterator every time.
class Deck:
def __init__(self, cards): self.cards = cards
def __iter__(self): return iter(self.cards)>>> d = Deck(["A", "K", "Q"])
>>> list(d), list(d)
(['A', 'K', 'Q'], ['A', 'K', 'Q'])Now every loop calls __iter__, which builds a brand-new iterator over the underlying list, so the deck can be walked as many times as you like.
The rule to carry: if your object holds data and should be looped over repeatedly, make it an iterable whose __iter__ returns a fresh iterator. Return self from __iter__ only when the object genuinely is a one-pass cursor. That single-pass cursor is worth getting comfortable with, because it is the exact shape of the next chapter — a generator is an iterator you get without writing __next__ at all.
A generator is a function you can pause
Put a print on the first line of a generator function and call it. Predict what prints.
def watch(sensor):
print(f" watch({sensor!r}) body running")
while True:
yield f"{sensor} reading">>> g = watch("cpu")
>>> type(g).__name__
'generator'Nothing printed. Calling watch("cpu") ran not one line of the body — not even the print at the top. It handed back a generator object. The body starts running only when you ask for a value:
>>> next(g)
watch('cpu') body running
'cpu reading'A yield anywhere in a function's body changes what the function is. Calling it no longer runs it; it builds a paused frame — the object that holds a running function's local variables and its position in the code — and returns a generator wrapped around that frame. next(g) resumes the frame, runs it to the next yield, hands back the yielded value, and suspends again with every local intact.
"With every local intact" is the point, and you can watch it. yield is two-way: the value on its right goes out to the caller, and whatever the caller sends in becomes the value the yield expression evaluates to inside the generator. A running total, driven entirely by sent values:
def running_total():
total = 0
while True:
n = yield total # send 'total' out, receive the next value as 'n'
total += n>>> g = running_total()
>>> next(g) # run to the first yield ("prime" it)
0
>>> g.send(10)
10
>>> g.send(5)
15Between calls the frame is frozen with its variables in place, which the generator will show you directly:
>>> g.gi_frame.f_locals
{'total': 15, 'n': 5}total is still 15 while the generator sits idle, because the frame was suspended, not torn down. That is what makes a generator different in kind from a function that returns a list: it is a resumable computation with memory, parked between values.
A generator moves through four states, which inspect.getgeneratorstate will name for you:
┌────────── next() / send(v) / throw() ──────────┐
│ resumes the frame │
▼ │
gen() ─▶ CREATED ───▶ RUNNING ────── yield ──────▶ SUSPENDED ──────┘
│
└── body returns · raises · close() ──▶ CLOSED
(further next() → StopIteration)send is the general form of next — next(g) is g.send(None) — and two more methods drive the frame from outside. throw raises an exception at the paused yield, which the generator can catch and recover from; close raises GeneratorExit to shut it down.
def guarded():
try:
yield "working"
except ValueError as e:
print(" caught inside:", e)
yield "recovered">>> g = guarded(); next(g)
'working'
>>> g.throw(ValueError("bad input"))
caught inside: bad input
'recovered'The exception was injected into the frame at the suspended yield, caught by the generator's own try, and turned into a recovery value. You cannot throw into a list.
This is the correction to the model most people carry. A generator is not "a lazy list." It is a one-shot cursor over a frame: it runs once and is then spent, it is not re-iterable, and — unlike a list — it can receive values and exceptions. It is also its own iterator, so iter(g) is g, which means iterating it twice gives you the values once and then nothing:
>>> squares = (n*n for n in range(3))
>>> iter(squares) is squares
True
>>> list(squares), list(squares)
([0, 1, 4], [])To hand a value-stream off to another generator, yield from delegates — it runs the sub-generator to exhaustion, forwarding each value, and forwarding sent values and exceptions too, which is what makes it more than a loop:
>>> def chained():
... yield from ["a", "b"]
... yield from ["c"]
>>> list(chained())
['a', 'b', 'c']One sharp edge lives right here, because a workhorse of the standard library is generator-based and lazy. itertools.groupby groups consecutive equal keys, reading its input one item at a time, so it only ever groups runs that are already adjacent. Feed it unsorted data and it silently splits a group in two:
>>> rows = [("open", 1), ("closed", 2), ("open", 3)]
>>> {k: [v for _, v in g] for k, g in groupby(rows, key=lambda r: r[0])}
{'open': [3], 'closed': [2]}The first ("open", 1) and the later ("open", 3) are not next to each other, so groupby emits open twice, and building a dict keeps only the last — the 1 is gone, with no error. Sort by the same key first and the runs line up:
>>> rows.sort(key=lambda r: r[0])
>>> {k: [v for _, v in g] for k, g in groupby(rows, key=lambda r: r[0])}
{'closed': [2], 'open': [1, 3]}The lazy, one-item-at-a-time reading that makes groupby cheap is the same thing that makes it demand sorted input. And that suspend-and-resume frame — a computation that yields control and later picks up exactly where it left off — is the machinery the next chapter runs an event loop on top of.
await is yield with a loop behind it
Call an async def function. Like a generator, it runs nothing:
async def fetch(url):
await asyncio.sleep(0.1)
return f"body of {url}">>> c = fetch("/status")
>>> type(c).__name__
'coroutine'Calling fetch("/status") did not fetch anything. It returned a coroutine object: a paused frame, the same idea as a generator's, wrapped for a different driver. Throw it away without running it and Python tells you it never happened:
>>> del c # never awaited
RuntimeWarning: coroutine 'fetch' was never awaitedThe reference is blunt about it:
Note that simply calling a coroutine will not schedule it to be executed.
So what runs it? An event loop: a scheduler that runs one coroutine at a time on one thread and switches between them only at await points. await x suspends the current coroutine's frame and hands control back to the loop until x is ready. It is the same suspend-and-resume you saw with yield, the loop now standing in for the caller that decides who resumes next. asyncio.run starts a loop and drives your coroutine to the end:
>>> asyncio.run(fetch("/status"))
'body of /status'One coroutine at a time is not yet useful; it is a slower function call. The point is running many at once. asyncio.gather wraps several coroutines as Tasks and schedules them together:
Tasks are used to schedule coroutines concurrently. When a coroutine is wrapped into a Task with functions like asyncio.create_task the coroutine is automatically scheduled to run soon.
Three jobs that each await asyncio.sleep for 0.3, 0.1, and 0.2 seconds. One after another that is 0.6 seconds. Predict the total under gather:
>>> asyncio.run(concurrent())
job 1 finished at 0.30s
job 2 finished at 0.10s
job 3 finished at 0.20s
total: 0.30s (sum of delays = 0.60s)0.30 seconds: the longest single delay, not the sum. While job 1 sits in its await, the loop is free to run jobs 2 and 3, and while they wait, job 1's timer keeps counting. They overlap because each await is a point where the task voluntarily steps aside. You can watch control bounce between two tasks at every await:
>>> asyncio.run(interleave())
A0 B0 A1 B1 A2 B2Each await asyncio.sleep(0) in A yields to the loop, which lets B take a step, which yields back: cooperative, in lockstep, on one thread.
That "voluntarily steps aside" is the entire deal, and it is also how async breaks. The loop can switch only at an await. Put a synchronous blocking call in a coroutine — an ordinary time.sleep, a blocking HTTP call, a heavy CPU loop — and there is no await for the loop to switch at, so the one thread is held and every other task waits. The same shape as before, one 0.3-second worker beside two 0.05-second quick tasks, run both ways:
awaiting worker — await asyncio.sleep(0.30) hands the thread back to the loop
time → 0.00 0.05 0.30
worker [──────── awaiting, thread free for others ─────────]
quick 1 [0.05] finished 0.05s
quick 2 [0.05] finished 0.05s
total: 0.30s the quick tasks run while the worker waits
blocking worker — time.sleep(0.30) holds the one thread; nothing else moves
time → 0.00 0.30 0.35
worker [════════ time.sleep, THREAD BLOCKED ═══════]
quick 1 [0.05] finished 0.35s
quick 2 [0.05] finished 0.35s
total: 0.35s the quick tasks could not start until the thread came free>>> asyncio.run(run_case(awaiting_worker))
quick 1 finished at 0.05s
quick 2 finished at 0.05s
awaiting total: 0.30s
>>> asyncio.run(run_case(blocking_worker))
quick 1 finished at 0.35s
quick 2 finished at 0.35s
blocking total: 0.35sWith await asyncio.sleep(0.30) the worker releases the thread and the quick tasks finish at 0.05s, right on their own timers. With time.sleep(0.30), one line's difference, the worker holds the thread for the full 0.3s, the quick tasks cannot even start until it returns, and they finish at 0.35s. One blocking call serialised everything.
This is where two wrong models die. Async is not parallelism: one thread, one core, tasks taking turns. Marking a CPU-bound loop async makes it no faster, because it never awaits and so never yields. And a blocking call inside a coroutine is not merely un-helped: it is actively harmful, because it freezes every other task on the loop, as the numbers show. Async pays off for I/O-bound work — network requests, disk, database calls — where tasks spend most of their time waiting, and the loop turns that idle time into progress for others.
Two practical notes. Anything blocking has an async-friendly counterpart — await asyncio.sleep, an async HTTP client, or loop.run_in_executor to push genuinely blocking work onto a thread pool — and the whole skill is never blocking the loop. And since Python 3.11, asyncio.TaskGroup is the structured replacement for bare gather: it runs its tasks concurrently and, if one raises, cancels the rest and propagates the error rather than leaving tasks orphaned.
async with asyncio.TaskGroup() as tg:
tg.create_task(fetch("/a"))
tg.create_task(fetch("/b"))The frame that suspends at await and resumes where it left off is the generator's frame from the previous chapter, wearing a loop. What it never does is add a second thread, which is the subject the next chapter has been building toward.
The GIL doesn't make your code thread-safe
The canonical warning about Python threads runs like this: four threads each add one to a shared counter a million times, and because += is not atomic, you lose updates and finish short of four million. Everyone has seen the example. Predict the result:
counter = 0
def worker():
global counter
for _ in range(1_000_000):
counter += 1
# start 4 threads, join them all, then read countertrial 0: expected 4000000, got 4000000, lost 0
trial 1: expected 4000000, got 4000000, lost 0
...
trial 5: expected 4000000, got 4000000, lost 0Nothing lost, not once in six trials. Force the interpreter to switch threads as often as it can, a 1-microsecond switch interval instead of the default, and it still loses nothing:
forced 1us interval #0: expected 2000000, got 2000000, lost 0
forced 1us interval #4: expected 2000000, got 2000000, lost 0The threads really are interleaving; a separate check that appends each thread's tag to a shared list shows them trading off: XYXYXYXYXY. So the textbook example, on CPython 3.11, does not reproduce. That is not "the GIL makes your code safe." The truth is stranger and more useful. To reach it you need to know what the GIL is and where it lets go.
The Global Interpreter Lock is one mutex a thread must hold to execute Python bytecode and to touch interpreter internals, above all every object's reference count. Every object has an identity, from the first chapter; it also carries a count of how many names point at it, bumped on assignments and calls and freed when it reaches zero. Two threads updating the same refcount at once would corrupt it, freeing objects early or leaking them. The GIL makes that impossible by letting one thread run Python at a time.
A thread holds the lock, runs, and gives it up only at eval-breaker checkpoints: points where the interpreter pauses to ask whether anything else (another thread, a signal) is waiting. sys.getswitchinterval() is 0.005, so a thread that wants the GIL sets a request flag, and after 5 milliseconds the running thread honours it at its next checkpoint. Not mid-instruction, but at the next checkpoint. Where those checkpoints fall is the whole chapter. Disassemble one increment:
one iteration of counter += 1 (4 threads share `counter`)
LOAD_GLOBAL counter ┐
LOAD_CONST 1 │ read → modify → write:
BINARY_OP += │ no eval-breaker checkpoint between these
STORE_GLOBAL counter ┘
JUMP_BACKWARD ────────▶ checkpoint here (the loop back-edge)
a hand-off falls BETWEEN whole increments
4 × 1,000,000 → 4,000,000, zero lost — even forced to 1µs switchingcounter += 1 is four bytecodes — read the global, load 1, add, store the global — with no checkpoint among them. The only checkpoint in the loop is JUMP_BACKWARD, the jump back to the top for the next pass. CPython checks the eval-breaker flag there and after calls; the C source runs a macro, CHECK_EVAL_BREAKER(), which hands the GIL to a waiting thread when the flag is set. That checkpoint sits between whole increments, never between the read and the store inside one. So on 3.11 a bare counter += 1 is accidentally atomic, not because += is indivisible but because the interpreter never offers to switch in the middle of it.
Move the checkpoint and the safety evaporates. Widen the increment so a function call sits between reading and writing — t = counter; t = plus_one(t); counter = t — and disassemble again:
one iteration of t = counter; t = plus_one(t); counter = t
LOAD_GLOBAL counter ◀── read
STORE_FAST t
LOAD_GLOBAL plus_one
LOAD_FAST t
CALL ────────▶ checkpoint here (a call)
a hand-off can fall INSIDE the window
STORE_FAST t
LOAD_FAST t
STORE_GLOBAL counter ◀── write (may clobber another thread's update)
forced 1µs → ~50% lost; default 5ms → a HeisenbugA CALL is a checkpoint. A thread can be suspended there, after reading counter and before writing it back, while another thread reads the same stale value; both then store, and one update is gone. Under forced switching:
read-modify-write with a call in the window, forced 1us switches:
#0: expected 400000, got 191124, lost 208876
#2: expected 400000, got 176007, lost 223993Over half the updates lost. At the default 5-millisecond interval it becomes a Heisenbug, sometimes clean, sometimes not:
same at DEFAULT 5ms interval:
#0: expected 400000, got 400000, lost 0
#1: expected 400000, got 344224, lost 55776One run loses nothing, the next loses 55,776, and nothing in the code changed. Here is the statement to carry: the GIL serialises bytecode execution, but it does not make your compound operations atomic. Whether a read-modify-write is safe depends on where checkpoints happen to fall relative to it, which you cannot see and must not build on. For correctness use a threading.Lock, a queue.Queue, or an already-atomic primitive like itertools.count. Never the GIL.
If the GIL does not buy safety, what does it cost? Parallelism. Because one thread runs Python at a time, CPU-bound threads take turns instead of running at once:
serial (2x) 0.986s
2 threads 1.055s <- no speedup; the GIL serialises CPU-bound threads
2 processes 0.565s <- ~1.7x; separate processes, separate GILsTwo threads doing heavy arithmetic are no faster than doing the work one after the other, a touch slower in fact, from the cost of trading the lock. Two processes nearly halve the time, because each has its own interpreter and its own GIL. That is the decision rule: I/O-bound work goes to threads or async, since a thread blocked on I/O releases the GIL for others; CPU-bound work goes to multiprocessing, or to a C extension that drops the GIL while it computes.
Finally, the thing you will read about but cannot run here. Starting with 3.13, CPython ships an experimental free-threaded build with the GIL removed:
The free-threaded mode is experimental and work is ongoing to improve it: expect some bugs and a substantial single-threaded performance hit.
It is a separate build, python3.13t; you check it with sys._is_gil_enabled; its single-threaded overhead was around 40% in 3.13; and PEP 779 moved it to "supported but optional" in 3.14. Optional, off by default, still slower on one thread. On the 3.11 you are running here the GIL is unconditional, and hasattr(sys, "_is_gil_enabled") is False. The conclusion outlasts every version: do not infer thread safety from the GIL. Lock what threads share.
Type hints are data the interpreter ignores
Annotate a parameter str, call it with an int, and watch nothing happen:
>>> def greet(name: str) -> str:
... return name
>>> greet(123)
123No error. The int went straight through a function that says it takes a str and returns a str, and came back an int. Annotations are not checks. They are ordinary objects the interpreter stores and otherwise ignores:
>>> greet.__annotations__
{'name': <class 'str'>, 'return': <class 'str'>}They sit in __annotations__ as data, nothing more. A list[int] will hold strings without complaint, because the [int] is a description you wrote, not a constraint the runtime enforces:
>>> scores: list[int] = ["not", "ints"]
>>> scores
['not', 'ints']And because a parameterised generic like list[int] is something a static tool checks rather than something the runtime tracks, you cannot even ask about it at runtime:
>>> isinstance([1], list[int])
TypeError: isinstance() argument 2 cannot be a parameterized genericIf the interpreter ignores hints, who reads them? A separate program — a static type checker such as mypy or pyright — that parses your source, works out the types, and reports mismatches without running anything. The split is total: the checker runs at your desk and blocks the commit; the interpreter runs in production and never consults a hint. That is the model to hold. A green type check and a working program are two independent facts.
One place the runtime does touch annotations is when you ask it to. Annotations can be written as strings — a forward reference, needed when a class refers to itself before it has finished being defined — and typing.get_type_hints resolves those strings back to the real objects:
>>> class Node:
... next: "Node"
... val: int
>>> Node.__annotations__
{'next': 'Node', 'val': <class 'int'>}
>>> get_type_hints(Node)
{'next': <class '__main__.Node'>, 'val': <class 'int'>}next was stored as the string 'Node' and resolved to the class only when asked. It is the same string-annotation trick that let TYPE_CHECKING break an import cycle back in chapter 5.
Because the checker is a separate tool, which tool you run matters, and the two common ones disagree out of the box in a way that bites. Take a file with one error inside an unannotated function and one inside a typed function:
def helper(): # no annotations on the function
x: int = "not an int" # a clear error, inside an untyped function
return x
def typed() -> int:
return "also wrong" # an error, inside a typed functionmypy 1.19.1, with no flags, finds one of the two:
checker_diff.py:2: note: By default the bodies of untyped functions are not checked
checker_diff.py:6: error: Incompatible return value type (got "str", expected "int")
Found 1 error in 1 fileThe error inside helper is a note, not an error, because mypy does not check the body of a function that has no annotations, until you pass --check-untyped-defs. pyright 1.1.408 checks every body by default and finds both:
checker_diff.py:2 - error: Type "Literal['not an int']" is not assignable to declared type "int"
checker_diff.py:6 - error: Type "Literal['also wrong']" is not assignable to return type "int"In a large codebase that gap is the difference between whole functions going silently unchecked and every line examined. Pick a checker, learn its defaults, and turn on --check-untyped-defs if you run mypy over code that is not fully annotated.
The last thing that decides what you can write is the version, because the typing language has grown one release at a time. On the 3.11 here you have built-in generics list[int] (3.9), X | Y unions (3.10), and Protocol (3.8). But the compact type-parameter syntax from PEP 695 — def first[T](items: list[T]) -> T and type Alias = ... — arrived in 3.12 and is a parser change, so it cannot be backported: you will read it in modern library source and cannot run it here. Two names are missing outright from 3.11's typing, override (3.12) and TypeIs (3.13). Everything else from 3.12 onward is reachable today through the typing_extensions backport; PEP 695's syntax is the one wall. What a checker accepts is a fact about the interpreter version and the tool, not about the language in the abstract, which is the frame for the generics the final chapter takes on.
Generics get interesting at variance and protocols
Dog is a subclass of Animal, so a list[Dog] is a list[Animal] and you can pass one where the other is wanted. Predict what mypy says:
class Animal: ...
class Dog(Animal): ...
def feed_all(animals: list[Animal]) -> None:
animals.append(Animal())
dogs: list[Dog] = [Dog()]
feed_all(dogs)variance.py:9: error: Argument 1 to "feed_all" has incompatible type "list[Dog]"; expected "list[Animal]"
variance.py:9: note: "list" is invariant -- see https://mypy.readthedocs.io/en/stable/common_issues.html#variance
variance.py:9: note: Consider using "Sequence" instead, which is covariantRejected. list[Dog] is not a list[Animal], even though Dog is an Animal. The reason sits in the body of feed_all: it calls animals.append(Animal()). If the checker let you pass a list[Dog], that append would drop a plain Animal — perhaps a Cat — into a list the rest of your program believes holds only dogs. To keep that impossible, a mutable container is invariant: list[Dog] and list[Animal] are unrelated types, neither usable for the other.
The fix mypy suggests names the way out. Sequence is the read-only view of a list — it has no append — so nothing can be smuggled in through it, and it is safe to treat a sequence of dogs as a sequence of animals. Sequence is covariant: Sequence[Dog] is a Sequence[Animal]. That gives a rule you can apply without memorising a table: a parameter you only read from takes a read-only type (Sequence, Iterable, Mapping on its values) and accepts subtypes; a parameter you also write to must be invariant (list, dict) and takes the exact type only. Type the argument by what the function does to it.
The generics those examples use are built from a type variable — a placeholder a call binds to a concrete type. TypeVar comes in two flavours people mix up. A bound accepts any subtype of a ceiling:
Num = TypeVar("Num", bound=float)
def clamp(x: Num, lo: Num, hi: Num) -> Num: ...error: Value of type variable "Num" of "clamp" cannot be "str"int satisfies bound=float and str does not, so clamp(5, 0, 10) checks and clamp("a", "b", "c") is an error — and because Num is one variable, the return type is whatever concrete type came in. Constraints are a different thing: a fixed menu, and each call picks exactly one option rather than a common supertype:
S = TypeVar("S", str, bytes)
def join(a: S, b: S) -> S: ...error: Value of type variable "S" of "join" cannot be "Sequence[object]"join("x", "y") is fine; join("x", b"y") fails, because S must resolve to str or bytes for the whole call, and no single choice fits a str and a bytes at once.
The last piece is how a type can be satisfied without being inherited. A Protocol describes a shape — a set of methods and attributes — and any class with that shape matches it, whether or not it has ever heard of the protocol. This is duck typing made checkable:
@runtime_checkable
class Closeable(Protocol):
def close(self) -> None: ...
class Socket:
def close(self) -> None: ...>>> isinstance(Socket(), Closeable)
True
>>> Socket.__bases__
(<class 'object'>,)Socket matches Closeable structurally — it has a close method — while its only base is object. Nothing was declared; the shape is the contract. That is how you type "anything with a close method" without forcing every such class to import and inherit a base.
One trap rides along with @runtime_checkable, which is what lets isinstance test a protocol at all. The runtime check is shallow: it verifies the attribute names exist, not their signatures, nor even that they are callable. A class whose close is an integer passes:
>>> class Liar:
... close = 42
>>> isinstance(Liar(), Closeable)
TrueLiar has an attribute named close, so the runtime isinstance says yes, though calling Liar().close() would fail. A static checker reads the protocol properly and rejects close = 42 as not matching def close(self) -> None; the runtime check only counts names. Use @runtime_checkable as a quick "has these attributes" gate, and lean on the static checker for the guarantee that they have the right shapes.
That is the top of the language, and it rests on the bottom. A protocol is checkable because a class is an object with a known set of attributes; variance is a fact about what append does to a list, which is a method looked up on the list's type; a TypeVar is data in an annotation the interpreter stores and ignores. Every mechanism in this guide — a name bound in a namespace dict, a half-built module, a dunder dispatched on the type, a frame that suspends at yield and again at await, a lock handed off between bytecodes — is the same small handful of objects seen from a different angle. You started with type(type) is type. Everything since has been what follows from taking that literally.
Further reading
Each of these is the authoritative version of something a chapter here compressed — the CPython documentation and the PEPs this guide was checked against.
- The Python Language Reference — Data model. The dunder protocols, special-method lookup, and descriptors stated exhaustively: the spine of chapters 6 through 8.
- The import system. How a file becomes a cached module object, and the
sys.modulesordering behind chapter 5. - The execution model. Binding, scope, and the class-body rule from chapter 3.
- Descriptor HowTo Guide. The clearest single account of data versus non-data descriptors — the cascade of chapter 7, worked in full.
- Coroutines and Tasks. The asyncio reference for chapter 12: coroutines, tasks, and
TaskGroup. typingand the typing specification. Generics, variance, and protocols from chapters 14 and 15, and the record of which feature landed in which release.- PEP 703 and the free-threading HowTo. The GIL-free build that chapter 13 points at but cannot run on 3.11.
- PEP 695. The 3.12 type-parameter syntax you will read in modern library source but cannot write on 3.11.