Container vs flat iterables

An iterable is anything Python can call iter() on — lists, files, generators, database cursors. Every for loop works the same on the surface. The difference is what happens underneath: does it build the whole result in memory first, or does it produce values one at a time as you consume them?

Think of a list like a DVD — the whole movie exists on the disc, you can skip to any chapter, rewind, and watch it again. A generator is a live broadcast — you get what is playing right now, you cannot rewind, and when the broadcast ends there is nothing left to tune into. Both play video. One materializes everything upfront; the other streams.

Core answer

Use a concrete sequence when you need repeated passes, random access, slicing, or a stable snapshot. Use an iterator or generator when one-pass streaming is enough and memory pressure matters.

# [CURRENT - 3.10-3.14] Works on Python 3.x
def read_lines(path):
    with open(path, encoding="utf-8") as fp:
        for line in fp:
            yield line.rstrip("\n")

Generators produce values lazily. That lowers peak memory usage because the whole result does not exist at once.

# [CURRENT - 3.10-3.14] Works on Python 3.x
numbers = (int(part) for part in "1,2,3".split(","))
print(list(numbers))
print(list(numbers))  # already exhausted

Mechanism and internals

The iterator protocol is two operations:

iter(obj) asks for an iterator
next(it) asks for the next value until StopIteration

A for loop is syntax around those calls.

# [CURRENT - 3.10-3.14] Works on Python 3.x
values = iter([10, 20])
print(next(values))
print(next(values))

Step Through a for Loop

Walk through how Python traverses a concrete list, binds the loop variable, updates an accumulator, and finishes with the computed result.

Generator functions are more than "functions that yield." On CPython they keep a suspended execution frame alive between iterations: local variables, the current instruction position, and references to still-live objects remain attached to the generator object until it finishes or is discarded.

That is why generators are memory-efficient for result size, but not free: each live generator still retains its frame state and anything reachable from it.

# [CURRENT - 3.10-3.14] Works on Python 3.x
def chunks(lines):
    buffer = []
    for line in lines:
        buffer.append(line)
        if len(buffer) == 3:
            yield tuple(buffer)
            buffer.clear()

In that example, buffer remains alive inside the suspended generator between yield points.

CPython internals

On CPython, generators are implemented as a special kind of frame object attached to a generator object (Objects/genobject.c). When a generator function is called, Python does not execute any of the function body. Instead, it creates a PyGenObject that holds:

a reference to the function's code object (gi_code)
a reference to the function's frame (gi_frame), which contains the local variables, the stack, the instruction pointer, and the block stack
the running/suspended/finished state (gi_running)
the current instruction offset (the f_lasti field in the frame)

When next(gen) is called, CPython's evaluation loop (Python/ceval.c) resumes execution of the frame from the last suspended position (stored in f_lasti). The frame's local variables, including buffer in the example above, remain allocated in the frame's fast-locals array. That is why a suspended generator's memory includes all referenced objects reachable from its locals.

This is fundamentally different from a container or a flat sequence. A list of n items allocates all n references immediately. A generator that yields n items allocates only the generator object and its frame (typically a few hundred bytes on CPython 3.12), regardless of how many items it will produce.

The gi_yieldfrom field on the generator object tracks the sub-iterator when yield from is used, creating a chain of suspended frames.

Generator-based coroutines and native coroutines both rely on the same suspension model. The frame is the "suspension unit" — the language creates one frame per active generator, and the frame's liveness determines the memory footprint. The yield from construct (and await in native coroutines) chains these frames together without additional allocation.

Container, flat, and streaming models

There are three materially different storage models:

container sequences such as list and tuple: store references to Python objects
flat sequences such as bytes, bytearray, and array.array: store values compactly
streaming iterables such as generators: store control state, not the whole output

# [CURRENT - 3.10-3.14] Works on Python 3.x
from array import array
row = ["Ana", 42, {"active": True}]
temperatures = array("h", [21, 22, 20, 19])
print(row)
print(temperatures)

If you turn a stream into list(stream), you trade the streaming model for full materialization immediately.

Version context

The iterator protocol and generator semantics are stable Python 3 behavior. Modern type hints often use collections.abc.Iterable, Iterator, and Sequence. Current project guidance targets Python 3.10-3.14. Python 3.9 and below are End-of-Life.

CPython-specific notes:

the generator frame layout is defined in Include/internal/pycore_frame.h
the generator object is defined in Objects/genobject.c
the frame's f_lasti field tracks the precise instruction offset for resumption

Edge cases and gotchas

Do not call len() or slice an arbitrary iterable unless the API promises a sized sequence. Many iterables cannot answer length without consuming themselves, and many iterators cannot be restarted.

# [CURRENT - 3.10-3.14] Works on Python 3.x
def first_match(rows, predicate):
    for row in rows:
        if predicate(row):
            return row
    return None

itertools.tee() looks like cloning an iterator, but it does so by caching produced values. If one branch lags far behind the other, memory usage can grow toward full materialization.

list() snapshots are useful at API boundaries, but they can silently exhaust a generator and allocate unbounded memory if the source is large or infinite.

Be explicit about consumption semantics. If a function consumes an iterator, document that fact. Hidden one-shot behavior is a production bug factory.

Production usage

Use these annotation rules:

accept Iterable[T] when you only need to loop once
accept Sequence[T] when you need indexing, length, or repeated passes
return an iterator when laziness is part of the contract
return a concrete container when callers need replayability and ownership

# [OLDER / 3.9, CURRENT - 3.10-3.14] Works on Python 3.9+
from collections.abc import Iterable
def non_empty(lines: Iterable[str]):
    for line in lines:
        stripped = line.strip()
        if stripped:
            yield stripped

Materialize deliberately:

# [CURRENT - 3.10-3.14] Works on Python 3.x
def normalized_snapshot(lines):
    return [line.strip() for line in lines if line.strip()]

That version is heavier in memory but easier to reuse and debug because the caller owns a stable list.

For compact storage choices, see . For slice-producing sequences, see .

Further depth

Core answer

Use a concrete sequence when you need repeated passes, random access, slicing, or a stable snapshot. Use an iterator or generator when one-pass streaming is enough and memory pressure matters.

# [CURRENT - 3.10-3.14] Works on Python 3.x
def read_lines(path):
    with open(path, encoding="utf-8") as fp:
        for line in fp:
            yield line.rstrip("\n")

Generators produce values lazily. That lowers peak memory usage because the whole result does not exist at once.

# [CURRENT - 3.10-3.14] Works on Python 3.x
numbers = (int(part) for part in "1,2,3".split(","))
print(list(numbers))
print(list(numbers))  # already exhausted

Mechanism and internals

The iterator protocol is two operations:

iter(obj) asks for an iterator
next(it) asks for the next value until StopIteration

A for loop is syntax around those calls.

# [CURRENT - 3.10-3.14] Works on Python 3.x
values = iter([10, 20])
print(next(values))
print(next(values))

Step Through a for Loop

Walk through how Python traverses a concrete list, binds the loop variable, updates an accumulator, and finishes with the computed result.

That is why generators are memory-efficient for result size, but not free: each live generator still retains its frame state and anything reachable from it.

# [CURRENT - 3.10-3.14] Works on Python 3.x
def chunks(lines):
    buffer = []
    for line in lines:
        buffer.append(line)
        if len(buffer) == 3:
            yield tuple(buffer)
            buffer.clear()

In that example, buffer remains alive inside the suspended generator between yield points.

CPython internals

a reference to the function's code object (gi_code)
a reference to the function's frame (gi_frame), which contains the local variables, the stack, the instruction pointer, and the block stack
the running/suspended/finished state (gi_running)
the current instruction offset (the f_lasti field in the frame)

The gi_yieldfrom field on the generator object tracks the sub-iterator when yield from is used, creating a chain of suspended frames.

Container, flat, and streaming models

There are three materially different storage models:

container sequences such as list and tuple: store references to Python objects
flat sequences such as bytes, bytearray, and array.array: store values compactly
streaming iterables such as generators: store control state, not the whole output

# [CURRENT - 3.10-3.14] Works on Python 3.x
from array import array
row = ["Ana", 42, {"active": True}]
temperatures = array("h", [21, 22, 20, 19])
print(row)
print(temperatures)

If you turn a stream into list(stream), you trade the streaming model for full materialization immediately.

Version context

CPython-specific notes:

the generator frame layout is defined in Include/internal/pycore_frame.h
the generator object is defined in Objects/genobject.c
the frame's f_lasti field tracks the precise instruction offset for resumption

Edge cases and gotchas

Do not call len() or slice an arbitrary iterable unless the API promises a sized sequence. Many iterables cannot answer length without consuming themselves, and many iterators cannot be restarted.

# [CURRENT - 3.10-3.14] Works on Python 3.x
def first_match(rows, predicate):
    for row in rows:
        if predicate(row):
            return row
    return None

itertools.tee() looks like cloning an iterator, but it does so by caching produced values. If one branch lags far behind the other, memory usage can grow toward full materialization.

list() snapshots are useful at API boundaries, but they can silently exhaust a generator and allocate unbounded memory if the source is large or infinite.

Be explicit about consumption semantics. If a function consumes an iterator, document that fact. Hidden one-shot behavior is a production bug factory.

Production usage

Use these annotation rules:

accept Iterable[T] when you only need to loop once
accept Sequence[T] when you need indexing, length, or repeated passes
return an iterator when laziness is part of the contract
return a concrete container when callers need replayability and ownership

# [OLDER / 3.9, CURRENT - 3.10-3.14] Works on Python 3.9+
from collections.abc import Iterable
def non_empty(lines: Iterable[str]):
    for line in lines:
        stripped = line.strip()
        if stripped:
            yield stripped

Materialize deliberately:

# [CURRENT - 3.10-3.14] Works on Python 3.x
def normalized_snapshot(lines):
    return [line.strip() for line in lines if line.strip()]

That version is heavier in memory but easier to reuse and debug because the caller owns a stable list.

For compact storage choices, see . For slice-producing sequences, see .

Further depth

Python in Depth

Container vs flat iterables

Python in Depth