Python Generators and Iterators Tutorial: yield, next() and Lazy Evaluation

You can loop over a list, but why do iter(), next() and yield exist, and why does a generator become empty after one pass? The iterator protocol uses three values, a manual iterator can be replaced with a generator, and lazy execution proceeds value by value. An iterator stores its current state, produces values on demand, and signals exhaustion when no values remain. For broader programming study, explore the Coding & Skill Development Courses catalogue.

1. Iterable, iterator and generator: separate the three ideas

An iterable is an object that can produce an iterator. A list is iterable. An iterator is a one-way cursor that produces the next value when asked. A generator is a convenient kind of iterator, created by a generator function or generator expression.

scores = [72, 84, 91]
cursor = iter(scores)

print(iter(cursor) is cursor)
print(next(cursor))
print(next(cursor))
print(list(cursor))
print(next(cursor, "done"))

The outputs are True, 72, 84, [91], and done. cursor is the iterator produced by scores, and iter(cursor) returns that same cursor. The first two next() calls consume 72 and 84; list(cursor) consumes the remaining 91. After exhaustion, two-argument next() returns its default. Without that default, it raises StopIteration.

2. See the iterator protocol with a positive step range

class StepRange:
    def __init__(self, start, stop, step):
        self.current = start
        self.stop = stop
        self.step = step

    def __iter__(self):
        return self

    def __next__(self):
        if self.current >= self.stop:
            raise StopIteration
        value = self.current
        self.current += self.step
        return value

steps = StepRange(2, 8, 2)
print(next(steps))
print(list(steps))
print(next(steps, "done"))

The output is 2, [4, 6], and done. State changes as current: 2 -> 4 -> 6 -> 8. The first call consumes 2, list(steps) consumes 4 and 6, and the final call sees the stop boundary at 8. This compact teaching class assumes a positive step.

The protocol is precise: __iter__() returns an iterator, and __next__() either returns one value or raises StopIteration. This object returns itself from __iter__(), so it is one-pass. A reusable container normally returns a fresh iterator rather than resetting a cursor that is already active.

3. Replace the class machinery with yield

def running_totals(values):
    total = 0
    for value in values:
        total += value
        yield total

totals = running_totals([4, 7, 2])
print(next(totals))
print(next(totals))
print(list(totals))

The output is 4, 11, and [13]. Calling running_totals([4, 7, 2]) creates the generator without running its body. The first next() sets total to 0, reads 4, computes 0 + 4 = 4, and pauses at yield 4. The second resumes with total still 4, computes 4 + 7 = 11, and pauses. Then list() computes 11 + 2 = 13, collects 13, and reaches the loop's end.

yield emits a value while preserving local state for the next request. return, or reaching the end of the function, ends the generator. The generator stores its suspended state, not a list of all future results.

4. Generator expressions evaluate on demand

def square(n):
    print(f"computing {n}")
    return n * n

squares = (square(n) for n in range(1, 4))
print("ready")
print(next(squares))
print(list(squares))

The lines appear in this order: ready, computing 1, 1, computing 2, computing 3, [4, 9]. Creating the generator expression does not call square. The first next() requests only n = 1; list() then requests n = 2 and n = 3 and collects their results.

By contrast, [square(n) for n in range(1, 4)] computes all three squares while building the list. If you want a structured route through Python concepts, MCQs and coding questions, see the Python Course: Concepts, MCQs and Coding.

5. Compose a one-item-at-a-time pipeline

def readings_at_least(values, minimum):
    for value in values:
        if value >= minimum:
            yield value

def celsius_to_fahrenheit(values):
    for celsius in values:
        yield celsius * 9 / 5 + 32

readings = [12, 7, 15, 4, 10]
selected = readings_at_least(readings, 10)
converted = celsius_to_fahrenheit(selected)
print(list(converted))

The output is [53.6, 59.0, 50.0]. In source order, 12 gives 12 * 9 / 5 + 32 = 53.6; 7 is rejected; 15 gives 59.0; 4 is rejected; 10 gives 50.0.

The pull starts at the consumer. list(converted) asks the conversion generator, which asks the filter generator, which advances the source only until it can yield the next accepted value.

def flatten(rows):
    for row in rows:
        yield from row

print(list(flatten([[2, 4], [7], [9, 11]])))

The result is [2, 4, 7, 9, 11]. Here yield from row delegates iteration to each row. It is not recursive by itself.

6. Avoid exhaustion and unbounded consumption

next([10, 20]) fails because a list is iterable but is not an iterator. Create cursor = iter([10, 20]); then next(cursor) is 10.

Exhaustion causes a subtler bug. After values = (n * n for n in [2, 3, 4]), sum(values) is 4 + 9 + 16 = 29, and list(values) is then []. For a second pass, create a new generator or materialise finite data once. list(generator) deliberately consumes everything; it is not a harmless preview.

Replacing yield with return changes the contract too. In def first_only(values):, a loop body containing return value returns only 4 for [4, 7, 2]; it does not produce three values.

from itertools import count, islice

evens = count(0, 2)
print(list(islice(evens, 5)))
print(next(evens))

The output is [0, 2, 4, 6, 8] and then 10. Never call list(evens) on this unbounded source.

7. Trace iterator state in coding questions

Typical tasks ask you to identify iterable versus iterator, predict next() calls, explain an empty second pass, or rewrite an eager collection as a generator pipeline. Predict these answers:

• For g = (n + 1 for n in [1, 2, 3]), the first next(g) is 2, sum(g) over the remaining values is 7, and next(g, "done") is done.

• sum(n for n in range(4)) is 0 + 1 + 2 + 3 = 6.

• For g = (n * 2 for n in [3, 4]), next(g) is 6, list(g) is [8], and a second list(g) is [].

8. Exercises, the short version and next step

1. Write odd_squares(values) as a generator. For [2, 3, 4, 5], its list result must be [9, 25].

2. Create steps = StepRange(3, 10, 3), call next(steps) once to get 3, then confirm that list(steps) is [6, 9].

3. Feed flatten([[1, 2], [3], [4, 5]]) into a generator expression that keeps even values. It should produce [2, 4].

Five rules matter: an iterable produces an iterator; next() advances one state; yield pauses a generator function; generator expressions delay work; and every one-pass iterator can be exhausted. Test exact output before adding a pipeline stage.

For a structured next route, continue with DSA Using Python. The earlier Python Decorators and Generators: yield and Closures Traced connects generators to closures and decorator behaviour; this tutorial instead develops the iterator protocol, custom iterators and pull-based lazy pipelines.