seon99il

Python

List vs Generator in Python: Why Use Generators When Both Work in a For Loop?

3 min read

If you’ve been using Python for a while, you’ve probably noticed that both lists and generators can be used in a for loop almost interchangeably.

For example, these two snippets look almost the same:

# Using a list
nums = [i for i in range(10)]
for n in nums:
    print(n)

# Using a generator
nums_gen = (i for i in range(10))
for n in nums_gen:
    print(n)

Both will print numbers 0 through 9. So why would anyone bother using a generator? The answer lies in memory efficiency, evaluation strategy, and real-world use cases. Let’s dig deeper.

Fundamental Difference Between Lists and Generators

List

  • Stores all values in memory at once.
  • Example: [i for i in range(1_000_000)] → immediately allocates memory for 1 million numbers.
  • Pros: Random access is possible (nums[500] works).
  • Cons: Large datasets can consume huge amounts of memory.

Generator

  • Produces values on demand, one at a time (lazy evaluation).
  • Example: (i for i in range(1_000_000)) → nothing is stored; values are generated as needed.
  • Pros: Extremely memory-efficient, works with infinite streams.
  • Cons: No random access (nums_gen[500] raises an error).

Memory Usage Comparison

One of the clearest differences is memory consumption:

import sys

nums_list = [i for i in range(1_000_000)]
nums_gen = (i for i in range(1_000_000))
print("List size:", sys.getsizeof(nums_list))
print("Generator size:", sys.getsizeof(nums_gen))

Possible output:

List size: 8697464
Generator size: 112
  • The list consumes around 8.7 MB for 1 million numbers.
  • The generator is always tiny (112 bytes), regardless of dataset size.

This demonstrates that generators can handle massive datasets without overwhelming memory.

Evaluation Strategy: Eager vs Lazy

  • List: All values are computed and stored immediately (eager evaluation).
  • Generator: Values are produced only when requested (lazy evaluation).

This distinction makes generators especially powerful for infinite sequences.

def infinite_numbers():
    n = 0
    while True:
        yield n
        n += 1

gen = infinite_numbers()
for n in gen:
    if n > 10:
        break
    print(n)

Output:

0
1
2
3
4
5
6
7
8
9
10

A list can never represent an infinite sequence, but a generator can.

Real-World Use Cases

1. Large Log File Processing

In data engineering, logs often span gigabytes. Loading them into a list would crash memory, but a generator can handle them line by line.

def read_logs(file_path):
    with open(file_path, "r") as f:
        for line in f:
            yield line.strip()

for log in read_logs("access.log"):
    if "ERROR" in log:
        print(log)

This approach never loads the entire file into memory.

2. Streaming Data (Kafka, Sockets, etc.)

Streaming data is unbounded. Lists can’t handle it, but generators fit perfectly.

import time

def stream_data():
    n = 0
    while True:
        yield f"message {n}"
        n += 1
        time.sleep(1)  # simulate data arrival every second

for msg in stream_data():
    print(msg)
    if "5" in msg:
        break

Output:

message 0
message 1
message 2
message 3
message 4
message 5

This pattern underlies many real-time data consumers, such as Kafka consumers.

3. Data Transformation Pipelines

Generators make it easy to build data pipelines without intermediate storage.

def numbers():
    for i in range(10):
        yield i

def squared(seq):
    for num in seq:
        yield num ** 2

def even_only(seq):
    for num in seq:
        if num % 2 == 0:
            yield num

pipeline = even_only(squared(numbers()))
for val in pipeline:
    print(val)

Output:

0
4
16
36
64

Each stage processes values lazily, without creating large intermediate lists.

When Lists Are Better (I think)

Generators are not always the right choice. Lists are better when:

  1. Random access is needed
    • nums[100] works for lists, not generators.
  2. Multiple iterations are required
    • A generator is exhausted after one pass; lists can be reused.
  3. Dataset is small enough to fit in memory
    • Lists are often more convenient for quick access.

Summary

  • List: Stores everything in memory → fast random access, higher memory usage.
  • Generator: Produces items on demand → memory-efficient, suitable for streams and pipelines.

Use generators for:

  • Large or infinite datasets
  • Stream processing (logs, Kafka, sockets)
  • Transformation pipelines

Use lists for:

  • Random access
  • Reusing datasets
  • Small-scale data

So while lists and generators may look interchangeable in a for loop, they differ fundamentally in evaluation strategy and memory usage.

Knowing when to use each can make your Python code not only correct but also scalable and efficient.

Previous posts

View all posts →

Comments