List vs Generator in Python: Why Use Generators When Both Work in a For Loop?
3 min read
If you’ve been using Python for a while, you’ve probably noticed that both lists and generators can be used in a for loop almost interchangeably.
For example, these two snippets look almost the same:
# Using a list
nums = [i for i in range(10)]
for n in nums:
print(n)
# Using a generator
nums_gen = (i for i in range(10))
for n in nums_gen:
print(n)
Both will print numbers 0 through 9. So why would anyone bother using a generator? The answer lies in memory efficiency, evaluation strategy, and real-world use cases. Let’s dig deeper.
Fundamental Difference Between Lists and Generators
List
- Stores all values in memory at once.
- Example:
[i for i in range(1_000_000)]→ immediately allocates memory for 1 million numbers. - Pros: Random access is possible (
nums[500]works). - Cons: Large datasets can consume huge amounts of memory.
Generator
- Produces values on demand, one at a time (lazy evaluation).
- Example:
(i for i in range(1_000_000))→ nothing is stored; values are generated as needed. - Pros: Extremely memory-efficient, works with infinite streams.
- Cons: No random access (
nums_gen[500]raises an error).
Memory Usage Comparison
One of the clearest differences is memory consumption:
import sys
nums_list = [i for i in range(1_000_000)]
nums_gen = (i for i in range(1_000_000))
print("List size:", sys.getsizeof(nums_list))
print("Generator size:", sys.getsizeof(nums_gen))
Possible output:
List size: 8697464
Generator size: 112
- The list consumes around 8.7 MB for 1 million numbers.
- The generator is always tiny (112 bytes), regardless of dataset size.
This demonstrates that generators can handle massive datasets without overwhelming memory.
Evaluation Strategy: Eager vs Lazy
- List: All values are computed and stored immediately (eager evaluation).
- Generator: Values are produced only when requested (lazy evaluation).
This distinction makes generators especially powerful for infinite sequences.
def infinite_numbers():
n = 0
while True:
yield n
n += 1
gen = infinite_numbers()
for n in gen:
if n > 10:
break
print(n)
Output:
0
1
2
3
4
5
6
7
8
9
10
A list can never represent an infinite sequence, but a generator can.
Real-World Use Cases
1. Large Log File Processing
In data engineering, logs often span gigabytes. Loading them into a list would crash memory, but a generator can handle them line by line.
def read_logs(file_path):
with open(file_path, "r") as f:
for line in f:
yield line.strip()
for log in read_logs("access.log"):
if "ERROR" in log:
print(log)
This approach never loads the entire file into memory.
2. Streaming Data (Kafka, Sockets, etc.)
Streaming data is unbounded. Lists can’t handle it, but generators fit perfectly.
import time
def stream_data():
n = 0
while True:
yield f"message {n}"
n += 1
time.sleep(1) # simulate data arrival every second
for msg in stream_data():
print(msg)
if "5" in msg:
break
Output:
message 0
message 1
message 2
message 3
message 4
message 5
This pattern underlies many real-time data consumers, such as Kafka consumers.
3. Data Transformation Pipelines
Generators make it easy to build data pipelines without intermediate storage.
def numbers():
for i in range(10):
yield i
def squared(seq):
for num in seq:
yield num ** 2
def even_only(seq):
for num in seq:
if num % 2 == 0:
yield num
pipeline = even_only(squared(numbers()))
for val in pipeline:
print(val)
Output:
0
4
16
36
64
Each stage processes values lazily, without creating large intermediate lists.
When Lists Are Better (I think)
Generators are not always the right choice. Lists are better when:
- Random access is needed
nums[100]works for lists, not generators.
- Multiple iterations are required
- A generator is exhausted after one pass; lists can be reused.
- Dataset is small enough to fit in memory
- Lists are often more convenient for quick access.
Summary
- List: Stores everything in memory → fast random access, higher memory usage.
- Generator: Produces items on demand → memory-efficient, suitable for streams and pipelines.
Use generators for:
- Large or infinite datasets
- Stream processing (logs, Kafka, sockets)
- Transformation pipelines
Use lists for:
- Random access
- Reusing datasets
- Small-scale data
So while lists and generators may look interchangeable in a for loop, they differ fundamentally in evaluation strategy and memory usage.
Knowing when to use each can make your Python code not only correct but also scalable and efficient.