Iterable in Python is any object that can be looped over. In order for an iterable to be looped over, an iterable needs to be converted to an iterator using the __iter__() method. Once created, it is actually the iterator that gets iterated over. This is also what happens internally when you run a for-loop.
Let’s understand iterators and iterables very clearly.
This covers:
1. What exactly is an iterator and iterable?
2. How to tell the difference between iterator and iterble?
3. What exactly happens when you run a for-loop in Python?
4. How to create a class based iterator?
There is a minor difference between an iterable and an iterator. For example, the List is an iterable but not an iterator.
Let’s understand the difference clearly, so you can write python code that is more efficient, and will enable you to see solutions to problems in a way you might not have thought through before.
Many of the python objects that we have seen so far are ‘Iterables’. List, string, tuples etc are iterables.
What is an iterable?
An iterable is basically a Python object that can be looped over. This means, lists, strings, tuples, dicts and every other object that can be looped over is an iterable.
See this for-loop for example.
# items that appear on the RHS of the for-loop is an iterable
for i in [1,2,3,4,5]:
print(i)
Output
1
2
3
4
5
So what really happens when you run a for loop?
An iterable defines an __iter__() method which returns an iterator. This means, everytime you call the iter() on an iterable, it returns an iterator.
# Get iterator from iterable
iterator_from_list = iter([1,2,3,4,5])
type(iterator_from_list)
#> list_iterator
The iterator in turn has __next__() method defined.
So, whenever you use a for-loop in Python, the __next__() method is called automatically to get each item from the iterator, thus going through the process of iteration.
In a similar way, you can loop over strings, tuples, dictionaries, files, generators (which we will cover next) etc.
How to tell if an object can be looped over or is an iterable?
You can tell whether an object is iterable or not by the presence of the __iter__() dunder method.
So, technically speaking, any python object that defines a __iter__() method, is an iterable. This is a special method, aka, a ‘Dunder method’ or ‘magic method.’
# check the methods of list
L = [1, 2, 3]
print(dir(L))
Output:
#> ['__add__', '__class__', '__class_getitem__', '__contains__', '__delattr__', '__delitem__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getitem__', '__gt__', '__hash__', '__iadd__', '__imul__', '__init__', '__init_subclass__', '__iter__', '__le__', '__len__', '__lt__', '__mul__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__reversed__', '__rmul__', '__setattr__', '__setitem__', '__sizeof__', '__str__', '__subclasshook__', 'append', 'clear', 'copy', 'count', 'extend', 'index', 'insert', 'pop', 'remove', 'reverse', 'sort']
You can find the __iter__ method in the list above. Likewise, you can find that method in every other iterable.
We are now clear with what an iterable is.
So, what is an iterator?
What is an iterator?
You know that you will get an iterator by calling the iter() on an iterable.
Iterator is an iterable that remembers its state. Which means, it’s a python object with a state so it remembers where it is during iteration.
Like how an iterable has a __iter__() method, which gives an iterator, an iterator defines a __next__() method that makes the iteration possible.
S = "Roger"
print(type(S))
Output:
#> <class 'str'>;
# Create iterator.
T = S.__iter__()
# or
# T = iter(S)
print(type(T))
Output
#> <class 'str_iterator'>;
We now have an iterator. It must have a state, so the next time it is iterated, it will know how to get the next value.
It does it using the dunder __next__() method.
So, technically, an iterator is an object that has executed the dunder methods: __iter__ and __next__.
# T is an iterator so it must have a __next__() method. Look for __next__.
print(dir(T))
Output
#> ['__class__', '__delattr__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__iter__', '__le__', '__length_hint__', '__lt__', '__ne__', '__new__', '__next__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__setstate__', '__sizeof__', '__str__', '__subclasshook__']
print(next(T))
print(next(T))
print(next(T))
print(next(T))
Output
#> R
#> o
#> g
#> e
When you call next(), it is actually calling the dunder method __next__() in the background.
print(T.__next__())
Output
#> r
After it has exhasted all the items, it errors our with StopIteration on further calling __next__().
# StopIteration Error!
T.__next__()
That means the iterator has been exhausted.
Also notice, the list T also contains the __iter__() method, which makes it return the same iterable, instead.
Creating your own iterator object
Python allows you to create your own iterator object. All you need to do is to define the __iter__() and __next__() methods, along with the constructor (__init__) ofcourse.
with open("textfile.txt", mode="r", encoding="utf8") as f:
for i in f:
print(i)
Output:
#> Amid controversy over ‘motivated’ arrest in sand mining case,
#> Punjab Congress chief Navjot Singh Sidhu calls for ‘honest CM candidate’.
#> Amid the intense campaign for the Assembly election in Punjab,
#> due less than three weeks from now on February 20, the Enforcement Directorate (ED)
#> on Friday arrested Bhupinder Singh ‘Honey’, Punjab Chief Minister
#> Charanjit Singh Channi’s nephew, in connection with an illegal sand mining case.
#> He was later produced before a special court and sent to ED custody till February 8.
#> Sensing an opportunity to plug his case as CM face, Punjab Congress chief
#> Navjot Singh Sidhu said the Congress must choose an ‘honest’ person as
#> its Chief Ministerial face for the upcoming polls.
class ReadText(object):
"""A iterator that iterates through the lines of a text file
and prints the list of words in each line."""
def __init__(self, file, end):
self.file = open(file, mode="r", encoding="utf-8")
self.current = 0
self.end = end
def __iter__(self):
return self
def __next__(self):
if self.current < self.end:
self.current += 1 return(self.file.__next__())
else:
raise StopIteration
F = ReadText("textfile.txt", 8)
# Check iteration values
F.current, F.end
#> (0, 8)
# print the corpus vectors
for line in F:
print(line)
Output:
#> Amid controversy over ‘motivated’ arrest in sand mining case,
#> Punjab Congress chief Navjot Singh Sidhu calls for ‘honest CM candidate’
#> Amid the intense campaign for the Assembly election in Punjab,
#> due less than three weeks from now on February 20, the Enforcement Directorate (ED)
#> on Friday arrested Bhupinder Singh ‘Honey’, Punjab Chief Minister
#> Charanjit Singh Channi’s nephew, in connection with an illegal sand mining case.
#> He was later produced before a special court and sent to ED custody till February 8.
#> Sensing an opportunity to plug his case as CM face, Punjab Congress chief
# Check iteration values again
F.current, F.end
#> (8, 8)
Practice Exercises on Iterators:
Q1: Make changes to the code so that it returns a list of words in each line.
Q2: Write an iterator class that reverses a string
Solution 2:
class Reverse:
"""Iterator for looping over a sequence backwards."""
def __init__(self, data):
self.data = data
self.index = len(data)
def __iter__(self):
return self
def __next__(self):
if self.index == 0:
raise StopIteration
self.index = self.index - 1
return self.data[self.index]
rev = Reverse('Mighty Monkey')
rev = Reverse('Mighty Monkey')
for char in rev:
print(char)
Output:
#> y
#> e
#> k
#> n
#> o
#> M
#> y
#> t
#> h
#> g
#> i
#> M
Awesome! That’s how you create an iterator on your own.
Now, if this feels cumbersome, you can create and work with Generators.
