Iterable in Python is any object that can be looped over. In order for an iterable to be looped over, an iterable needs to be converted to an iterator using the
__iter__() method. Once created, it is actually the iterator that gets iterated over. This is also what happens internally when you run a for-loop.
Let’s understand iterators and iterables very clearly.
1. What exactly is an iterator and iterable?
2. How to tell the difference between iterator and iterble?
3. What exactly happens when you run a for-loop in Python?
4. How to create a class based iterator?
There is a minor difference between an iterable and an iterator. For example, the List is an iterable but not an iterator.
Let’s understand the difference clearly, so you can write python code that is more efficient, and will enable you to see solutions to problems in a way you might not have thought through before.
Many of the python objects that we have seen so far are ‘Iterables’. List, string, tuples etc are iterables.
What is an iterable?
An iterable is basically a Python object that can be looped over. This means, lists, strings, tuples, dicts and every other object that can be looped over is an iterable.
See this for-loop for example.
# items that appear on the RHS of the for-loop is an iterable for i in [1,2,3,4,5]: print(i)
1 2 3 4 5
So what really happens when you run a for loop?
An iterable defines an
__iter__() method which returns an iterator. This means, everytime you call the
iter() on an iterable, it returns an iterator.
# Get iterator from iterable iterator_from_list = iter([1,2,3,4,5]) type(iterator_from_list)
The iterator in turn has
__next__() method defined.
So, whenever you use a for-loop in Python, the
__next__() method is called automatically to get each item from the iterator, thus going through the process of iteration.
In a similar way, you can loop over strings, tuples, dictionaries, files, generators (which we will cover next) etc.
How to tell if an object can be looped over or is an iterable?
You can tell whether an object is iterable or not by the presence of the
__iter__() dunder method.
So, technically speaking, any python object that defines a
__iter__() method, is an iterable. This is a special method, aka, a ‘Dunder method’ or ‘magic method.’
# check the methods of list L = [1, 2, 3] print(dir(L))
#> ['__add__', '__class__', '__class_getitem__', '__contains__', '__delattr__', '__delitem__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getitem__', '__gt__', '__hash__', '__iadd__', '__imul__', '__init__', '__init_subclass__', '__iter__', '__le__', '__len__', '__lt__', '__mul__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__reversed__', '__rmul__', '__setattr__', '__setitem__', '__sizeof__', '__str__', '__subclasshook__', 'append', 'clear', 'copy', 'count', 'extend', 'index', 'insert', 'pop', 'remove', 'reverse', 'sort']
You can find the
__iter__ method in the list above. Likewise, you can find that method in every other iterable.
We are now clear with what an iterable is.
So, what is an iterator?
What is an iterator?
You know that you will get an
iterator by calling the
iter() on an iterable.
Iterator is an iterable that remembers its state. Which means, it’s a python object with a state so it remembers where it is during iteration.
Like how an iterable has a
__iter__() method, which gives an iterator, an iterator defines a
__next__() method that makes the iteration possible.
S = "Roger" print(type(S))
#> <class 'str'>;
# Create iterator. T = S.__iter__() # or # T = iter(S) print(type(T))
#> <class 'str_iterator'>;
We now have an iterator. It must have a state, so the next time it is iterated, it will know how to get the next value.
It does it using the dunder
So, technically, an iterator is an object that has executed the dunder methods:
# T is an iterator so it must have a __next__() method. Look for __next__. print(dir(T))
#> ['__class__', '__delattr__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__iter__', '__le__', '__length_hint__', '__lt__', '__ne__', '__new__', '__next__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__setstate__', '__sizeof__', '__str__', '__subclasshook__']
print(next(T)) print(next(T)) print(next(T)) print(next(T))
#> R #> o #> g #> e
When you call
next(), it is actually calling the dunder method
__next__() in the background.
After it has exhasted all the items, it errors our with
StopIteration on further calling
# StopIteration Error! T.__next__()
That means the iterator has been exhausted.
Also notice, the list
T also contains the
__iter__() method, which makes it return the same iterable, instead.
Creating your own iterator object
Python allows you to create your own iterator object. All you need to do is to define the
__next__() methods, along with the constructor (
with open("textfile.txt", mode="r", encoding="utf8") as f: for i in f: print(i)
#> Amid controversy over ‘motivated’ arrest in sand mining case, #> Punjab Congress chief Navjot Singh Sidhu calls for ‘honest CM candidate’. #> Amid the intense campaign for the Assembly election in Punjab, #> due less than three weeks from now on February 20, the Enforcement Directorate (ED) #> on Friday arrested Bhupinder Singh ‘Honey’, Punjab Chief Minister #> Charanjit Singh Channi’s nephew, in connection with an illegal sand mining case. #> He was later produced before a special court and sent to ED custody till February 8. #> Sensing an opportunity to plug his case as CM face, Punjab Congress chief #> Navjot Singh Sidhu said the Congress must choose an ‘honest’ person as #> its Chief Ministerial face for the upcoming polls.
class ReadText(object): """A iterator that iterates through the lines of a text file and prints the list of words in each line.""" def __init__(self, file, end): self.file = open(file, mode="r", encoding="utf-8") self.current = 0 self.end = end def __iter__(self): return self def __next__(self): if self.current < self.end: self.current += 1 return(self.file.__next__()) else: raise StopIteration F = ReadText("textfile.txt", 8)
# Check iteration values F.current, F.end #> (0, 8)
# print the corpus vectors for line in F: print(line)
#> Amid controversy over ‘motivated’ arrest in sand mining case, #> Punjab Congress chief Navjot Singh Sidhu calls for ‘honest CM candidate’ #> Amid the intense campaign for the Assembly election in Punjab, #> due less than three weeks from now on February 20, the Enforcement Directorate (ED) #> on Friday arrested Bhupinder Singh ‘Honey’, Punjab Chief Minister #> Charanjit Singh Channi’s nephew, in connection with an illegal sand mining case. #> He was later produced before a special court and sent to ED custody till February 8. #> Sensing an opportunity to plug his case as CM face, Punjab Congress chief
# Check iteration values again F.current, F.end #> (8, 8)
Practice Exercises on Iterators:
Q1: Make changes to the code so that it returns a list of words in each line.
Q2: Write an iterator class that reverses a string
class Reverse: """Iterator for looping over a sequence backwards.""" def __init__(self, data): self.data = data self.index = len(data) def __iter__(self): return self def __next__(self): if self.index == 0: raise StopIteration self.index = self.index - 1 return self.data[self.index] rev = Reverse('Mighty Monkey')
rev = Reverse('Mighty Monkey') for char in rev: print(char)
#> y #> e #> k #> n #> o #> M #> y #> t #> h #> g #> i #> M
Awesome! That’s how you create an iterator on your own.
Now, if this feels cumbersome, you can create and work with Generators.