Lecture 4: Iterators, generators and decorators

In this lecture, we will examine such important topics as containers and related concepts of iterators and generators. To conclude, we will look at decorators.

Python Containers

 Let's start in order. A container is a data type designed to store elements and provide a set of operations to work with them. The containers themselves and, as a rule, their elements are stored in memory. In Python, there are many different containers, among which are well-known ones: list, deque, set, frozensets, dict, defaultdict, OrderedDict, Counter, tuple, namedtuple, str. Technically, an object is a container when it provides the ability to determine the presence or absence of a specific element in it. Note that although most containers provide the ability to extract any element from them, the presence of this ability alone does not make an object a container, but merely an iterable object.

Iterable Objects

As mentioned above, most containers in Python are iterable. In addition, many other data types are also iterable objects, such as files, sockets, and the like. Unlike containers, which usually contain a finite number of elements, an iterable object can essentially represent an infinite source of data. By definition, an iterable object is any object that can provide an iterator. This iterator, in turn, returns individual elements. Generally speaking, this sounds a bit strange, but it is nevertheless very important to understand the difference between an iterator and an iterable object. Let's now look at an example.

x = [1, 2, 3]
y = iter(x)
z = iter(x)
next(y)
1
next(y)
2
next(z)
1

The iter function returns an iterator based on the container. Here, x is an iterable object, while y and z are two separate instances of the iterator producing values from the iterable object x. As we can see, both instances y and z maintain their state between calls to next. In this example, a list is used as the data source for the iterator, but strictly speaking, this is not necessary. In practice, to reduce the amount of code, classes of iterable objects implement both methods: __iter__ and __next__, with __iter__ returning an instance of the class. Thus, the class is simultaneously an iterable object and an iterator of itself. However, it is still considered best practice to return a separate object as an iterator.

Iterators

Now about the iterators themselves. So, an iterator is some auxiliary object that returns the next element each time the __next__ function is called with this object passed as an argument. Thus, any object that provides a __next__ method is an iterator, returning the next element when this method is called, and it doesn't matter exactly how this happens. In total, an iterator is a kind of factory that produces values. Whenever you ask it for "the next value", it knows how to do this because it preserves its internal state between calls to it. There are countless examples of using iterators.

For example, all functions of the itertools library return iterators. Some of them generate infinite sequences.

from itertools import count
counter = count(start=1)
print(next(counter))
# Output: 1
print(next(counter))
# Output: 2

Others create infinite sequences from finite ones. For example, the cycle method generates a repeating infinite sequence from the sequence ['red', 'white', 'blue'].

from itertools import cycle

colors = ['red', 'white', 'blue']
color_cycle = cycle(colors)

for _ in range(10):
    print(next(color_cycle))

# Output:
# red
# white
# blue
# red
# white
# blue
# red
# white
# blue
# red

Another useful method is islice. It returns a finite iterator from any large or even infinite sequence.

from itertools import cycle, islice

unlimited_numbers = cycle([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
limited_numbers = islice(unlimited_numbers, 3, 6)

for number in limited_numbers:
    print(number, end=' ')

# Output: 3 4 5

Generators

In general, a generator is a special, more elegant case of an iterator. Using a generator, you can create iterators, like the one we examined above, using more concise syntax, without creating a separate class with __iter__ and __next__ methods. Let's clarify a bit. Any generator is an iterator (but not vice versa). Consequently, any generator is a 'lazy factory' returning sequence values on demand.

Here's an example of a Fibonacci sequence iterator implemented as a generator.

def fib():
    a, b = 0, 1
    while True:
        yield a
        a, b = b, a + b

Note that fib is a regular function, nothing special. However, it lacks an return operator that returns a value. In this case, the return value of the function will be a generator (which is provided by the yield operator). That is, essentially, an iterator is a value factory that maintains a state between calls to it. Now, when the fib function is called, an instance of the generator will be created and returned. At this point, no code inside the function has been executed yet, and the generator is waiting to be called.

Then, the created generator instance can be passed as an argument to the islice() function, which also returns an iterator, so no code of the fib() function is executed yet.

from itertools import islice
gen = fib()
result = islice(gen, 0, 10)

Finally, you can convert this to a list by calling list with the iterator returned by the islice() function as an argument. For list to be able to build a list object based on the received argument, it needs to get all the values from this argument. To do this, list performs sequential calls to the next method of the iterator returned by the islice call, which in turn performs sequential calls to next in the gen iterator instance.

from itertools import islice

gen = fib()
result = islice(gen, 0, 10)

print(list(result))

# Output: [1, 1, 2, 3, 5, 8, 13, 21, 34, 55]

Decorators

While it's better to study the theory for the previous topics in today's lecture, here we'll start immediately with a practical example. For instance, we have a dictionary user with a username and access level, which in our case is "guest". There's also a function get_admin_password that returns a password. If we call the function, we'll get the admin password, but at the same time, the access level shouldn't theoretically allow this to happen.

user = {'name': 'Alex', 'access_level': 'guest'}

def get_admin_password():
    return '1234'

print(get_admin_password())

# Output: 1234

The simplest solution for this task is to write a conditional construct.

user = {'name': 'Alex', 'access_level': 'guest'}

def get_admin_password():
    return '1234'

if user['access_level'] == 'admin':
    print(get_admin_password())

Everything would be fine, but we only protected a specific function call, while we can still call this same function elsewhere and get the password. The function itself is still not protected.

user = {'name': 'Alex', 'access_level': 'guest'}

def get_admin_password():
    return '1234'

if user['access_level'] == 'admin':
    print(get_admin_password())

print(get_admin_password())

# Output: 1234

It becomes obvious to put the check in the function itself. Again, it seems that everything is fine, but it's not.
If you need to write more 'protected' functions in the future, you'll have to add the check each time, and that's not a good solution.

user = {'name': 'Alex', 'access_level': 'guest'}

def get_admin_password():
    if user['access_level'] == 'admin':
        return '1234'

print(get_admin_password())

# Output: None

In such situations, decorators are needed because they allow you to modify existing functions without changing their code.
The essence of decorators is that they take a function as input and return a new function with modified behavior.
Now we'll write a universal function make_secure that will be responsible for all security logic.

The new function will take any other function as an input argument. Then, right in the body of this function, we declare a new secure_function (yes, this can be done).
Inside the body of the new function, we check the user's access level. If it equals 'admin', we return the result of calling the function that make_secure took as input.
At the end, we return the new function. Note that we don't put parentheses because we're not calling it (but returning it).

def make_secure(func):
    def secure_funcrion():
        if user['access_level'] == 'admin':
            return func()
    
    return secure_funcrion

Next, in the second-to-last line, we redefine the get_admin_password variable. Initially, it stored the original function value, but now it essentially stores the new secure_funcrion.
Now the get_admin_password function will only return a value if the access level is appropriate.
And most importantly, make_secure will allow us to make any function secure, as required.

user = {'name': 'Alex', 'access_level': 'guest'}

def get_admin_password():
    ...

def make_secure(func):
    ...

get_admin_password = make_secure(get_admin_password)
print(get_admin_password())

# Output: None

In general, this is roughly what the simplest decorator looks like. It would be possible to return any other function, but then the whole point of the decorator would be lost.
Let's now add one more small detail to understand what's happening in the program.

...

def make_secure(func):
    def secure_funcrion():
        if user['access_level'] == 'admin':
            return func()
        else:
            return f'No admin permissions for {user["name"]}'
    
    return secure_funcrion

@ Syntax

Instead of writing make_secure(get_admin_password), we can make it even simpler: use the @ syntax, as shown below. Such notation is equivalent to what we wrote earlier.

user = {'name': 'Alex', 'access_level': 'guest'}

def make_secure(func):
    ...

@make_secure
def get_admin_password():
    return '1234'

print(get_admin_password())

# Output: No admin permissions for Alex

By the way, there's another problem. Since it's a new function, its internal name is also new.
Moreover, if the function had any documentation, it would also be lost.

user = {'name': 'Alex', 'access_level': 'guest'}

def make_secure(func):
    ...

@make_secure
def get_admin_password():
    return '1234'

print(get_admin_password.__name__)

# Output: secure_function

But the solution is quite simple: you can use the wraps decorator from the functools module. It substitutes certain arguments, docstring, and names so that the function doesn't change.
We wrap the function that is returned as the new one with this decorator. As an argument, we pass the function we're initially decorating.
This trick isn't necessary, but it's highly desirable in every decorator.

from functools import wraps

user = {'name': 'Alex', 'access_level': 'guest'}

def make_secure(func):
    @wraps(func)
    def secure_funcrion():
        if user['access_level'] == 'admin':
            return func()
        else:
            return f'No admin permissions for {user["name"]}'
    return secure_funcrion

@make_secure
def get_admin_password():
    ...

print(get_admin_password.__name__)

# Output: get_admin_password

Functions with Parameters being Decorated

Let's modify the logic of the original function a bit more: now it will also accept a panel from which the password is being requested.

@make_secure
def get_admin_password(panel):
    if panel == 'admin':
        return '1234'
    elif panel == 'billing':
        return 'super_secure_password'

Now we need to make secure_function accept this argument. But there's a problem: the make_secure decorator was planned to be universal. And other functions that need to be secured might take different parameters.
This is exactly why it's common to use *args, **kwargs. This allows accepting an unlimited number of arguments and passing them back to the decorated function.

from functools import wraps

user = {'name': 'Alex', 'access_level': 'admin'}

def make_secure(func):
    @wraps(func)
    def secure_funcrion(*args, **kwargs):
        if user['access_level'] == 'admin':
            return func(*args, **kwargs)
        else:
            return f'No admin permissions for {user["name"]}'
    return secure_funcrion

@make_secure
def get_admin_password(panel):
    ...

print(get_admin_password('billing'))
# Output: super_secure_password

Decorators with Parameters

Let's start by making the task a bit more complex. Let's say we have two different functions get_admin_password and get_user_password. Both should be secured, but the first one should only be accessible to admins, while the second one should be accessible to users.
This means that if the user's access level is 'admin', they can get the password from the first function but not from the second. If the access level is 'user', then it's exactly the opposite.

...

@make_secure
def get_admin_password():
    return '1234'

@make_secure
def get_user_password():
    return 'qwerty'

print(get_admin_password())
print(get_user_password())

Now pay attention to a small detail. In the @make_secure notation, we don't write parentheses because we're calling the function, and therefore we can't pass any parameters. However, this decorator function is still applied to the decorated function.
Now we'll write a new function that will accept some parameters and return the actual decorator.

Here we've added another level of functions and an accepted argument.
Essentially this is the same decorator that returns a decorator when called... But technically, it's more correct to call this a decorator factory. But if you don't want to overcomplicate things, just remember this technique as a 'decorator with parameters'.

def make_secure(access_level):
    def decorator(func):
        @wraps(func)
        def secure_funcrion(*args, **kwargs):
            ...
        
        return secure_funcrion
    return decorator

Next, we'll slightly modify the conditional construct, since now we need to compare with the passed access_level, not just with 'admin'.

def make_secure(access_level):
    def decorator(func):
        @wraps(func)
        def secure_funcrion(*args, **kwargs):
            if user['access_level'] == access_level:
                return func(*args, **kwargs)
            else:
                return f'No {access_level} permissions for {user["name"]}'
        return secure_funcrion
    return decorator

Now it's very important to put parentheses after applying the decorators and passing the corresponding arguments.
The notation @make_secure(...) will return a new decorator, which we previously declared a decorator.

def make_secure(access_level):
    ...

@make_secure('admin')
def get_admin_password():
    return '1234'

@make_secure('user')
def get_user_password():
    return 'qwerty'

Let's change the 'access_level' in the dictionary and see how everything works.
Overall, this is already almost a production-ready decorator that could be used even in real projects. So it's not that complicated after all

...

user = {'name': 'Alex', 'access_level': 'user'}

...

print(get_admin_password())
print(get_user_password())

# Output:
# No admin permissions for Alex
# qwerty

I think now you should understand everything about decorators, as we've covered almost all topics related to them. But there's one more thing - applying multiple decorators to a single function at once.
Let's see what happens when we apply several decorators at once.

def first_decorator(func):
    def wrapped():
        print('Inside first_decorator product')
        return func()
    return wrapped

def second_decorator(func):
    def wrapped():
        print('Inside second_decorator product')
        return func()
    return wrapped

@first_decorator
@second_decorator
def decorated():
    print('Finally called...')

decorated()

# Output:
# Inside first_decorator product
# Inside second_decorator product
# Finally called...

The same thing, but without the syntactic sugar in the form of @.

decorated = first_decorator(second_decorator(decorated))
decorated()

# Output:
# Inside first_decorator product
# Inside second_decorator product
# Finally called...

We can see that the first decorator was called first, then the second one. Let's break this down in detail.
The second_decorator function returns a new wrapped function, thus the function is replaced with wrapped inside second_decorator. After that, the first_decorator is called, which takes the wrapped function obtained from the second_decorator and returns another wrapped function, replacing decorated with it.
Therefore, the final decorated function is the wrapped function from first_decorator calling the function from second_decorator.

Here's another example of applying decorators. Notice that the tags first appear in the same order as the decorators, and then in reverse order. This happens because the decorators are called one inside another.

def bold(func):
    def wrapped():
        return f'<b>{func()}</b>'
    return wrapped

def italic(func):
    def wrapped():
        return f'<i>{func()}</i>'
    return wrapped

@bold
@italic
def hello():
    return 'hello'

print(hello())

# Output: <b><i>hello</i></b>