Cognilabs

🏁 Everything is an object in Python. And Python’s Dunder Methods (short for “double underscores”) are one of the most core concepts in the language. They are actually a language’s ‘original’ capabilities, all of the other concepts we know so far are constructed using these methods. Using them we can also define the behavior of objects for various operations. They are also called as Magic Methods. In this article, we introduce all of magic methods and give examples with explanations.

✅ We can categorize the magic methods into these groups:

1. Object Initialization and Finalizing
   • __init__(self, [...]): Initializes a new object instance.
   • __new__(cls, [...]): Controls the creation of a new instance.
   • __del__(self): Deletes of Finalizes the object instance.
2. Comparison
   • __eq__(self, other): Defines behavior for the equality operator, ==.
   • __ne__(self, other): Defines behavior for the inequality operator, !=.
   • __lt__(self, other): Defines behavior for the less than operator, <.
   • __le__(self, other): Defines behavior for the less than or equal to operator, <=.
   • __gt__(self, other): Defines behavior for the greater than operator, >.
   • __ge__(self, other): Defines behavior for the greater than or equal to operator, >=.
3. Arithmetic
   • __add__(self, other): Implements addition.
   • __sub__(self, other): Implements subtraction.
   • __mul__(self, other): Implements multiplication.
   • __truediv__(self, other): Implements division (/).
   • __floordiv__(self, other): Implements integer division (//).
   • __mod__(self, other): Implements modulo operation (%).
   • __pow__(self, other[, modulo]): Implements the behavior of power operator (**).
   • __divmod__(self, other): Implements the divmod() function.
4. Reflected Arithmetic
   • __radd__(self, other): Implements addition when the left operand does not support addition.
   • __rsub__(self, other): Implements subtraction when the left operand does not support subtraction.
   • __rmul__(self, other): Implements multiplication when the left operand does not support multiplication.
   • __rtruediv__(self, other): Implements true division when the left operand does not support true division.
   • __rfloordiv__(self, other): Implements floor division when the left operand does not support floor division.
   • __rmod__(self, other): Implements modulo when the left operand does not support modulo.
   • __rpow__(self, other): Implements power function when the left operand does not support power.
   • __rmatmul__(self, other): Implements matrix multiplication when the left operand does not support matrix multiplication.
5. Augmented Assignment
   • __iadd__(self, other): Implements in-place addition (i.e., +=).
   • __isub__(self, other): Implements in-place subtraction (i.e., -=).
   • __imul__(self, other): Implements in-place multiplication (i.e., *=).
   • __itruediv__(self, other): Implements in-place true division (i.e., /=).
   • __ifloordiv__(self, other): Implements in-place floor division (i.e., //=).
   • __imod__(self, other): Implements in-place modulo (i.e., %=).
   • __ipow__(self, other): Implements in-place exponentiation (i.e., **=).
   • __ilshift__(self, other): Implements in-place left shift (i.e., <<=).
   • __irshift__(self, other): Implements in-place right shift (i.e., >>=).
   • __iand__(self, other): Implements in-place bitwise AND (i.e., &=).
   • __ixor__(self, other): Implements in-place bitwise XOR (i.e., ^=).
   • __ior__(self, other): Implements in-place bitwise OR (i.e., |=).
   • __imatmul__(self, other): Implements in-place matrix multiplication (i.e., @=).
6. Type Conversion
   • __repr__(self): Official string representation of the object.
   • __str__(self): Informal or nicely printable string representation of the object.
   • __bool__(self): Implements type conversion to bool.
   • __int__(self): Implements type conversion to int.
   • __float__(self): Implements type conversion to float.
   • __complex__(self): Implements type conversion to complex.
   • __bytes__(self): Defines behavior for when bytes() is called on an instance.
   • __format__(self, format_spec): Defines behavior for when an object is formatted using the format() function.
7. Container
   • __len__(self): Returns the number of items in the container.
   • __getitem__(self, key): Allows accessing an item using the subscription syntax.
   • __setitem__(self, key, value): Allows setting an item using the subscription syntax.
   • __delitem__(self, key): Allows deleting an item using the subscription syntax.
   • __iter__(self): Should return an iterator for the container’s items.
   • __reversed__(self): Should return a reverse iterator for the container’s items.
   • __contains__(self, item): Should return True if the item is in the container, False otherwise.
   • __next__(self): Advances to the next item in the sequence. If there are no further items, it should raise StopIteration. This is typically used within an iterator object.
   • __missing__(self, key): Defines the behavior for when a requested key is not found in the dictionary or other collection.
   • __length_hint__(self): Returns an estimation of the number of remaining items in an iterable. This is optional and primarily used to optimize memory allocations when converting iterables to lists.
8. Context Management
   • __enter__(self): Enter the runtime context related to this object.
   • __exit__(self, exc_type, exc_value, traceback): Exit the runtime context related to this object.
9. Attribute Access
   • __getattr__(self, name): Defines behavior when an attribute lookup has not found the attribute in the usual places.
   • __getattribute__(self, name): Defines behavior for attribute access.
   • __setattr__(self, name, value): Defines behavior for attribute assignment.
   • __delattr__(self, name): Defines behavior for deleting an attribute.
10. Asynchronous operations
    • __await__(self): Must return an iterator which is itself awaited in an asynchronous context. Used to make objects awaitable.
    • __aenter__(self) and __aexit__(self, exc_type, exc_value, traceback): These methods are used to define asynchronous context managers, similar to __enter__ and __exit__ for synchronous context managers but designed for use with the async with statement.
    • __aiter__(self) and __anext__(self): These methods are used to define asynchronous iterators, which can be used in an async for loop.
11. Callability
    • __call__(self, [*args, **kwargs]): Allows an instance of a class to be called as a function.

There are much more available actually, but we listed all of the very important and needed methods in a single article.

---

Enough with a list and a general definitions. Now, let’s consider examples.

1. Object Initialization and Finalizing

In the following example, we tried implementing a Resource class with restriction of creating instances only once. This pattern is also called Singleton:COPYCOPY

class Resource:
    _instance_count = 0 

    def __new__(cls):
        # Limit the creation of instances to only one
        if cls._instance_count == 0:
            print("Creating the object")
            instance = super(Resource, cls).__new__(cls)
            cls._instance_count += 1
            return instance
        else:
            print("Instance already created, returning existing instance")
            return None

    def __init__(self):
        print("Initializing the resource")
        self.data = "Resource data"

    def __del__(self):
        print("Cleaning up the resource")
        Resource._instance_count -= 1

# Usage
resource1 = Resource()  # Object creation and initialization
print(resource1.data)

resource2 = Resource()  # Attempt to create another instance

# Output when the program ends or when 'del resource1' is explicitly called
del resource1

Here, we are declaring a private attribute _instance_count to keep track of the number of instances and restricting that to 1. Then, we are overriding __new__() , __init__() and __del__() methods by specifying our own logic.

2. Comparison

This time, we implement a Card class to represent the usage of comparison magic methods.COPYCOPY

class Card:
    suits = ('Clubs', 'Diamonds', 'Hearts', 'Spades')
    ranks = {'2': 2, '3': 3, '4': 4, '5': 5, '6': 6, '7': 7, '8': 8, '9': 9, '10': 10, 'J': 11, 'Q': 12, 'K': 13, 'A': 14}

    def __init__(self, rank, suit):
        if rank in self.ranks and suit in self.suits:
            self.rank = rank
            self.suit = suit
        else:
            raise ValueError("Invalid rank or suit")

    def __repr__(self):
        return f"Card('{self.rank}', '{self.suit}')"

    def __eq__(self, other):
        return self.rank == other.rank and self.suit == other.suit

    def __ne__(self, other):
        return not self.__eq__(other)

    def __lt__(self, other):
        if self.ranks[self.rank] == self.ranks[other.rank]:
            return self.suits.index(self.suit) < self.suits.index(other.suit)
        return self.ranks[self.rank] < self.ranks[other.rank]

    def __le__(self, other):
        return self.__lt__(other) or self.__eq__(other)

    def __gt__(self, other):
        return not self.__le__(other)

    def __ge__(self, other):
        return not self.__lt__(other)

# Usage
card1 = Card('Q', 'Hearts')
card2 = Card('Q', 'Spades')
card3 = Card('J', 'Spades')
card4 = Card('Q', 'Hearts')

print(card1 == card2)  # False
print(card1 != card3)  # True
print(card1 < card2)   # True
print(card1 <= card4)  # True
print(card1 > card3)   # True
print(card1 >= card2)  # False

It is clear that we cannot just compare the Card instances as we do ints. Or check for which one is bigger with >= or <=, right? Actually, these comparison operators call the dunder methods we specified above implicitly. Hence, now we can easily compare two cards according to their ranks and suits, as we defined our own logic of comparison.

3. Arithmetic

Do you believe that even very fundamental operations like + and - are also actually a dunder method? In Python, yes. For example, when we do 1 + 1, __add__() method is called in built-in int class (namely first 1) and then the result is obtained. As you can see, even Python’s very basic operators are far far away from low-level or actual additions.COPYCOPY

class Fraction:
    def __init__(self, numerator, denominator):
        if denominator == 0:
            raise ValueError("Denominator cannot be zero.")
        self.numerator = numerator
        self.denominator = denominator

    def __repr__(self):
        return f"Fraction({self.numerator}, {self.denominator})"

    def __add__(self, other):
        new_numerator = self.numerator * other.denominator + self.denominator * other.numerator
        new_denominator = self.denominator * other.denominator
        return Fraction(new_numerator, new_denominator)

    def __sub__(self, other):
        new_numerator = self.numerator * other.denominator - self.denominator * other.numerator
        new_denominator = self.denominator * other.denominator
        return Fraction(new_numerator, new_denominator)

    def __mul__(self, other):
        new_numerator = self.numerator * other.numerator
        new_denominator = self.denominator * other.denominator
        return Fraction(new_numerator, new_denominator)

    def __truediv__(self, other):
        new_numerator = self.numerator * other.denominator
        new_denominator = self.denominator * other.numerator
        if new_denominator == 0:
            raise ValueError("Division by zero.")
        return Fraction(new_numerator, new_denominator)

    def __floordiv__(self, other):
        result = self.__truediv__(other)
        return Fraction(result.numerator // result.denominator, 1)

    def __mod__(self, other):
        result = self.__truediv__(other)
        return Fraction(result.numerator % result.denominator, 1)

    def __pow__(self, power, modulo=None):
        if modulo is not None:
            return Fraction(pow(self.numerator, power, modulo), pow(self.denominator, power, modulo))
        return Fraction(self.numerator ** power, self.denominator ** power)

    def __divmod__(self, other):
        return (self.__floordiv__(other), self.__mod__(other))

# Usage
f1 = Fraction(1, 2)
f2 = Fraction(3, 4)

print(f1 + f2)  # Fraction(10, 8)
print(f1 - f2)  # Fraction(-2, 8)
print(f1 * f2)  # Fraction(3, 8)
print(f1 / f2)  # Fraction(4, 6)
print(f1 // f2)  # Fraction(0, 1)
print(f1 % f2)  # Fraction(4, 6)
print(f1 ** 2)  # Fraction(1, 4)
print(divmod(f1, f2))  # (Fraction(0, 1), Fraction(4, 6))

In this example, we tried implementing Fraction class and the functionality to perform simple operations on fractions. We specified our own logic for each of the dunder method and now walya! You now have Fraction data type in Python!

4. Reflected Arithmetic

Reflected arithmetic methods are special methods that provide support for arithmetic operations where the left operand does not support the corresponding operation but the right operand does. These methods are useful for interoperability between different types or classes.

In this example, we defined a CustomNumber class and specified right addition and right subtraction special methods:COPYCOPY

class CustomNumber:
    def __init__(self, value):
        self.value = value

    def __radd__(self, other):
        return other + self.value

    def __rsub__(self, other):
        return other - self.value

    def __repr__(self):
        return f"CustomNumber({self.value})"

# Test the CustomNumber class with reflected operations
cn = CustomNumber(10)

# Reflected addition
print(5 + cn)  # Output: 15

# Reflected subtraction
print(20 - cn)  # Output: 10

In the first example, 5 + cn cannot be added simply by using __add__() method of 5 (int). Hence, the interpreter calls the __radd__() method of cn (CustomNumber). So, 5+cn is equivalent to cn.__radd__(5). Subtraction and any other such methods have this behaviour.

5. Augmented Assignment

These are special methods used to customize the behavior of compound assignments, which combine a binary operation and an assignment. In this example, we tried implementing +=, +- and +* operators for Vector objects, just like we have for integers:COPYCOPY

class Vector:
    def __init__(self, elements):
        self.elements = list(elements)

    def __repr__(self):
        return f"Vector({self.elements})"

    def __iadd__(self, other):
        if len(self.elements) != len(other.elements):
            raise ValueError("Vectors must be of same length")
        self.elements = [a + b for a, b in zip(self.elements, other.elements)]
        return self

    def __isub__(self, other):
        if len(self.elements) != len(other.elements):
            raise ValueError("Vectors must be of same length")
        self.elements = [a - b for a, b in zip(self.elements, other.elements)]
        return self

    def __imul__(self, scalar):
        self.elements = [a * scalar for a in self.elements]
        return self

# Usage
v1 = Vector([1, 2, 3])
v2 = Vector([4, 5, 6])

v1 += v2
print(v1)  # Vector([5, 7, 9])

v1 -= v2
print(v1)  # Vector([1, 2, 3])

v1 *= 3
print(v1)  # Vector([3, 6, 9])

We defined a logic that when an operation is performed on Vector object, as it is constructed from a list here, the operation is performed on each element of that list. Hence, we can achieve the ‘simulation’ of addition, subtraction and multiplication of vectors using augmented assignments.

6. Type Conversion

Type conversions are much related to temperatures, right? From Celsius to Farenheit or even Kelvin. Thus, we tried implementing Temperature class to represent type conversion dunder methods:COPYCOPY

class Temperature:
    def __init__(self, celsius):
        self.celsius = celsius

    def __repr__(self):
        return f"Temperature({self.celsius}°C)"

    def __int__(self):
        # Convert Celsius to nearest whole number Fahrenheit
        return int((self.celsius * 9 / 5) + 32)

    def __float__(self):
        # Return the temperature in Celsius as a float
        return float(self.celsius)

    def __complex__(self):
        # Represent the temperature with the imaginary part set to zero
        return complex(self.celsius, 0)

    def __bool__(self):
        # Return False if the temperature is at absolute zero, else True
        return not self.celsius == -273.15

    def __str__(self):
        # User-friendly string representation
        return f"{self.celsius}° Celsius"

    def __bytes__(self):
        # Convert the string representation to bytes
        return str(self).__bytes__()

# Usage
temp = Temperature(25)

print(int(temp))     # 77
print(float(temp))   # 25.0
print(complex(temp)) # (25+0j)
print(bool(temp))    # True
print(str(temp))     # 25° Celsius
print(bytes(temp))   # b'25° Celsius'

Here, we did not much customize the type conversion logic, instead we used the built-in functions like float() (which also calls the built-in dunder method __float__() under the hood).

7. Container

Most important category I think. Since, every iterable collection in Python has __iter__() , __next__() and such dunder methods that make them collection or iterable. In this example, we implemented CustomList class that simulates a custom list and we made that iterable using these dunder methods:COPYCOPY

class CustomList:
    def __init__(self, initial_data=None):
        self.data = initial_data if initial_data is not None else []
        self.index = 0

    def __repr__(self):
        return f"CustomList({self.data})"

    def __iter__(self):
        self.index = 0
        return self

    def __next__(self):
        if self.index < len(self.data):
            result = self.data[self.index]
            self.index += 1
            return result
        else:
            raise StopIteration

    def __length_hint__(self):
        return len(self.data) - self.index

class CustomDict(dict):
    def __missing__(self, key):
        return f"{key} not found!"

# Usage for CustomList
clist = CustomList([1, 2, 3, 4, 5])
iterator = iter(clist)
print(next(iterator))  # 1
print(next(iterator))  # 2
print(list(iterator))  # [3, 4, 5]
print(clist.__length_hint__())  # 2 (after consuming three elements)

# Usage for CustomDict
cdict = CustomDict({'a': 1, 'b': 2})
print(cdict['c'])  # c not found!

8. Context Management

This example will create a temporary directory when the context is entered and delete it when the context is exited, showcasing a common use case in file system management.COPYCOPY

import os
import shutil
import tempfile

class TempDirectoryManager:
    def __enter__(self):
        # Create a temporary directory and store its path
        self.temp_dir = tempfile.mkdtemp()
        print(f"Created temporary directory at {self.temp_dir}")
        return self.temp_dir  # This can be used by the with-statement

    def __exit__(self, exc_type, exc_val, exc_tb):
        # Clean up the directory when exiting the context
        shutil.rmtree(self.temp_dir)
        print(f"Deleted temporary directory at {self.temp_dir}")
        return False  # Allow any exception to propagate

# Usage example
with TempDirectoryManager() as temp_dir:
    # User can do anything with temp_dir here
    with open(os.path.join(temp_dir, 'example.txt'), 'w') as f:
        f.write("Hello, world!")

    # Check what's inside temp_dir
    print("Contents of the temporary directory:", os.listdir(temp_dir))

# After exiting the block, the temporary directory is cleaned up

As soon as with statement is used, the __enter__() method is called and context manager starts working. It creates a temporary directory and then we can do our operations in that directory. After a code inside with is completed or error occurs, __exit__() method is called.

9. Attribute Access

How we access to attributes inside classes? Do this happen by chance? No, as you expected. We have get, set and del special attribute methods that define this behaviour.COPYCOPY

class LoggedAttrs:
    def __init__(self):
        self._storage = {}

    def __getattr__(self, name):
        value = self._storage.get(name, f"{name} not found")
        print(f"Accessing: {name} -> {value}")
        return value

    def __setattr__(self, name, value):
        if name == "_storage":
            super().__setattr__(name, value)
        else:
            print(f"Setting: {name} -> {value}")
            self._storage[name] = value

    def __delattr__(self, name):
        if name in self._storage:
            print(f"Deleting: {name}")
            del self._storage[name]
        else:
            print(f"{name} not found, nothing to delete")

# Usage
obj = LoggedAttrs()
obj.x = 10  # Setting: x -> 10
print(obj.x)  # Accessing: x -> 10
del obj.x  # Deleting: x

# Trying to access or delete non-existing attributes
print(obj.y)  # Accessing: y -> y not found
del obj.y  # y not found, nothing to delete

In this example, we defined a test class LoggedAttrs to simulate what happens when we assign a value to an attribute under the hood. Here, obj.x = 10 calls obj.__setattr__(x, 10) method, print(obj.x) calls print(obj.__getattr__(x))function and del obj.x calls obj.__delattr__(x) method.

10. Asynchronous operations

As Python introduced asynchronous programming features with asyncio, they constructed them using additional special dunder methods. In this example, I’ll demonstrate an asynchronous context manager and iterator. Let’s imagine we’re asynchronously reading lines from a simulated “sensor” that produces data over time.COPYCOPY

import asyncio

class AsyncSensorReader:
    def __init__(self, data):
        self.data = data
        self.index = 0

    async def __aenter__(self):
        print("Opening sensor reader")
        return self

    async def __aexit__(self, exc_type, exc_val, exc_tb):
        print("Closing sensor reader")
        return False  # Propagate exceptions if any

    def __aiter__(self):
        return self

    async def __anext__(self):
        if self.index < len(self.data):
            value = self.data[self.index]
            await asyncio.sleep(1)  # Simulate an asynchronous operation, e.g., waiting for data
            self.index += 1
            return value
        else:
            raise StopAsyncIteration

async def read_sensor_data():
    data = ["Temperature: 20°C", "Temperature: 21°C", "Temperature: 22°C"]
    async with AsyncSensorReader(data) as reader:
        async for line in reader:
            print(line)

# Run the async function
asyncio.run(read_sensor_data())

In this code, __aenter__ is called when entering the context using async with. It initializes the context. __aexit__ is called when exiting the context, and it is used to clean up resources. __aiter__ simply returns the object itself, which must have an __anext__ method. __anext__ is an asynchronous method that you use to fetch the next item in the iterator. If there are no more items, it raises StopAsyncIteration.

11. Callability

This single method makes a class callable, namely a function. We know that, in Python, all functions are actually a first-class objects and for this reason they are objects and functions at the same time!COPYCOPY

class Adder:
    def __init__(self, initial_value=0):
        self.value = initial_value

    def __call__(self, add_value):
        """ Increase the stored value by add_value and return the result. """
        self.value += add_value
        return self.value

# Usage
adder = Adder(10)  # Create an instance of Adder, initialized with value 10
print(adder(5))  # Call the instance like a function, output will be 15
print(adder(3))  # Output will be 18

# adder can be used in a place where a callable is expected
print(list(map(adder, [1, 2, 3])))  # Output will be [19, 21, 24], as it adds to the previous value

We tried implementing Adder class that constantly adds numbers when one gives as arguments. We defined a __call__() and specified what happens when the object is called as function, means what it will return. You can see, we may construct our own behaviour for an object and make them functions completely as we want.

Reference material: https://www.pythonmorsels.com/every-dunder-method

---

Although, this is not all dunder methods (they are so many, believe me!), we tried considering as much as possible with practical examples. The main idea behind this article generally is to understand how Python is constructed under the hood, at least on 1st level of low layer. And we should now enough understanding that even Python doesn’t use direct physical memories or pointers like C/C++, or static types like in Java, it implemented all of these using classes perfectly.

🏁 That’s it for today, thanks!

By Samandar Komilov