In place operations and objects

Q&A DQ Courses
#1

I was reading https://lerner.co.il/2019/06/06/why-do-python-lists-let-you-a-tuple-when-you-cant-a-tuple/
which explains the difference between += and + in python.

Below is a truncated version (without def add)

class Foo(object):

    def __init__(self, x):
        self.x = x

    def __iadd__(self, other):
        print("In __iadd__")
        self.x = self.x + other.x
        # why must return this?
        return self      

f1 = Foo(10)
f2 = Foo(20)
f1 += f2

print(f1.x)

My question is why must i return self? I thought the previous self.x = self.x + other.x would have already editted f1’s x attribute value to 30 and the f1 in the global scope should print 30 rather than None? (I know not having return statement is equivalent to return None). Where is the return statement returning to? I find it easier to understand non in-place operations like x = x + 1, where the returned object is bound to the Left Hand Side. With in-place operations like f1 += f2, there is no Left Hand Side?

An extension question is do all in-place operations require this “return self” pattern? (I know sklearn’s model objects and pandas methods on their data structures often do this)

Looking for a deeper explanation than “this is just how python is designed”

#2

Here’s a good answer on stackoverflow:

def __iadd__(self, other):
    self.number += other.number
    return self

In addition to what’s correctly given in answers above, it is worth explicitly clarifying that when __iadd__ is overriden, the x += y operation does NOT end with the end of __iadd__ method.

Instead, it ends with x = x.__iadd__(y) . In other words, Python assigns the return value of your __iadd__ implementation to the object you’re “adding to”, AFTER the implementation completes.

This means it is possible to mutate the left side of the x += y operation so that the final implicit step fails. Consider what can happen when you are adding to something that’s within a list:

>>> x[1] += y # x has two items

Now, if your __iadd__ implementation (a method of an object at x[1] ) erroneously or on purpose removes the first item ( x[0] ) from the beginning of the list, Python will then run your __iadd__ method) & try to assign its return value to x[1] . Which will no longer exist (it will be at x[0] ), resulting in an ÌndexError .

Or, if your __iadd__ inserts something to beginning of x of the above example, your object will be at x[2] , not x[1] , and whatever was earlier at x[0] will now be at x[1] and be assigned the return value of the __iadd__ invocation.

Unless one understands what’s happening, resulting bugs can be a nightmare to fix.

The outcome of += might mutate the left hand side object. Therefore it is always good practice to return object if the object is mutated.

Here is one similar example, where you failed to change the value of a variable. The assignment change or reference change is not reflected back to the global variable x.

x = [124545]
def a(t):
     t = "1"
a(x)
>>> x
[124545]

Here’s what you really want - to change value of x.

x = [124545]
def a(t):
     t = "1"
     return t
x = a(x)
>>> x
"1"

The reference to 5 is not updated back to global t.
1 and 5 are different objects. In order for the function to update object reference, you need to have a return statement.

t = 1
def a(t):
     t = 5

a(t)
>>> 1
1

global t is updated with a new reference to 5.

t = 1
def a(t):
     t = 5
     return t
t = a(t)
>>> t
5

For function that modifying object (that are mutable) in place, the object will be updated and changed, since the global variable x still references the same object.

x = [124545]
def a(t):
     t[0] = "1"
a(x)
>>> x
["1"]

If you are changing reference to a different object, then you must have a return statement in the function that modified the variable intended.

If the object reference is not change but you are modifying its value, then you may not need a return statement.

To reinforce the concept about modifying objects and its references within function, you can read a good post on stackoverflow - make sure you read till the end.

#3

Thanks. The missing piece of my puzzle was

it ends with x = x.__iadd__(y)

Now i know where the return statement goes to