Nonlinear activation functions ReLu activation functions

Q&A DQ Courses
,
#1

source page:

I was looking around for info about how Relu functions work, and stumbled into a comparison of more efficient versions.

I figured I should try some out and see what happens.

import matplotlib.pyplot as plt
import numpy as np
x = np.linspace(-2, 2, 20)

def relu(x):
    return np.maximum(x, 0)


def alt_relu(a):         #my edit
    a[a<0]=0             #my edit
    return a             #my edit

relu_y = relu(x)
print(x)
print(relu_y)
print(alt_relu(x))       #my edit
plt.plot(x,relu_y)
plt.show()
plt.plot(x,alt_relu(x))  #my edit
plt.show()               #my edit

well, as soon as the alt_relu is called at all, it changes the base array, even if you do something like

b=a
b[b<0] = 0
return b

it still screws up the whole base array

[-2.         -1.78947368 -1.57894737 -1.36842105 -1.15789474 -0.94736842
 -0.73684211 -0.52631579 -0.31578947 -0.10526316  0.10526316  0.31578947
  0.52631579  0.73684211  0.94736842  1.15789474  1.36842105  1.57894737
  1.78947368  2.        ]
[0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.10526316 0.31578947
 0.52631579 0.73684211 0.94736842 1.15789474 1.36842105 1.57894737
 1.78947368 2.        ]
[0.         0.         0.         0.         0.         0.
 0.         0.         0.         0.         0.10526316 0.31578947
 0.52631579 0.73684211 0.94736842 1.15789474 1.36842105 1.57894737
 1.78947368 2.        ]

any help understanding how this is happening would be appreciated.

edit: it doesn’t show the screw up, but as soon as you run it, the graphs show the weirdness better.

edit2: moving the print(alt_relu(x)) further down, makes the first graph come out correct, but the second graph is screwed up. this is weird.

edit3: moving ‘print(alt_relu(x))’ above the first ‘print(x)’ cause the print to show that it’s screwed up.

1 Like
#2

You can read about python name binding to understand why this does not create a copy of the data, just another variable pointing to same numpy array.

So the base array is getting overwritten, how do you stop it from being overwritten? (create a copy).

2 Likes