How is the cost function in matrix form derived?

Screen Link:
https://app.dataquest.io/m/238/ordinary-least-squares/2/cost-function

J ( a ) = 1/n ( X a − y ) T ( X a − y )

Not able to understand how this Cost function is derived from the previous value of y i.e y = Xa − ϵ , from here we can get the Mean Squared Error as ϵ = Xa - y but how the Cost function( the mean squared error here) is derived from this? Please, anyone, clarify this. Thanks.

1 Like

Roughly speaking,

\epsilon = Xa - y

The above is already provided to us.

The mean squared error is given by, by definition -

\frac{1}{n} \sum(\hat{y} − y)^2

We know that

\epsilon = \hat{y} − y

So, the mean squared error is -

\frac{1}{n}\sum\epsilon^2

And from the equation at the top, that would give us -

\frac{1}{n} \sum(Xa - y)^2

Now, X is a matrix, and a and y are vectors. So, Xa - y is going to be a vector as well.

Consider an example vector -

m = \begin{bmatrix} 1\\2\\3 \end{bmatrix}

What is the dot product of m with itself?

m \cdot m = 1*1 + 2*2 + 3*3 = 14

Do you notice a pattern here? What if we did the following -

\sum(m)^2

We would get -

1^2 + 2^2 + 3^2 = 14

They are the same operations. There is one more way to represent a dot product -

m^Tm

Where, m^T is the transpose of m. So, we can say -

\sum(m)^2 = m^Tm

Coming back to our equation \rightarrow \frac{1}{n} \sum(Xa - y)^2, we can rewrite this to -

\frac{1}{n} (Xa - y)^T(Xa - y)

3 Likes

Wow, what a lucid and great explanation, thanks a lot doctor :slight_smile: Cheers !!
Just a small suggestion- If you include this in the Lesson, will really help many learners, thanks.

1 Like

Glad I could help!

I don’t work for dataquest, but you can provide feedback to them using the Contact Us button in the top-right of this page.

Also, please note that I modified your question title a little bit so that it’s helpful for any students who come across it in the future.