Regression Models

Regression models are the best way to get introduced to the simplest concepts in machine learning. These include:

• the idea of an algorithm generating a model
• the idea of a model "fitting" data
• the idea of a loss function
• the mechanism of gradient descent

I've seen other resources introduce gradient descent when they introduce neural nets (AKA "multi-layered perceptrons"), but I think this makes it too complicated. Isolated in the simplest environment, gradient descent becomes much easier to quickly understand.

While legendary ML educator Andrew Ng has since come out with a newer specialization via deeplearning.ai, the older version of this course retains a certain simple charm that I think lends itself to multiple a-ha moments learning the basics.

I remember the wonder I felt when I first saw the three-dimensional loss space of a two variable linear regression model. Something about seeing it on Ng's simpler contour map in the original lessons made it especially sticky and rewarding.

You can find a playlist with the old Ng videos here (opens in a new tab).

Week 1 - Intro to ML, Linear Regression, and Linear Algebra

Week 2 - Gradient Descent and Polynomial Regression

You should only watch the week 2 videos up to (and including) lecture 5.

Week 3 - Logistic Regression and Some Regularization

One additional resource that will prove useful is learning in greater detail what Euler's number is and how it works to create the sigmoid function, an essential component of logistic regression.

Because Ng uses the outdated programming language Octave, I've also included a few videos from Python Engineer to help you get started with Python. Coding linear regression from scratch directly after learning it is a great way to solidify the concepts.

Python Engineer

In this one, you code an entire linear regression algorithm from scratch, which trains and then outputs its final parameters. You do this without the use of any libraries other than NumPy. It really helped things click.

And here you'll code logistic regression, which is very similar.