Logistic regression is a machine learning algorithm used for classification tasks. It is an extension of the linear classifier we studied in the previous section and is used to predict the probability of a binary outcome. In this article we will discuss the basics of logistic regression, the mathematics behind it and its implementation in Python.
What is Logistic Regression?
Logistic regression is a method for classifying data into discrete outcomes. For example, it can be used to predict whether an email is spam or not. Logistic regression is a linear method, but with an added non-linearity. The functional form of logistic regression is as follows:
a = wTx
Where a is the dot product between the weights w and the input vector x. a can be any real number between negative infinity and positive infinity.
In logistic regression, the output of the model is passed through the logistic function or sigmoid, which outputs values between 0 and 1. Graphically, a sigmoid looks like this:
Mathematically, the output of the model is as follows:
y = sigmoid(wTx)
The sigmoid is defined as:
sigmoid(a) = 1 / (1 + exp(-a))
In Python, this can be implemented as:
def sigmoid(a):
return 1 / (1 + np.exp(-a))
The output of the sigmoid can be interpreted as the probability of being the positive class. In other words:
p(y=1 | x) = sigmoid(wTx)
The left side is read as “the probability that y equals 1 given x”.
If the model falls exactly on the boundary of the two classes, then the probability of being the positive class is 0.5. This makes sense as the boundary line indicates that the model could go either way.
Variations of Logistic Regression
In addition to the logistic function, there are other variations of the logistic regression algorithm. These include the hyperbolic tangent (tanh) and the softmax function.
The tanh function is a non-linear function that outputs values between -1 and 1. It has the same shape as a sigmoid, but a different scale. The output of the tanh cannot be considered a probability.
The definition of tanh is as follows:
tanh(a) = (exp(a) – exp(-a)) / (exp(a) + exp(-a))
The softmax function is used for multiclass classification tasks with more than two classes. It ensures that the probabilities of all possibilities sum to 1. For example, it can be used to predict the probability of an image being a cat, dog or rabbit.
The output of the softmax is as follows:
p(y=1 | x) = exp(a1) / (exp(a1) + exp(a0))
p(y=0 | x) = exp(a0) / (exp(a1) + exp(a0))
In Python, the softmax can be implemented as:
def softmax(a):
return np.exp(a) / np.sum(np.exp(a))
Conclusion
Logistic regression is a machine learning algorithm used for classification tasks. It is an extension of the linear classifier, which adds a non-linearity to the model. The output of the model is passed through a logistic function, which outputs values between 0 and 1. This can be interpreted as the probability of being the positive class.
In addition to the logistic function, there are other variations of the logistic regression algorithm, such as the hyperbolic tangent and the softmax function. The softmax function is used for multiclass classification tasks with more than two classes. It ensures that the probabilities of all possibilities sum to 1.
Logistic regression is a powerful algorithm with many applications. It is important to understand the mathematics and implementation of the algorithm in order to use it effectively.