CS229a - Week 3

Logistic Regression

Classification and Representation

Classification 예제들

Binary classification problem

0: Negative Class, 1: Positive Class

Using Linear regression for Classification problem

Logistic Regression Model

Logistic function or Sigmoid function


Equation, g = sigmoid function


Interpretation of Hypothesis output

Decision Boundary

Sigmoid function이 g의 인자가 0보다 크면 1, 0보다 작으면 0

, , 일 때, g 함수의 인자가 0보다 크면 1


Logistic Regression Model

Cost function

Training set

m examples


Cost function 일반화

Linear regression cost function

Linear regression cost Function을 Logistic regression의 Hypothesis에 사용하면 가 non-convex 함수가 됨, 그래서 Local optima 도 많고 Gradient Descent를 제대로 실행할 수 없음


Logistic regression cost function

When , vs :


When , vs :


Logistic regression의 cost function은 y = 0일 때, hypothesis의 결과가 1이 나오면 cost 값이 무한대로 가고 y = 1일 때, hypothesis의 결과가 0이 나오면 cost 값이 무한대로 나옴

Simplified Cost Function and Gradient Descent

Note: y = 0 or 1 always

Plugging-in the definition for full cost function

Vectorized implementation:

To fit parameters theta:

To make a prediction given new x (inference):

Gradient Descent, Repeat(simultaneously update all theta_j):

Calculate the derivative term, Repeat(simultaneously update all theta_j):

Vectorized implementation, abstract j features:

XXX: 아래 식 추론 경로는 아직 모름

Advanced optimization

Sophisticated optimization algorithm

Linear regression Example in Octave

% costFunction
function [jVal, gradient] = costFunction(theta)

jVal = (theta(1)-5)^2 + (theta(2)-5)^2;

gradient = zeros(2,1)
gradient(1) = 2*(theta(1)-5);
gradient(2) = 2*(theta(2)-5);

% Main
options = optimset('GradObj', 'on', 'MaxIter', 100);
initialTheta = zeros(2,);

[functionVal, exitFlag] = fminunc(@costFunction, initialTheta, options)

Logistic regression Example in Octave

% costFunction
function [jVal, gradient] = costFunction(theta)

jVal = % code to compute J(theta)
gradient(1) = % derivative for theta_0, CODE#1
gradient(2) = % derivative for theta_1, CODE#2


Multi-class Classification(one-vs-all)


One-vs-all: Train a logistic regression classifier for each class i

Inference, pick the class i that maximizes:


The Problem of Overfitting


Addressing over-fitting

Cost function

High order degree polynomial, Suppose we penalize and make , really small


Regularization: penalize the parameter values being large

Regularization example: Housing problem with Linear regression

Add regularization term at the end of Cost function:

Regularized linear regression

Gradient descent, Repeat until convergence:

First term: learning rate and lambda is small it’s value is less than 1(i.g. 0.99)

Normal equation

If ,

Using regularization also takes care of any non-invertibility issues of the X transpose X matrix as well

Regularized logistic regression

Over-fitting logistic regression


Gradient descent, Repeat(linear regression과 모양은 비슷하지만, hypothesis가 다름):

Apply Advanced optimization

function [jVal, gradient] = costFunction(theta)
  jVal = % code to compute J(theta)

  gradient(1) = % code to compute partial theta_0 of J(theta)
  gradient(n + 1) = % code to compute partial theta_n of J(theta)




You probably know quite a lot more machine learning right now than frankly, many of the Silicon Valley engineers out there having very successful careers. You know, making tons of money for the companies. Or building products using machine learning algorithms. So, congratulations. - Andrew Ng

XXX: Andrew Ng는 강의에서 묘한 웃음으로 말했는데, 뭘 의미하는 걸까?

XXX: Week 3 Programming 숙제에서는 Sigmoid Function, Cost Function, Predict 구현 했음. 이전처럼 Gradient Descent는 직접 구현하지 않고 Octave에 있는 fminunc를 사용했다. 수식에 맞게 Vector와 Matrix를 잘 조립하는게 중요했는데, size 함수로 하나씩 찍어보면서 맞춰야 했다.

%  Set options for fminunc
options = optimset('GradObj', 'on', 'MaxIter', 400);

%  Run fminunc to obtain the optimal theta
%  This function will return theta and the cost
[theta, cost] = ...
  fminunc(@(t)(costFunction(t, X, y)), initial_theta, options);