Logistic Regression Analysis

Reference

1. https://zhuanlan.zhihu.com/p/44591359

2. https://www.sciencedirect.com/topics/medicine-and-dentistry/logistic-regression-analysis

---

Linear Regression

$$f(x)=w^{T}x$$

Sigmoid Function

$$\sigma (x)=\frac{1}{1+e^{-x}}$$

Logistic Regression

$$y=\sigma (f(x))=\frac{1}{1+e^{-w^{T}x}}$$

---

Detail

$$\begin{array}{rl}& P(y=1\mid w,x)=\frac{1}{1+e^{-w^{T}x}}=p\\ & P(y=0\mid w,x)=1-\frac{1}{1+e^{-w^{T}x}}=1-p\end{array}$$$$P(y_i\mid x_i)=p^{y_i}(1-p{)}^{1-y_i}$$

Event $\{(x_1,y_1),(x_2,y_2),\dots ,(x_n,y_n)\}$

$$\begin{array}{rl}{P}_{total}& =\prod _{i=1}^{n}P(y_i\mid x_i)\\ & =\prod _{i=1}^n{p}^{y_i}(1-p{)}^{1-y_i}\end{array}$$$$\begin{array}{rl}F(w)& =\ln P_{total}\\ & =\sum _{i=1}^n{y}_i\ln p+(1-y_i)\ln (1-p)\end{array}$$

Problem: Find $w^{\ast }$

$$w^{\ast }=\arg \underset{w}{max}F(w)=-\arg \underset{w}{max}F(w)$$

---

Solvation

$$\begin{array}{rl}\mathrm{\nabla }F(w)& =\mathrm{\nabla }(\sum _{i=1}^n(y_i\ln p+(1-y_i)\ln (1-p)))\\ & =\sum _{i=1}^n(y_i{\ln }^{\prime }p+(1-y_i){\ln }^{\prime }(1-p))\\ & =\sum _{i=1}^n((y_i\frac{1}{p}{p}^{\prime })+(1-y_i)\frac{1}{1-p}(1-p{)}^{\prime })\\ & =\sum _{i=1}^n(y_i(1-p)x_i-(1-y_i)p{x}_i)\\ & =\sum _{i=1}^n(y_i-p)x_i\\ & =\sum _{i=1}^n(y_i-\frac{1}{1+e^{-w^T{x}_i}})x_i\end{array}$$