User:MPopov (WMF)/Notes/Log transform in regression

Linear regression

The multiple linear regression model is often written as:

$${\begin{array}{rcl}y_{i}&=&\beta _{0}+\beta _{1}x_{1i}+\ldots +\beta _{p}x_{pi}+\epsilon _{i},~i=1,\ldots ,n\\\epsilon _{i}&{\overset {\text{iid}}{\sim }}&{\mathcal {N}}(0,\sigma ^{2})\end{array}}$$

Where $\epsilon _{i}$ is the independent, identically distributed error term and is usually assumed to come from a Normal distribution with mean 0 and standard deviation $\sigma$.

The quantity $\beta _{0}+\beta _{1}x_{1}+\ldots +\beta _{p}x_{p}$ is $\mathbb {E} (y|\mathbf {x} )$, the expected value of $y$ given the values of $\mathbf {x} =(x_{1},\ldots ,x_{p})$.

After the intercept $\beta _{0}$, the coefficients $\beta _{1},\ldots ,\beta _{p}$ represent changes in the expected value of $y$ from unit changes in the corresponding $x_{1},\ldots ,x_{p}$.

For example, suppose we had 2 predictors $x_{1},x_{2}$ and we held the value of $x_{2}$ constant while increasing $x_{1}$ by 1 unit:

$${\begin{array}{rcl}\mathbf {x} _{\text{old}}&=&(k,c)\\\mathbf {x} _{\text{new}}&=&(k+1,c)\\y_{\text{old}}&=&\beta _{0}+\beta _{1}k+\beta _{2}c\\y_{\text{new}}&=&\beta _{0}+\beta _{1}(k+1)+\beta _{2}c\\&=&\beta _{0}+\beta _{1}k+\beta _{1}+\beta _{2}c\\y_{\text{new}}-y_{\text{old}}&=&\beta _{1}\end{array}}$$

Binary (0/1) indicators are especially nice because when $x=0$ then the corresponding $\beta$ doesn't contribute to expected value of $y$, but when $x=1$ then y changes by the corresponding $\beta$. So if $x$ is "received new UI/UX" then the corresponding $\beta$ represents the effect of receiving the new UI/UX on whatever metric/KPI $y$ is.

Log transformation

Sometimes it's necessary to transform either the response/outcome variable $y$ and/or some of the $p$ regressors/predictors $x_{1},\ldots ,x_{p}$.

If we apply the transformation to the dependent variable $y$ we end up with:

$$\log y=\beta _{0}+\beta _{1}x_{1}+\ldots +\beta _{p}x_{p}+\epsilon$$

where $\log$ is usually the natural log ($\log _{n}$ aka $\ln$). Changes in regressors/predictors correspond to changes in expected value of the response/outcome on the log scale, which can pose difficulties. To talk about changes on the original scale, we have to apply the inverse of the natural log: $\exp$. Then

$${\begin{array}{rcl}\exp(\mathbb {E} [\ln y|\mathbf {x} ])&=&\exp(\beta _{0}+\beta _{1}x_{1}+\ldots +\beta _{p}x_{p})\\&=&e^{(\beta _{0}+\beta _{1}x_{1}+\ldots +\beta _{p}x_{p})}\\&=&e^{\beta _{0}}e^{\beta _{1}x_{1}}\cdots e^{\beta _{p}x_{p}}\end{array}}$$

Notice how exponent of the summed terms becomes product of exponentiated terms, turning what used to be additive changes in expected value to multiplicative changes in expected value.

To see the effect of a unit increase in $x_{1}$, we repeat the example from earlier but this time with a log-transformed $y$:

$${\begin{array}{rcl}\mathbf {x} _{\text{old}}&=&(k,c)\\\mathbf {x} _{\text{new}}&=&(k+1,c)\\\ln y_{\text{old}}&=&\beta _{0}+\beta _{1}k+\beta _{2}c\\\ln y_{\text{new}}&=&\beta _{0}+\beta _{1}(k+1)+\beta _{2}c\\\ln y_{\text{new}}-\ln y_{\text{old}}&=&\beta _{1}\\\exp(\ln y_{\text{new}}-\ln y_{\text{old}})&=&e^{\beta _{1}}\\{\frac {y_{\text{new}}}{y_{\text{old}}}}&=&e^{\beta _{1}}\\y_{\text{new}}&=&e^{\beta _{1}}y_{\text{old}}\end{array}}$$

Log-Log transformation

If we apply the transformation to both the dependent and the independent variable, we end up with:

$$\ln(y)=\beta _{0}+\beta _{1}\ln(x_{1})+\beta _{2}x_{2}+\epsilon$$

Suppose we increase $x_{1}$ by one percent (1%):

$${\begin{array}{rcl}\mathbf {x} _{\text{old}}&=&(k,c)\\\mathbf {x} _{\text{new}}&=&(1.01k,c)\\\ln(y_{\text{old}})&=&\beta _{0}+\beta _{1}\ln(k)+\beta _{2}c\\\ln(y_{\text{new}})&=&\beta _{0}+\beta _{1}\ln(1.01k)+\beta _{2}c\\&=&\beta _{0}+\beta _{1}\ln(1.01)+\beta _{1}\ln(k)+\beta _{2}c\\\ln(y_{\text{new}})-\ln y_{\text{old}}&=&\beta _{1}\ln(1.01)\\\exp(\ln y_{\text{new}}-\ln y_{\text{old}})&=&e^{\beta _{1}\ln(1.01)}=e^{\ln(1.01^{\beta _{1}})}\\{\frac {y_{\text{new}}}{y_{\text{old}}}}&=&1.01^{\beta _{1}}\\y_{\text{new}}&=&1.01^{\beta _{1}}y_{\text{old}}\end{array}}$$

Therefore a 1% increase in $x_{1}$ corresponds to a $1.01^{\beta _{1}}$ multiplicative change, which will be less than 1 (a decrease) if $\beta _{1}$ if negative and will be more than 1.01 (an increase) if $\beta _{1}$ is positive. The closer to 0 that $\beta _{1}$ is, the closer to 1 (no change) $1.01^{\beta _{1}}$ will be.