15. Regression assumptions and diagnostics
Module items¶
R Script file code¶
-
[[Copy the code]] below ➜ Paste into [[RStudio console]] ➜ Hit Enter
-
source(url("https://raw.githubusercontent.com/ttezcann/ssric-reg/refs/heads/main/docs/assets/attachments/data/0-packages-data.R")); (function(f="15-regression-assumptions-diagnostics.R"){if(!file.exists(f)){download.file("https://raw.githubusercontent.com/ttezcann/ssric-reg/refs/heads/main/docs/assets/attachments/modules/15.-regression-assumptions-diagnostics/15-regression-assumptions-diagnostics.R",f,mode="wb");file.edit(f)}else{download.file("https://raw.githubusercontent.com/ttezcann/ssric-reg/refs/heads/main/docs/assets/attachments/modules/15.-regression-assumptions-diagnostics/15-regression-assumptions-diagnostics.R",gsub(".R","-original.R",f),mode="wb");file.edit(gsub(".R","-original.R",f))}})()- When this R script file opens in a new tab, [[Save R script file|save your previous R script file(s)]], and
- Close the previous tabs (R Script files), which you can find later in the [[Files tab]].
- When this R script file opens in a new tab, [[Save R script file|save your previous R script file(s)]], and
-
Lab assignment¶
Regression analysis assumptions and diagnostics
Sample lab assignment¶
In this assignment, you will run the exact same codes provided in the R script file without making any changes. Therefore, there’s no sample lab assignment file.
Suggested reading¶
Hernán, Miguel A., John Hsu, and Brian Healy. 2019. “A Second Chance to Get Causal Inference Right: A Classification of Data Science Tasks.” Chance 32(1):42–49. doi:10.1080/09332480.2019.1579578
Learning outcomes¶
- Learn about five important checks in regression analysis
- Identify the issues posed by homoscedasticity and heteroscedasticity
- Identify the issues posed by multicollinearity
- Identify the issues posed by curvilinear relationships
- Identify the issues posed by non-normally distributed errors
- Identify response categories with fewer than 10% of the cases
- Learn the diagnostic tools:
- Performance package
- Scatterplot matrix
- Revise a regression model and compare the results before and after revision
[[Linear regression assumptions]] - Overview¶
- Assumption 1: Homoscedasticity
- The errors our model makes should be about the same size across different predicted values.
- When the size of the errors changes, we call it heteroscedasticity.
- The errors our model makes should be about the same size across different predicted values.
- Assumption 2: Multicollinearity
- The factor variables should not be too strongly correlated with each other.
- When they are, it becomes difficult to separate their relationships with the outcome variable.
- The factor variables should not be too strongly correlated with each other.
- Assumption 3: Linear relationship
- There should be a linear relationship between the outcome variable and continuous factor variables.
- This means that a straight line should describe the relationship reasonably well.
- There should be a linear relationship between the outcome variable and continuous factor variables.
- Assumption 4: Normal distribution
- The continuous variables used should display approximately normal distribution;
- Most people's values should be clustered around the middle, not at the very high or very low ends.
- The continuous variables used should display approximately normal distribution;
- Check 5: At least 10% of the cases
- Enough sample size for each category of the dummy variables;
- In this class, we check whether the categories have at least 10% of the responses.
- Enough sample size for each category of the dummy variables;
Assumption 1: Homoscedasticity¶
- [[Homoscedasticity]] means that the errors in our regression model have about the same spread across different predicted values.
- An error is the difference between an actual value and the value predicted by the model.
- Linear regression aims to create a straight-line model that best fits the data.
- Several reasons may contribute to the [[heteroscedasticity]] issue:
- Outliers: Extreme values in data can lead to heteroscedasticity.
- Nonlinear relationships: When the relationship between the independent and dependent variables is nonlinear (i.e., curvilinear), it can lead to heteroscedasticity.
-
Omitted variables: If important variables are omitted from the regression model, they can lead to heteroscedasticity.
-
Addressing the heteroscedasticity
- For this module, we will try removing some variables and then check the model again. Removing variables does not always solve this issue.
-
Assumption 2: Multicollinearity¶
- [[Multicollinearity]] occurs when two or more factor variables in a regression model are strongly correlated with each other.
- This makes it difficult to separate their individual relationships with the outcome variable.
- Several solutions exist for [[multicollinearity]] issue:
- Removing one of the strongly correlated variables
- Creating an index variable using strongly correlated variables
- Centering variables (subtracting the mean value from each observation)
-
Lasso regression (L1 Regularization)
-
Addressing the multicollinearity issue
- For this module, we will remove some strongly related variables and check the model again.
-
- This makes it difficult to separate their individual relationships with the outcome variable.
Assumption 3: Linear relationship¶
- The term "[[linearity]]" in linear regression means that a straight line can reasonably describe the relationship between a continuous factor variable and the outcome variable.
- A [[curvilinear relationship]] between a continuous factor and an outcome variable may not be described well by a straight line.
- Several solutions exist for [[curvilinear relationship]] issue:
- Recoding the variable into categorical
- Polynomial regression
- Adding interaction terms
-
Trying a different way to represent the relationship in the model
-
Addressing the curvilinear relationship
- For this module, we will try removing some variables and check the model again. In types of modeling, curved relationships can also be included in regression.
-
- A [[curvilinear relationship]] between a continuous factor and an outcome variable may not be described well by a straight line.
Assumption 4: Normal distribution¶
- In linear regression, the model's errors should have an approximately [[normal distribution]] for the usual statistical tests.
-
For example, the variable below has an nonnormal distribution:
-
-
Several ways of dealing with a [[nonnormal distribution]] include:
- Recoding the variable into categorical
- Logarithmic transformation (log(x))
- Square root transformation (sqrt(x))
- Inverse transformation (1/x)
-
Adding Polynomial terms (x^2)
-
Addressing the nonnormal distribution
- For this module, we will address the nonnormal distribution issue by removing the problematic variables from the model.
-
Check 5: At least 10% of the cases¶
-
For this class, we use [[10% of the cases]] as a guideline for identifying small response categories.
- Before creating dummy variables, we check the frequency distributions to see whether some categories have fewer than 10% of the cases.
-
Let's check the frequency table of
classvariable.- We can create dummy variables for each category, but "Upper class" has only 4.13% of valid responses. For this exercise, we will consider combining it with another category.
-
-
Respondents' subjective class identification (Variable label)
-
value value label frq raw.prc valid.prc cum.prc 1 Lower class 446 11.19 11.31 11.31 2 Working class 1587 39.81 40.25 51.56 3 Middle class 1747 43.83 44.31 95.87 4 Upper class 163 4.09 4.13 100.00 5 No class 0 0.00 0.00 100.00 NA NA 43 1.08 NA NA
-
-
-
Several solutions exist for having less than [[10% of the cases]] issue:
- Removing the variable from the model
- Collapsing the rare category into an adjacent one (e.g., merging "Upper class" with "Middle class")
- Dropping cases in the rare category from the sample (use cautiously; may introduce bias)
- Treating the variable as continuous if it is ordinal and the distribution is otherwise reasonable
GSS example: Predicting social life index score¶
-
We'll use [[computing]] to create an [[index variable]] for our outcome variable,
sociallife_index.-
flowchart LR subgraph C0[Factor variables] direction TB A[Respondents' socio-economic index score] B[Respondents' education in years] D[Respondents' personal income] F[Respondents' family income] G[Respondents' occupational prestige score] H[Number of children respondents have] I[Level of finding life exciting] end subgraph O0[Outcome variable - Index] E[Social life index score<br><br> The mean of: <br><br> 1: Frequency of social evening with relatives <br><br> 2: Frequency of social evening with neighbors] end A -.->|Association?| E B -.->|Association?| E D -.->|Association?| E F -.->|Association?| E G -.->|Association?| E H -.->|Association?| E I -.->|Association?| E
-
-
The first two variables are to create
sociallife_indexvariable.-
Variable name Variable label Variable type Question wording and response categories socrelFrequency of social evening with relatives Ordinal ✅ RECODE How often do you spend a social evening with relatives?
(1: Almost daily; 2: Once or twice a week; 3: Several times a month; 4: About once a month; 5: Several times a year; 6: About once a year; 7: Never)socommunFrequency of social evening with neighbors Ordinal ✅ RECODE How often do you spend a social evening with neighbors?
(1: Almost daily; 2: Once or twice a week; 3: Several times a month; 4: About once a month; 5: Several times a year; 6: About once a year; 7: Never)educRespondents' education in years Continuous What is the highest year of school you completed?
(Min: 0, Max: 20)conincRespondents' family income Continuous What is your family income in dollars?
(Min: $281.5, Max: $139,024.4)conrincRespondents' personal income Continuous What is your income in dollars?
(Min: $281.5; Max, $123,761.9)prestg10Respondents' occupational prestige score Continuous Respondent's occupational prestige score (calculated)
(Min: 16, Max: 80)childsNumber of children respondents have Continuous How many children do you have?
(Min: 0, Max: 8)lifeLevel of finding life exciting Ordinal, RECODE In general, do you find life exciting, pretty routine, or dull?
(1: Exciting; 2: Routine; 3: Dull)
-
[[Recoding]] and [[computing]] #code¶
-
[[Working code]]
-
- [[Find this working code in the R script file]].
- [[Highlighting and running]] this code will create three more variables.
- [[Find this working code in the R script file]].
-
[[Dummy variable]] #code¶
-
[[Working code]]
-
- [[Find this working code in the R script file]].
- [[Highlighting and running]] this code will create three more variables.
- [[Find this working code in the R script file]].
-
[[Linear regression]] #code (Model 1)¶
-
[[Working code]]
-
- [[Find this working code in the R script file]].
- [[Highlighting and running]] this code will generate the output below (which will appear in the [[viewer tab]] of RStudio).
- [[Find this working code in the R script file]].
-
[[Linear regression]] #output (Model 1)¶
-
Social life index score
-
Factors Coeff. std. Coeff. p (Intercept) 3.69
(0.34)-0.00
(0.03)<0.001*** Respondents' socio-economic index score 0.00
(0.00)0.04
(0.07)0.505 Respondents' education in years -0.04
(0.02)-0.07
(0.04)0.076 Respondents' personal income 0.00
(0.00)0.00
(0.05)0.950 Respondents' family income -0.00
(0.00)-0.06
(0.05)0.254 Respondents' occupational prestige score -0.00
(0.01)-0.03
(0.06)0.618 Number of children respondents have 0.02
(0.03)0.02
(0.04)0.537 Finding life exciting 1.02
(0.22)0.36
(0.08)<0.001*** Finding life routine 0.40
(0.21)0.15
(0.08)0.056 Observations 794 R² / R² adjusted 0.062 / 0.052
-
[[Linear regression]] #interpretation (Model 1)¶
-
Linear regression interpretation sample
- First section: The significance levels
- Finding life exciting has a statistically significant association with social life index score since the p-value is less than 0.05.
- Respondents' socio-economic index score, respondents' education in years, respondents' personal income, respondents' family income, respondents' occupational prestige score, number of children respondents have, and finding life routine are not statistically significant factors of social life index score since their p-values are greater than 0.05.
- Second section: The explanation of coefficients
- Finding life exciting is associated with 1.02 points higher predicted social life index score than finding life dull.
- Third section: The explanation of standardized coefficients
- The largest standardized association with social life index score is for finding life exciting (std. Coeff=0.36).
- Fourth section: The explanation of adjusted R-squared
- The adjusted R-squared value indicates that the model, which includes socio-economic index score, education in years, personal income, family income, occupational prestige score, number of children, finding life exciting, and finding life routine, accounts for 5.2% of the variation in social life index score.
- First section: The significance levels
Assessing the assumptions¶
[[Performance diagnostic]] #code¶
-
[[Working code]]
-
- This code will help us check the first three assumptions.
- [[Find this working code in the R script file]].
- [[Highlighting and running]] this code will generate the output below (which will appear in the [[plots tab]] of RStudio).
-
[[Performance diagnostic]] #output¶

- The output shows:
- Assumption (1) [[Homoscedasticity]] diagnostic
- Assumption (2) [[Multicollinearity]] diagnostic
- Assumption (3) [[Linearity]] diagnostic
- The output shows:
Assumption 1: Homoscedasticity¶
- Homogeneity of variance (Reference line should be flat and horizontal):

- The plot suggests that the spread of errors is not the same across predicted values, so the current model is not homoscedastic.
- We'll run the following code for more details:
[[Homoscedasticity]] #code (Model 1)¶
-
[[Working code]]
-
- [[Find this working code in the R script file]].
- [[Highlighting and running]] this code will generate the output below (which will appear in the [[viewer tab]] of RStudio).
- [[Find this working code in the R script file]].
-
[[Homoscedasticity]] #output (Model 1)¶

- Since the p-value is less than 0.05, this test provides evidence of heteroscedasticity in Model 1.
Assumption 2: Multicollinearity¶
- [[VIF]]:
- The Variance Inflation Factor (VIF) is a measure used to detect the presence and severity of multicollinearity in a regression analysis.
- Lower VIF values are generally better. A VIF of 1 means no multicollinearity with the other factors.
- Family income (
coninc) and personal income (conrinc) are related concepts. Occupational prestige (prestg10) and socio-economic index score (sei10) are also related.- We can use these variables together, but strong relationships between them may make it harder to interpret their separate associations.
- We can use these variables together, but strong relationships between them may make it harder to interpret their separate associations.
- We'll run the following code for more details:
[[Multicollinearity]] #code (Model 1)¶
-
[[Working code]]
-
- [[Find this working code in the R script file]].
- [[Highlighting and running]] this code will generate the output below (which will appear in the [[viewer tab]] of RStudio).
- [[Find this working code in the R script file]].
-
[[Multicollinearity]] #output (Model 1)¶
-
Term VIF VIF 95% CI adj. VIF Tolerance Tolerance 95% CI sei10 3.58 [3.19, 4.03] 1.89 0.28 [0.25, 0.31] educ 1.49 [1.37, 1.65] 1.22 0.67 [0.61, 0.73] conrinc 2.35 [2.12, 2.63] 1.53 0.43 [0.38, 0.47] coninc 2.33 [2.10, 2.61] 1.53 0.43 [0.38, 0.48] prestg10 2.81 [2.52, 3.16] 1.68 0.36 [0.32, 0.40] childs 1.03 [1.00, 1.44] 1.01 0.98 [0.69, 1.00] exciting 4.98 [4.42, 5.65] 2.23 0.20 [0.18, 0.23] routine 4.92 [4.36, 5.58] 2.22 0.20 [0.18, 0.23] - Several VIF values are higher than 2. The highest are for
exciting(4.98) androutine(4.92), so we should pay attention to these two variables.
- Several VIF values are higher than 2. The highest are for
Assumption 3: Linearity¶
- Linearity (Reference line should be flat and horizontal):

- The reference line is not flat. It curves downward as predicted values increase.
- This suggests that the model may not capture some curved relationships very well.
- The reference line is not flat. It curves downward as predicted values increase.
- To further see this issue, we'll use [[scatterplot graph matrix]].
[[Scatterplot graph matrix]] #code (Model 1)¶
-
[[Working code]]
-
- [[Find this working code in the R script file]].
- [[Highlighting and running]] this code will generate the output below (which will appear in the [[plots tab]] of RStudio).
- [[Find this working code in the R script file]].
-
[[Scatterplot graph matrix]] #output (Model 1)¶

- The highlighted scatterplots suggest curvilinear (nonlinear) relationships rather than straight-line relationships.
- Specifically, the relationships between
sociallife_indexand some socioeconomic variables (sei10,educ,conrinc, andconinc) appear to be curved.- In other words, the relationship may change at different values of these variables. This suggests that a straight-line model may not describe it well.
Assumption 4: Normal distribution¶
- We'll use [[scatterplot graph matrix]] to look at the distributions of the variables. However, the normality assumption in regression concerns the model's errors, not the factor variables themselves.
[[Scatterplot graph matrix]] #code (Model 1)¶
-
[[Working code]]
-
- [[Find this working code in the R script file]].
- [[Highlighting and running]] this code will generate the output below (which will appear in the [[plots tab]] of RStudio).
- [[Find this working code in the R script file]].
-
[[Scatterplot graph matrix]] #output (Model 1)¶

- Most histograms look approximately normal, but
childsis skewed (nonnormal).
- Most histograms look approximately normal, but
Check 5: At least 10% of the cases¶
- We created dummy variables for
lifevariable without checking the frequency distribution.
[[Frequency table]] #code¶
-
[[Working code]]
-
- [[Find this working code in the R script file]].
- [[Highlighting and running]] this code will generate the output below (which will appear in the [[viewer tab]] of RStudio).
- [[Find this working code in the R script file]].
-
[[Frequency table]] #output¶
-
Level of finding life exciting
-
value value label frq raw.prc valid.prc cum.prc 1 Exciting 971 24.36 36.42 36.42 2 Routine 1519 38.11 56.98 93.40 3 Dull 176 4.42 6.60 100.00 NA NA 1320 33.12 NA NA - Here the "Dull" category has only 6.6% of valid responses (less than 10%).
- For this exercise, we will merge it with "Routine".
- Here the "Dull" category has only 6.6% of valid responses (less than 10%).
-
[[Dummy variable]] #code¶
-
[[Working code]]
-
- [[Find this working code in the R script file]].
- [[Highlighting and running]] this code will create two more variables.
- [[Find this working code in the R script file]].
-
Fixing the model (Model 2)¶
[[Linear regression]] #code (Model 2)¶
-
[[Working code]]
-
- [[Find this working code in the R script file]].
- [[Highlighting and running]] this code will generate the output below (which will appear in the [[viewer tab]] of RStudio).
- [[Find this working code in the R script file]].
-
[[Linear regression]] #output (Model 2)¶
-
Social life index score
-
Factors Coeff. std. Coeff. p (Intercept) 4.01
(0.19)-0.00
(0.03)<0.001*** Respondents' education in years -0.04
(0.01)-0.08
(0.03)0.004** Finding life exciting 0.60
(0.08)0.20
(0.03)<0.001*** Observations 1313 R² / R² adjusted 0.043 / 0.041
-
[[Linear regression]] #interpretation (Model 2)¶
-
Linear regression interpretation sample
- First section: The significance levels
- Respondents' education in years and finding life exciting have statistically significant associations with social life index score since their p-values are less than 0.05.
- Second section: The explanation of coefficients
- One additional year of respondents' education is associated with 0.04 points lower predicted social life index score.
- Finding life exciting is associated with 0.60 points higher predicted social life index score than finding life routine or dull.
- Third section: The explanation of standardized coefficients
- The largest standardized association with social life index score is for finding life exciting (std. Coeff=0.20), followed by respondents' education in years (std. Coeff=-0.08).
- Fourth section: The explanation of adjusted R-squared
- The adjusted R-squared value indicates that the model, which includes respondents' education in years and finding life exciting, accounts for 4.1% of the variation in social life index score.
- First section: The significance levels
Assessing the assumptions¶
[[Performance diagnostic]] #code (Model 2)¶
-
[[Working code]]
-
- [[Find this working code in the R script file]].
- [[Highlighting and running]] this code will generate the output below (which will appear in the [[plots tab]] of RStudio).
- [[Find this working code in the R script file]].
-
[[Performance diagnostic]] #output (Model 2)¶
[[Homoscedasticity]] #code (Model 2)¶
-
[[Working code]]
-
- [[Find this working code in the R script file]].
- [[Highlighting and running]] this code will generate the output below (which will appear in the [[viewer tab]] of RStudio).
- [[Find this working code in the R script file]].
-
[[Homoscedasticity]] #output (Model 2)¶

- Since the p-value is greater than 0.05, this test does not find statistically significant evidence of heteroscedasticity in Model 2.
[[Multicollinearity]] #code (Model 2)¶
-
[[Working code]]
-
- [[Find this working code in the R script file]].
- [[Highlighting and running]] this code will generate the output below (which will appear in the [[viewer tab]] of RStudio).
- [[Find this working code in the R script file]].
-
[[Multicollinearity]] #output (Model 2)¶
-
Term VIF VIF 95% CI adj. VIF Tolerance Tolerance 95% CI educ 1.02 [1.00, 1.45] 1.01 0.98 [0.69, 1.00] exciting_new 1.02 [1.00, 1.45] 1.01 0.98 [0.69, 1.00] -
Both VIF values are 1.02, which suggests little multicollinearity in Model 2.






