Skip to content

11. Linear regression basics

Module items

R Script file code

  • [[Copy the code]] below ➜ Paste into [[RStudio console]] ➜ Hit Enter 

    • source(url("https://raw.githubusercontent.com/ttezcann/ssric-reg/refs/heads/main/docs/assets/attachments/data/0-packages-data.R")); 
(function(f="11-linear-regression.R"){if(!file.exists(f)){download.file("https://raw.githubusercontent.com/ttezcann/ssric-reg/refs/heads/main/docs/assets/attachments/modules/11.-linear-regression-basics/11-linear-regression.R",f,mode="wb");file.edit(f)}else{download.file("https://raw.githubusercontent.com/ttezcann/ssric-reg/refs/heads/main/docs/assets/attachments/modules/11.-linear-regression-basics/11-linear-regression.R",gsub(".R","-original.R",f),mode="wb");file.edit(gsub(".R","-original.R",f))}})()
      • When this R script file opens in a new tab, [[Save R script file|save your previous R script file(s)]], and
        • Close the previous tabs (R Script files), which you can find later in the [[Files tab]].

Lab assignment

Linear regression

Sample lab assignment

Sample: Linear regression

Suggested reading


  • Shmueli, Galit. 2010. “To Explain or to Predict?” Statistical Science 25(3):289–310. doi:10.1214/10-STS330

Learning outcomes

  1. Define linear regression analysis and its purpose in statistical modeling
  2. Identify situations in which linear regression is appropriate
  3. Differentiate between explanatory, predictive, and descriptive modeling goals in regression analysis
  4. Interpret regression coefficients, reporting how a one-unit difference in each factor is associated with the outcome
  5. Interpret standardized coefficients to compare the relative size of associations for factor variables measured in different units
  6. Interpret the adjusted R-squared value
  7. Identify the effect of confounding variables on modeling

[[Regression]] definition

  • Regression is a statistical technique used to model relationships between variables.

    • In regression, we examine how [[factor variable]]s are related to the [[outcome variable]].

    • Like [[correlation analysis]], regression does not by itself establish cause and effect. In regression, we specify which variable is the outcome and which variables are the factors. Here is a correlation table as an example:

    • tab_corr (gss[c("age", "childs")],  
p.numeric = T, triangle="lower")
    • Respondents' age Number of children respondents have
      Respondents' age
      Number of children respondents have r = 0.297
      p < 0.001***

    • Significant (p < 0.05) and weak (|r| < 0.3) correlation interpretation sample

      There is a statistically significant correlation between respondents' age and number of children respondents have since the p-value is less than 0.05.

      This correlation is positive and weak since the r-value is 0.297.

      This means that as respondents' age increases, the number of children respondents have tends to increase.

    • In regression analysis, respondents' age is the factor variable, and the number of children is the outcome variable.

      • Having more children does not make a respondent older. That is why we use age to help explain differences in the number of children, rather than treating the number of children as a factor that determines age.
        • Otherwise, we might argue that having children increases respondents' age, or not having children magically decreases respondent's age.

[[Types of regression modeling based on outcome variable]]

  • There are two types of regression modeling based on outcome variable:
    1. [[Linear regression]]
      • The [[outcome variable]] is
        • [[Continuous]].
      • There can be one or more [[factor variable]]s.
    2. [[Logistic regression]]
      • The [[outcome variable]] is
        • [[Binary]] (coded with two values, which we will discuss later).
      • There can be one or more [[factor variable]]s.

[[Types of regression modeling based on modeling type]]

  • Regression models can be used for three different purposes. A model can have more than one purpose:
    1. [[Explanatory modeling]]
      • These models use existing research and theory to choose factors that may help explain the outcome variable.
        • They are theory-driven.
        • An explanatory regression model can test relationships suggested by a theory.
    2. [[Predictive modeling]]
      • The goal is to predict the outcome variable using the values of the factor variables. For example, we might use several variables to predict personal income.
        • Predictive models focus on how well the model predicts an outcome.
    3. [[Descriptive modeling]]
      • Descriptive models summarize the relationships found in the data without claiming that one variable causes another.
        • We select variables to explore patterns in the data.
        • These models describe patterns, even when our main goal is not prediction or a causal explanation.
      • We will see some examples of regression analysis and outputs in the following section.

Example 1: Hospitalization days

  • [[Predictive modeling]] and [[explanatory modeling]] can be used to study hospital stays. For example, we can predict how long a patient may stay and examine which patient characteristics are associated with longer or shorter stays.
    • Imagine working in a hospital, calculating hospitalization days to arrange available beds for the next month.
    • A regression model might predict hospitalization days using the following variables, among many others:
      • Seriousness of the operation: A continuous variable that quantifies the complexity and inherent risks of the operation on a scale (0: not serious; 100: very serious).
      • Age: The age of the patient in years.
      • Healthy lifestyle: A continuous variable that provides a measure of health lifestyle (0: extremely poor; 100: extremely healthy).
      • Previous hospitalizations: The total number of days spent in the hospital in the past due to medical procedures or illnesses.
  • Here's what such regression analysis table looks like:

  • Hospitalization days

    • Factors Coeff. std. Coeff. p
      (Intercept) -2.50
      (0.85)      		 0.00
      (0.01)      		 0.003**
      Seriousness of the operation 0.30
      (0.01)      		 0.35
      (0.01)      		 < 0.001***
      Age 0.12
      (0.01)      		 0.18
      (0.01)      		 < 0.001***
      Healthy lifestyle -0.22
      (0.01)      		 -0.23
      (0.01)      		 < 0.001***
      Previous hospitalizations 0.25
      (0.01)      		 0.32
      (0.01)      		 < 0.001***
      Observations 7995
      R² / R² adjusted 0.65 / 0.65

  • Formula:

    • Predicted hospitalization days = -2.50 + (0.30 × seriousness of the operation) + (0.12 × age) + (-0.22 × healthy lifestyle) + (0.25 × previous hospitalizations)
    • The formula starts at -2.50 days (the intercept, which applies when all factors equal zero). Each factor then changes the predicted number of days. The intercept is not a realistic hospital stay for this example.
  • Example calculation:
    • Say a patient has:
    • Seriousness of the operation = 40, Age = 88, Healthy lifestyle = 54, Previous hospitalizations = 6
      • Plug each value into the formula:

  • Formula table

    • Step Calculation Result
      Start with the baseline -2.50 days
      Seriousness of the operation (0.30 × 40) adds +12 days
      Age (0.12 × 88) adds +10.56 days
      Healthy lifestyle (-0.22 × 54) subtracts -11.88 days
      Previous hospitalizations (0.25 × 6) adds +1.50 days
      Predicted hospitalization days = 9.68 days

  • Each factor shifts the predicted hospitalization days up or down:

    • Each one-point increase in seriousness of the operation is associated with 0.30 more predicted days.
    • Each additional year of age is associated with 0.12 more predicted days.
    • Each one-point increase in healthy lifestyle is associated with 0.22 fewer predicted days.
    • Each additional day of previous hospitalizations is associated with 0.25 more predicted days.

  • Warning

    • In practice, this calculation is done for all 7,995 patients at once; not manually one by one.
      • This example shows how to calculate predictions from coefficients.

Example 2: Car insurance rates

  • [[Predictive modeling]] and [[explanatory modeling]] can be used to study car insurance rates. We can predict rates from city characteristics and examine which characteristics are associated with higher or lower rates.
    • Why do car insurance rates vary between different states and even between cities within the same state?
  • A regression model might predict car insurance rates using the following variables, among many others:
    • Population of the city: Larger populations may lead to higher accident rates due to increased traffic.
    • Accidents per 1000 people: Reflects the overall risk associated with the city.
    • Quality of roads: A city-wide average rating of road conditions.
    • Crime rate: Higher crime rates, especially theft and vandalism, lead to higher insurance premiums.
    • Weather patterns: Cities with severe weather conditions could experience more weather-related accidents (0:bad, 100:ideal).
    • Traffic congestion levels: Cities with higher levels of congestion experience more fender benders.
    • Average commute time: Longer commutes can increase the probability of accidents occurring.
  • Here's what such regression analysis table looks like:

  • Car insurance rate

    • Factors Coeff. std. Coeff. p
      (Intercept) 1452.87
      (50.00)      		 0.00
      (0.01)      		 < 0.001***
      Population of the city 0.0015
      (0.000019)      		 0.41
      (0.01)      		 < 0.001***
      Accidents per 1000 people 2.35
      (0.05)      		 0.24
      (0.01)      		 < 0.001***
      Quality of roads -1.78
      (0.04)      		 -0.21
      (0.01)      		 < 0.001***
      Crime rate 0.97
      (0.03)      		 0.17
      (0.01)      		 < 0.001***
      Weather patterns -0.045
      (0.00198)      		 -0.12
      (0.01)      		 < 0.001***
      Traffic congestion levels 0.215
      (0.0067)      		 0.17
      (0.01)      		 < 0.001***
      Average commute time 0.28
      (0.01)      		 0.16
      (0.01)      		 < 0.001***
      Observations 19495
      R² / R² adjusted 0.45 / 0.45

  • Formula:

    • Predicted rate = 1452.87 + (0.0015 × population) + (2.35 × accidents per 1000) + (-1.78 × road quality) + (0.97 × crime rate) + (-0.045 × weather patterns) + (0.215 × traffic congestion) + (0.28 × average commute time)
    • The formula starts at $1,452.87 (the intercept, which applies when all factors equal zero). Each city characteristic then changes the predicted rate.
  • Example calculation:
    • Say a city has:
    • Population = 500,000, Accidents per 1000 = 8, Road quality = 60, Crime rate = 12, Weather patterns = 45, Traffic congestion = 7, Average commute time = 30
      • Plug each value into the formula:

  • Formula table

    • Step Calculation Result
      Start with the baseline $1,452.87
      Population (0.0015 × 500,000) adds +$750.00
      Accidents per 1000 (2.35 × 8) adds +$18.80
      Road quality (-1.78 × 60) subtracts -$106.80
      Crime rate (0.97 × 12) adds +$11.64
      Weather patterns (-0.045 × 45) subtracts -$2.03
      Traffic congestion (0.215 × 7) adds +$1.51
      Average commute time (0.28 × 30) adds +$8.40
      Predicted insurance rate = $2,134.39

  • What each variable does to the prediction:

    • One additional person in population is associated with $0.0015 more in the predicted rate.
    • One additional accident per 1,000 people is associated with $2.35 more.
    • A one-point improvement in road quality is associated with $1.78 less.
    • A one-point increase in crime rate is associated with $0.97 more.
    • A one-point improvement in weather patterns is associated with $0.045 less.
    • A one-point increase in traffic congestion is associated with $0.215 more.
    • One additional minute of average commute time is associated with $0.28 more.

[[Linear regression]] specifics

  • Linear regression table provides:
    • [[Coefficient]]: The estimated difference in the outcome variable for a one-unit increase in a factor variable, while the other factor variables remain the same. For example, one additional accident per 1,000 people is associated with a $2.35 higher predicted insurance rate.
      • [[Standard error]]: A measure of uncertainty in an estimated coefficient. Smaller standard errors generally mean more precise estimates. Standard errors appear in parentheses beneath the coefficients.
    • [[Standardized coefficient]]: The coefficient is rescaled to allow comparison across factor variables measured in different units. The larger its absolute value, the stronger its association with the outcome variable in the model. Standard errors appear in parentheses below the standardized coefficients.
      • This is useful because different variables are measured differently.
        • Age is measured in years.
        • Education is measured in years.
        • Income may be measured in dollars.
        • Occupational prestige may be measured with a score.
    • [[p-value]]: Helps us assess evidence against the starting assumption that a factor has no association with the outcome, after accounting for the other factors in the model. When the p-value is less than 0.05 (p < 0.05), we call that coefficient statistically significant.
    • [[Adjusted R-squared]]: Shows how much of the variation in the outcome variable is explained by the regression model. It also helps us see whether adding more factor variables improves the model. We report adjusted R-squared as a percentage.

GSS example: Predicting personal income (conrinc)

Proposed relationship

  • In this analysis, we examine whether these factors are associated with higher or lower personal income. Regression alone cannot tell us that they cause changes in income.

    • flowchart LR
subgraph F1["Factor variable (coded categories)"]
    A[Residence at age 16]
end

subgraph F2["Factor variable"]
    B[Age]
end

subgraph F3["Factor variable"]
    C[Occupational prestige]
end

subgraph F4["Factor variable"]
    D[Education]
end

subgraph O["Continuous outcome variable"]
    E[Personal income]
end

A -.->|Association?| E
B -.->|Association?| E
C -.->|Association?| E
D -.->|Association?| E

  • [[Predictive modeling]] and [[descriptive modeling]] will be used to predict respondents' personal income based on the population density of residence during their adolescence years, age, and occupational prestige score, and their education in years.

    • This modeling is descriptive in nature as well, since the variables have not been derived from existing literature but are based on logically constructed connections.
    • We will examine whether each factor is associated with personal income, the size of its coefficient, and how its estimate changes after we add more factors.
    • Regression analysis will answer the following question:
      • Which factor variables are statistically significantly associated with respondents’ personal income?
        • How much higher or lower is predicted income for a one-unit difference in each factor, after accounting for the other factors?
        • Which factors have the largest standardized coefficients in this model?

Find the variables in Variables in GSS page

  1. We want to make sure that conrinc and res16, age, prestg10, and educ are continuous variables or usable ordinal variables (Ordinal ✅).

  2. [[Search]] the variable names in Variables in GSS page.

    • Variable name Variable label Variable type Question wording and response categories
      conrinc Respondents' personal income Continuous What is your income in dollars?

      (Min: $281.5; Max, $123,761.9)
      res16 Population density of residence during adolescence years Ordinal ✅ Which of the categories on this card comes closest to the type of place you were living in when you were 16 years old?

      (1: Country, nonfarm; 2: Farm; 3: Town less than 50K; 4: 50K to 250K; 5: Big city, suburb; 6: City greater than 250K)
      age Respondents' age Continuous What is your age?

      (Min: 18, Max: 89)
      prestg10 Respondents' occupational prestige score Continuous Respondent's occupational prestige score (calculated)

      (Min: 16, Max: 80)
      educ
      		Respondents' education in years Continuous What is the highest year of school you completed?

      (Min: 0, Max: 20)

[[Linear regression]] with 1 factor #code

  • [[Model code]]

    • 1
2
      model1 <- lm(outcome_here ~ factor1_here, data = gss)
tab_model(model1, show.std = T, show.ci = F, collapse.se = T)

  • [[Working code]]

    • 1
2
      model1 <- lm(conrinc ~ res16, data = gss)
tab_model(model1, show.std = T, show.ci = F, collapse.se = T)
      • Note that we'll introduce the factor variables one by one.
      • Line 1: We put conrinc here ➜ outcome_here and res16 here ➜ factor1_here.
        • Outcome variable variable first; factor variable second.
      • Line 2: Check the first argument in line 1: model1. If this is model1, then we should use model1 in line 2.
        • This needs to be model1, otherwise this code won't work.
        • model1 ➜ model1; model2 ➜ model2; model_orange ➜ model_orange.
          • [[Find this working code in the R script file]].
            • [[Highlighting and running]] this code will generate the output below (which will appear in the [[viewer tab]] of RStudio).

[[Linear regression]] with 1 factor #output

  • Respondents' personal income

    • Factors Coeff. std. Coeff p
      (Intercept) 31243.32
      (1929.26)      		 -0.00
      (0.02)      		 0.001***
      Population density of residence during
      adolescence years      		 1449.97
      (457.75)      		 0.07
      (0.02)      		 0.002**
      Observations 2340
      R² / R² adjusted 0.004 / 0.004
      • Population density of residence during adolescence years is statistically significantly associated with personal income (p < 0.05).
      • A one-step increase in population density of residence during adolescence years is associated with $1,449 higher predicted personal income.
        • For example, moving from category 5 to category 6 changes predicted income by about $1,449.
      • Now, let's add the second factor variable, age

[[Linear regression]] with 2 factors #code

  • [[Model code]]

    • 1
2
      model2 <- lm(outcome_here ~ factor1_here + factor2_here, data = gss)
tab_model(model2, show.std = T, show.ci = F, collapse.se = T)

  • [[Working code]]

    • 1
2
      model2 <- lm(conrinc ~ res16 + age, data = gss)
tab_model(model2, show.std = T, show.ci = F, collapse.se = T)
      • Line 1: We put conrinc here ➜ outcome_here; res16 here ➜ factor1_here; age here ➜ factor2_here.
        • Outcome variable variable first; factor variable second.
      • Line 2: Check the first argument: model2. If this is model2, then we should use model2 here.
        • This needs to be model2, otherwise this code won't work.
          • [[Find this working code in the R script file]].
            • [[Highlighting and running]] this code will generate the output below (which will appear in the [[viewer tab]] of RStudio).

[[Linear regression]] with 2 factors #output

  • Respondents' personal income

    • Factors Coeff. std. Coeff p
      (Intercept) 17269.65
      (2839.75)      		 -0.00
      (0.02)      		 0.001***
      Population density of residence during
      adolescence years      		 1515.16
      (456.70)      		 0.07
      (0.02)      		 0.001***
      Respondents' age 308.91
      (45.49)      		 0.14
      (0.02)      		 0.001***
      Observations 2303
      R² / R² adjusted 0.023 / 0.023
      • Population density of residence during adolescence years and respondents' age are statistically significantly associated with personal income (p < 0.05) in this model.
      • A one-step increase in population density of residence during adolescence years is associated with $1,515 higher predicted personal income, after accounting for age.
      • One additional year of age is associated with $309 higher predicted income, after accounting for residence category. In this model, a 40-year-old is predicted to earn $3,089 more than a 30-year-old with the same residence category.
      • Now, let's add the third factor variable, prestg10.

[[Linear regression]] with 3 factors #code

  • [[Model code]]

    • 1
2
      model3 <- lm(outcome_here ~ factor1_here + factor2_here + factor3_here, data = gss)
tab_model(model3, show.std = T, show.ci = F, collapse.se = T)

  • [[Working code]]

    • 1
2
      model3 <- lm(conrinc ~ res16 + age + prestg10, data = gss)
tab_model(model3, show.std = T, show.ci = F, collapse.se = T)
      • Line 1: We put conrinc here ➜ outcome_here and res16 here ➜ factor1_here; age here ➜ factor2_here; prestg10 here ➜ factor3_here.
        • Outcome variable variable first; then, factor variables separated by plus (+).
        • Check the first argument: model3. If this is model3, then we should use model3 here.
      • Line 2: Check the first argument: model3. If this is model3, then we should use model3 here.
        • This needs to be model3, otherwise this code won't work.
          • [[Find this working code in the R script file]].
            • [[Highlighting and running]] this code will generate the output below (which will appear in the [[viewer tab]] of RStudio).

[[Linear regression]] with 3 factors #output

  • Respondents' personal income

    • Factors Coeff. std. Coeff p
      (Intercept) -16380.05
      (3167.04)      		 -0.00
      (0.02)      		 0.001***
      Population density of residence during
      adolescence years      		 938.46
      (424.60)      		 0.04
      (0.02)      		 0.027*
      Respondents' age 224.50
      (42.34)      		 0.10
      (0.02)      		 0.001***
      Respondents' occupational prestige score 881.86
      (45.69)      		 0.37
      (0.02)      		 0.001***
      Observations 2278
      R² / R² adjusted 0.160 / 0.159
      • Population density of residence during adolescence years, respondents' age, and respondents' occupational prestige score are statistically significantly associated with personal income (p < 0.05) in this model.
        • The coefficient for residence category has become smaller after adding occupational prestige.
      • One additional year of age is associated with $225 higher predicted income. For example, a 40-year-old is predicted to earn $2,245 more than a 30-year-old.
      • One additional point of occupational prestige is associated with $882 higher predicted income. For example, a respondent with a prestige score of 80 is predicted to earn $8,819 more than someone with a score of 70.
      • Finally, let's add our fourth, and final factor variable, educ.

[[Linear regression]] with 4 factors #code

  • [[Model code]]

    • 1
2
      model4 <- lm(outcome_here ~ factor1_here + factor2_here + factor3_here + factor4_here, data = gss)
tab_model(model4, show.std = T, show.ci = F, collapse.se = T)

  • [[Working code]]

    • 1
2
      model4 <- lm(conrinc ~ res16 + age + prestg10 + educ, data = gss)
tab_model(model4, show.std = T, show.ci = F, collapse.se = T)
      • Line 1: We put conrinc here ➜ outcome_here; res16 here ➜ factor1_here; age here ➜ factor2_here; prestg10 here ➜ factor3_here; educ here ➜ factor4_here.
        • Outcome variable variable first; then, factor variables separated by plus (+).
      • Line 2: Check the first argument: model4. If this is model4, then we should use model4 here.
        • This needs to be model4, otherwise this code won't work.
          • [[Find this working code in the R script file]].
            • [[Highlighting and running]] this code will generate the output below (which will appear in the [[viewer tab]] of RStudio).

[[Linear regression]] with 4 factors #output

  • Respondents' personal income

    • Factors Coeff. std. Coeff p
      (Intercept) -41471.83
      (3964.93)      		 -0.00
      (0.02)      		 0.001***
      Population density of residence during
      adolescence years      		 499.80
      (417.99)      		 0.02
      (0.02)      		 0.232
      Respondents' age 196.07
      (41.51)      		 0.09
      (0.02)      		 0.001***
      Respondents' occupational prestige score 649.05
      (50.12)      		 0.27
      (0.02)      		 0.001***
      Respondents' education in years 2624.02
      (256.54)      		 0.22
      (0.02)      		 0.001***
      Observations 2272
      R² / R² adjusted 0.197 / 0.196
      • One additional year of age is associated with $196 higher predicted personal income. In this model, a 40-year-old is predicted to earn $1,961 more than a 30-year-old with the same values on the other factors.
      • One additional point of occupational prestige is associated with $649 higher predicted personal income. A respondent with a score of 80 is predicted to earn $6,491 more than someone with a score of 70.
      • One additional year of education is associated with $2,624 higher predicted personal income. A respondent with 20 years of schooling is predicted to earn $26,240 more than someone with 10 years.
      • Population density of residence during adolescence years is not a statistically significant factor of respondents' personal income since the p-value is greater than 0.05.

The presence of [[confounding variable]]

  • What changed?
    • In earlier models, population density of residence during adolescence years was statistically significantly associated with personal income.
      • After adding education, its coefficient became smaller and its p-value became greater than 0.05 (p > 0.05).
  • What does this mean?
    • Some of the relationship between population density of residence during adolescence years and income may overlap with the relationship between education and income.
      • This may help explain why the coefficient changes when we account for education. However, these results alone do not establish why the change happened. The number of observations also changes across models, which can affect the estimates.
  • What is a confounding variable?
    • A confounding variable is related to both the factor we are studying and the outcome. If we leave it out, the estimated relationship may be misleading.
      • For example, education may help us understand the relationship between population density of residence during adolescence years and income. We would need more evidence to determine whether it acts as a confounder in this study.
  • What does this mean, again?
    • Our first model examined population density of residence during adolescence years without accounting for education.
      • Our final model also includes education, age, and occupational prestige. The population density of residence during adolescence years coefficient is smaller.
    • When the p-value is greater than 0.05 (p > 0.05), we do not have enough evidence to conclude that the population density of residence during adolescence years coefficient differs from zero in the final model.
      • This is why adding more relevant variables can help us better describe relationships in regression analysis.
  • In short:
    • Population density of residence during adolescence years was statistically significant in earlier models but is no longer statistically significant in the final model.

    • This change may be related to the addition of education and other factor variables.

      • flowchart LR
A[Confounder factor <br/> Population density]
B[Education]
C[Income]

A -->|Association| B
B -->|Association| C
A -.->|No direct effect| C

[[Linear regression interpretation specifics]]

[[Reporting of coefficients]]

  • When reporting the [[coefficient]]s of the factor variables:
    • We ensure that the sentence includes the [[unit of analysis|ref]] (one unit, score, year, dollars, etc.) of both the factor variable and the outcome variable.
      • Factor variable: (health - Perceived personal health quality - (1: Excellent; 2: Very Good; 3: Good; 4: Fair; 5: Poor))
      • Outcome variable: (sei10 - Respondents' socio-economic index score - (Min: 9, Max: 92.8))
        • In this hypothetical example, a one-unit increase in the health category is associated with a 12.45-point difference in respondents' socio-economic index score. The direction depends on the coefficient's sign.
      • Factor variable: (age - Respondents' age - (Min: 18, Max: 89))
      • Outcome variable: (polviews - Respondents' conservatism level - (1: extremely liberal; 7: extremely conservative))
        • In this hypothetical example, each additional year of age is associated with 1.78 more points on the conservatism scale.
      • Factor variable: (maeduc - Respondents' mothers' education in years - (Min: 0, Max: 20))
      • Outcome variable: (conrinc - Respondents' personal income - (Min: $281.5; Max, $123,761.9))
        • In this hypothetical example, each additional year of respondents' mothers' education is associated with $2,857 higher predicted personal income.
  • In the final regression model, each coefficient shows how a one-unit difference in a factor is associated with predicted personal income, after accounting for the other factors:
    • What is the association for one higher population density of residence during adolescence years?
      • Population density of residence during adolescence years is not a statistically significant factor of respondents' personal income since the p-value is greater than 0.05.
    • What is the association for one more year of age?
      • Each additional year is associated with $196 higher predicted personal income.
    • What is the association for one more point of occupational prestige?
      • Each additional point is associated with $649 higher predicted personal income.
    • What is the association for one more year of education?
      • Each additional year is associated with $2,624 higher predicted personal income.
  • The education coefficient looks largest, but the factors are measured in different units.
    • Is it?

  • Education in years (min: 0; max: 20)

    • Education ranges from 0 to 20 years; that gives 21 possible whole-number values.

      • flowchart LR
    A1["0"] -->|$2,624| A2["1"] --> A3["..."] --> A4["12"] -->|$2,624| A5["13"] --> A6["..."] --> A7["20"]

  • Occupational prestige score (min: 16; max: 80)

    • The range of occupational prestige score is 16-80; there are 65 different responses. 

      • flowchart LR
    A1["16"] -->|$649| A2["17"] --> A3["..."] --> A4["66"] -->|$649| A5["67"] --> A6["..."] --> A7["80"]
        • Coefficients depend on the units used to measure each variable. A one-year difference in education is not the same kind of change as a one-point difference in occupational prestige.
        • If we measured parental income in dollars instead of thousands of dollars, its coefficient would look much smaller even though the relationship would be the same.
          • Therefore, raw coefficients should not be used to rank factors measured in different units.
            • We CAN’T directly say "education has a stronger relationship with income than occupational prestige does" just because its raw coefficient is larger.

[[Reporting of standardized coefficients]] (std. Coeff.)

  • To compare the relative strength of factor variables, we use [[standardized coefficient]].
  • Standardized coefficients put all variables on the same scale.
    • This way, we can compare variables even if they are measured differently and with different ranges.
      • Population density is measured with categories (1-6).
      • Age is measured in years (18-89).
      • Occupational prestige is measured with a prestige score (16-80).
      • Education is measured in years (0-20).
  • Therefore, standardized coefficients answer a slightly different question:
    • Not “How much does one unit increase income?”
    • But “Which factor has the largest standardized association with income in this model?”
  • Based on standardized coefficients In the final model:
    • "0.27" ➜ Occupational prestige score has the largest standardized coefficient.
    • "0.22" ➜ Education has the second largest.
    • "0.09" ➜ Age has a smaller standardized coefficient.
    • "0.02" ➜ Residence category has the smallest standardized coefficient. When the p-value is greater than 0.05 (p > 0.05), it is not statistically significant.
  • This is why we should not compare coefficients directly.
    • Education has a larger raw coefficient than occupational prestige:
      • Education coefficient = 2,624
      • Occupational prestige coefficient = 649
    • But the standardized coefficient of occupational prestige is larger:
      • Occupational prestige std. Coeff. = 0.27
      • Education std. Coeff. = 0.22
    • This means that occupational prestige has a larger standardized association with personal income in this model.
  • We saw a similar issue in the car insurance rate example.
    • The coefficient for population was very small:
      • Population coefficient = 0.0015
      • At first, this looks like population has almost no association with insurance rates.
        • However, the standardized coefficient for population was the highest:
          • Population std. Beta = 0.41
    • This happens because population has a very large range.
      • A one-person difference corresponds to only a small difference in the predicted rate.
      • But a one-standard-deviation difference in city population can be large.
        • Therefore, a small raw coefficient can still correspond to a relatively large standardized association.
  • In short:
    • [[Coefficient]]s show the estimated association for a one-unit increase in a factor, after accounting for other factors.
    • [[Standardized coefficient]]s show the relative size of associations when factors are measured in different units.
      • To compare the sizes of these associations, we look at the absolute values of standardized coefficients, not raw coefficients.
      • In the final model, this is how we would rank the statistically significant factors by their standardized coefficients.
        • The largest standardized association with respondents’ personal income is for respondents' occupational prestige score (std. Coeff=0.27), followed by respondents' education in years (std. Coeff=0.22), and respondents' age (std. Coeff=0.09).

[[Reporting of adjusted R-squared]] (R² adjusted)

  • The [[adjusted R-squared]] value summarizes how much of the variation in the outcome is accounted for by the entire model, while adjusting for the number of factor variables.
    • It helps us compare models that include different numbers of factors.
    • The adjusted R-squared value should be reported as a percentage.
  • Here's a shortcut for converting a number with decimals to a percentage:
    • Move the dot two times to the right:
      • 0.004 ➜ 0.4%
      • 0.023 ➜ 2.3%
      • 0.159 ➜ 15.9%
      • 0.196 ➜ 19.6%
  • In our regression models:
    • Model 1 adjusted R-squared: 0.004 ➜ 0.4%
    • Model 2 adjusted R-squared: 0.023 ➜ 2.3%
    • Model 3 adjusted R-squared: 0.159 ➜ 15.9%
    • Model 4 adjusted R-squared: 0.196 ➜ 19.6%
  • This means that every additional factor variable improved the subsequent regression models.
    • Model 2 explains more of personal income than Model 1.
    • Model 3 explains more of personal income than Model 2.
    • Model 4 explains more of personal income than Model 3.
  • In the final model, the adjusted R-squared is 0.196.
    • This means that population density of residence during adolescence years, age, occupational prestige score, and education account for 19.6% of the variation in respondents' personal income in this model.
    • The remaining variation (80.4%) is not accounted for by this model. It may involve other factors, measurement differences, or chance.

[[Linear regression]] #interpretation

  • Respondents' personal income

    • Factors Coeff. std. Coeff p
      (Intercept) -41471.83
      (3964.93)      		 -0.00
      (0.02)      		 0.001***
      Population density of residence during
      adolescence years      		 499.80
      (417.99)      		 0.02
      (0.02)      		 0.232
      Respondents' age 196.07
      (41.51)      		 0.09
      (0.02)      		 0.001***
      Respondents' occupational prestige score 649.05
      (50.12)      		 0.27
      (0.02)      		 0.001***
      Respondents' education in years 2624.02
      (256.54)      		 0.22
      (0.02)      		 0.001***
      Observations 2272
      R² / R² adjusted 0.197 / 0.196

  • Linear regression interpretation sample

    • First section: The significance levels
      • Respondents' age, respondents' occupational prestige score, and respondents' education in years have statistically significant associations with respondents' personal income since their p-values are less than 0.05.
      • Population density of residence during adolescence years is not a statistically significant factor of respondents' personal income since the p-value is greater than 0.05.
    • Second section: The explanation of [[coefficient]]s with [[unit of analysis]]
      • One additional year of respondents' age is associated with $196 higher predicted personal income.
      • One additional point of respondents' occupational prestige score is associated with $649 higher predicted personal income.
      • One additional year of respondents' education is associated with $2,624 higher predicted personal income.
    • Third section: The explanation of [[standardized coefficient]]s
      • The largest standardized association with respondents’ personal income is for respondents' occupational prestige score (std. Coeff=0.27), followed by respondents' education in years (std. Coeff=0.22), and respondents' age (std. Coeff=0.09).
    • Fourth section: The explanation of [[adjusted R-squared]] value
      • The adjusted R-squared value indicates that the model, which includes population density of residence during adolescence years, age, occupational prestige, and education, accounts for 19.6% of the variation in respondents' personal income.

  • Linear regression interpretation template

    • First section: The significance levels
      • [[Variable label]] of significant [[factor variable]] 1, variable label of significant factor variable 2, variable label of significant factor variable 3... have statistically significant associations with variable label of [[outcome variable]] since their p-values are less than 0.05.
      • [If any]: Variable label of nonsignificant factor variable 4 is not a statistically significant factor of variable label of outcome variable since the p-value is greater than 0.05.
    • Second section: The explanation of [[coefficient]]s with [[unit of analysis]]
      • One additional [year/point/unit/score/dollar of factor variable 1] of variable label of significant factor variable 1 is associated with [coefficient with outcome variable unit] higher/lower predicted variable label of outcome variable.
      • One additional [year/point/unit/score/dollar of factor variable 2] of variable label of significant factor variable 2 is associated with [coefficient with outcome variable unit] higher/lower predicted variable label of outcome variable.
      • One additional [year/point/unit/score/dollar of factor variable 3] of variable label of significant factor variable 3 is associated with [coefficient with outcome variable unit] higher/lower predicted variable label of outcome variable.
    • Third section: The explanation of [[standardized coefficient]]s
      • The largest standardized association with variable label of outcome variable is for variable label of the strongest factor variable (std. Coeff=0.xx), followed by variable label of the second strongest factor variable (std. Coeff=0.xx), and variable label of the third strongest factor variable (std. Coeff=0.xx)...
    • Fourth section: The explanation of [[adjusted R-squared]] value
      • The adjusted R-squared value indicates that the model, which includes variable label of factor variable 1, variable label of factor variable 2, variable label of factor variable 3..., accounts for [adjusted R-squared as a percentage] of the variation in variable label of outcome variable.

  • Interpretation explanation

    • First section: The significance levels
      • We identify the variable labels of the factor variables that have statistically significant associations with the outcome variable.
        • When the p-value is less than 0.05, we say the factor variable has a statistically significant association with the outcome variable.
        • When the p-value is greater than 0.05, we say the factor variable is not a statistically significant factor of the outcome variable.
    • Second section: The explanation of coefficients
      • We use the Coeff. column to explain how a one-unit increase in each statistically significant factor variable is associated with higher or lower predicted values of the outcome variable.
        • We include the units of both the factor variable and outcome variable (year, point, unit, score, dollar, etc.).
        • Note: We do not include nonsignificant factor variables in this section.
    • Third section: The explanation of standardized coefficients
      • We use the std. Coeff. column to identify the factor variables with the largest standardized associations with the outcome variable.
        • We list the statistically significant factor variables from the largest to the smallest absolute standardized coefficient.
        • Note: We do not include nonsignificant factor variables in this section.
    • Fourth section: The explanation of adjusted R-squared
      • We report the adjusted R-squared as a percentage to explain how much of the variation in the outcome variable is accounted for by the model.
        • We list all factor variables included in the model, including those that are not statistically significant.