Bayesian Measurement Models

MIT | 17.S952 | Fall 2023 | Th 9–11 | Prof. Devin Caughey

Last updated: October 29, 2023

Overview

This course covers quantitative measurement from a Bayesian perspective. It focuses on the specification of measurement models linking observed data (i.e., manifest indicators) to unobserved constructs (i.e., latent variables) of interest. For estimation of these models, we will rely primarily on the Bayesian software environment Stan, as called from R, though we will occasionally touch on other R-based methods. The goal is to get students comfortable specifying and estimating “bespoke” measurement models tailored for particular applications. The course applies this basic framework to a large range of problems and topics, including hierarchical models, factor analysis, item response theory, latent class analysis, ecological inference, network data, and text analysis. Each is covered only in enough depth to provide a sense of what a Bayesian approach to the problem might look like. A solid command of generalized linear models and the theory of likelihood and Bayesian inference (i.e., the material in 17.804) is preferred, but the only prerequisites are 17.800 and 17.802 or their equivalent.

Contact

Instructor Email address Office Office hours
Devin Caughey E53-463 Thursdays 11–12

Course website: https://canvas.mit.edu/courses/21390

Materials

Texts

Readings are an important component of the course. The lectures assume detailed familiarity with the assigned texts, so before each session make sure to give them a close read. Please read them in the order in which they appear on the syllabus.

Required

We will read large parts of the following texts. All other readings will be posted on the course website or can be freely accessed online.

  • McElreath (2020) — accessible online with MIT credentials
  • Lauderdale (2022) — draft freely available online
  • Stan Development Team (2022) — freely available online

Suggested

We will read selections from the following texts, but each is a valuable resource that at some point you may want to read in its entirety.

  • Skrondal and Rabe-Hesketh (2004) — PDF posted
  • Jackson (2008) — PDF posted
  • Jackman (2009) — PDF posted
  • Gelman et al. (2014) — accessible online with MIT credentials

Software

You will make extensive use of the following programs and packages. Some have a lengthy and somewhat cumbersome installation process, so please get started on installing them as soon as you can.

Assignments

Grades in this class are based on three components:

  • Article presentation and general class participation (10%): In addition to participating productively in class discussions throughout the term, each student is expected to give one presentation summarizing an applied paper and relating it to the topics of that session.
  • Homework exercises (45%): Students are expected to complete 9 weekly homework exercises, each worth 5% of their course grade. The primary focus of these assignments will be using R and Stan to implement measurement models and then interpreting and assessing the results.
  • Research paper (45%): The capstone assignment of this course is a research project that employs methods covered in the class. Co-authoring is permitted but not required. The project involves several components due at the following times:
    • Session 5: Submit project idea(s), having identified potential data sources and perhaps performed some exploratory analysis (5% of course grade).
    • Session 9: With any coauthors, submit brief description of proposed project, including a descriptive analysis of the dataset (summary statistics, plots, etc.) and an explanation of the concepts you propose to measure and the methods you anticipate using to do so (5% of course grade).
    • Session 12 or 13: Give a short conference-style presentation of your project to the class (10% of course grade).
    • One week after Session 13: Submit final paper, revised to incorporate feedback from presentations (25% of course grade).

Schedule

1. Thursday, September 7: Measurement

Topics

  • representational vs. pragmatic measurement
  • measurement models
  • measurement error, validity, and reliability
  • consequences of mismeasurement
  • fair measurement

Required readings

  • Lauderdale (2022), pp. 13–114 (chap. 1–6)
  • Jacobs and Wallach (2021)
  • Ansolabehere, Rodden, and Snyder (2008)

Additional resources

2. Thursday, September 14: Bayesian inference

TODO: Install software

Topics

  • Fundamentals of Bayesian statistics
  • Stan and brms
  • The workflow of of Bayesian inference
  • Bayesian regression
  • Regression as measurement

Required readings

  • Lauderdale (2022), pp. 135–158 (chap. 8)
  • McElreath (2020), chap. 1–5, 7, and 9
  • Betancourt (2020)

Additional resources

  • Jackman (2009), chap. 1–2

3. Thursday, September 21: Latent variables and response models

TODO: Submit problem set #1 (Stan and brms basics)

Topics

  • Generalized linear models
  • Latent variables
  • Bradley–Terry models

Required readings

  • McElreath (2020), chap. 10–12
  • Skrondal and Rabe-Hesketh (2004), chap. 1–2
  • Lauderdale (2022), pp. 115–134 (chap. 7)
  • Zucco, Batista, and Power (2019)

Additional resources

4. Thursday, September 28: Hierarchical models

TODO: Submit problem set #2 (Bradley–Terry models)

Topics

  • Hierarchical priors
  • Hierarchical/multilevel GLMS
  • Multilevel regression and poststratification (MRP)

Required readings

  • McElreath (2020), chap. 13
  • Gelman et al. (2014), chap. 15
  • Skrondal and Rabe-Hesketh (2004), sec. 3.2
  • Lax and Phillips (2009)

Additional resources

5. Thursday, October 5: Scale measurement with metrical indicators

TODO: Submit problem set #3 (MRP)

Topics

  • Unsupervised measurement
  • GLLAMM general factor model
  • Bayesian factor analysis
  • Principal components
  • Identification with one factor

Required readings

  • Lauderdale (2022), pp. 201–224 (chap. 11)
  • Jackman (2009), pp. 435–354 (part of chap. 9)
  • Bishop (2006), pp. 559–86 (most of chap. 12)
  • Skrondal and Rabe-Hesketh (2004), sec. 3.3.1–3.3.3 and 4.1–4.4
  • Pan and Xu (2018)

Additional resources

6. Thursday, October 12: Scale measurement with categorical indicators

TODO: Submit problem set #4 (factor analysis)

Topics

  • Item response theory
  • Spatial models of choice
  • Mixed factor models
  • Identification in multiple dimensions

Required readings

  • Lauderdale (2022), pp. 225–236 (chap. 12)
  • Clinton, Jackman, and Rivers (2004)
  • Quinn (2004)
  • Tsai and Lin (2017)

Additional resources

7. Thursday, October 19: Structural models

TODO: Submit problem set #5 (item response theory)

  • Hierarchical latent variable models
  • Dynamic linear models
  • Forecasting
  • Estimating effects on latent variables

Required readings

  • Jackman (2009), pp. 471–488 (sec. 9.4)
  • Zhou (2019)
  • Caughey and Warshaw (2015)
  • Kołczyńska and Bürkner (2023)

Additional resources

8. Thursday, October 26: Model specification (discussion session)

TODO: Submit problem set #6 (dynamic models)

Topics

  • Differential item functioning
  • Joint scaling
  • Dimension selection
  • Latent variables as covariates
  • Bayesian casual inference

Required readings

  • Jessee (2016)
  • Bølstad (2020)
  • Marble and Tyler (2022)
  • Pang, Liu, and Xu (2022)

Additional resources

  • King et al. (2004)

9. Thursday, November 2: Class measurement

TODO: Submit descriptive analysis of research project

Topics

  • Clustering
  • Gaussian mixture models
  • Latent class models
  • Hidden Markov models

Required readings

Additional resources

10. Thursday, November 9: Missing and mismeasured data

TODO: Submit problem set #7 (mixture models)

Topics

  • Errors in variables
  • Propagating uncertainty
  • Multiple (over)imputation
  • Joint modeling

Required readings

  • McElreath (2020), chap. 15
  • Blackwell, Honaker, and King (2017)
  • Treier and Jackman (2008), esp. pp. 215–16
  • Tai, Yue, and Solt (2022)
  • Claassen (2022)

Additional resources

  • Knox, Lucas, and Cho (2022)

11. Thursday, November 16: Ecological inference

TODO: Submit problem set #8 (measurement error)

Topics

  • The ecological fallacy
  • Ecological inference models
  • Dynamic and hierarchical EI
  • Combining aggregate and individual-level data

Required readings

  • Wakefield (2004)
  • Corder and Wolbrecht (2006)
  • Caughey and Wang (2019)

Additional resources

  • Glynn and Wakefield (2010)

Thursday, November 23: NO CLASS (Thanksgiving)

12. Thursday, November 30: Network analysis

Topics

  • Network concepts
  • Exponential random graph models
  • Latent space/factor models
  • Additive and multiplicative effects (AMEN) models

Required readings

  • Jackson (2008), chap. 1–4, esp. pp. 3–17 (sec. 1.1–1.2), 20–43 (sec. 2.1–2.2), and 77–86 (sec. 4.1–4.2)
  • Young, Cantwell, and Newman (2021)
  • Hoff (2021)
  • Dorff, Gallop, and Minhas (2020)

Additional resources

13. Thursday, December 7: Text analysis

TODO: Submit problem set #9 (AMEN models)

Topics

  • Text as data
  • Naive Bayes models
  • Latent Dirichlet allocation
  • Structural topic models
  • Word embeddings

Required readings

  • Grimmer and Stewart (2013)
  • Roberts, Stewart, and Airoldi (2016)
  • Lauretig (2019)

Additional resources

  • Silge and Robinson (2017)

References

Ansolabehere, Stephen, Jonathan Rodden, and James M. Snyder Jr. 2008. “The Strength of Issues: Using Multiple Measures to Gauge Preference Stability, Ideological Constraint, and Issue Voting.” American Political Science Review 102 (2): 215–32.
Bertsou, Eri, and Daniele Caramani. 2022. “People Haven’t Had Enough of Experts: Technocratic Attitudes Among Citizens in Nine European Democracies.” American Journal of Political Science 66 (1): 5–23.
Betancourt, Michael. 2020. “An Introduction to Stan.” March 1, 2020. https://betanalpha.github.io/assets/case_studies/stan_intro.
Bishop, Christopher M. 2006. Pattern Recognition and Machine Learning. Springer.
Blackwell, Matthew, James Honaker, and Gary King. 2017. “A Unified Approach to Measurement Error and Missing Data: Overview and Applications.” Sociological Methods & Research 46 (3): 303–41.
Bølstad, Jørgen. 2020. “Capturing Rationalization Bias and Differential Item Functioning: A Unified Bayesian Scaling Approach.” Political Analysis 28 (3): 340–55.
Broockman, David E., and Benjamin E. Lauderdale. 2023. “‘Moderates’.” 10.31219/osf.io/ma2fs.
Bürkner, Paul-Christian. 2017. “brms: An R Package for Bayesian Multilevel Models Using Stan.” Journal of Statistical Software 80 (1): 26–35. https://doi.org/10.18637/jss.v080.i01.
Caughey, Devin, and Mallory Wang. 2019. “Dynamic Ecological Inference for Time-Varying Population Distributions Based on Sparse, Irregular, and Noisy Marginal Data.” Political Analysis 27 (3): 388–96. https://doi.org/10.1017/pan.2019.4.
Caughey, Devin, and Christopher Warshaw. 2015. “Dynamic Estimation of Latent Opinion Using a Hierarchical Group-Level IRT Model.” Political Analysis 23 (2): 197–211. http://dx.doi.org/10.1093/pan/mpu021.
Claassen, Christopher. 2022. “Including Measurement Uncertainty in Time-Series, Cross-Sectional Analyses: The Case of Mood and Democracy.” http://chrisclaassen.com/docs/Claassen_dynamic_measurement_error.pdf.
Clinton, Joshua, Simon Jackman, and Douglas Rivers. 2004. “The Statistical Analysis of Roll Call Data.” American Political Science Review 98 (2): 355–70.
Corder, J. Kevin, and Christina Wolbrecht. 2006. “Political Context and the Turnout of New Women Voters After Suffrage.” Journal of Politics 68 (1): 34–49.
Dorff, Cassy, Max Gallop, and Shahryar Minhas. 2020. “Networks of Violence: Predicting Conflict in Nigeria.” Journal of Politics 82 (2): 476–93.
Fowler, Anthony, Seth J. Hill, Jeffrey B. Lewis, Chris Tausanovitch, Lynn Vavreck, and Christopher Warshaw. 2023. “Moderates.” American Political Science Review 117 (2): 643–60.
Gelman, Andrew. 2014. “How Bayesian Analysis Cracked the Red-State, Blue-State Problem.” Statistical Science 29 (1): 26–35.
Gelman, Andrew, John B. Carlin, Hal S. Stern, David B. Dunson, Aki Vehtari, and Donald B. Rubin. 2014. Bayesian Data Analysis. 3rd ed. Boca Raton, FL: CRC Press.
Glynn, Adam N., and Jon Wakefield. 2010. “Ecological Inference in the Social Sciences.” Statistical Methodology 7 (3): 307–22.
Grimmer, Justin, and Brandon M. Stewart. 2013. “Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts.” Political Analysis 21 (3): 267–97.
Hand, David J. 2016. Measurement: A Very Short Introduction. Oxford University Press.
Hoff, Peter. 2021. “Additive and Multiplicative Effects Network Models.” Statistical Science 36 (1): 34–50.
Jackman, Simon. 2009. Bayesian Analysis for the Social Sciences. Hoboken, NJ: Wiley.
Jackson, Matthew O. 2008. Social and Economic Networks. Princeton University Press.
Jacobs, Abigail Z., and Hanna Wallach. 2021. “Measurement and Fairness.” In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (FAccT ’21), 375–85. New York: ACM. https://doi.org/10.1145/3442188.3445901.
Jessee, Stephen A. 2016. “(How) Can We Estimate the Ideology of Citizens and Political Elites on the Same Scale?” American Journal of Political Science 60 (4): 1108–24.
King, Gary, Christopher J. L. Murray, Joshua A. Salomon, and Ajay Tandon. 2004. “Enhancing the Validity and Cross-Cultural Comparability of Measurement in Survey Research.” American Political Science Review 98 (1): 191–207.
Knox, Dean, Christopher Lucas, and Wendy K. Tam Cho. 2022. “Testing Causal Theories with Learned Proxies.” Annual Review of Political Science 25: 1–23. https://doi.org/10.1146/annurev-polisci-051120-111443.
Kołczyńska, Marta, and Paul-Christian Bürkner. 2023. “Modeling Public Opinion over Time: A Simulation Study of Latent Trend Models.” Journal of Survey Statistics and Methodology. https://doi.org/10.1093/jssam/smad024.
Lauderdale, Benjamin E. 2022. “Pragmatic Social Measurement.” https://uclspp.github.io/POLS0013/readings/pragmatic-social-measurement.pdf.
Lauretig, Adam. 2019. “Identification, Interpretability, and Bayesian Word Embeddings.” In Proceedings of the Third Workshop on Natural Language Processing and Computational Social Science, 7–17. Minneapolis, MN: NLP+CSS; Association for Computational Linguistics. https://doi.org/10.18653/v1/W19-2102.
Lax, Jeffrey R., and Justin H. Phillips. 2009. “How Should We Estimate Public Opinion in the States?” American Journal of Political Science 53 (1): 107–21.
Marble, William, and Matthew Tyler. 2022. “The Structure of Political Choices: Distinguishing Between Constraint and Multidimensionality.” Political Analysis 30 (3): 328–45.
McElreath, Richard. 2020. Statistical Rethinking: A Bayesian Course with Examples in R and Stan. 2nd ed. Chapman & Hall/CRC. https://learning.oreilly.com/library/view/statistical-rethinking-2nd/9780429639142/.
Pan, Jennifer, and Yiqing Xu. 2018. “China’s Ideological Spectrum.” Journal of Politics 80 (1): 254–73.
Pang, Xun, Licheng Liu, and Yiqing Xu. 2022. “A Bayesian Alternative to Synthetic Control for Comparative Case Studies.” Political Analysis 30 (2): 269–88.
Park, Jong Hee. 2010. “Structural Change in u.s. Presidents’ Use of Force.” American Journal of Political Science 54 (3): 766–82.
Quinn, Kevin M. 2004. “Bayesian Factor Analysis for Mixed Ordinal and Continuous Responses.” Political Analysis 12 (4): 338–53.
Roberts, Margaret E., Brandon M. Stewart, and Edoardo M. Airoldi. 2016. “A Model of Text for Experimentation in the Social Sciences.” Journal of the American Statistical Association 111 (515): 988–1003.
Silge, Julia, and David Robinson. 2017. Text Mining with R: A Tidy Approach. O’Reilly. https://www.tidytextmining.com.
Skrondal, Anders, and Sophia Rabe-Hesketh. 2004. Generalized Latent Variable Modeling: Multilevel, Longitudinal, and Structural Equation Models. Boca Raton, FL: CRC Press.
Stan Development Team. 2022. “Stan User’s Guide.” https://mc-stan.org/docs/stan-users-guide/.
Tai, Yuehong “Cassandra”, Hu Yue, and Frederick Solt. 2022. “Democracy, Public Support, and Measurement Uncertainty.” American Political Science Review. https://doi.org/10.1017/S0003055422000429.
Treier, Shawn, and Simon Jackman. 2008. “Democracy as a Latent Variable.” American Journal of Political Science 52 (1): 201–17.
Tsai, Tsung-han, and Chang-chih Lin. 2017. “Modeling Guessing Components in the Measurement of Political Knowledge.” Political Analysis 25 (4): 483–504.
Wakefield, Jon. 2004. “Ecological Inference for 2 \(\times\) 2 Tables.” Journal of the Royal Statistical Society. Series A (General) 167 (3): 385–445.
Young, Jean-Gabriel, George T. Cantwell, and M. E. J. Newman. 2021. “Bayesian Inference of Network Structure from Unreliable Data.” Journal of Complex Networks 8 (6).
Zhou, Xiang. 2019. “Hierarchical Item Response Models for Analyzing Public Opinion.” Political Analysis 27 (4): 481–502.
Zucco, Cesar, Mariana Batista, and Timothy J. Power. 2019. “Measuring Portfolio Salience Using the Bradley–Terry Model: An Illustration with Data from Brazil.” Research & Politics 6 (1): 1–8. https://doi.org/10.1177/2053168019832089.