2020 Census Quality Indicators:
February 2021 Public Update

In October 2020, the American Statistical Association released the 2020 Census Quality Indicators, a report written by a task force of census experts with a plan to shed light on the quality, accuracy, and coverage of the 2020 Census counts. The report proposed various measures of quality, accuracy, and coverage spanning the multiple decennial census components to be applied to the 2020 data as it became available. A primary report recommendation was for qualified external researchers to be granted access to the primary data so they can conduct independent analyses of data quality.

In this communication, the first of anticipated biweekly updates on quality indicator developments, we provide a progress report on the status of our recommendations, including the adoption and use of the quality indicators, next steps, and the scope of the work. We do not have results to report at this point, but will communicate a timeline for reporting results as soon as we can.

The US Census Bureau welcomed the ASA’s efforts to assist in assessing 2020 Census data quality. The bureau offered to allow three ASA member researchers with special sworn status (SSS) to access and apply quality indicators to confidential, internal 2020 Census data. While the ASA is free to report on process aspects of this work, the actual substantive results of researchers’ analyses are necessarily subject to the Census Bureau’s disclosure avoidance system review. The bureau is also providing technical assistance to the researchers, including analysis and IT support, to the extent feasible in these extraordinarily busy times for the bureau.

The three researchers are Paul Biemer, Robert Fay, and Joseph Salvo. See below for their bio sketches. (Please refrain from contacting these researchers directly about their work on this project so they can carry out their work objectively and unhindered.)

To inform users about the quality of the 2020 Census and assist in their efforts to assess the fitness for use of its products, the researchers will address the following questions:

  • Do the quality indicators (QIs) suggest a high risk of state-level coverage error (undercounts/overcounts) in the 2020 Census?
  • Which states have the highest risks of coverage error?
  • Which of the following sources of census error are the likely causes of the high risks?
    • Master Address File (MAF) creation and/or revisions
    • Self-response phase
    • Nonresponse follow-up
    • Post-data collection data processing
  • Which QIs provide the most compelling evidence for these conclusions?

Most of the planned analyses will use corresponding 2010 data as a basis for comparison to identify 2020 Census outliers or anomalies. State-level tables will be computed separately for each of the 50 states, DC, and the nation. Anomalies identified in these tables could lead to additional sub-state analyses to identify the root causes of the anomalies.

Thus far, the planned analyses involve the following:

  1. Population Estimates (2020 Vintage Census), 2020 Administrative Records (ADREC) Census Estimates, and 2020 Census Counts by State
  2. Distribution of Type of Enumeration (e.g., self-response, administrative record imputation, proxy enumeration, etc.)
  3. Vacant Units
  4. Data for Key Characteristics (name, birthdate, sex, race, and ethnicity)
  5. Missing Data for Key Housing Unit (HU) Characteristics
  6. Returns with/without a Census ID
  7. Characteristics of Persons Responding by Alternate Modes and Protocols
  8. Master Address File (MAF) Change Analysis
  9. Analysis of Post-Data Collection Data Processing (DaP) Outcomes
  10. Analysis of responses submitted without a census identification number

The intent of applying these indicators is to provide information about the quality of the 2020 Census—from apportionment to redistricting—in as timely a manner as possible. The 2020 Census Quality Indicators do not include consideration of such issues as the bureau’s new disclosure avoidance system. The task force welcomes and recommends more thorough analyses by other groups (e.g., the National Academies Committee on National Statistics or a congressionally appointed body) to develop a fuller understanding of the quality of the 2020 Census; consider broader issues; and inform an accurate, fair, and efficient 2030 Census.

With SSS status, Biemer, Fay, and Salvo will be able to analyze data that cannot be disclosed but that they will attempt to summarize to a level to meet the disclosure standards for publication. To be clear, the researchers’ roles are to carry out their analysis and report their findings to the ASA for subsequent public distribution. They have not been asked to address issues they identify or make policy recommendations.

Contact: Steve Pierson, ASA Director of Science Policy, pierson@amstat.org.

Bio sketches:

  • Paul Biemer is distinguished fellow of statistics at RTI International with more than 40 years of experience in error modeling and evaluation for official statistics and is intimately familiar with decennial census processes and systems. He spent eight years as an employee of the Census Bureau, where he worked on error evaluation projects for the 1980 Census. He served as a bureau statistical consultant for the 1990, 2000, and 2010 decennial censuses. He also has consulted extensively on the quality of official statistics with a foreign government, including Statistics Sweden, Statistics Canada, Australian Bureau of Statistics, Statistics New Zealand, IAB (Germany), Central Bureau of Statistics (The Netherlands), and City, University of London. Biemer has authored or edited eight books in survey and census methodology. He is a fellow of the ASA and AAAS and an elected member of the ISI. He holds a few awards for his contributions to the fields of statistics and survey methodology, including the W.S. Connor Award, H.O. Hartley Award, Morris Hansen Award, and Roger Herriot Award.
  • Robert Fay is a senior statistical fellow at Westat, Inc. and assistant adjunct professor at the University of Maryland. He retired from his position as senior mathematical statistician at the Census Bureau in 2008, after a 34-year career there that included extensive work on coverage measurement for the 1980, 1990, and 2000 censuses. Fay earned his PhD in statistics from The University of Chicago and currently serves on the Advisory Committee on Statistical Methods for Statistics Canada. He is a fellow of the ASA and the AAAS, and his awards include the Roger Herriot Award from the American Statistical Association and the Gold Medal Award from the US Department of Commerce.
  • Joseph J. Salvo is New York City’s chief demographer. He has testified before Congress, been an adviser to the Census Bureau, served on panels at the National Academy of Sciences, and, most recently, was an expert witness in the citizenship lawsuit headed by the New York Attorney General. Salvo is presently participating in a national effort to monitor and evaluate the 2020 Census and the use of methods to protect the confidentiality of census data while protecting its utility for data users. He earned MA and PhD degrees from Fordham University, is a recipient of the Sloan Public Service Award from the Fund for the City of New York, and is a fellow of the American Statistical Association.