NAEP and Inattentional Blindness: You Can’t Manage What You Don’t Measure

If you have 40 seconds, please indulge me for a brief brain experiment. Watch the first half minute of the video below of college students playing in a circle. Your task? Count how many times the students wearing white shirts pass the basketball.

If you guessed 15 passes, you win!

But did you see the gorilla?!?

For two decades, in multiple countries and contexts, as part of repeated research studies, thousands of audiences from diverse backgrounds have watched this video for the first time. A stunning 50% become so distracted trying to count the passes that they completely miss something extraordinary. As the researchers, Christopher Chabris and Daniel Simons, note, “Halfway through the video, a female student wearing a full-body gorilla suit, walked into the scene, stopped in the middle of the players, thumped her chest, and then walked off, spending about nine seconds on-screen.”

Don’t believe me? Watch again.

In the world of neuroscience, this phenomenon of being oblivious to the obvious is called “inattentional blindness.” This occurs any time we as human beings fail to notice a fully visible but unexpected object because our attention was on another task, event, or activity.

Inattentional blindness is an important concept to keep in mind as we all await the April 10 release of the 2017 National Assessment of Educational Progress (NAEP) for reading and mathematics for fourth and eighth grades. The U.S. Department of Education releases results bi-annually to help inform decisions about how to improve the education system in our country.

If prior results are any guide, reading achievement results will likely remain stagnant. Nevertheless, there will be reams of analysis of certain subgroups, especially highlighting the stubborn achievement gaps within the mesmerizing categories of race and income percentile. For example, as a NAEP teaser, Fordham Institute produced 35 charts of analysis, the vast majority of which focused on racial differences in outcomes for African American, Hispanic, and white students.

Yet you’re unlikely to see NAEP analyses of achievement differences by family structure, even though we know the stability of the family within which children are raised matters monumentally to their educational outcomes.

Take a recent study from the Institute for Women’s Policy Research, which finds that the number of single mothers in college has doubled over the last decade to nearly 2.1 million. Despite their best efforts to create a better future for themselves and their children, only 28% of single mothers who entered college between 2003 and 2009 earned a degree or certificate within six years—an outcome that adversely affects both mother and child. For college-going women without children, it was 57%.

Indeed, evidence on the adverse effects of fragile families on children is widely accepted. Even studies cited by those who believe promotion of marriage is an ineffective strategy to wage in the war against poverty correctly assert that “decades of research show that children raised in single-parent homes fare worse on a wide range of outcomes (e.g., poverty, educational attainment, nonmarital and teen childbearing) than children raised by two biological parents. The poverty rates of single-parent households are particularly striking.”

Consider, too, recent words from the University of North Carolina at Chapel Hill. In announcing a grant to study the link between poverty and the high-risk impact on processes that facilitate learning, the school declared that:

Early-life experiences that occur between the prenatal period through age 2 have major impacts on the trajectories of children's early cognitive development, including the neural substrates that support these abilities. Traditional investments toward children in the one-to-two years prior to their enrollment in kindergarten may be too late.

And according to the CDC, there were 1,016,040 births to women aged 25 and under in 2016. Seventy-one percent of these births were to unmarried women, and 42% were to women who were having between their second and eighth child. There are always exceptions, but the magnitude in the number of multiple, non-marital births to typically unprepared young women and men, and the decades-long rise in the rate of single-parenthood, create a much greater risk of unstable home environments and fragile families that frequently correlate with child poverty, chronic student absenteeism, and the kind of toxic stress that begins in utero and impairs long-term academic advancement of children.

Alongside the spirited debate about the causality between single-parent households and child poverty, there is new evidence of the impact of absent fathers, particularly on boys. After studying more than 1 million children born in Florida between 1992 and 2002, MIT researcher David Autor found that

Relative to their sisters, boys born to low-education and unmarried mothers, raised in low-income neighborhoods, and enrolled at poor-quality public schools have a higher incidence of truancy and behavioral problems throughout elementary and middle school, exhibit higher rates of behavioral and cognitive disability, perform worse on standardized tests, are less likely to graduate high school, and are more likely to commit serious crimes as juveniles.

And Raj Chetty and Nathaniel Hendren just released a comprehensive study titled “Race and Economic Opportunity in the United States” to understand why “racial disparities in income and other outcomes are among the most visible and persistent features of American society.” Some have framed the findings as evidence of the punishing reach of racism for black boys, but the research itself suggests a more nuanced, inclusive recognition of family structure as a key contributor to racial disparities. The study states that

higher rates of father presence among low-income black households are associated with better outcomes for black boys...Black father presence at the neighborhood level predicts black boys' outcomes irrespective of whether their own father is present or not, suggesting that what matters is...community-level factors associated with the presence of fathers, such as role-model effects or changes in social norms.

In other words, fathers matter.

So, what does all this mean for NAEP and understanding the factors that truly drive education outcomes? To be clear, NAEP data are a vital and invaluable resource. The test remains the largest continuing and nationally representative assessment of what our nation’s students know and can do. It has served as a national yardstick of student achievement since 1969.

NAEP includes information on special student demographic groups, such as race, ethnicity, socioeconomic status, gender, disability, and limited English proficiency. But reporting by these main measurement groupings—though critical—has the unintended effect of fueling the fixation on those categories as the primary bases for comparison. This is particularly true in the case of race, where the differences in outcomes (e.g., the Hispanic-white achievement gap) are perceived to be caused by race-related reasons (e.g., racism), which often elicit only race-related interventions, as opposed to other factors complementing those forces. What gets measured gets managed. And what does not get measured gets ignored.

Enter family structure in a full-body gorilla suit, undetected due to obsessive counting in each of the other NAEP demographic groupings.

Information on the stability of family structure is hard to track, especially if self-reported. Nevertheless, I challenge the technical experts at the National Center for Education Statistics (NCES) to incorporate within NAEP groupings some kind of proxy measure for family structure or stability that already exists and for which the government has already collected data. What about the age of parents? Or the marital status of parents at the time of birth? Perhaps there is a way to make use of (anonymized) tax records, as has been done with the Equality of Opportunity Project.

Unlike gender or race, family structure is not a static measure that is easily discernible. It changes over time. Take for example the powerful analysis done by Maria Cancian and Daniel R. Meyer in their study titled “Implications of Complex Families on Poverty and Child Support Policy.” They monitored 7,169 first-born children of unmarried mothers in Wisconsin between 1997 and 2007. See figure 1, from the study below.

Over the decade, 10-year-old children of unmarried parents lived in increasingly complex family structure, as both the biological father and mother had more children in and outside of their relationship.

One suggestion from noted psychologist and researcher Nicholas Zill is that parents of a representative subsample of NAEP-tested students could be contacted by Internet, phone, or mail to gather information on parent education, family income, welfare receipt, marital status, etc. These data could then enrich the NAEP tabulations to better estimate family structure and stability. This would add to the cost of the program, of course, but it would also strengthen the utility of the enterprise and perhaps shed new light on why educational progress has slowed.

As the data cited above indicate, family structure and stability are undeniably significant influencers on a host of child outcomes, including academic achievement. NAEP researchers would do our nation a great service if they developed a common metric that could estimate the effects of these forces on educational progress.

Virtually 100% of the research subjects who failed to witness the invisible gorilla upon first viewing immediately saw it the second time around. They usually express disbelief that they initially missed it. But once confronted with the truth, the viewers could no longer ignore the reality right before their eyes.

Leaders in education reform and the researchers at NCES have both the capacity and the responsibility to elevate family structure as a critical prism through which we evaluate our country’s educational progress, on par with race, socioeconomic status, and the other key groupings.

We do not yet know whether 2018 will bring more disappointing, unchanged NAEP scores, particularly in reading. But we can predict with great certainty that education reformers and policymakers will engage in handwringing about outcome disparities between races, classes, and genders. And that they’ll ignore a force at least as fundamental to human development. So, let 2020 be an opportunity to create a new baseline to measure progress by a critical indicator that has been hidden in plain sight for decades: family structure.

If we really want to cure our blindness and understand why our children are not making the progress we seek, we must make this essential and predictive measure invisible no more.

Ian Rowe is a senior visiting fellow at the Thomas B. Fordham Institute and the CEO of Public Prep.

Editor's Note: This article originally appeared on the Thomas B. Fordham Institute's Flypaper Blog and it has been reprinted here with permission.