No, the SAT doesn’t just “measure income”

I started this post to refute some specific arguments, but I changed my mind midstream and decided to add a lot more material than I initially envisioned. This is best viewed as being akin to a FAQ (Frequently Refuted Objections – FRO?) relating to the standardized tests and their use in higher education.

table of contents

• 1) SAT and income are not perfectly correlated
• 2) Income correlates well with lots of tests
• 3) mean GPA, academic rigor, and the significance HSGPA varies with SES
• 4) Even controlling for SES, different racial groups see different average scores
• 5) SES & B-W cognitive gaps start very young and keep growing
• 6) The B-W test score gaps are poorly explained by schools
• 7) The differences in “school quality” are generally overstated
• 8) The SAT’s association with family income is almost entirely mediated by family education
  
• 8) SES does not significantly mediate the predictive power of the SAT
• 9) Some have misconstrued the Univ. of Calif. data to argue otherwise
• 10) Nor are these tests biased against minorities
• 11) The predictive power of the SAT is vastly under-estimated
• 12) The strength of this prediction does not weaken past freshman year
• 13) Holding HSGPA constant the SAT offers significant incremental validity
• 14) The SAT is well correlated with IQ tests
• 15) Lumosity’s cognitive tests show strong correlations with SAT across universities
• 16) SAT test prep has little effect
• 17) Test prep rates do not vary all that much
• 18) SATs predict graduation rates between schools too
• 19) When schools systematically discount standardized tests the effects are very obvious
• 20) Heritability explains a great deal of these systematic SES relationships

SAT and income are not perfectly correlated

The SAT is certainly modestly correlated with parental income, but it is simply not true that the SAT is nothing more than a measure of family income.

I will briefly plot the 2011 SAT reading scores by income level to illustrate that the r**2 is considerably less than one.

There is significant overlap across the entire income distribution:

Box and whisker plot of simulated test scores

Simulated distribution by actual test taker counts

On the relationship between negative home owner equity and racial demographics

There are a large and growing number of popular media articles alleging racial discrimination in the mortgage market.   It is simply assumed that if lenders are less willing to extend credit to blacks or make loans in “black neighborhoods” as often or with similarly favorable terms as they do whites or “white neighborhoods” that this apt to be explained by explicit racism or (subconscious) bias.  These naive arguments persist despite tremendous evidence that there are observable and unobservable differences that have profound effects on credit risk.

I will briefly describe some of this evidence before making my own modest contribution using data from zillow.com and the US census. You can click here if you are familiar with this literature already and wish to skip ahead to my analysis.

1- Blacks have much lower credit scores (e.g., FICO) 

source

The difference in between the white and black means is about 1 standard deviation.

On the effects of wealth on the B-W gaps, a response to questions posed by a commenter

Max, a commenter, asked:

Could you do an analysis on racial differences in educational outcomes after controlling for parental education, parental occupation, household wealth, neighborhood wealth, neighborhood education, single parent status, native language etc.? I’ve seen you control for family income and parental education (and occasionally both), but I’ve never seen you control for more beyond that (perhaps I’ve missed something!). In Chapter 16 of Affirmative Action for the RichThe Future of Affirmative Action, Dalton Conley of New York University used the Panel Study of Income Dynamics to show that parental wealth (not income) and parental education are the best predictors of college completion, which means that they may also be good predictors of other educational outcomes. He also discussed the data showing that racial wealth gaps are much larger than racial income gaps, which implies that wealth could account for a larger portion of the achievement gap than income. Could you do a similar analysis for IQ? The reason I’m asking for all this is that Carnevale and Strohl control for all of these factors and are left with a very small race effect: http://www.tcf.org/assets/downloads/tcf-CarnevaleStrivers.pdf

I have heard this bit about wealth before.  I am deeply skeptical that wealth can mediate much, if any, of the B-W gap.

Before I dig into this though, let’s take a very brief look at some of the studies Max cited (see here and here) :

A quick post on gun related homicides

While I am personally ambivalent about gun ownership and suspect it plays an incremental role in relative differences between countries/regions/etc (holding other things roughly equal), I thought I’d add some perspective into this argument about the presumed causality of gun ownership on homicide rates.

Mother Jones analysis includes gun suicides and makes no attempt to correct for even course-grain racial/ethnic confounds.

---

I downloaded the 2009-2013 data for gun-related homicides by race/ethnicity from the CDC’s WONDER database  and compared it to the gun ownership rate data via wikipedia.

I do not feel like doing a lengthy analysis here and now, but suffice it to say that once you remove suicides and race/ethnicity from the equation the case gets much weaker.

There is no evidence of a positive correlation here for blacks and hispanics (if anything somewhat negative).

There is a positive correlation for non-hispanic whites (r=0.45), but it pales in comparison to the racial/ethnic differences here.  To put this into perspective, amongst non-hispanic whites (the bulk of the gun owners in most states), states with the highest gun ownership rates have just 1 death per 100,000 more than states with the lowest rates (on average).

On SAT, ACT, IQ, and other psychometric test correlations

Although I touched on this topic briefly before, I seem to have forgotten some relevant plots and data and don’t feel like editing the last one, so this post is somewhat redundant…

Both the SAT and ACT are quite well correlated with IQ tests, so much so that you can make good guesses about someone’s IQ if you know their SAT/ACT score and vice versa.

On sex differences within California public schools

A year or two ago I read an article which demonstrated that sex differences in math and reading achievement are inversely related within and across countries on the PISA tests.  The smaller the male-female math gap, the larger the verbal gap and vice versa.  This tends to support the view that there are innate underlying differences in average abilities and interests between the sexes that strongly influence these patterns.

 

 

 

NLSY97: Assessing systematic patterns in relationships between IQ and other test scores by SES, GPA, and more

*WARNING*: This is just a dump of plots (for sharing purposes) until I get the time to do some actual thoughtful analysis of this topic…. :-)

 

 

 

 

On educational attainment rates and income as a causative factor

As I mentioned to Robert VerBruggen in his latest piece on educational attainment and income, I do not believe that economic concerns are a major cause of differences in education attainment rates by income.

I previously analyzed this and related issues using the ELS:2002 data, but I decided to extend my analysis with NLSY97 and clarify my views, now that I have marshaled a fair amount of data to support my arguments.

First

Although the IQ (ASVAB) is an excellent predictor and generally mediates these differences fairly well, there are other systematic differences that are not fully accounted for when you control for IQ.   High SES people, whether measured by income or educational attainment, typically have higher GPAs even with the same test scores.

(these differences would likely to be larger still if I did this as a composite SES index using education, income, occupational prestige, etc)

IQ, test scores, GPA, income, and related correlations from NLSY97

Diving into the NLSY97 dataset a bit right now and I thought I’d share some plots pertaining to IQ/achievement tests, income, GPA, educational attainment, and more.   Nothing here is particularly novel, if you’ve looked for this sort of data before, but sometimes it’s nice to have additional independent analysis or alternative presentations of the data.

---

High school GPA-test score relationships

HS GPA by ASVAB in percentiles (IQ test)

HS GPA by PIAT (Peabody Individual Achievement Test, IQ test admin. in early childhood)

Understanding socioeconomic mobility

Although it may not seem like it at first blush, given the apparently modest correlations, the socioeconomic figures that I blogged about earlier largely agree with the published data on economic mobility.

They are measuring income (or earnings) whereas I am measuring with the ELS SES index (which is an equally weighted average of the respondents own earnings as of 2011, educational attainment, and occupational prestige), but the systematic income differences between classes (however measured) are almost certainly virtually fully mediated by this more comprehensive SES measure.

ELS data, students 2011 SES by 2002 SES of parents

The correlation between parent SES and child SES is 0.35.  This may not sound like much, but if you bin child SES and parent SES into quintiles the mobility estimates are very similar to the widely publicized economic mobility estimates.

The average person born at the top of the SES distribution has little chance of ending up at the bottom and vice versa, but there is clearly a great deal of mobility that happens amongst less extreme groupings.

[Note: I didn’t correct these figures for oversampling, so I won’t claim they’re a perfect representation of reality, but they are generally pretty close in practice and they still are good for illustrative purposes.]

Mobility delta (child SES – parent SES) by parent SES

On average, as compared to their parents, high SES people are downward mobile and low SES people are upward mobile.  This may seem counter-intuitive to some, but this makes sense because r < 1, i.e., there is non-trivial mobility, and this must be true for there to be meaningful relative mobility.  Of course, just because there is mobility doesn’t imply that all people have an equal chance of ending up in any place in the SES distribution.

As I mentioned in my last post, there are other differences between groups besides just propensity to end up in particular SES bins and many of these differences are highly predictive of mobility — see test scores, HS GPA, etc.  Indeed, they almost fully mediate outcomes, so talking in terms of “mobility” here can be a bit misleading because relative starting position (parent SES) tells you relatively little about what is likely once you have better information (e.g., test scores, HS GPA, etc).