Everything* you ever wanted to know about perceived income but were afraid to ask

This is a follow-up to my previous post. For context, you may wish to read this first. In that post I discussed how a plot from a Guardian piece (based on a policy paper) made the claim that German earners tend to misjudge themselves as being closer to the mean or, in the authors’ own words, “everyone thinks they’re middle class“. Now last week, I simply looked at this in the simplest way possible. What I think this plot shows is simply the effect of transforming a normal-ish data distribution into a quantile scale. For reference, here is the original figure again:

The column on the left aren’t data. They simply label the deciles, 10% brackets of the income distribution. My point previously was that if you calculate the means of the actual data for each decile you get exactly this line-squeeze plot that is shown here. Obviously this depends on the range of the scale you use. I simply transformed (normalised) the income data into a 1-10 scale where the maximum earner gets a score of 10 and everyone else is below. The point really is that in this scenario this has absolutely nothing to do with perceiving your income at all. It is simply plotting the normalised income data and you produce a plot that is highly reminiscent of the real thing.

Does the question matter?

Obviously my example wasn’t really mimicking what happens with perceived income. By design, it wasn’t supposed to. However, this seems to have led to some confusion about what my “simulation” (if you can even call it that) was showing. A blog post by Dino Carpentras argues that what matters here is how perceived income was measured. Here I want to show why I believe this isn’t the case.

First of all, Dino suggested that if people indeed reported their decile then the plot should have perfectly horizontal lines. Dino’s post already includes some very nice illustrations of that so I won’t rehash that here and instead encourage you to read that post. A similar point was made to me on Twitter by Rob McCutcheon. Now, obviously if people actually reported their true deciles then this would indeed be the case. In this case we are simply plotting the decile against the decile – no surprises there. In fact, almost the same would happen if they estimated the exact quantile they fall in and we then average that (that’s what I think Rob’s tweet is showing but I admit my R is too rusty to get into this right now).

My previous post implicitly assumed that people are not actually doing that. When you ask people to rate themselves on a 1-10 scale in terms of where their income lies, I doubt people will think about deciles. But keep in mind that the actual survey asked the participants to rate exactly that. Yet even in this case, I doubt that people are naturally inclined to think about themselves in terms of quantiles. Humans are terrible at judging distributions and probability and this is no exception. However, this is an empirical question – there may well be a lot of research on this already that I’m unaware of and I’d be curious to know about it.

But I maintain that my previous point still stands. To illustrate, I first show what the data would look like in these different scenarios if people could indeed judge their income perfectly on either scale. The plot below is showing what I used in my example previously. This is a distribution of (simulated) actual incomes. The x-axis shows the income in fictitious dollars. All my previous simulation did was to normalise so the numbers/ticks on the x-axis are changed to be between 1-10 but all the relationships remain the same.

But now let us assume that people can judge their income quantile. This comes with a big assumption that all survey respondents even know what that means, which I’d doubt strongly. But let’s take that granted that any individual is able to report accurately what percentage of the population earns less than them. Below I plot that on the y-axis against the actual income on the x-axis. It gives you the characteristic sigmoid shape – it’s a function most psychophysicists will be very familiar with: the cumulative Gaussian.

If we averaged the y-values for each x-decile and plotted this the way the original graph did, we would get close to horizontal lines. That’s the example I believe Rob showed in his tweet above. However, Dino’s post goes further and assumes people can actually report their deciles (that is, answer the question the survey asked perfectly). That is effectively rounding the quantile reports into 10% brackets. Here is the plot of that. It still follows the vague sigmoid shape but becomes sharply edged.

If you now plotted the line squeeze diagram used in the original graph, you would get perfectly horizontal lines. As I said, I won’t replot this; there really is no need for it. But obviously this is not a realistic scenario. We are talking about self-ratings here. In my last post I already elaborated on a few psychological factors why self-rating measures will be noisy. This is by no means exhaustive. There will be error on any measure, starting from simple mistakes in self-report or whatever. While we should always seek to reduce the noise in our measurements, noisy measurements are at the heart of science.

So let’s simulate that. Sources of error will affect the “perceived income” construct at several levels. The simplest we can do to simulate it is an error on how much the individual thinks their actual income is – we take each person’s income and add a Gaussian error. I used a Gaussian with SD=$30,000. That may be excessive but we don’t really know that. There is likely error in how high people think their income is relative to their peers and general spending power. Even more likely, there must be error on how they rate themselves on the 1-10 decile scale. I suspect that when transformed back into actual income this will be disproportionally larger than the error on judging their own income in dollars. It doesn’t really matter in principle.

Each point here is a simulated person’s self-reported income quantile plotted against their actual income. As you can see while the data still follow the vague sigmoid shape, there is a lot of scatter in people’s “reported” quantiles compared to what it actually should be. For clarity, I added a colour code here which denotes the actual income decile each person belongs to. The darkest blue are the 10% lowest earners and the yellow bracket is the top earners.

Next I round people’s income to simulate their self-reported deciles. The point of this is to effectively transform the self-reports into the discrete 1-10 scale that we believe the actual survey respondents used (I still don’t know the methods and if people were allowed to score themselves a 5.5 for instance – but based on my reading of the paper the scale was discrete). I replot these self-reported deciles using the same format:

Obviously, the y-axis will now again cluster in these 10 discrete levels. But as you can see from the colour code, the “self-reported” decile is a poor reflection of the actual income bracket. While a relative majority (or plurality) of respondents scoring themselves 1 are indeed in the lowest decile, in this particular example some of them are actual top earners. The same applies to the other brackets. Respondents thinking of themselves as perfectly middle class in decile 5 actually come more or less equally from across the spectrum. Now, again this may be a bit excessive but bear with me for just a while longer…

What happens when we replot this with our now infamous line plots? VoilĂ , doesn’t this look hauntingly familiar?

The reason for this is that perceived income is a psychological measure. Or even just a real world measure. It is noisy. The take-home message here is: It does not matter what question you ask the participants. People aren’t computers. The corollary of that is that when data are noisy the line plot must necessarily produce this squeezing effect the original study reported.

Now you may rightly say, Sam, this noise simulation is excessive. That may well be. I’ll be the first to admit that there are probably not many billionaires who will rate themselves as belonging to the lowest decile. However, I suspect that people indeed have quite a few delusions about their actual income. This may be more likely to affect the people in the actual middle range perhaps. So I don’t think the example here is as extreme as it may appear at first glance. There are also many further complications, such as that these measures are probably heteroscedastic. The error by which individuals misjudge their actual income level in dollars is almost certainly greater for high earners. My example here is very simplistic in assuming the same amount of error across the whole population. This heteroscedasticity is likely to introduce further distortions – such as the stronger “underestimation” by top earners compared to the “overestimation” by low earners, i.e. what the original graph purports to show.

In any case, the amount of error you choose for the simulation doesn’t affect the qualitative pattern. If people are more accurate at judging their income decile, the amount of “squeezing” we see in these line plots will be less extreme. But it must be there. So any of these plots will necessarily contain a degree of this artifact and thus make it very difficult to ascertain if this misestimation claimed by the policy paper and the corresponding Guardian piece actually exists.

Finally, I want to reiterate this because it is important: What this shows is that people are bad at judging their income. There is error on this judgement, but crucially this is Gaussian (or semi-Gaussian) error. It is symmetric. Top earner Jeff may underestimate his own income because he has no real concept of how the other half** live. In contrast, billionaire Donny may overestimate his own wealth because of his fragile ego and he forgot how much money he wastes on fake tanning oil. The point is, every individual*** in our simulated population is equally likely to over- or under-estimate their income – however, even with such symmetric noise the final outcome of this binned line plot is that the bin averages trend towards the population mean.

*) Well, perhaps almost everything?

**) Or to be precise, how the other 99.999% live.

***) Actually because my simulation prevents negative incomes for the very lowest earners, the error must skew their perceived income upwards.

Matlab code for this simulation is available here.