Measuring Volatility, Blog 1 of 2: Statistical Riddles, Innovative Approaches, and Strength in Diversity

By David Mitchell

How many Americans experience substantial fluctuations in their monthly income? Seems like a simple enough question. But, like so many statistical riddles, it depends on who – and how – you ask.

Take, for example, the results from the Federal Reserve’s 2015 Survey on Household Economics and Decisionmaking, affectionately referred to by policy wonks as “The SHED.” The 2015 survey re-asked a question from 2013 on income volatility, giving the more than 5,000 respondents the following three options to “best describe” how they or their partner’s income changes from month to month in the past year: (1) roughly the same amount each month; (2) roughly the same most months, but some unusually high or low months during the year; or (3) often varies quite a bit from one month to the next. Nearly a third of respondents answered (2) or (3), a substantial portion of the population. But other studies have found that an even larger number experience volatility, which begs the question – why the discrepancy?

One possibility is that survey respondents are notoriously unreliable at recalling certain details about their financial lives. And this makes sense – do you really remember if that bonus from work hit your account in December or January? Or if you ended up getting those extra hours you asked for in February? The SHED is administered annually, at the end of each calendar year, so it stands to reason that respondents might be largely guessing about what happened many months prior. Even more regular – and comprehensive – surveys suffer from the same weakness. The Survey of Income and Program Participation (SIPP), for example, is a long-standing source of monthly income data for researchers, but it too relies on respondents’ own recollection about their finances and there is evidence that panel participants, who are quizzed every four months, are unable to provide reliable, granular information for anything but the most recent month.

Data that does not rely on self-reporting – like government administrative records or actual bank statements – are much more dependable. This is one reason why researchers have been so excited by the JP Morgan Chase Institute’s (JPMCI) recent use of reams of proprietary banking data from their customers (don’t worry, it’s anonymized). JPMCI found that, between 2012 and 2015, 55% of their customers (largely, but not perfectly, representative of the country as a whole) regularly experienced more than a 30% change in income – up or down – from one month to the next. Similarly, the U.S. Financial Diaries (USFD) combined in-depth and regular interviews of low- and moderate-income families with a review of their financial records to account for every dollar of the family’s budget. The resulting analysis determined that sample households experienced an average of 5.1 months out of the year in which income was more or less than 25% above average.

USFD’s choice to use 25% as the volatility threshold – as compared to JPMCI’s 30% – illustrates that there is no right way to answer the question. And the methodological distinctions do not end there: • Some researchers, like USFD, count the number of spikes and dips an average household experiences; some, like JPMCI, count the number of families that experience regular spikes or dips of a certain magnitude; some, like those at the Urban Institute, count the number of families that experience one major drop in income over a certain amount of time; and others calculate something called the coefficient of variation, which roughly amounts to the average deviation from individuals’ average incomes across an entire population.

• Some researchers count all forms of income, while others exclude tax refunds, since their immense lumpiness can overwhelm other trends in the data.
• Some researchers ask about all members of the household, but others, like the SHED, focus just on the respondent and his or her spouse or partner. Though more labor-intensive to collect, data on who shares resources with who and how fluctuations experienced by one family members effect the others provides a fuller picture.

Agreeing on a standard way to measure income volatility would be helpful in some regards – especially for making comparisons across certain demographic groups and over time – but there is also value in the current diversity. Americans’ financial lives are often messy and chaotic. Trying to simplify that complexity into neat data points will inevitably fail. The pioneering researchers in this area understand that, and the resulting mix of quantitative and qualitative data is helping to build a case for why volatility matters in ways a one-dimensional measure never could. Next week, I’ll be back with more on the challenge of measuring volatility, with a focus on income predictability, consumption volatility, and historical trends.