HOW IMPORTANT IS THE STOCK MARKET REALLY?

A correlational analysis of the stock market, the economy, and our happiness.

13 min readJan 8, 2021

Why do we care so much about the stock market? If you listen to Marketplace, you likely have listened to its host, Kai Ryssdal, explain that the stock market is not the economy. However, we give so much focus to the market — exemplified by the headlines above and the way politicians brag about strong markets during their tenures, that you might as well think the stock market is a vital part of the economy. With this question in my mind, I decided to look for data about the stock market and other economic indicators and find if there is any correlation between them.

I must say: I am not an economist and because of that I am 99.99% sure I am butchering some very technical ways to conduct this analysis. With that said, I am approaching this as a way to use very simple statistical analysis and visualizations to learn if there is any correlation between the returns of the market and how the economy behaves (which is what actually impacts the lives of most Americans) and answer a question that I have had in my mind for quite a while now.

Another thing: if you have taken an Econ101 class, you know that correlation does not imply causation. Any correlation we find must be taken with a grain of salt. Yet, we can learn some useful things from such correlations and have grounds for further exploration.

Our indicators

A brief intro to the data we will be looking at.

For the market, we will be looking at the S&P 500, which is the most followed stock index in the U.S.
For the economy, we will look at new orders of durable goods, unemployment rate and labor participation, and GDP.
We will lastly look on measures of happiness, given by consumer confidence, earning increases, and inequality.

With the indicators out of the way, let’s start looking at the data!

Analysis

PS: The original code that underlie this post is lengthy. Therefore, I will only show snippets of code that add some color and highlight something important about the process. You can access the full R markdown on my Github or access it on my website.

Most of what we will analyze can be accessed through quantmod. Whenever we need to use other sources, you will see a link to the original data file. Below, we go through common procedures which we will need to perform time and again for different indicators and our S&P data:

Import the necessary libraries and data: we start by downloading the data for the S&P 500 and our indicator. Below, that is done for Durable Goods (DGORDER).
Calculate quarterly and monthly returns to match the economic indicators. DGORDER is given in billions, therefore we calculate the growth in orders over the periods. If an indicator is given as a percentage, this step is not necessary.
Match the time for the observations: the first durable goods observation is from February 1992, so we match the S&P observations to that. The same will be done for other indicators where dates do not match.
Combine row names: at times, the observations are off from each other by a day. We need to then match row names so it is easier to plot a time-series or do correlational analysis.
Calculate correlations and graph: this is the part that will give us insights about the relationship between the stock market, the economic indicators, and our happiness.

Durable Goods Manufacturing

Durable goods are any goods that do not wear out quickly such as cars, appliances, electronics, etc. Durable goods are important because they are a gauge into how the economy is doing: you wouldn’t buy a car if you think the economic outlook is dire and the car manufacturer wouldn’t produce more cars.

library(quantmod)                       
library(ggthemes)                       
library(dplyr)                       
library(tidyr)                       
library(ggplot2)                       
library(readr)sp<-new.env()                      getSymbols('^GSPC',env=sp,from='1975-06-30',src = "yahoo")                                               GSPC<-sp$GSPC
GSPC.quarter<-periodReturn(GSPC,period = 'quarterly',from='1975-07-01',leading=TRUE,type = "log")getSymbols('DGORDER',src="FRED")                       new.orders<-DGORDER                       orders.month<-monthlyReturn(new.orders)                       orders.month<-orders.month[2:nrow(orders.month),]sp_4<-monthlyReturn(GSPC)                       
sp_4<-sp_4[201:(nrow(sp_4)-3),]                       sp_4<-as.matrix(sp_4)                       orders.month<-as.matrix(orders.month)                       rownames(sp_4)<-rownames(orders.month)                       sp_orders<-as.data.frame(cbind(sp_4,orders.month))                       colnames(sp_orders)<-c("sp500.returns","orders.growth")cor(sp_orders$sp500.returns,sp_orders$orders.growth)

The correlation between durable goods growth and the S&P 500 returns is 19.5%. This is not a strong correlation, but it is not insignificant either. This correlation might be due to people gauging economic growth from stock market growth. Therefore, whenever the market is up, people will buy and manufacturers will increase production (or the reverse).

However, is the stock market affecting durable goods orders or is the reverse happening? Strong manufacturing numbers could create more market confidence and therefore lead to higher returns and weak numbers could make the stock market drop.

To test this hypothesis, we can correlate manufacturing numbers at month t with S&P returns at at month t+1 and vice versa. For instance, manufacturing numbers which are released at the beginning of January are correlated with the returns for full month of January.

As suggested below, there is more evidence of the S&P leading to stronger manufacturing goods than the opposite. However, the correlation between S&P returns at month t and Durable goods at t+1 is reduced, which might suggest that both indicators are actually moving in tandem.

Correlation decreases on the second and third graph. S&P at time t and Orders at t+1 correlate at 9% and Orders at t and S&P at t+1 at 5%.

Unemployment and Labor Participation

The more unemployed people in a country, the worse we can expect its economic outcome to be. This indicator is accessed through the FRED (Federal Reserve Economic Data) through the code LRUN64TTUSQ156S:

getSymbols('LRUN64TTUSQ156S',src='FRED')
unrate<-LRUN64TTUSQ156S

PS: It is not possible to specify beginning dates when downloading data from the FRED through quantmod — meaning all available data is downloaded. So, we need to do the same sorting dates procedure highlighted at the Analysis section above.

Unemployment is a quarterly indicator and its correlation with the stock market is -3%. Though weak, this points to the general assumption that strong stock market returns lead to lower unemployment. Besides the positive sentiment of a strong market, this correlation is likely related to firms having more access to credit (or issuing stock in times of market growth) to fund new projects which will lead to more hiring in upturns (and the reverse in downturns).

The idea of looking at the labor participation rate comes from the fact that a strong economy might lead people to re-enter the job market and look for jobs. Even if we did not see any correlation of S&P returns and unemployment, we could see some correlation in the participation rate. This rate is also accessed through the FRED through the symbol CIVPART. The correlation here is almost zero (0.02%) as illustrated by the graph below.

Gross Domestic Product

Gross Domestic Product (GDP) is a measure of value of finished goods and services in an economy and it is the go-to measure of economic growth and success.

Some economists do disagree about focusing solely on GDP for measuring the economic success of a country. If you want to learn more about this measure and learn about alternatives, check out GDP: A Brief But Affectionate History by Diane Coyle.

Real GDP (in billions) can also be accessed by the FRED through symbol GDPC1. Below, let’s calculate GDP growth by quarter, match the dates which our S&P 500 observations, and calculate correlations:

getSymbols('GDPC1',from='1975-01-01',src='FRED')
GDP.quarter<-quarterlyReturn(GDPC1)
GDP.quarter<-GDP.quarter[115:nrow(GDP.quarter),]gdp<-as.matrix(GDP.quarter)
rownames(sp)<-rownames(gdp)
sp_gdp<-as.data.frame(cbind(sp,gdp))
colnames(sp_gdp)<-c("sp500.returns","gdp.growth")cor(sp,gdp)

The correlation here is the strongest yet at 32%. Now, we might want to think about the reverse correlation: what if GDP numbers are creating surges in the S&P500? (Investors get excited about strong numbers and go bullish on stocks). Let’s test this in a similar fashion to what we did for Durable Goods Manufacturing: matching GDP observations at quarter t to S&P observations at quarter t+1.

spx<-sp[2:nrow(sp),]
gdpx<-gdp[1:(nrow(gdp)-1)]
cor(spx,gdpx)

This yields a 2% correlation, a large drop from what we first observed (illustrated in the graphs below). Two possibilities on why this happens:

A quarter is a long time for the stock market. Though GDP might create excitement in investors and lead to gains, other events might come by and erase those gains.
By the time the GDP measures are released, other indicators have already given investors an idea about how the economy did for the quarter. That means that, by the time of the release, GDP growth has already been priced by the stock market.

Measures of Happiness

As mentioned above, GDP might not be the best measure of how an economy is doing. Plus, traditional economic measures might not actually capture how people are doing in a country. With that in mind, let’s look into how the stock market correlates with measures of happiness in an economy.

Consumer Confidence and Wages

Consumer Confidence is an indicator that measures optimism in a country. If people feel more optimist, we might presume they will spend more and be happier (a very materialistic assessment, I know). We will measure confidence through the University of Michigan Consumer Sentiment Survey (UMSENT).

The distribution looks sparse, but we do observe a 11% correlation between the two measures. That indicates customers do feel more confident when the stock market soars and less so when it drops.

Wage growth is another important measure that has been widely debated in the U.S. since the Great Recession. If people are not making enough money, they will be financially stressed and unhappier.

Below, let’s download the data for median wages (in dollars) and calculate their growth. Then, let’s match the observations for the S&P 500 with the observations for wage growth and calculate correlations.

getSymbols('LES1252881600Q',src='FRED')earn<-quarterlyReturn(LES1252881600Q)
sp_8<-GSPC.quarter[15:(nrow(GSPC.quarter)-1),]
earn<-as.matrix(earn)
sp_8<-as.matrix(sp_8)
rownames(sp_8)<-rownames(earn)
sp_earn<-as.data.frame(cbind(sp_8,earn))
sp_earn$time<-rownames(sp_earn)
colnames(sp_earn)<-c("sp500.returns","median.week.growth","time")cor(sp_earn$sp500.returns,sp_earn$median.week.earning)

Here we find that the correlation is -17%, meaning higher stock market returns are associated with decreases in wages. This might be a reflection of the system: as companies cut costs (one of which is wages), they are rewarded by investors with higher stock prices since there are expectations for higher profits. This is further illustrated in the graph to the right in which you see S&P returns frequently moving in reverse to wage growth.

The Gini Index, Inequality, and the Stock Market

The idea to look at inequality as a measure of happiness originated from the World Happiness Report. One of the measures of happiness used in the report is the Gini Index, a measure of inequality across the world. An index of 0 means all the wealth is equally distributed while a 1 means one individual holds all the wealth of the economy.

If you are interested in such measures of inequality, read also about the Lorenz Curve, which is used to calculate the Gini coefficient.

In addition to the Gini Index, we will get data from the Census Bureau about mean income across 5 income quintiles to measure how incomes have changed for the past decades.

To prepare our data, we will:

Download the data for the Gini index, 5 quintiles, and S&P yearly prices into dataframes from .csv files.
Transposed the Gini data using function t(). By transposing, we can easily match the dates of the observations and bind the dataframes into a single one.
Convert data types from character into numeric so we can do our calculations.
Correlate our Gini numbers and the five quintiles to the S&P 500 prices, using scale() to standardize all columns.

gini<-read.csv("gini.csv")
inequality<-read.csv("inequality.csv")
sp.prices<-read.csv("spprices.csv")
colnames(sp.prices)<-c("Year","Price")
gini<-as.data.frame(t(gini))
gini<-gini[2:nrow(gini),]
gini<-as.numeric(gini)inequality<-as.data.frame(t(inequality))
colnames(inequality)<-inequality[1,]
inequality<-inequality[2:nrow(inequality),]inequality_gini<-as.data.frame(cbind(inequality,gini))
rownames(inequality_gini)<-substr(rownames(inequality_gini),2,5)for(i in 1:ncol(inequality_gini)){
  inequality_gini[,i]<-as.numeric(inequality_gini[,i])
}inequality_gini2<-inequality_gini[1:(nrow(inequality_gini)-8),]
inequality_gini3<-as.data.frame(cbind(inequality_gini2,sp.prices))
inequality_gini3<-inequality_gini3[,-7]cor(scale(inequality_gini3$gini),scale(inequality_gini3$Price))
cor(scale(inequality_gini3$`Highest quintile`),scale(inequality_gini3$Price))
cor(scale(inequality_gini3$`Middle quintile`),scale(inequality_gini3$Price))
cor(scale(inequality_gini3$`Lowest quintile`),scale(inequality_gini3$Price))

The correlations follow:

There is a high correlation between the S&P 500 returns and the Gini index (also shown in the left graph below) and income across the major quintiles. The higher the income quintile, though, the strongest the correlation to the S&P 500.

To further analyze this income discrepancy, let’s simulate the growth of $10,000 dollar for each of the income brackets. To do that we will:

Change row names into numeric and arrange the observations by row numbers so we can graph the time series.
Create a vector with first value at 10 to represent the $10k we want to simulate and a NULL variable which will be used to bind the calculations for all 5 quintiles.
Create a for loop that calculates the change in income for each quintile. The loop will iterate through all rows calculating the change and multiplying that change by our previous income.
Bind the vector values for each quintile, reset the vector at 10 and move to the next quintile column.

rownames<-rownames(inequality_gini3)
rownames<-as.numeric(rownames)
rownames(inequality_gini3)<-rownames
inequality_gini4<-inequality_gini3%>%arrange(-row_number())quintile.growth<-NULL
growth.vector<-c(10)for (i in 1:(ncol(inequality_gini4)-1)){
  
  for (j in 2:nrow(inequality_gini4)){
    
    x<-growth.vector[(j-1)] * (inequality_gini4[j,i]/inequality_gini4[(j-1),i])
    growth.vector<-append(growth.vector,x)
  }
  
  quintile.growth<-cbind(quintile.growth,growth.vector)
  
  growth.vector<-c(10)
}quintile.growth<-as.data.frame(quintile.growth[,-6])
colnames(quintile.growth)<- c("low.quintile" , "second.quint" , "mid.quint" , "fourth.quint" , "high.quint")rownames(quintile.growth)<-rownames(inequality_gini4)
quintile.growth$year<-as.numeric(rownames(quintile.growth))

On the graph to the right, you see how the income for the top 20% of individuals grew much more than the other 80%. This is further illustration of the same information the Gini index is providing: there has been an increase in inequality in the U.S.. How is that related to the stock market?

To understand that, we need to analyze the changes in wealth tied up in the stock market. To capture and analyze that information, we will rely on the DFA: Distributional Financial Accounts from the Federal Reserve. The DFA show changes in wealth for the Top 1%, Next 9%, Next 40%, and Bottom 50% of households.

To match the yearly information we had on the S&P prices, I filtered the information on Excel to leave only observations on the first quarter of each year. Moreover, I also calculated the percentage of wealth in corporate equities and funds, the growth of the wealth tied up in corporate equities and funds, and the growth of $10,000 dollars in corporate equities and funds under each category.

In the code below, we will do some additional work to set up the data:

First, we clean up the data from the Excel file mentioned above and filter the S&P yearly prices data for observations at or after 1990.
We then calculate the returns for each year and then simulate the growth of a hypothetical $10,000 over the period.
We create two graphs: one for the changes in the percentage of wealth tied up in equities and funds, and another for the growth of $10,000 under the wealth brackets and the stock market.

wealth.growth<-read.csv("dfa-networth-levels-yearly.csv")
wealth.growth$Income<-wealth.growth$Income/1000
wealth.growth$Date<-as.numeric(substr(wealth.growth$Date,1,4))wealth.growth$Equity...of.wealth <- 
(wealth.growth$Equity...of.wealth) * 100
wealth.growth<-wealth.growth[,c(1,2,7,15,16,17)]
colnames(wealth.growth)<- c("Year" , "Category" , "Equities.wealth" , "share.of.wealth.equities","growth.of.equities","income")sp.prices[,1]<-as.numeric(sp.prices[,1])
sp.prices.90<-sp.prices%>%filter(sp.prices$Year>=1990)%>%arrange(Year)growth.90<-c(1)
for(i in 2:nrow(sp.prices.90)){
  x.90<-sp.prices.90[i,2]/(sp.prices.90[(i-1),2])
  growth.90<-append(growth.90,x.90)
}sp.prices.90<-cbind(sp.prices.90,growth.90)
sp.income<-c(10)for(i in 2:nrow(sp.prices.90)){
  y.90<-sp.income[(i-1)]*sp.prices.90[i,3]
  sp.income<-append(sp.income,y.90)
}
sp.prices.90<-cbind(sp.prices.90,sp.income)wealth.growth%>%filter(Category=="Bottom50")
graph17<-wealth.growth%>%ggplot(aes(x=Year,y=share.of.wealth.equities,color=factor(Category)))+ geom_line()+labs(title="Share of Wealth in Equities and Mutual Funds",x="Year", y="Share (%)",color=" ") + theme_classic() + guides(color=guide_legend(reverse=TRUE))ggplot(data=wealth.growth,aes(x=Year,y=income))+geom_line(aes(color=factor(Category)))+ theme_classic() + geom_step(data = sp.prices.90,aes(x=Year,y=sp.income),size=1)+
  labs(title="Growth of $10,000 under each bracket and Stock Market",x="Year", y="Income (in thousands)",color=" ") + guides(color=guide_legend(reverse=TRUE))

The graphs below are the result of the analysis:

On the graph to the right, the sharp line represents the growth of $10,000 tied up in the S&P 500 over the period.

The wealthier 10% of individuals have larger shares of their wealth tied up in the stock market than the bottom 90%.
The wealth tied up in the market for the top 1% and next 9% of individuals has oscillated but constantly increased over the years. There hasn’t been significant changes for the wealth in stocks and funds for the bottom 90% of individuals.
In downturns, everybody loses. In upturns, the wealth gap widens with the top 10% reaping most benefits. Since the Great Recession, the wealth gap has increased like never before.
The equity and funds wealth of the bottom 50% has moved in similar fashion to the S&P 500 until around 4 years ago when it went down in comparison to the market.

All of these findings lead to the curious pattern we see in the graph on the right: the wealth of the top 10% tied up in equities and funds grew at a faster rate than the S&P 500. We can speculate why that is:

Financial literacy tends to be higher among high-income individuals. Thus, the wealthier you are, the likelier you are to get help managing your assets, leading to better financial outcomes.
Wealthier individuals have access to leverage, derivatives, and investment strategies that enhance their gains in bull markets (and granted, lead to greater drops during downturns). These assets are mostly out of reach for the average person.

What can we conclude?

We have found that the stock market definitely correlates with many of the indicators we analyzed. From that, we can conclude that the stock market is definitely not irrelevant and we should pay attention to it.

What story do the correlations tell us?

At times, they tell a positive story. Markets moving in tandem with GDP and consumer sentiment suggests that the average person pay attention to the market to take clues about the economy, planning their spending in the future. This highlights an important role for the market: an easy to follow gauge for how the economy is doing. Moreover, the stock market growth might enable companies to expand, invest in new ventures and hire more workers, reducing the unemployment rate and leading to further growth.
Of course, there is a negative story, too. The reverse correlation between wage growth and the market highlights an structural problem to which the stock market is a contributor: pursuing the highest possible corporate profits will be always in conflict with the well-being of workers. For the last few years a movement emerged to fight back against this idea of shareholder capitalism. This leads to our findings about inequality as well: as the stock market grows, the wealthy benefit while the majority stays behind.
Lastly, the wealth of households has moved in tandem with the stock market. For this reason, it is important to notice