For many months, my pro bono work for non-profits has limited the time I've been able to invest in this site. To repair neglected and outdated material, I will be making several changes in November and December. Some rarely used pages will be removed. Pages visited more often will be updated, if needed. Please add a comment to this post, if you have feedback or suggestions.

Today, Mark Hulbert published a short article titled The Real Reason Why Value and Small-Cap Stocks Lag the Market.  It's a good article, as far as it goes, but overlooks some important points.

The gist of the article is this:

• In the past decade, the total return of small-cap stocks has lagged that of large-cap stocks, contrary to academic research that demonstrates the long-term superior performance of small-caps. The same is true of value stocks when compared to growth stocks over the past ten years.
• Historically, however, the long-term out-performance of small-cap and value stocks has not occurred in every time period of 1 year or 5 years or even a full decade. In fact, under-performance occurs more often than intuition might suggest.
• Therefore, investors should not abandon small-cap and value stocks merely because of their recent under-performance in the past 1, 5, and 10 year periods.

To these cogent points, I would add three more.

First, investors should not over-invest in small-cap or value stocks unless they plan to hold them for a long time, at least 10 years. For example, if you are investing in a college-savings fund for your child, it would be better (on average) to invest in broad market averages, rather than betting that the next 5 to 15 years will be one of the periods when small-cap and value stocks show their muscle.

Second, there is some evidence in academic research that the advantage of value over growth is stronger for smaller stocks, and that the value-advantage is bigger than the small-cap-advantage. Thus, in addition to planning for a very long holding period, investors should focus on the combination of small size plus good value, rather than on either factor in isolation. Because of this research, Vanguard in their recent ETF for U.S. value stocks (VFVA) has a disproportionate number of small and medium sized companies. The average market capitalization is $32B, and more than half of the holdings are mid-, small-, and micro-cap firms.  Other ETFs often do the opposite.  For example, the holdings in iShares VLUE have an average market cap of $215B, and 88% of them are large-cap companies.

Third, it not always obvious how to invest in "small" and "value" stocks as defined in the research. Funds and benchmarks often digress from the academic criteria. The S&P 600 Small-Cap benchmark, for example, has a bias toward small companies that are profitable. Thus, funds using this benchmark are perhaps most accurately described as "small quality" rather than simply "small-cap." In contrast, funds benchmarked to the CRSP or Russell definitions of "small" really are exactly that.

For "value" the problem is more pervasive. Often, "value" funds are better described as "value + quality" or "current value + expect earnings-growth" or some other composite terminology. In the original academic research, "value" meant simply a low price compared to book value. In contrast, the CRSP indices used by Vanguard define "value" by book to price plus four other metrics: forward earnings, historic earnings, price-to-dividend, and price-to-sales. Similarly, the MSCI benchmark for "Enhanced Value" uses book to price plus two more considerations: forward earnings and the ratio of a company's entire economic value to the cash it produces. Whether these divergences from the academic standards matter, favorably or unfavorably, is not the point. For investors, "value" in a fund's name simply cannot be taken at face value. To know what you are getting, you have to read the fine print.

The chart at the beginning of this post nicely illustrates these points. The combination of small value (as defined by Wilshire) has outperformed the total market since 1980. However, for the first 20 years of that period, small value stocks under-performed, largely because they were swamped by the dot-com bubble that peaked in early 2000. Not until the bottom of the 2008-2009 recession did they regain the upper hand.

While I prepare able2pay.com's 2017 update on the best investment firms, I'll post parts of my analysis here, one topic at a time. Today's topic is rebalancing.

To compare firms on how they rebalance their clients' portfolios, one needs a standard for what methods are best. A comprehensive study of various methods, published on the Investing section of this site's menu, concluded that over the long term, the best results rest on:

1. Rebalancing toward one's target percentages for stocks and bonds when they drift a fixed percentage from their target, rather than doing so by the calendar. In particular, a criterion of 3% or 4% drift has worked better, historically, than rebalancing monthly, quarterly, or yearly. In short, use a percentage trigger, not a calendar date, to decide when to rebalance.
2. Limiting the frequency of rebalancing by directing dividends, interest, withdrawals, and new deposits selectively toward target percentages, rather than reinvesting dividends and interest or making deposits and withdrawals in proportion to current or fixed allocations. For example, if stocks were currently 2% below your target, not yet at the 3% to 4% threshold for rebalancing, you would direct interest, dividends, and deposits toward your stock funds, and withdrawals toward your bond funds. This is income placement.
3. Relying not on human intervention but instead on an automated process to execute the rebalancing strategy. Doing it yourself, or relying on an adviser to do it for you, is prone to the lapses of attention and fallibility of judgment to which all of us are vulnerable. A decent computer algorithm can do it more systematically and more reliably.
4. Keeping the costs low. Rebalancing is never free; you pay with your own time or you pay fees to your investment firm. The fees may not be obvious, but they are there. Fees of 0.50% are too high, particularly for portfolios with 40% or less invested in stocks. Historical analysis of various portfolios and rebalancing methods implies that the biggest benefits of rebalancing are 1% or more, over the very long run, using the very best methods, for portfolios with 60% or more in stocks. On the other hand, weaker methods, such as quarterly rebalancing, have tended to return less than 0.50% in the long run, particularly when 40% or less is invested in stocks.

I used these four standards to evaluate the methods of rebalancing offered by the four firms in this year's analysis. In rank order, best first, here is how I rate them:

1. Betterment uses a fully automated process that does income placement and that rebalances with 3% drift as its trigger. The fee is 0.25% of your portfolio's value, deducted quarterly. For accounts over $100,000, this charge is higher than last year's 0.15%; for small, starter accounts, the current charge is lower than last year; and for very large accounts, the 0.25% charge is limited to the first $2,000,000. To be fair, Betterment would assert that the 0.25% fee covers more than rebalancing. For example, it supports online access to tools for setting financial goals, and it enables Betterment to find the best funds for a portfolio that fits your goals and time-horizon. Other firms would make similar argument for their fees. Still, Betterment's 0.25% is not the lowest available charge for rebalancing (that prize goes to one of Vanguard's methods).
2. Wealthfront also uses a fully automated process that does income placement and rebalances when drift exceeds a trigger, at a fee of 0.25%. It's a close call, but I rate them slightly below Betterment for one reason: transparency. They don't disclose the threshold for their trigger. If it were too small (below 2%) or too high (above 5%) then the benefit of rebalancing would be greatly diminished, according to the historical data.
3. Vanguard offers rebalancing in two ways. One option is to use their all-in-one funds, such as their target-date or LifeStrategy funds. These charge about 0.05%, a pittance, over the already low cost of the underlying index funds. However, the all-in-one funds essentially use continuous or monthly rebalancing, which has historically generated much weaker benefits than a 3% or 4% trigger. In a word, it's both the least expensive method and the least effective. A second option from Vanguard is to pay one of their advisers a fee of 0.30% to rebalance your funds manually, probably every quarter. Perhaps you could get them to do so with a percent-trigger, rather than quarterly; even then their costs would be higher and their methods less automated than Betterment and Wealthfront. The bottom line is that Vanguard's methods are better than do-it-yourself, but not the best available.
4. FutureAdvisor is part of Blackrock Solutions, whose parent company also owns the iShares family of ETFs. Although FutureAdvisor has products and attributes that make them attractive for some investors, rebalancing is not a strong point. They use a percentage trigger, which is good, but (like Wealthfront) they don't disclose their threshold. Worse yet, they charge 0.50%. If rebalancing were your only concern, any of the other firms reviewed here would be a better choice than FutureAdvisor.

In forthcoming posts, I'll compare the same firms on other dimensions, including tax-management, financial planning, and more.

A year ago, we published ratings of investment firms, with Betterment rated clearly the best, followed by Vanguard, Wealthfront, and Blackrock iShares, in that order, and with TIAA, Fidelity, and Schwab trailing behind. Much has changed, however, since early 2016. Betterment recently raised its fees on most accounts and introduced a personal adviser service similar to Vanguard's. Wealthfront added a good 529 college savings plan, eliminating a weak-spot that had affected their 2016 ratings. And Blackrock iShares appears to have refrained from meddling with the better aspects of Future Advisor, a firm they had acquired shortly before last year's review. The bottom line is that these four firms are more similar in products and pricing than they were a year ago. The choice between them may depend as much on your personal circumstances as on their specific offerings. A full re-analysis will be published here by mid-year.

Five years ago, iShares introduced ACWV, a fund of global stocks selected to be less volatile than average. Has the fund lived up to its forecast of providing returns equal to or better than worldwide stock markets, with less volatility?

In October 2011, iShares introduced a set of ETFs that tracked indexes developed by MSCI, the financial research and investment advisory firm. The most diversified of the new funds was iShares ACWV, which invested in low-volatility stocks across the globe. By design, it sought stocks that reasonably represented stock markets around the world, large and small; that approximated a standard mix of sectors such as energy, health, financials, industrials, technology, utilities, and others; and that would fluctuate in value less than average.

Theoretically, low volatility could be neutral. One can imagine that less rambunctious stocks might tend to rise or fall at the same diminished rate, gaining 5% when the broad market gains 10% and similarly losing 5% when the average loss is 10%. In reality, historical results show a favorable bias. When most stocks rise 10%, less volatile ones may gain, say, 7%; but when a 10% loss hits the broad market, less volatile stock may lose only 5%. Over a long period of time, then, less volatile stocks have tended to out-perform. They lag a bit in up-markets, but suffer far less in down-markets.

In MSCI's research, covering the period from 1988 to 2016, their global low-volatility index had a double advantage over an index of all stocks wordwide. It had better annual returns for the 28-year period, 8.75% vs. 7.26%, while also having a lower standard deviation (a measure of annual volatility), 11% vs 15%. Better returns and less risk? It seems too good to be true. Or to last forever. One can't help wondering if, as the investment world wises up, the benefits of low-volatility stocks may diminish.

Indeed, academic researchers have yet to agree on an explanation for the low-volatility effect, nor on a standard way to measure it. MSCI's method is one of several, in an area that remains new enough to capture a minor fraction of investors' dollars. It's the method I've chosen for this post because of the unique position of ACWV as having a five-year history, low costs, and global reach. (In subsequent posts, I'll review other good options.)

The table below shows results for the period from November 1, 2011, to November 30, 2016, using monthly data for two funds: iShares ACWV for global low-volatility and Vanguard VT for total global stocks. Both are invested roughly half in the U.S. and half in other countries. On all counts, ACWV did better. It had superior annual returns; scored better (lower) on standard deviation and maximum drawdown, which are measures of volatility; and thus did better, as well, on the Sharpe ratio, which adjusts returns for the amount of risk taken.

Visually, these results can be seen in two charts. The first compares weekly data from October 18, 2011, to November 28, 2016, for ACWV and a balanced portfolio. The balance was 80% invested in VT and 20% in long-term Treasury bonds (Vanguard VGLT). The rationale for the 80-20 portfolio is that mixing stocks with bonds is a traditional way to reduce volatility. Furthermore, long-term Treasuries tend to be negatively correlated with stocks; when one zigs, the other zags. One might expect the mixture to resemble low-volatility stocks, generating good returns at reduced risk. An 80-20 ratio was chosen to match ACWV on volatility; it was rebalanced when the stock portion drifted above 83% or below 77%, as recommended in our article on rebalancing. 

For the first three years in the chart, ACWV and the mixed portfolio were indistinguishable. They grew the same amount, at the same steady pace. In 2015 and 2016, however, ACWV took the lead. Over the entire period, a bit longer than five years, ACWV's original $10,000 investment grew to $15,609 while the mixed portfolio grew to $14,357. The next chart shows how these two investment strategies compared on volatility, over the same period.

The chart measures volatility in a way that may be more intuitive than, say, the standard deviation. Each marker in the chart represents an occasion when, over a four-week period, a portfolio lost value. In effect, it shows the frequency with which you might have felt some anxiety, if you inspected your portfolio monthly. The portfolio invested 100% in ACWV had 18 such occasions, or about one every four months. The mixed 80-20 portfolio had 19 of them. Both had mostly small losses of 2% or less, and both kept the maximum monthly loss under 6%.

A key takeaway from this analysis is that investing in low-volatility stocks reduces churn by about the same amount as crafting an 80-20 mix of stocks and bonds. However, during periods when stocks are rising or flat, as occurred respectively from 2012 through 2014, and from 2015 to 2016, low-volatility stocks may generate equal or better returns than a classical mix of stocks and bonds.

Importantly, this five-year analysis failed to include a time when stocks fell extremely. If stocks were to drop as sharply as they did in 2008-2009, perhaps a mixed portfolio would fare better than low-volatility stocks, because of the negative correlation with which Treasuries tend to rise as stocks panic. Still, all indications are that both a fund like ACWV and a mixed portfolio would fall in value less sharply than the broad stock market.

In subsequent posts, I'll refine this analysis by considering some addtional questions:

• What about currency risk? In ACWV, roughly half the stocks are purchased in currencies other than the U.S. dollar. Some alternatives, notably a fund from Vanguard, hedge against changes in currency exchange rates, thus potentially reducing another source of volatility.
• What if you prefer less international exposure? In that case, better options may be to craft a simple portfolio of a few funds, some of which specialize in low-volatility stocks in the U.S. or in a particular international sector.
• Finally, what about other factors? Low-volatility is but one of several methods that research has shown to be effective for building a robust portfolio of investments. Is it prudent to combine low-volatility funds with other factors, such as value, size or momentum?

A bit later than expected, I plan to start posting my new findings on value and volatility in December and January. First up, around December 1, will be an analysis of low volatility stock-funds that have low costs, plus a track record of three to five years. Have they lived up to their forecast of returns equal to or better than an ordinary index, at lower risk?

In December, able2pay.com will have a fresh analysis of value and volatility in the stock and bond markets. This post is a preview of the topics to be covered.

Always timely, questions like these seem particularly germane now. Stocks are over-valued by many of the traditional metrics; interest rates on bonds imply dismal future returns; and cash has languished below inflation rates for a painfully long period. If no investment has obvious value, does something have to give, and if it does, is there any prudent shelter from the ensuing volatility?

Accurately measuring whether stocks are over- or under-valued is hard enough. Yet even if that problem were solved, others would remain. An indicator that predicts a big change in stock prices over the near term is likely to have a wild margin of error. Longer term, the error-rate may decline, leading to a safer, more confident prediction, but the price-change will also fade to a smaller size. Is there any way to have it all: calculate a credible benefit, capture it safely, and bank it soon?

Earlier posts on this topic proposed how best to measure whether stocks are over- or under-valued. To recap, three ideas are key.

• According to solid academic studies, Shiller's ten-year CAPE is a good starting point. Andrew Ang's Asset Management (chapter 8, section 5.2 ) explains CAPE's uniquely reliable, if somewhat modest, power as a predictor
• The standard measurement of CAPE can be improved by computing it over a longer period than ten years with a method that gradually reduces the weight of older data, and by applying logarithms to ensure that high and low readings are judged evenly.
• Some credence should be given to the possibility that CAPE has its own long-term trend. Predictions can be improve by filtering out this trend, at least partially.

"Corrected CAPE" is the term used here to designate a metric that applies all these ideas. To learn more, follow the links above.

As of early 2016, the corrected CAPE is rather high, approximately +0.4. A corrected CAPE this much above a neutral value of zero implies over-valued stocks, hence a risk that future returns from stocks will be less than normal. Given this reading, how much weakness in stock prices might one expect? Is a swift decline likely, or an extended period of lackluster gains, or what? The answer, it turns out, depends on whether the goal is to predict changes for next year, the next decade, or even farther into the future.

In the chart above, the CAPE Effect is a multiplier that tells how much to adjust the future returns of U.S. stocks, given a value of the corrected CAPE. The blue dots are actual data points from a comprehensive study covering the period from 1896 through 2015. The red line is a statistically fitted model used in our calculators.*

Given corrected CAPE's early 2016 value of +0.4, a one-year forecast is an effect of roughly 0.4 * 0.14 = 0.056 on a log scale. The spreadsheet function EXP(0.056) converts back to a percentage, 5.8%, which is the predicted reduction in real (inflation-adjusted) stock-returns over the next year, because of the current over-valuation. For the 1896-2015 period of the study, compound real returns were about 6.6% per year, so the year-ahead prediction is a return of 6.6% - 5.8%, or a meager 0.8% better than inflation, including reinvested dividends. With economists currently predicting an inflation rate of about 2%, the nominal return would be 2.8%. That's less than the historical return of ten-year Treasury bonds.

​If U.S. stocks were held longer, for the next 20 years, the chart predicts the effect would still be negative but less drastic, roughly half as large. The compound, inflation-adjusted return would be reduced about 2.6%. Instead of enjoying average real returns of 6.6% annually for two decades, one's portfolio of U.S. stocks would grow 6.5% - 2.6% = 4.0% per year, above inflation. That's not shabby. It's better than bonds, but below average for the long-term performance of stocks.

​What the chart says, in short, is that buying stocks today is like paying a premium price for a house when real-estate is hot. Even after holding the investment for many years, one's annualized profits may be tepid.

There's a problem, however, in making decisions like these. The next chart shows why. It presents the same data points (the blue dots), bracketed by their margins of error.** Statistically, most of the historical data falls between the upper and lower margins of error (the orange lines). A big gap between the lines implies great uncertainty; a narrow gap means the prediction is more trustworthy.

For stock investments held one year, the margin of error is extreme. While the average one-year effect might be alluring, actual one-year results have often been very different, in both directions. Evidently, a short-term forecast is both big and wild. It's like predicting tomorrow's weather to be a 90% chance of rain, maybe a steady downpour, maybe a brief drizzle.

At the other extreme, for a holding period of 30 to 40 years, the prediction is much more precise, thanks to a tiny margin of error. It's a safe forecast, but the effect has become small. Now the weather forecast is for a certainty of rain this season, totaling 5 to 6 inches, but there's no telling which days will be wet.

The ideal case would be to find the points on the chart where the range of uncertainty excludes a zero effect, which appears to happen after about three or four years, yet the effect-size remains large enough to matter, which seems possible up to the mid-twenties.

Our calculators quantify these ideas by giving a weight to the corrected CAPE.*** As the margin of error decreases, the weight goes up. At the same time, as the effect-size declines, the weight goes down. The statistical model finds an optimal way to calculate the weights, and, it so happens, the weight steadily increases from one to 24 years, then very gradually declines.  

For a concrete example, start with the chart below. It shows how our calculators simulate the portfolio of a hypothetical investor who matches this description:

• She has moderate preferences. Her goal is a compromise between maximizing gains and minimizing losses, and she is somewhat able to tolerate ups and downs in her investments.
• She plans to start spending her savings in two years, when she will retire. Given her health status and current age, she expects to continue spending for a total of 25 years.
• If she had children, she might add another five or ten years of spending, to allow her heirs to spend-down any residual savings after she dies. But she has no children and prefers to maximize her retirement budget by spending all her investments. Her home equity provides a safety net.

The calculator's statistical model, shown by the red line, sets the investor's exposure to stocks, given her preferences and the time remaining until invested funds will be spent.^ The blue bars are slices or portions of her portfolio, one for each year of anticipated spending.

Because this particular investor's preferences are moderate, the calculator assigns 65% as her maximum exposure to stocks, for investments held 20 years or longer. Anything not invested in stocks goes to bonds, at durations that depend on how soon the funds will be spent.

More than half the portfolio will be held 12 years or longer, and is therefor invested near the maximum level of 65% stocks and 35% bonds. The other half is invested with increasing caution, depending on proximity to the retirement date. As the investor ages and begins to spend her savings, the blue bars will, in effect, march to the left in the chart while staying under the model's red line, thus causing her portfolio to become more conservative over time. Right now, with retirement still two years away and many years of longevity on the horizon, her investments are, overall, about 60% in stocks.

That's without any adjustment for stocks being over- or under-valued. The next chart shows how much this investor's profile would have changed, when adjusted by the corrected CAPE, had the current date been any year between 1920 and 2015. Over those 96 years, 80% of the data fell between the lines shown as High CAPE and Low CAPE (10% were even higher, 10% even lower).

​As it happens, the corrected CAPE in early 2016 is approaching the value marked by the blue line, where High CAPE means that stocks are over-valued. In times like these, the hypothetical investor's profile gets shifted to the blue line, where the maximum invested in stocks is adjusted down to about 50%. On the other hand, if today's corrected CAPE were like the green line in the chart, with stocks a real bargain (Low CAPE), then more than 75% would be invested in stocks to be held for a decade or more. Notice, however, what happens for near-term holdings that will be spent in two or three years. Here, the calculator makes hardly any adjustment for CAPE.

If the holding period were extended well beyond 30 years, the adjustment would diminish somewhat because the effect of CAPE would very gradually fade. But it would not vanish completely in any normal lifetime. To give a specific example, if you had bought stocks near the top of the dot-com bubble in late 1999, when over-valuation of stocks peaked, your returns, even if you held the stocks for many decades, would likely be poorer than if you had purchased at a more normal price.

One final note is important. Some time in the next decade or two, stocks could take a tumble. If they do, the corrected CAPE may fall to a level that recommends a higher-than-normal exposure to stocks. If you adjust your stock allocation periodically, perhaps every quarter or year, according to the calculator's then-current recommendations, the result will be to move some of your holdings in and out of stocks as they become cheap or expensive, taking into account your evolving plans to spend your savings. In that sense, today's relatively expensive stocks are not a permanent penalty for your portfolio. They are a reason for caution that will last only until more optimism is warranted.

* The data for the study were monthly prices of the S&P 500, or a reasonable surrogate, and corresponding estimates of consumer inflation, from 1871 to 2015, as compiled by Robert Shiller. The period from 1871 to 1895 was used to initialize the values of the corrected CAPE. The plotted points in the chart are slopes from regressing LN(r) on LN(c), where r is the annualized real return on stocks, with dividends reinvested, and LN(c) is the corrected CAPE as described in an earlier post. The fitted line is a power function of the form s = -b * POWER( m, y ) where s is the slope; b and m are fitted constants; and y is the holding period in years.

** Statistically, the margin of error is the standard error of the least-squares estimated slope, on a natural-log scale.

*** The weight is k * s / e, where s is the slope; e, the standard error of the slope; and k, a fitted constant. All are on a natural-log scale. The weight is fitted to optimize the natural-log of the annualized real return.

^ The calculator's model for stock-allocations is an exponential function of the form:
p = q + u * ( 1 - EXP( - (y-1)/v) ),
where p is the portion allocated to stocks; q is a minimum allocation when there is one year left before spending starts; q + u is the upper limit or asymptote of the stock allocation; y is the holding period or "slice"; and v is a factor that controls the rate of rising from the minimum to the asymptote. As described here, this model worked better than other growth functions as a method of optimizing returns.

​

Measured by CAPE, stocks in the U.S. seem expensive now. But CAPE has an historical bias. In the early 20th century, it drifted to consistently low values; in recent decades, to persistently high ones. These trends occurred whether the stock market was booming or busting. After removing CAPE's apparent bias, stocks in 2016 seem nearly normal. Which of these opposing views is more credible?

As noted in an earlier post, CAPE is a uniquely reliable predictor of future stock prices. It tends to correctly forecast the long-term direction, up or down, although it does better for the distant future than for next month or even the next year or two. A refined version of CAPE, called LogrCAPE0.33%, is shown below for the years from 1896 through 2015.* 

Right now, the computed value of this version of CAPE is above 0.5, as indicate by the last plotted value at the right end of the blue line. That's higher than the market peaks in 1929 and the late 1960's, which were excellent times to cut back on stock-ownership. However, the long-term trend of CAPE, plotted with the red line, suggests that CAPE was depressed in value throughout the early 20th century, then rose to higher levels in recent decades. A majority of the below-zero observations occurred before 1950, yet virtually every value since the late 1980's has been above zero. An extension of the trend-line suggests that CAPE may continue to have high relative values for another decade or more.

It's reasonable to ask whether CAPE would be a better predictor if its long trend were taken into consideration. Accordingly, the next chart adjusts the LogrCAPE0.33% metric by removing the trend.** The picture is more balanced, with values above and below zero occurring in every quarter-century. And the new trend-line is flat (a statistical necessity). Looking at recent values, the selloff in 2008 and early 2009 now looks like a good buying opportunity, with the de-trended CAPE hitting a low of -0.5. The current, near-normal value raises no alarm to avoid stocks.

One reason to be skeptical of the de-trended CAPE is that many other indicators say stocks are over-valued and are thus likely to weaken in the future. But what if all the indicators, including CAPE, share the same historical bias? Perhaps something about global economies or contemporary finance has changed fundamentally since late in the last century and will persist for decades to come. Maybe investors have more confidence in long-term payoffs from owning stocks because they see value in more global trade, or in better management of national economies, or in reduced fear of world wars and pandemics. Who knows? Lending credence to this speculation is a telling statistic. In a comprehensive analysis of U.S. markets since 1896 for periods of owning stocks from 1 to 40 years, the de-trended CAPE explained more variance (26%) in real returns than did LogrCAPE0.33% (20%).***

A rational person could take either side of the argument. Our calculators strike a compromise. They report value-adjusted allocations to stocks and bonds by first averaging LogrCAPE0.33% and the de-trended CAPE, then estimating allocations and returns from that average. Additional details about the statistical machinery of the calculators are posted here.

* For improved statistical results, the refined version computes a very long-term 0.33% exponential average of companies' earnings, instead of the classical 10-year ordinary average. It then takes the natural logarithm this ratio: current, inflation-adjusted prices divided by the exponentially averaged earnings. In a slight improvement on the standard method, the current month's price adds in the inflation-adjusted value of one month of dividends.

** The trend-line is a log-linear cubic polynomial. The de-trended CAPE is simply LogrCAPE0.33% minus the value predicted by the least square fit of the cubic model.

*** The to-be-predicted variable in the analysis was the actual, inflation-adjusted, compound annual return of U.S. stocks, with dividends reinvested. The returns were computed for all holding periods from one to 20 years, plus all even-numbered holding periods from 22 to 40 years, from January 1896 through December 2015. In a least-squares regression, the observed returns were predicted from a combination of CAPE and the holding period, using a function of the form r = b * c * POWER( m, y ) + a, where r is the log-real-return; c is the de-trended CAPE or LogrCAPE0.33%; a, b, and m are fitted constants; and y is the holding period.

Shiller's CAPE is a sensible measure of whether U.S. stocks are over- or under-valued. But for statistical analysis, it has some weaknesses. Correcting them can improve the insights that CAPE may offer about future returns from stock investments.

Robert Shiller's CAPE is a well established, highly respected metric of the extent to which U.S. stocks are over- or under-valued, rooted in historical data that spans more than a century. Among its good points: CAPE is based on an idea which, on its face, makes a lot of sense. And it does a better job of predicting future stock prices than many other contenders (albeit, with some caveats). So, what's not to like?

First, consider the good points.

Essentially, CAPE compares a one-time snapshot to an historical trend. The snapshot is the price of stocks for large U.S. companies at a given time (the Standard and Poor's 500 or a reasonable surrogate). The trend is the average earnings of those companies for the preceding decade. Why use a 10-year average? Shiller's insight was that doing so smooths out the peaks and dips in earnings that companies experience during the normal boom and bust of economic cycles. Of course, over periods as long as a decade, inflation has to be considered, so CAPE adjusts all stock prices and earnings to express them in current dollars. In short, CAPE estimates whether, after making sensible adjustment for inflation, the current price of stocks is cheap or dear compared to companies' demonstrated ability to generate earnings. For a stock-buyer who wants to capture future earnings, it makes perfect sense.

​What's more, CAPE is a decent predictor, according to standard statistical tests. In Asset Management, Andrew Ang evaluated 14 possible predictors of future stock returns, over periods from 90 days to five years. CAPE was one of only two that gave statistically reliable predictions, and, of those two, only CAPE can be calculated contemporaneously. If you want to make a decent prediction, without waiting for future data, CAPE is the metric of choice.

Now, the weak points:

• Balance. When calculated as a simple ratio and used in that form to conduct statistical analysis, CAPE is imbalanced. It gives too much weight to very high values (extreme over-valuation) and too little weight to very low ones (extreme under-valuation).
• Persistence. Counting current earnings for exactly ten years, equally weighted throughout that period, is not optimal. The persistent effects of extreme conditions are better captured by averaging earnings in a different manner.
• Trend. CAPE's message may be clouded by its own long-term trends. The CAPE value that is purportedly normal today may differ from what was normal decades ago.

​For each of these problems, fortunately, there is a plausible correction.

To see how CAPE over-weights high values and under-weights low ones, consider the chart below. It shows historical values of CAPE, from January 1896 to December 2015, assigned to bins in what statisticians call a "histogram."

In this chart, the values are reasonably well balanced, although a slight elongation to the left has replaced PE10Y's extreme elongation to the right. Accordingly, the statistical skew for LogrPE10Y is -0.17, which is close to the ideal value of zero, though slightly negative. By using the median for the calculation, LogrPE10Y has exactly half its values below zero, and half above. Furthermore, a doubling of stock-prices would raise the metric from zero to about 0.7, while a halving would lower it by the same amount, to -0.7.

Notice that, in the chart for LogrPE10Y, halving of normal valuations has occurred a bit more often since 1896 than doubling of valuations. That's evident because the bars below -0.75 are somewhat higher than the ones above +0.75. From looking at the original, badly skewed chart for the ordinary PE10Y, you would never have seen that. In fact, you might have concluded just the opposite, because of PE10Y's long tail to the right. This observation is one way to appreciate the misleading nature CAPE's classical computation. 

In PE10Y, today's price is compared to ten years of annualized, inflation-adjusted earnings, with the earnings estimated each month. Every month's estimate gets the same weight. But why should ten-year-old earnings get the same weight as current earnings, while those from eleven years ago get no weight at all?

​Shiller's original idea was that a decade-long average filters out the business cycle. While it may do so ordinarily, there are exceptions. For example, a period without a recession, but slightly longer than ten years, occurred from February 1991 to April 2001. Furthermore, the impact of earnings on stock-prices may be more than economic. There may be a behavioral component, as well. Investors' memory of extreme events like the bull market from 1982 to 2000 or the crash from 1929 to 1932 may influence their perception of stock-values. Would a twenty-year CAPE, for example, be a better filter of very long bull markets? Would an extended series of weights that decline ever so gradually for a very long time resemble investors' lingering memory of fearsome crashes?

The following chart addresses these questions by depicting three different ways to compute the average earnings in CAPE.

Since 1896, CAPE has not varied randomly. Over periods of 20 years or so, it has tended to hover at negative, neutral, or positive levels, not wandering freely across the full range of potential values. The chart below demonstrates the non-random behavior, using LogrPE0.3% for CAPE. The plotted line is a fitted (cubic) polynomial. If CAPE were truly random, the line would be flat or nearly so.

The trend shown in the chart poses some ambiguity about the proper interpretation of CAPE. For example, the dot-com peak in 2000 was double the size of any previous peak, when measured on raw values of LogrPE0.3%. However, when measured by the excess above trend, the extremity of 2000 was about the same as the 1929 peak. Making the interpretation even more difficult, the trend as of 2016 could look different when the fitted line is recomputed a decade or two in the future. If the fitted line were projected into the future, it would forecast a leveling-off, with today's CAPE becoming the "new normal." But, as Mark Twain famously observed, forecasting is hazardous, especially when you are trying to predict the future. It's just not clear whether current stock-prices are almost normal (compared to the trend line) or extremely over-valued  (assuming zero to be normal).

Speculatively, one might view the long trend as a behavioral indicator. Perhaps investors get accustomed to high or low stock-valuations, causing the "normal" level of CAPE to rise during long-term bull markets and to fall during periods of prolonged economic distress.

Whatever the reason, the long trend suggests why the explained variance, reported in an earlier chart, is low. It's just 20% because the statistical model made no adjustment for the long trend. Realistically, invoking CAPE to predict future stock-returns or to design a portfolio is a rough approximation. Using it is better than ignoring it. And, although LogrPE0.3% is a "better CAPE," it offers modest improvements, not guarantees.

Follow-up posts examine how the "better CAPE" can be used judiciously to fine-tune an investment portfolio and to adjust realistic expectations of future returns. In doing so, the follow-ups give a deep-dive into the merits of de-trending CAPE and the inner workings of our calculators.

* In data of the sort used here, with overlapping time-periods, traditional statistical tests may be biased. Ang's book, in chapter 8, section 5.2, explains why. Thus, the absolute level of explained variance (R-Squared) should be taken with caution. That said, different methods of measuring CAPE were all subject to the same biases. Comparing the explained variance of the methods is probably correct directionally, if not in magnitude.
​
** Two sets of regression analyses were performed. In both sets, real (inflation adjusted) stock-returns were predicted from CAPE values, using monthly data from January 1896 to December 2015. For exponential averages, PE10Y in December 1880 was the starting value. Annualized stock-returns, with dividends reinvested, were computed for all annual holding periods from one to 20 years, and for all biennial holding periods from 22 to 40 years. In one set of analyses (reported here), the regression slopes and intercepts were forced to be constant across all holding periods. In a second set (to be covered in a future post), the regression parameters were allowed to vary for each holding period.