The Benefit of Hindsight: New rules allow ETFs to market back-tested data, but the practice is mired in murky practices that don’t benefit most investors


The Benefit of Hindsight


New rules allow ETFs to market back-tested data, but the practice is mired in murky practices that don’t benefit most investors.

Hindsight is 20/20. So it’s easy to see why investors might be kicking themselves for not investing in the PowerShares S&P 500 Low Volatility Portfolio (ticker: SPLV) in 2008. Over the past five years, the index this exchange-traded fund tracks has returned 11.7% a year — considerably better than the S&P 500’s 8.5%. There’s just one hitch: That index didn’t exist five years ago, and those returns are mostly hypothetical.Those “historical” returns for the Low Volatility index, which launched in April 2011, are the result of back-testing — going back in time and calculating how an index would have done had it been around years ago. This widely used practice has played a key role in helping new ETFs — many of which track indexes with no real performance records — to grow quickly. In a little more than two years, the PowerShares ETF has swelled to $4.7 billion, making it one of the fastest-growing ETFs. Back-testing has also contributed a groundswell of new, increasingly niche indexes over the years, which, of course, were designed to create a groundswell of new, increasingly niche ETFs.

The Financial Industry Regulatory Authority has, until recently, prohibited ETF sponsors from sharing back-tested index data with investors. No such rule applied to the index provider, however, so with a wink and a nudge, ETF sponsors could direct investors to the indexers to find back-tested returns. Most index companies, which aren’t regulated, make those numbers readily available, though the exact methodology or constituents behind those results aren’t always as easy to access.

In April, Finra gave ETF companies the green light to share back-tested index results with institutional investors (not retail investors), but with many stipulations on how they’re presented. The guidelines are spelled out in a letter to Brad Swenson, chief compliance officer at ALPS Distributors, which has its own proprietary ETFs and distributes other companies’ products, including PowerShares QQQ.

“Enough institutions were asking for [pre-inception index performance] data that it prompted us to engage in dialogue with Finra to establish some criteria that, should ETF sponsors choose to utilize it, they could,” says Swenson.

The criteria, if adopted, could clarify these often-murky numbers. They also shine a spotlight on the role that back-testing plays in the development of new indexes and the marketing of their corresponding ETFs.

As demand for ETFs has exploded, index companies have evolved from neutral market observers to, in many cases, market makers.

“Most of the indexes that have launched in the past three years are segments of existing products; they’re slicing the market or taking it further afield with rules-based products,” says Todd Rosenbluth, director of ETF research for S&P Capital IQ. “In certain cases, the index didn’t exist until there was a want for the ETF.”

And that’s assuming there was actually a want for the ETF. There are currently 1,488 ETFs tracking a total of 1,131 unique indexes, according to IndexUniverse. About half of them have been launched within the past four years. And it’s hard to discern demand — 674 have less than $50 million in assets.

To be fair, just because an index is new, doesn’t mean it’s spurious. The Low Volatility index is a straightforward concept — it tracks the 100 least volatile stocks in the S&P 500 — and so far, its performance has proven to be consistent with its promise.

IT’S CLEAR THAT MOST INDEXES these days are being created for the sole purpose of launching an ETF, which makes the issue of back-testing data all the more prominent. The median time between the inception of an index and a related ETF has been compressed from at least three years in 2000 to just 77 days in 2011, according to a Vanguard study published last year.

This phenomenon wouldn’t be possible without back-testing, says Joel Dickson, the senior ETF strategist with Vanguard who led the study. “Back-tested index data allow for a ready-made history on a product that has not been tested live,” he says. Not surprisingly, performance numbers help sell new funds, even if ETF providers aren’t themselves trumpeting the results. According to Dickson’s study, ETFs associated with back-tested indexes took in far more cash in the first six months of inception than those with no such data.

It’s possible that there would have been demand for these asset classes anyway. Many new ETFs come about because investors are keen to gain exposure to any area that has been doing well. Still, with the benefit of hindsight, index providers can slice and dice the indexes, adjust weightings and methodologies to optimize returns. “I’ve never seen a back-test that looks bad,” says Kevin Mahn, president and chief investment officer of Hennion & Walsh Asset Management in Parsippany, N.J. “We’re going to assume that if [ETF sponsors] are going to launch a product, it’s because they’ve tested a hypothesis and it’s gone well.”

Back-testing in and of itself isn’t problematic, says Dave Nadig, president of ETF analytics at IndexUniverse, which does independent analysis of ETFs. “Back-testing is a useful tool, because in finance we have no lab for conducting experiments,” he says. “This is our lab.”

Where it is worrisome, though, is when results are taken at face value. Investors should know better than to base investment decisions on past performance, but this axiom is more imperative when the results are cooked up in a financial lab.

“Back-tested data is even worse than past performance because it almost guarantees survivorship bias,” says Nadig.

Ideally, back-tested data start with what an index would have looked like at the beginning of a time period, and work forward, but it’s a cumbersome process that still leaves plenty of room for interpretation. As such, most index companies start with the present and work backward, says Nadig. The result is survivorship bias. An index company creating a new financial index today, for example, probably wouldn’t include Lehman Brothers in its back-testing. “Yet, had you created the index 10 years ago, you would have certainly included it,” he says.

In Dickson’s analysis of 370 indexes with at least six months of back-filled data and six months of live data between 2000 and 2011, 87% of indexes outperformed the broad U.S. stock market during the time the back-filled data were used. Yet only 51% outperformed the broad U.S. market after the index was launched. Looking at it from another angle, on average the indexes outperformed the broad stock market 10.3% a year based on five years of back-filled returns. But over the five years following their inceptions, on average, the indexes lagged the broad market by nearly 1% a year.

EVEN WHEN BACK-TESTING is done meticulously, there are still differences between theory and reality. “When a major index announces changes,” says Rosenbluth, “we see those underlying stocks move or up or down as a result. I don’t believe we’re going to see the impact of that in a back-tested environment.”


Moreover, because new ETFs tend to piggyback on investment trends du jour, there is the added risk that the biggest upswing has already occurred. The new ETFs that have come to market in recent years are the result of “huge selection bias,” says Dickson. “Investors need to be aware that they are seeing a subset of strategies that happen to be tilted toward products that have done well recently.”

When the MAC Global Solar Energy Index, which is tracked by Guggenheim Solar ETF (TAN), launched in 2008, its back-tested returns reflected a golden time for solar stocks. Yet, over the past three years, the index has fallen nearly 30% a year on average.

In another example, fundamentally weighted indexes tend to tilt toward small and midsize companies, says Dickson. “In many cases, their back-tested data start in 2000, which in hindsight was a great time to invest in smaller companies,” he says. “But you would have never launched a product like this in 2000.”

The Finra guidelines won’t solve the problem of selection bias. But, armed with better information on what’s behind an index, investors may be better equipped to understand whether back-tested data are a realistic snapshot of how an index will perform in different markets.

MOST INDEX COMPANIES ADHERE to industry standards when presenting past results, says Nadig. If they took too many liberties, after all, it would undermine their credibility. Still, key information, such as the day-to-day changes in constituents, isn’t widely available; investors typically need to pay for that information. Meanwhile, there are plenty of ways indexes can paint hypothetical historical results in a positive light, such as by cherry-picking the timelines they highlight. “We’ve seen index presentations where performance goes back 3½ years,” says Dickson.

Under the Finra guidelines, ETF providers that want to share back-tested data with institutional investors — which include registered investment advisors, large retirement plans, and consultants — can’t use arbitrary timelines. They’ll need to show pre-inception performance going back at least 10 years and updated through the most recent quarter. For indexes that have been around at least a year, back-tested results need to be accompanied by actual performance data. Their results must also take into account any fees or other charges levied by the fund. Perhaps most importantly, they need to provide the methodology used to create the index with enough detail that an investor with access to readily available market data could reconstruct it. In other words, if the index contains an element of “secret sauce,” its back-tested results can’t be used in marketing the fund.

Individual investors — who stand to benefit the most from better transparency — won’t be privy to the cleaned-up results. The guidelines make it clear that financial intermediaries can’t share the data with clients or plan participants. Nevertheless, Finra’s clarifications could ultimately benefit individuals. “Having better back-tested data in the hands of financial advisors or consultants may help individuals avoid some of the pitfalls,” says Brandon Thomas, chief investment officer of Envestnet, which provides products, services, and technology to wealth advisors.

OF COURSE, GIVEN ALL THESE HOOPS, Dickson and others don’t think most ETF providers will be in any hurry to provide their own back-tested data to investors — at least until institutional investors start expecting it. Why would they, when it’s available from the index companies? “They can do the exact thing they’ve been doing with no regulation,” he says.

One exception may be with so-called self-indexers, which include the likes of IndexIQ and WisdomTree. Because they play the dual role of index creator and ETF sponsor, prior to April they couldn’t point investors to a third-party indexer’s back-tested results when discussing their products. Now that they can share pre-inception performance data, albeit with institutional investors, they’re better positioned to compete with traditional ETF sponsors, says Nadig. Ironically, this could open the floodgates for even more niche indexes and ETFs.

That’s an even better reason for investors to be suspicious of new products — and their hypothetical results. “You really need to dig deeper and ask, ‘Am I seeing this because it has a good investment case, or are they creating the investing case after observing a good period of performance?’ ” says Dickson. “In most cases, I think it’s the latter.”


About bambooinnovator
Kee Koon Boon (“KB”) is the co-founder and director of HERO Investment Management which provides specialized fund management and investment advisory services to the ARCHEA Asia HERO Innovators Fund (, the only Asian SMID-cap tech-focused fund in the industry. KB is an internationally featured investor rooted in the principles of value investing for over a decade as a fund manager and analyst in the Asian capital markets who started his career at a boutique hedge fund in Singapore where he was with the firm since 2002 and was also part of the core investment committee in significantly outperforming the index in the 10-year-plus-old flagship Asian fund. He was also the portfolio manager for Asia-Pacific equities at Korea’s largest mutual fund company. Prior to setting up the H.E.R.O. Innovators Fund, KB was the Chief Investment Officer & CEO of a Singapore Registered Fund Management Company (RFMC) where he is responsible for listed Asian equity investments. KB had taught accounting at the Singapore Management University (SMU) as a faculty member and also pioneered the 15-week course on Accounting Fraud in Asia as an official module at SMU. KB remains grateful and honored to be invited by Singapore’s financial regulator Monetary Authority of Singapore (MAS) to present to their top management team about implementing a world’s first fact-based forward-looking fraud detection framework to bring about benefits for the capital markets in Singapore and for the public and investment community. KB also served the community in sharing his insights in writing articles about value investing and corporate governance in the media that include Business Times, Straits Times, Jakarta Post, Manual of Ideas, Investopedia, TedXWallStreet. He had also presented in top investment, banking and finance conferences in America, Italy, Sydney, Cape Town, HK, China. He has trained CEOs, entrepreneurs, CFOs, management executives in business strategy & business model innovation in Singapore, HK and China.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: