Ad scientists; Simple tests can overstate the impact of search-engine advertising

Ad scientists; Simple tests can overstate the impact of search-engine advertising

Jul 13th 2013 |From the print edition


SEARCH for a term like “tennis balls” using Google, Bing or Yahoo, and two types of link appear. The majority form a long list of “organic” results. Companies pay the search engines nothing for these. But those at the very top and on the right-hand side of the screen are paid links, a form of advertising that accounts for most of the revenue of search engines. These search ads appear to solve a puzzle that has preoccupied advertisers since John Wanamaker, the 19th-century founding father of marketing, reportedly declared: “Half the money I spend on advertising is wasted; the trouble is I don’t know which half.” But new research shows that the simple measures often used to assess the impact of search ads may be exaggerating their effectiveness.Establishing cause and effect in offline advertising is hard. Ads are difficult to target: space on billboards and in newspapers is seen by lots of shoppers. Some of these eyeballs are worth spending money on; others, either because they belong to existing customers or to people who never will be, are not. And even when big ad campaigns are followed by strong sales, the intuitive conclusion—that rising sales are the result of good ads—can be misleading. Advertising budgets often rise in good times so that spending and sales grow together, even if the advertisements are useless. The ads and the sales have a common cause—strong demand—but may have no causal link.

Internet advertising seems to offer a solution to both these problems. First, internet search ads are targeted: the links that search engines show are based on a combination of the search term a user has typed in and his browsing history. Second, because firms can track whether visitors to their websites come from search-engine links they have paid for, they can work out whether ads convert into sales.

Not so fast. Spurious correlations are also rife in the online world, as a 2011 paper* by Randall Lewis, Justin Rao and David Reiley, a trio of economists then working for Yahoo, shows. Individuals use the web in a lumpy way. On some days lots of sites are visited and many purchases made; on others usage is lighter. This makes comparisons across time unhelpful. On a high-activity day people will tend to perform a lot of searches (and see lots of ads) as well as make many purchases. The relationship between the ads and the purchases looks causal, but may not be.

To test this problem of “activity bias”, the authors recruited volunteers online and split them into two groups. The first group watched a video promoting Yahoo, and the other group watched a political broadcast. The first group used Yahoo around three times more after seeing the ad, giving the impression it was very influential. But the control group—those subjected to a bout of politics but no Yahoo promotion—also used Yahoo a lot more. Both groups happened to be in an active period of internet use. This is why they were recruited in the first place and why they used Yahoo more than in previous periods. Lumpy internet use created a false sense of advertising impact.

The problem of activity bias means that in order to assess the effect of search ads, a proper control group is needed. A 2013 study by Chris Nosko of Chicago University and Steven Tadelis of the University of California, Berkeley, shows how such a test can be designed. Together with Thomas Blake of eBay they examined how important it was for the auction site to buy ads that appear when the term “eBay” is used in a search (“eBay tennis socks”, for example). In March 2012 they switched off eBay’s brand advertisements on Yahoo and Bing, but kept paying for them on Google as a control.

The finding was striking. When the sponsored ad was turned off, search-engine users simply switched to the first “organic” link that mentioned eBay. Overall, the site retained 99.5% of its traffic. Users who type in a brand-specific search are already trying to navigate to eBay’s site. Even if they appear lower down, free search results work just as well as ones that are paid for.

Calling Mr Draper

Firms like eBay don’t just pay for adverts when their brand is mentioned, of course. They place ads in response to millions of other words that indicate the presence of a potential customer. So a second test also investigated ads associated with non-branded keywords (“tennis socks”, for instance). The researchers tracked spending on ads and the number of “attributed sales” (sales made within 24 hours of clicking on a paid Google link) over time. A simple correlation analysis showed a familiar result: ads and sales tend to rise and fall together. A 10% increase in spending seems to raise revenues by 9%. The ads appear to work.

To check these results the authors split America into 210 geographical segments. A third were picked at random, with all Google advertising switched off. Of the rest, the researchers selected regions where patterns of internet activity closely resembled those where the ads were turned off. This allowed them to isolate sales variations that were caused by ads, rather than lumpy activity. The isolated impact is far smaller: a 10% increase in ad spending raises revenues by just 0.5%. (Results for users who had never previously used eBay were stronger, however, suggesting that firms with lesser-known brands may gain more from ads.)

Bosses should still take Wanamaker’s fear seriously: a rise in sales after an ad campaign does not automatically mean that the ads worked. But it also shows how the online world is getting closer to solving the conundrum he posed. Far from being an industry where cause and effect remain murky, online advertising may yet become one area where the dismal science can predict how to get costs down and profits up.


Industry and Trade: A Study of Industrial Technique and Business Organization; and of Their Influences on the Conditions of Various Classes and Nations“, A. Marshall, MacMillan and Co, 1919

Here, there, and everywhere: correlated online behaviors can lead to overestimates of the effects of advertising”, by R.A. Lewis, J.M. Rao and D.H. Reiley, Proceedings of the 20th international conference on World wide web, 2011

Consumer Heterogeneity and Paid Search Effectiveness: A Large Scale Field Experiment”, by Thomas Blake, Chris Nosko and Steven Tadelis, NBER, 2013

About bambooinnovator
Kee Koon Boon (“KB”) is the co-founder and director of HERO Investment Management which provides specialized fund management and investment advisory services to the ARCHEA Asia HERO Innovators Fund (, the only Asian SMID-cap tech-focused fund in the industry. KB is an internationally featured investor rooted in the principles of value investing for over a decade as a fund manager and analyst in the Asian capital markets who started his career at a boutique hedge fund in Singapore where he was with the firm since 2002 and was also part of the core investment committee in significantly outperforming the index in the 10-year-plus-old flagship Asian fund. He was also the portfolio manager for Asia-Pacific equities at Korea’s largest mutual fund company. Prior to setting up the H.E.R.O. Innovators Fund, KB was the Chief Investment Officer & CEO of a Singapore Registered Fund Management Company (RFMC) where he is responsible for listed Asian equity investments. KB had taught accounting at the Singapore Management University (SMU) as a faculty member and also pioneered the 15-week course on Accounting Fraud in Asia as an official module at SMU. KB remains grateful and honored to be invited by Singapore’s financial regulator Monetary Authority of Singapore (MAS) to present to their top management team about implementing a world’s first fact-based forward-looking fraud detection framework to bring about benefits for the capital markets in Singapore and for the public and investment community. KB also served the community in sharing his insights in writing articles about value investing and corporate governance in the media that include Business Times, Straits Times, Jakarta Post, Manual of Ideas, Investopedia, TedXWallStreet. He had also presented in top investment, banking and finance conferences in America, Italy, Sydney, Cape Town, HK, China. He has trained CEOs, entrepreneurs, CFOs, management executives in business strategy & business model innovation in Singapore, HK and China.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: