To Change Tunes, a Nod Will Be as Good as a Wink; Automakers, including Mercedes-Benz and Ford, are cautiously installing gesture sensing systems

To Change Tunes, a Nod Will Be as Good as a Wink


image002 image001-1

Drawings from a 2013 Microsoft patent application suggests gestures that would serve, from left, as commands to lower and raise the audio volume and a request for more information; A Google patent filing proposes this gesture as a command for the audio system to play.CreditUnited States Patent and Trademark Office

Hand gestures from other drivers are a lot like trash talk at playground basketball games: provocative distractions from an otherwise engaging activity.

But gestures may one day find a higher calling, making driving safer and more enjoyable, judging by the amount of work being done to develop technology that would read hand and head motions as computer commands.

Automakers, including Mercedes-Benz and Ford, are cautiously installing gesture sensing systems. Others, like Honda and Hyundai, have applied for gesture patents. Microsoft and Apple have long-range projects with automakers; both companies, along with Google, have filed for related patents, have purchased gesture-systems makers or both.

Commands initiated by hand movements are no Harry Potter fantasy, but a reality found in many homes today. Microsoft’s Kinect interface, the infrared-based system in the Xbox 360 console, enables players to control games solely with body movements.

The most visible example of automotive gesture recognition is the foot-swipe, which will open the rear hatch of a Ford Escape when someone carrying the key fob wags a foot under the rear bumper. Most proposed gestures are obvious enough — hands swiping or pointing, palms facing up or down — but some involve pantomimes like cupping an ear. Coming next are waves, winks and nods that are read by video or infrared cameras and ultrasound sensors.

Gesture technology, which falls under the heading of natural user interfaces, will be more than just another gadget. It will operate comfort and entertainment features, and even then, would always be backed up with buttons and voice commands. Researchers say that only about a dozen signs should be used; more would be onerous.

“It is excruciatingly crucial not to be driven by technology” when it comes to human-machine interfaces, said Parrish Hanna, Ford’s global director for H.M.I. Gestures have to address a “pain point,” he said, not just create marketing buzz.

The foot-swipe isn’t as intuitive as pointing, Mr. Hanna said, but it addresses a pain point — getting armloads of groceries into a car. “As a consumer, once I know about it, I really appreciate it,” he said.

Behind the wheel, simple gestures that are instantly recognized by a car’s computer would help keep drivers’ eyes on the road. Gestures also work better in noisy situations that can foil voice commands.

Computer giants like Apple, Google and Microsoft see a future here, starting with the licensing fees charged for each vehicle running their operating systems.

Apple, which made gestures on touch screens a must-have feature when it introduced the iPhone, recently made news when it struck deals with several automakers for the use of its CarPlay control system. Apple executives have talked — and talked — about “iOS for the Car,” which would put the Siri personal-assistant app and other Apple technology on four wheels. More than a dozen carmakers have signed on to the effort, but so far, Honda is taking the lead.

Late last year, Apple bought PrimeSense, the company that licensed to Microsoft the technology behind early Kinect Xbox 360 sensors. (Microsoft had moved on to homegrown sensor technology before the buyout.) Besides the patents Apple got with its purchase, it has its own gesture patent and a patent application for using your gaze to control a computer. Apple is also recruiting in this area: Last August, it placed an ad for an “iOS Car Experience Engineer,” and in November it sought a gesture-recognition engineer.

Microsoft may be struggling for traction with its mobile strategy, but the company does have advantages in its Kinect technology. The bigger story is its impressive list of industry partnerships. Besides the work with Ford on the Sync system, it has partnered with Kia and Fiat, and its software is embedded in BMW, Nissan, Honda and Aston Martin vehicles as well. And it has been reported to be working on gesture- and face-recognition products.

Still, it is Google that has the most to gain from putting gesture systems in cars. According to company documents, 90 percent of Google’s revenue comes from advertising. One of the best ways to increase ad revenue is to make sure ads are seen by people most likely to click on them. That means collecting data on people using Google’s browser, Maps, Gmail and other products. Consumer behavior in cars is all but virgin territory for marketers, and Google is positioned to cash in on it.

Google has multiple efforts underway that could lead it to an automotive future. Last fall, it bought Flutter, a maker of software that enables people with a webcam to operate Netflix and other apps using signs.

Google seems to be comfortable pushing the limits of what a gesture should control. In a patent application, the company proposes using gestures not only to operate media systems but windshield wipers and cruise controls as well.

For carmakers, gestures are arguably sexier than voice systems or touch screens. Focused as designers are on creating a unique driving experience, gestures are an attractive solution.

BMW engineers and designers are experimenting with six gestures as part of the automaker’s ConnectedDrive electronics package, expected to be available within the next two model years.

This could be part of a broader effort. The research unit of Mercedes-Benz in North America has been hiring engineers for tasks including 3-D tracking, a necessity in reading gestures. The company’s new V-Class vans, due in Europe this year, will have a 3-D sensor that can tell the difference between a command entered on the car’s touch pad and an accidental brush.

A Mercedes design study introduced in 2011, the F125, was a more ambitious experiment in gesture technology. That concept had, among other things, a retractable display screen and gullwing doors that were operated by gestures.

In the United States, Ford’s foot-swipe most likely will not be a goodbye wave for the company. It has a patent application for in-cabin systems using a video camera to capture seven signs. Ford is also working with Microsoft to outfit its cars with Kinect.

General Motors has been working on gestures with Carnegie Mellon University researchers since at least 2002. To date, the best it has to offer domestically is a sensor in its Cadillac User Experience media package that, when approached by a hand, reveals a slate of favorite apps on a touch screen.

G.M.’s Advanced Technical Center in Shanghai, on the other hand, has developed a gesture-and-voice feature called DiDi Weibo to help Chinese drivers participate in social media on the road.

In South Korea, Hyundai has applied for a patent for gesture-recognition technology, and it’s working with two companies to create a two-layer gesture option.

In the first layer, you press a button behind the steering wheel and look at a feature icon in a heads-up display. Releasing the button activates the chosen feature. The second layer is a gesture system that manipulates an interactive screen in the brow of the dash. One demo of the feature shows someone opening video poker games on the screen.

Hyundai has not confirmed a release date, but it is expected to see production within the next couple of model years.

Taken together, all of these efforts might seem meek, but the real critics of gestures in cars, oddly, are those closest to the subject: interface designers.

“It’s too much work, too crude and too inaccurate” for important tasks, said Chris Noessel, director of interaction design at Cooper Studio, a product design firm based in San Francisco. Gesture recognition is getting attention now because of popular movies like “Iron Man” and “Minority Report,” he said.

For the time being, at least, drivers won’t look like the inflatable “air dance” figures outside tire stores, their boneless arms in perpetual motion. But the vocabulary of gestures probably will grow, if less profanely.


About bambooinnovator
Kee Koon Boon (“KB”) is the co-founder and director of HERO Investment Management which provides specialized fund management and investment advisory services to the ARCHEA Asia HERO Innovators Fund (, the only Asian SMID-cap tech-focused fund in the industry. KB is an internationally featured investor rooted in the principles of value investing for over a decade as a fund manager and analyst in the Asian capital markets who started his career at a boutique hedge fund in Singapore where he was with the firm since 2002 and was also part of the core investment committee in significantly outperforming the index in the 10-year-plus-old flagship Asian fund. He was also the portfolio manager for Asia-Pacific equities at Korea’s largest mutual fund company. Prior to setting up the H.E.R.O. Innovators Fund, KB was the Chief Investment Officer & CEO of a Singapore Registered Fund Management Company (RFMC) where he is responsible for listed Asian equity investments. KB had taught accounting at the Singapore Management University (SMU) as a faculty member and also pioneered the 15-week course on Accounting Fraud in Asia as an official module at SMU. KB remains grateful and honored to be invited by Singapore’s financial regulator Monetary Authority of Singapore (MAS) to present to their top management team about implementing a world’s first fact-based forward-looking fraud detection framework to bring about benefits for the capital markets in Singapore and for the public and investment community. KB also served the community in sharing his insights in writing articles about value investing and corporate governance in the media that include Business Times, Straits Times, Jakarta Post, Manual of Ideas, Investopedia, TedXWallStreet. He had also presented in top investment, banking and finance conferences in America, Italy, Sydney, Cape Town, HK, China. He has trained CEOs, entrepreneurs, CFOs, management executives in business strategy & business model innovation in Singapore, HK and China.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: