Most qualified data scientists don’t want to work for a company where they are the only one doing this sort of thing. “If they’re the only data scientist, they don’t see a lot of career growth”

Will the Next Nate Silver Please Stand Up?

BY KLINT FINLEY

Ever since Nate Silver made a splash with his freakishly accurate election predictions, all sorts of companies have been looking for their own rock-star data scientists. The trouble is that these people are hard to come by — few can blend computer science with applied mathematics in a way that produces truly effective data science — and for many companies, it’s not even clear that they really need this kind of expertise. Shashi Upadhyay, CEO of analytics outfit Lattice Engines, which helps companies tackle data science, has seen this issue firsthand. “Customers ask us: do we need to hire data scientists?” he says. “It’s a question that’s been debated a lot: should the chief marketing officer of the future be a data scientist?” Lattice Engines certainly has a stake in the game. If companies hire their own data scientists, they might not need the company’s cloud-based marketing and sales analytics tools. So Upadhyay and company decided to do some research to answer questions like, “Which industries are hiring data scientists?” and “Where are they located?”LATTICE ENGINES DATA SCIENTIST STUDY

Top cities housing data scientists:

· Greater New York City Area
· San Francisco Bay Area
· Houston, Texas Area
· Greater Minneapolis-St. Paul Area
· Greater Chicago Area

Top employers of data scientists:
· Cognizant Technology Solutions
· Thomson Reuters
· IBM
· Google
· Tata Consultancy Services

Top industries employing data scientists (U.S.):
· IT/Software
· Financial Services
· Telecommunications
· Hospital & Health Care
· Pharmaceuticals

Industries data scientists work in before becoming data scientists:
· IT and Services/Computer Software
· Higher Education
· Financial Services
· Telecommunications
· Pharmaceuticals

Who’s hiring the most data scientists:
· Deloitte (1,042 job postings)
· General Dynamics (1040)
· UnitedHealth Group (989)
· National Guard (872)
· SAIC (622)

To answer these questions, the company’s own data scientists scoured the web for job listings — on job boards and company websites — to get a sense of the demand for data scientists. They then scooped up publicly searchable personal data, such as LinkedIn profiles, to determine where existing data scientists are living and working. Then they weighted the listings and profiles depending on keywords used to determine which ones were dealing with what most people consider “data science.”

“A lot of people call themselves data scientists without being in the middle of the latest big data technology like Hadoop or Hbase,” says Upadhyay. “They were statisticians and now they’re calling themselves data scientists.”

Some of the answers were predictable: lots of data scientists live in New York and San Francisco, and financial services companies hire a lot of them. But some of the results were less expected.

One of the big surprises was that consulting companies were some of the biggest employers of data scientists. “When a profession is still new and people are asking questions about whether they need to hire, it’s not usual for that profession to turn up in consulting companies first,” Upadhyay says.

The top companies hiring data scientists were: Cognizant Technology Solutions, Thomson Reuters, IBM, Google, and Tata Consultancy Services. “We think ‘data scientist’ and we think Google, Facebook and LinkedIn, but Facebook and LinkedIn don’t even make the list,” he says.

The top locations for data scientists were: the Greater New York City Area, the San Francisco Bay Area, the Houston, Texas Area, the Greater Minneapolis-St. Paul Area and the Greater Chicago Area. The top industries were: IT/software, financial services, telecommunications, hospital and health care, and pharmaceuticals.

“Houston is an oil and gas center and they tend to do a lot of exploration, a lot of analysis in house,” Upadhyay says. He says the numbers they found reflect a certain maturation of the field of data science. “It’s a different group from the group that pioneered it. All the action is taking place inside these consulting firms in New York City and Houston and not so sexy verticals.”

Lattice Engines couldn’t find much data about the educational background of data scientists — too few public profiles included educational details to be statistically relevant. But Upadhyay did offer some anecdotal evidence. “It’s skewed towards undergrads, but the tail is a bit longer for this group than you would expect,” he says. “Most data scientists only have bachelor’s degrees and learned on the job, but if you were looking at a normal engineering pool, 95 percent of them would have only a bachelor’s degree. Data scientists are far more likely than engineers to have a doctorate.”

That squares with other research. Most of the top ranked data scientists in Kaggle’s competitions don’t have PhDs. And of course, Nate Silver himself only has a bachelor’s degree in economics.

So how about it then? Should you hire a data scientist? “It’s a very competitive market. There are many open positions which are not filling,” says Upadhyay. “So I’ve been telling customers that if they want to build a data science team they need to either over pay, which is the model in the Valley, or commit to training people and giving them a career path.

He doesn’t believe most qualified people want to work for a company where they are the only one doing this sort of thing. “If they’re the only data scientist, they don’t see a lot of career growth,” he says. “They seem like engineers, but they actually have different career goals. People like to go to places where they’ve seen other people like themselves succeed. So data scientists go to places like LinkedIn and Facebook. But if you don’t have a data scientist already and you just expect them to keep their heads down and do analysis all day, that tends not to be attractive to them.”

The other alternative, he says, not surprisingly, is hiring a company like Lattice Engines to do your data science for you. That may seem self-serving, but the market does seem to be shifting that way. From consulting firms to web-based analytics tools to data science competitions like those held by Kaggle, there’s an entire industry out there trying to provide data science services to companies that don’t have the talent in-house.

“Either go whole hog and build a full team, or don’t bother, at least not right now,” Upadhyay says.