The old image of looking for a needle in a haystack doesn't even come close to the realities of modern life. Simply put, the world is drowning in data. Every item scanned at every grocery store, every keystroke made on every website, every transaction made at every bank -- all these generate a digital record.
But then what? Increasingly, organizations are seeing opportunity where formerly they saw only useless or inaccessible data. Thanks to increasingly sophisticated computers and new algorithms, trained analysts are looking for meaningful patterns among the endless data to aid senior executives in making all kinds of decisions. For example:
* Which cell phone customers are going to drop their service plan before their contract expires?
* Which credit card transactions are fraudulent?
* Which homeowners are likely to default on their mortgages within the next six months?
* Which patients are likely to develop a specific illness in the future based on today's medical imaging results?
Organizations in business, health care, government, the military and beyond are turning to data mining and predictive analytics to try to answer questions like these.
"Data mining is a subspecialty of the broad field of informatics," says Robert Hendry, adjunct professor of informatics and technology management systems at the Illinois Institute of Technology in Chicago. "Predictive analytics is a process where you look at history to make a prediction about the future. Data mining is the technique you would use to do that."
For Suzanne Fogel, chair of the marketing department at DePaul University, data mining is about knowing how to bring order to chaos. "Businesses and other enterprises are generating amazing amounts of digital information every minute of every day," she says. "Unless you know how to find the information, it's just sitting there unused. Now, because of better analytical tools that have been developed, we can pull it out and use it in ways we never thought possible."
The U.S. Bureau of Labor Statistics' Occupational Outlook Handbook does not yet categorize data mining as a separate occupation. But it is listed among common responsibilities for the job of occupation research analyst, where the employment outlook is particularly rosy. Employment of operations research analysts is expected to grow 22 percent over the 2008-2018 period, much faster than average:
"Advancements in computing capabilities and analytical software have made it faster and cheaper for analysts to solve problems. As problem solving becomes cheaper and faster, more firms will have the ability to employ analysts," according to the BLS.
Bomshad Mobasher, a DePaul computer science professor, has seen this firsthand.
"There's quite a shortage in industry of people who can do data mining or high-level analytics," he says. "I recently attended a conference where many companies were trying to recruit, because there is such strong demand for data mining expertise right now. The demand is particularly strong for those with graduate degrees, but even those with bachelor's degrees will find many opportunities."
Now that businesses ranging from mid-sized operations to behemoths like Google and AT&T are eager to hire qualified people to mine their data mysteries, universities have answered the call.
"It used to be that universities excelled at producing people with either business analyst skills -- 'please write software to solve these problems'-- or those with advanced computer skills but not much else -- 'here's the software you requested'", says Hendry, who is also IIT's chair of informatics curriculum. "In the past few years, universities have heeded industry's requests to train people who can blend technical skills with analytical skills by revamping their curricula."
The region's most visible effort has occurred at DePaul, where the College of Computing and Digital Media launched the new Center for Data Mining and Predictive Analytics just this month. The university is also offering a new, 13-course master's degree in predictive analytics that couples the necessary computer courses with training in marketing and communications.
"It doesn't do you any good to find out something interesting and new if you can't tell people about it. Once you have an insight, you must be able to talk about it compellingly," says Fogel. "That's what training in marketing provides."
No dedicated program
DePaul's new Center has been garnering plenty of media attention but Mobasher points out that students can gain the technical skills they need for informatics careers at any number of local schools. "Anyone at a school with good business and computer science courses could put something together," he says. "We've just done that work for students already."
At IIT, for example, students have several ways of getting data mining training.
David Grossman, associate professor of computer science, believes in keeping things simple. "This advice may not be popular, but I always suggest getting a degree in computer science, then adding business training to that," he says. "You can always take business electives, do a business minor or earn an MBA later, but data mining has a huge core of computer science and math and if you don't know these disciplines thoroughly, you are prone to dramatic failure."
Grossman, who has been teaching data mining courses since 2003, likens the situation to that of automobiles 100 years ago. "You used to have to understand how a car works to drive one because the technology was so new and prone to breakdowns. Now you don't even have to know how to shift manually in order to drive a car," he says. "In data mining, our tools are not so mature that they will work well without someone knowing how they work."
IIT students can also seek out the information technology management department, where Hendry teaches. "We offer a bachelor's degree in information technology with an available five-course specialization in informatics," he says, estimating that perhaps a quarter of the department's declared IT majors are completing this specialization.
But no matter what you call it or where you obtain your training, those who succeed in data mining share a few traits. "Data mining appeals to those who are curious, love to work with information and are quick to see patterns," says DePaul's Fogel.
"They tend to love puzzles and are able to work very independently."
Does this mean people who want to work in data mining and other branches of informatics should aspire to fulfill the stereotype of a computer nerd with poor people skills? Not so fast, Fogel says: "Nobody works in a vacuum. No matter what you do, you have to be able to communicate about it with others."
Special Advertising Section
Digging for information gold with data mining
As companies generate unprecedented amounts of data theyre gaining new insights
Digging for information gold with data mining (photos.com)