Definition
Data mining is the technology of digging through huge piles of data stored on computers to find useful rules, hidden patterns, and signals that help predict the future, things people couldn't spot on their own. Just like miners dig minerals out of the ground (that's where the word 'mining' comes from), data mining pulls valuable, gem-like knowledge out of a giant heap of information that looks useless at first glance.
How to Dig Gold Out of a Pile of Sand
The receipts we get every day, our web search history, and the locations our phones record all pile up endlessly, like a mountain of dirt. On its own, the pile looks messy and useless. But inside it hide useful rules, like 'what kind of person buys what together on a rainy day.'
Data mining is the work of running this big-data dirt pile through a very fine sieve. It's even used to find buying rules between products that seem unrelated. A famous story says that shoppers who bought diapers at a store also bought beer (though some people point out the story has been exaggerated).
A human eye could never spot connections in a ledger with hundreds of millions of rows. But a computer, using statistics and analysis rules, can snap up the meaningful hidden patterns in an instant.
A Closer Look: How Does It Find Hidden Patterns?
To be more precise, data mining is nothing like a simple search or a summary table of statistics. It doesn't look for answers to questions you already have. It's an exploration that starts from not even knowing what's hiding in there.
Data mining relies on four main techniques. The first is 'clustering,' which groups customers who share similar traits. The second is 'classification,' which decides where a new piece of data belongs. The third is 'association analysis,' which finds links like 'people who buy product A also buy product B.' The fourth is 'anomaly detection,' which flags behavior that is completely different from the usual.
Thanks to these techniques, financial companies can catch stolen credit card payments in real time, and video platforms can recommend new videos you'll love with uncanny accuracy.
How Is It Different from Machine Learning?
People often mix up data mining with artificial intelligence or 'machine learning.' The two are like cousins. They're closely related and share many tools, but they focus on slightly different goals.
The main goal of machine learning is to let a computer learn from data and build a smart AI brain that can predict the future or make decisions on its own, in place of a person. Data mining, on the other hand, puts more focus on digging out knowledge and insights that humans can understand from huge amounts of data.
In the end, data mining works like a compass. It discovers new facts in vast records of the past and present, helping people make wiser and better decisions.
Common misconceptions
Data mining is the same as typing keywords into a search box to find the information you want.
A search (query) looks for specific data that already exists. Data mining explores what rules and relationships are hiding between pieces of data, and uses that to uncover new knowledge.
Where you meet it
A convenience store finds that on rainy days, sales of a certain hot drink jump along with umbrellas, so it rearranges its shelves.
A bank spots an overseas payment that looks nothing like a customer's usual locations or times, and blocks the card approval at the moment of purchase.
In one sentence
Data mining uses computer algorithms to dig gems, in the form of hidden patterns and rules, out of huge piles of data.