Data Cleansing

Carrots fresh from the soil are covered in dirt, tiny bugs, and wilted leaves. Toss them straight into the pot without washing, and you end up with a gritty, unpalatable mess. Raw data collected from the real world is exactly the same.

Humans make typos during manual entry, systems glitch and duplicate records, and crucial fields are frequently left empty. If you feed this contaminated information directly into a computer, even the smartest artificial intelligence will spit out nonsense.

Computer science has a famous saying for this: 'garbage in, garbage out.' Just as a world-class chef cannot create a gourmet meal from spoiled ingredients, the success of any data project hinges on preparing clean ingredients.

Ultimately, data cleansing is the crucial prep work of washing away impurities and keeping only the fresh produce before you cook up AI.

Data cleansing is the concrete work of turning messy information into a usable resource. The process varies depending on the dataset and often requires multiple passes, but common steps include:

A primary task is handling missing values. For instance, if an age field is left blank in a survey, teams decide whether to fill it with the group average or drop it from the statistical model entirely. If you replace it with an average, you need to clean up absurd errors and duplicates beforehand so the mean does not get skewed. Inconsistent formats must also be standardized—such as unifying phone numbers when some rows use hyphens and others contain only digits.

Nonsensical inputs must be carefully rooted out as well. When errors like a human age of 500 or an exam score of -20 appear, analysts verify the source to correct them, convert them to blanks, or flag them separately. Teams also remove duplicate records so only a single entry remains, such as when a customer's order is logged twice by mistake.

Only after filling missing values, standardizing formats, and wiping out errors and duplicates is data finally ready for computers to learn from with confidence.

To be more precise, data cleansing is not just mindlessly deleting every unusual value that looks odd.

Among outliers—data points that deviate drastically from the norm—some are simple typos, but others conceal vital real-world signals. For example, a transaction amount 100 times larger than usual could be a system glitch, or it could be actual credit card fraud. Rather than wiping these entries out immediately, analysts examine the context and establish rigorous statistical criteria to handle them.

In fact, multiple industry surveys reveal that data scientists spend a significant portion of their work hours on data preparation and cleaning. While designing fancy AI algorithms might look glamorous, all that effort goes to waste if the underlying data is dirty.

Trustworthy AI can only emerge from trustworthy data, making cleansing both the foundation and the backbone of data science.