Outlier

Imagine you asked 10 students in your class how much daily allowance they get. Nine of them get around 5,000 won (about $4). But one friend wrote down 1,000,000 won (about $700). Now work out the average, and it comes to more than 100,000 won (about $70) per person. That is a strange number that fits none of the nine students.

A value that sits far from the rest and is unusually large or small like this is called an outlier. Outliers have a habit of badly distorting the average. One extreme value can drag the total way up or way down. That is why you should never trust the average alone when you look at statistics.

If an outlier is throwing off the average, bring in another measure. Line up all the numbers from smallest to largest and pick the one right in the middle. That is the median. In our example, the median still stays near 5,000 won, so it shows the real picture without being pulled around by the outlier.

If you ignore how spread out the data is and trust only the average, you can badly misread reality. A single odd data point can completely cover up the true character of a whole group.

How do you tell a value that is just a bit large from a true outlier that needs special handling? You need a clear rule. Statisticians skip the eyeballing and use objective math tools such as the box plot and quartiles.

First, line up the data from smallest to largest. Then measure the width of the range that holds the middle 50 percent of the data. Using that width, set up invisible fences above and below. Any data point outside those fences is officially flagged as a possible outlier. It is a strict mathematical test, not a gut feeling.

For bell-shaped data, called a normal distribution, statisticians use a tool called the standard deviation. It measures how widely the data spreads around the average. A value more than 3 standard deviations from the average is suspected to be an outlier. Under this rule, only the tiny handful of values outside about 99.7 percent of the data get flagged.

Once you set up a clear rule like this, a computer can find outliers on its own, even among millions of data points. The same step is often used to filter out unwanted noise before an AI learns from a huge pile of data.

To be more precise, finding an outlier does not mean you should always delete it or ignore it. Outliers come from very different causes depending on the situation.

The first cause is a simple measurement slip or typing error. Say someone types 600 kilograms instead of 60 kilograms in a weight box by hitting the wrong key. That value is a plain error that gets in the way of your analysis. You should fix it through data cleaning, or leave it out of the analysis.

The second cause, though, can be a real event that is very rare and special. In 1985, the British Antarctic Survey reported a sharp drop in the ozone layer over Antarctica from ground measurements. Meanwhile, the satellite data held values so low that they had been marked 'questionable quality.' Those values were capturing that very drop. Like a fraudulent payment in finance, or one rare-disease patient's reaction to a drug, an outlier can be a decisive clue that leads to a new discovery.

In the end, an outlier is a test of the analyst's insight. Before you wipe it out, ask why that number was born. That habit is where good scientific inquiry begins.