Anomaly Detection: Finding Unusual Events in Data

Most of the time, data looks normal. The interesting moments are the rare exceptions: a card payment far from your usual pattern, a pump vibrating more than usual, a sudden jump in failed logins, a sensor stuck at one value. Anomaly detection is the practice of finding such unusual observations automatically.

What is an anomaly?

An anomaly (or outlier) is an observation that deviates significantly from what is expected. Not every anomaly is bad; it may signal a problem, an opportunity or just a rare but legitimate event.

Types of anomalies:

Point anomalies: a single value is unusual (a transaction of 2,000,000 rupees on an account that normally spends 5,000).
Contextual anomalies: a value is unusual in context but normal otherwise (30 degrees Celsius is fine in summer but strange in January in Kathmandu).
Collective anomalies: a group of values is unusual together (a flat line in a sensor reading, or a sudden series of small payments).

Why it is hard

Anomalies are rare, so there are few or no labelled examples.
"Normal" changes over time (seasonality, growth, new behaviour).
High dimensionality: many variables make unusual combinations hard to see.
Noise: random variation looks like anomalies.
Cost of mistakes: false alarms exhaust attention, and missed anomalies cause damage.
Definition ambiguity: experts may disagree on what counts as unusual.

Approaches

1. Statistical methods

Z-score: how many standard deviations a value is from the mean. Flag if beyond about 3. Assumes roughly normal data.
Interquartile range (IQR): flag values below Q1 − 1.5×IQR or above Q3 + 1.5×IQR. More robust to skew.
Moving averages and control charts: compare a value to a rolling average and spread.
Seasonal decomposition: remove trend and seasonality, then look for unusual residuals.

import numpy as np

values = np.array([102, 98, 105, 99, 101, 97, 103, 240, 100, 96])

q1, q3 = np.percentile(values, [25, 75])
iqr = q3 - q1
lower, upper = q1 - 1.5 * iqr, q3 + 1.5 * iqr

anomalies = values[(values < lower) | (values > upper)]
print("Range:", round(lower, 1), "to", round(upper, 1), "| anomalies:", anomalies)

2. Machine-learning methods (unsupervised)

Isolation Forest: builds random trees; anomalies are easier to isolate, so they have shorter average path lengths. Fast and effective for tabular data.
Local Outlier Factor (LOF): compares the density around a point with its neighbours.
One-class SVM: learns a boundary around normal data.
Clustering-based: points far from any cluster centre are suspicious.
PCA reconstruction error: points that cannot be well reconstructed from principal components are unusual.

import numpy as np
from sklearn.ensemble import IsolationForest

rng = np.random.default_rng(42)
normal = rng.normal(loc=[50, 5], scale=[5, 1], size=(500, 2))    # amount, items
odd = np.array([[200, 1], [10, 30], [180, 2]])                    # a few unusual transactions
X = np.vstack([normal, odd])

model = IsolationForest(contamination=0.01, random_state=0).fit(X)
labels = model.predict(X)                  # -1 = anomaly, 1 = normal
scores = model.decision_function(X)        # lower = more anomalous

flagged = np.where(labels == -1)[0]
print("Flagged rows:", flagged.tolist())

The contamination parameter is your guess of the anomaly fraction. It is an important knob that determines how many items are flagged, so tune it with domain experts.

3. Supervised methods

If you have labelled anomalies (confirmed fraud cases, known failures), you can train classifiers, handling severe class imbalance with weighting, resampling and metrics like precision-recall. They detect known patterns well but may miss new kinds.

4. Deep learning methods

Autoencoders: compress and reconstruct normal data; high reconstruction error suggests an anomaly.
LSTM or transformer forecasters: predict the next values and flag large prediction errors in time series.
Variational autoencoders and GAN-based methods for complex data like images.

Deep methods suit large, complex data (images, logs, multivariate sensor streams) but are harder to tune and explain.

Anomaly detection in time series

Use methods that consider the temporal context: compare with forecasts, use seasonal baselines, look at changes in level or variance, and handle gaps and delays. A simple but effective approach is to forecast the expected value and flag points where the actual value falls outside a prediction interval.

Setting thresholds and handling alerts

Most methods give a score. Choosing the threshold is a business decision balancing false positives and false negatives.

Start conservative to avoid alert fatigue.
Rank by score and review the top items first.
Provide context in alerts: what was unusual, compared with what, recent history.
Allow analysts to mark alerts as true or false, and use this feedback to improve.
Group related alerts into incidents.

Applications

Finance: fraud and unusual transactions
Cybersecurity: suspicious logins, network traffic, malware behaviour
Manufacturing and utilities: predictive maintenance from vibration, temperature or power signals
IT operations: server metrics and error spikes
Healthcare: abnormal vital signs or lab results (as decision support)
Retail and e-commerce: pricing errors, unusual returns, bot traffic
Data quality: detecting data entry mistakes, sensor failures and pipeline problems

Evaluating performance

Without labels, evaluation is difficult. Options:

Inject synthetic anomalies into normal data and test detection.
Review the top-ranked alerts with domain experts and compute precision at top k.
Use historical incident logs as weak labels.
Monitor operational outcomes: how many alerts led to action?

Best practices

Understand the domain and what anomalies matter.
Clean and prepare data, handle missing values and scale features.
Model normal behaviour per segment: different customers or machines have different baselines.
Combine methods and rules; simple rules based on business logic catch many issues.
Update models as behaviour evolves.
Keep humans in the loop for decisions.
Document and explain alerts so teams trust them.

Frequently Asked Questions

Which algorithm should I start with?

For tabular data, try simple statistics and Isolation Forest. For time series, start with seasonal baselines and forecast-based residuals.

How do I choose contamination?

Estimate from experience or past incidents, then tune by reviewing flagged items. It is a starting point, not a truth.

Can anomaly detection run in real time?

Yes. Many models score single events quickly. Streaming systems update statistics continuously.

What is the difference between novelty detection and outlier detection?

Outlier detection finds unusual points in the training data itself. Novelty detection trains on clean normal data and flags new points that differ.

Conclusion

Anomaly detection finds the rare events that matter by modelling what normal looks like and flagging deviations. Start simple, tune thresholds with domain experts, evaluate with real feedback and keep refreshing your models.