Feature Engineering: Giving Models Better Inputs

A machine-learning model can only learn from the information you give it. Feature engineering is the craft of turning raw data into inputs that make the underlying patterns easier to learn. In many real projects with tabular data, thoughtful features improve results more than switching to a more complicated algorithm.

What is a feature?

A feature is a measurable property used as an input to a model: a customer's age, the number of purchases last month, the length of a message, the hour of the day. The label (target) is what you want to predict. Raw data columns are not always the best features. Combining and transforming them can reveal more.

Step 1: understand and clean the data

Before creating anything new:

• Check types and ranges. Ages of 250 or negative prices indicate errors.
• Handle missing values. Options include removing rows, filling with median or most frequent value, or using a model-based imputer. Adding an "is missing" indicator can itself be a useful feature.
• Remove duplicates.
• Fix inconsistent categories such as "Kathmandu", "kathmandu" and "KTM".
• Look at distributions with histograms to spot outliers and skew.

Step 2: scale numeric features

Many algorithms (linear models, k-nearest neighbours, support vector machines, neural networks) are sensitive to the scale of features. Tree-based models are mostly insensitive.

• Standardisation: subtract the mean and divide by the standard deviation.
• Min-max scaling: squeeze values into the range 0 to 1.
• Log transform: compresses long right tails, such as income or price, so models treat large values more sensibly.

import numpy as np
import pandas as pd
from sklearn.preprocessing import StandardScaler

df = pd.DataFrame({"income": [20000, 35000, 40000, 1200000], "age": [22, 35, 41, 52]})
df["log_income"] = np.log1p(df["income"])
df[["age_scaled", "log_income_scaled"]] = StandardScaler().fit_transform(df[["age", "log_income"]])
print(df.round(2))

Always fit scalers on training data only, then apply them to validation and test data.

Step 3: encode categorical variables

Models need numbers, so categories must be converted.

• One-hot encoding: one binary column per category. Good for nominal categories with few values.
• Ordinal encoding: map ordered categories (low, medium, high) to 0, 1, 2.
• Target or frequency encoding: replace a category with its average target value or its count. Powerful for high-cardinality columns such as zip codes, but easy to leak. Use cross-validation to compute it safely.
• Embeddings: learned vectors for categories in neural networks.

import pandas as pd

df = pd.DataFrame({"city": ["Kathmandu", "Pokhara", "Kathmandu", "Butwal"]})
encoded = pd.get_dummies(df, columns=["city"], prefix="city")
print(encoded)

Step 4: extract information from dates and times

A raw timestamp is rarely useful. Extract:

• Year, month, day of week, hour
• Is weekend or public holiday (festival periods such as Dashain matter for retail)
• Days since an event, such as days since last purchase
• Cyclical encoding for time of day: use sine and cosine so that 23:00 and 01:00 are close

import numpy as np
import pandas as pd

df = pd.DataFrame({"ts": pd.to_datetime(["2025-03-14 23:30", "2025-03-15 00:30"])})
df["hour"] = df["ts"].dt.hour
df["hour_sin"] = np.sin(2 * np.pi * df["hour"] / 24)
df["hour_cos"] = np.cos(2 * np.pi * df["hour"] / 24)
df["is_weekend"] = df["ts"].dt.dayofweek >= 5
print(df)

Step 5: create interactions and ratios

Sometimes the relationship is in a combination.

• Price per square foot = price / area
• Debt-to-income ratio
• Click-through rate = clicks / impressions
• Difference between two measurements
• Products of features, for example age times income

Domain knowledge is the best guide to which combinations make sense.

Step 6: aggregate behaviour

For customer or transaction data, summarise history: number of orders in the last 30 days, average order value, maximum spend, time since last visit. Rolling-window features are powerful for fraud, churn and forecasting, but be careful to compute them using only information available at prediction time.

orders = pd.DataFrame({
    "customer": ["A", "A", "B", "A", "B"],
    "amount": [500, 700, 300, 200, 900],
})
features = orders.groupby("customer")["amount"].agg(["count", "mean", "max"]).add_prefix("order_")
print(features)

Step 7: text and other data

• Text: word counts, TF-IDF, message length, number of exclamation marks, embeddings from language models.
• Images: pretrained model embeddings as features.
• Location: distance to city centre, region, clusters.

Step 8: select features

More features are not always better. Irrelevant or redundant ones add noise and risk overfitting. Options:

• Remove features with near-zero variance or very high correlation with another.
• Use model-based importance, such as tree feature importances or permutation importance.
• Try regularisation (L1) to shrink useless weights to zero.
• Evaluate by cross-validation, adding or removing groups of features.

Avoid leakage

A strong feature that looks too good may be leaking the target. Ask for every feature: will this value exist at the time of prediction? Compute statistics on training data only, and use time-aware splits for time series.

Putting it together with a pipeline

Pipelines keep preprocessing consistent and prevent leakage:

from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import OneHotEncoder, StandardScaler
from sklearn.ensemble import GradientBoostingClassifier

numeric = ["age", "income"]
categorical = ["city"]

preprocess = ColumnTransformer([
    ("num", Pipeline([("impute", SimpleImputer(strategy="median")),
                      ("scale", StandardScaler())]), numeric),
    ("cat", Pipeline([("impute", SimpleImputer(strategy="most_frequent")),
                      ("onehot", OneHotEncoder(handle_unknown="ignore"))]), categorical),
])

model = Pipeline([("prep", preprocess), ("clf", GradientBoostingClassifier())])
# model.fit(X_train, y_train); model.score(X_test, y_test)

Frequently Asked Questions

Is feature engineering still needed with deep learning?

For images, audio and text, networks learn features themselves. For tabular business data, engineered features still often matter a lot.

How do I know if a feature helps?

Compare cross-validated performance with and without it, and look at feature importance, but remain sceptical of leakage.

How many features are too many?

There is no fixed limit. Watch for overfitting, long training times and instability. Feature selection helps.

Should I scale features for tree-based models?

Usually not necessary, since trees split on thresholds and are unaffected by monotonic scaling.

Conclusion

Feature engineering means cleaning data and building inputs that expose the signal: scaled numbers, encoded categories, time features, ratios and aggregated history. Use domain knowledge, avoid leakage and test every change with proper validation.