Does Feature Engineering Still Matter in the Age of Gradient Boosting?

Feature engineering - the craft of transforming raw data into more informative representations - is often described as the most impactful (and most underrated) step in any machine learning pipeline. With modern ensemble methods like XGBoost and Random Forest capable of learning complex patterns automatically, a reasonable question arises: is handcrafted feature engineering still worth the effort?

1. Introduction & Hypothesis

This experiment was designed to answer that question rigorously across two very different problem domains: clinical diagnosis (breast cancer detection) and financial market prediction. Rather than relying on anecdote or intuition, we ran a controlled study comparing model performance on raw versus engineered feature sets, using identical models, identical cross-validation strategy, and a full suite of evaluation metrics.

Thoughtful feature engineering - grounded in domain knowledge - consistently improves model performance, even when the model is already sophisticated enough to extract non-linear patterns on its own.

The hypothesis going in: domain-informed feature engineering will yield measurable and consistent gains across all four model classes - not just simple linear models, but also powerful tree ensembles. The results, as you will see, largely confirmed this - with some fascinating domain-specific nuances.

2. The Experiment

Datasets

To check the importance we considered two sectors which show the most overall volatility and outliers:

• Medical: The Breast Cancer Wisconsin dataset (569 samples, 212 malignant / 357 benign) served as the medical domain benchmark. It is a well-characterised, real-world dataset with interpretable biological features.
• Financial: For the financial domain, synthetic OHLCV market data was generated using Geometric Brownian Motion (GBM) with regime shifts, simulating ~10 years of daily trading data (2,451 days; 1,333 up days, 1,118 down days). The prediction task was binary: next-day direction.

Feature Engineering Approach

For the medical dataset, the raw 10 mean measurement features were expanded into 60 features by incorporating all 30 original measurements, polynomial interaction terms (radius_x_texture, area_x_concavity), and composite diagnostic scores such as Malignancy_Score.

For financial data, raw OHLCV prices - which are non-stationary - were transformed into 26 stationary, economically meaningful signals: daily returns, momentum indicators, RSI, MACD, Bollinger Bands, volume ratios, and rolling volatility.

Models & Evaluation

Four classifiers were evaluated: Logistic Regression (linear baseline), Decision Tree (interpretable non-linear), Random Forest (ensemble bagging), and XGBoost (gradient boosted ensemble). All were evaluated using 5-fold stratified cross-validation. Five metrics were tracked: Accuracy, F1 Score, ROC-AUC, Precision, and Recall.

3. Medical Domain Results

Across all four models and all five metrics, feature engineering delivered consistent, measurable improvements - with only a single minor exception (Decision Tree Recall: -0.005).

Logistic Regression - a model that cannot learn non-linear patterns on its own - benefited the most in absolute terms, gaining +3.34% in accuracy and +5.21% in recall. XGBoost, despite being capable of learning complex interactions, still improved meaningfully, suggesting that explicit domain-coded features provide a signal quality boost that automatic feature interaction learning cannot fully replicate.

In cancer diagnosis, recall (sensitivity) - the fraction of actual malignancies correctly identified - is the metric that costs lives when it fails. The engineered feature set pushed XGBoost recall from 0.906 to 0.962: a gain that translates to missing ~6% fewer malignant cases. That is clinically significant.

4. Financial Domain Results

The financial dataset told a more nuanced story. Predicting next-day market direction is an inherently noisy problem - the base accuracy for the best models hovers around 60%. Feature engineering did not uniformly help here; its impact was strongly model-dependent.

Linear models actually degraded slightly with engineered features. The raw OHLCV prices, while non-stationary, still contain level information that a linear model can partially exploit. When transformed into returns and oscillators, the signal becomes more noisy from a linear perspective. Ensemble models, however, flourished: XGBoost ROC-AUC jumped from 0.591 to 0.671 - a substantial gain in financial ML terms.

Feature engineering in financial markets is not universally beneficial - it requires matching the feature type to the model's inductive bias. Stationary, non-linear technical signals are best exploited by ensemble models with sufficient depth to capture regime interactions.

5. Understanding the 'Why' via SHAP

Knowing that performance improved is valuable. Knowing why it improved is what makes knowledge transferable. To answer this, we applied SHAP (SHapley Additive exPlanations) analysis to the medical dataset for both Logistic Regression and XGBoost.

SHAP assigns each feature a contribution value for each individual prediction, grounded in cooperative game theory. The mean absolute SHAP value across all samples gives a reliable ranking of global feature importance.

The contrast is striking. On raw features, mean area dominates with a SHAP importance of 332 - the model leans heavily on a single scalar. With engineered features, importance distributes meaningfully across features that jointly encode tumor aggressiveness: worst area, radius_x_texture (large irregular nuclei), and area_x_concavity (tumor shape irregularity). The model now reasons about the problem the way a pathologist would.

XGBoost abandons its previously dominant signal (mean area) and instead leads with Malignancy_Score - a composite feature synthesising multiple raw measurements into a clinically grounded index - followed by ratio features that quantify the relative change between worst and mean measurements (a proxy for intra-tumour heterogeneity). This is a known biomarker for aggressive cancer. XGBoost could not have derived this ratio cleanly without having it pre-computed.

Even models that can theoretically learn any transformation benefit from being handed the right representation. Feature engineering is not about compensating for model weakness - it is about encoding domain expertise that raw data cannot express.

What happens when the model's inductive bias mismatches the engineered features?