Correlation Beats Complexity: What 20+ Models Taught Me About Learning Signal

Machine learning comparisons often focus on which model wins. This experiment asked a different question: how much does the amount of predictive signal matter compared to model complexity?

To answer that, I evaluated a broad collection of regression and classification algorithms while systematically increasing the correlation between features and target (ρ = 0.0 → 1.0).

Models evaluated:

• *Regression:* Linear Regression, Ridge, Lasso, ElasticNet, Decision Tree, Random Forest, Extra Trees, Gradient Boosting, KNN, SVR, and MLP.
• *Classification:* Logistic Regression, Gaussian Naive Bayes, Decision Tree, Random Forest, Extra Trees, Gradient Boosting, AdaBoost, KNN, and MLP.

Metrics were evaluated on both training and test sets across every correlation level.

The Hypothesis

The intuition before running the experiment was straightforward:

• Higher correlation should improve every model.
• More complex models should benefit earlier from weak signal.
• Ensemble methods should generalize better than individual trees.
• Regularized linear models should trade variance for bias.

The results largely confirmed these expectations - but with several surprises.

Result 1 - Correlation Dominates Model Choice

Across almost every regression metric (MSE, RMSE, MAE, R²) and every classification metric (accuracy, precision, recall, F1, ROC-AUC), increasing correlation produced nearly monotonic improvements.

• At ρ = 0, most models were close to baseline.
• By ρ ≈ 0.6, nearly every modern model had become highly predictive.
• By ρ = 1, almost every algorithm reached essentially perfect performance.

The biggest improvement came not from switching algorithms, but from increasing available signal.

Result 2 - Complexity Helps Early, Then Stops Mattering

The aggregated complexity tiers summarize this nicely. Low-, medium-, and high-complexity families separate clearly when signal is weak - high-complexity models consistently extract more useful structure at low correlation.

However, once correlation exceeds roughly 0.6, the curves rapidly converge. By ρ = 0.8–1.0, virtually every family performs almost identically. When the data contain strong signal, simple models become remarkably competitive.

Result 3 - Single Trees Memorize. Ensembles Generalize.

Decision Trees achieved nearly zero training error and perfect classification accuracy regardless of correlation. The test curves, however, improved only gradually - textbook overfitting.

Random Forests and Extra Trees dramatically reduced this gap. Gradient Boosting produced one of the smallest train-test differences throughout the experiment while maintaining excellent accuracy. The ensembles consistently provided the best balance between flexibility and generalization.

Does the Sign of Correlation Matter, or Just Its Strength?

Every feature-selection tutorial teaches some version of the same move: compute the correlation matrix, sort by value, keep the top features. In practice, "sort by value" quietly means "sort by positive value." A feature at ρ = +0.85 gets kept without a second thought. A feature at ρ = −0.85 - exactly as strong a relationship, just pointed the other way - sometimes gets dropped by a corr > threshold filter that never took an absolute value in the first place.

The implicit assumption is that positive and negative correlation are not quite the same currency. This experiment tests whether that's true, extended past ρ = 0 → 1 and out to ρ = −1.

Result 4 - Neural Networks Were Surprisingly Conservative

Both MLP regression and MLP classification behaved much more like Linear Regression than Decision Trees. Training and testing curves almost overlapped. Instead of aggressively fitting the training data, the network generalized consistently across all correlation values. This stability was one of the more interesting findings.

The Unexpected Story: The MAPE Hump

Every regression metric behaved exactly as expected except one: MAPE.

Rather than decreasing steadily with increasing correlation, most models exhibited a clear hump around ρ ≈ 0.4. Meanwhile, RMSE and MAE decreased while R² increased - the predictions were objectively improving.

The hump exists because MAPE divides by the true target. When targets become small, even modest absolute errors translate into disproportionately large percentage errors. This experiment illustrates why MAPE should never be interpreted in isolation.

Key Findings

• Signal Superiority: Increasing feature-target correlation consistently improves every meaningful evaluation metric.
• Capacity Convergence: High-capacity models learn useful structure sooner but lose their advantage as signal increases.
• Ensemble Efficiency: Decision Trees dramatically overfit; ensemble trees mitigate this behavior.
• Linear Longevity: Linear models remain highly competitive whenever the underlying relationship is informative.
• Metric Nuance: MAPE can produce misleading conclusions because of its sensitivity to small target values.

Conclusion

The largest gains did not come from replacing Linear Regression with Gradient Boosting or replacing Logistic Regression with a neural network. They came from increasing the amount of information available to learn.

Model complexity matters - but only after the data provide something worth learning. If there is one lesson from these experiments, it is that improving feature quality, reducing noise, and increasing signal will usually deliver larger returns than endlessly searching for a more sophisticated algorithm.