Exploratory Data Analysis

Find Problems Before They Become Expensive

Data quality issues discovered during EDA take minutes to fix. The same issues discovered after a week of training cost far more — in compute, time, and credibility with stakeholders.

• Label leakage: a feature that directly encodes the target variable
• Train/test contamination: the same entity appearing in both splits
• Silent joins gone wrong: many-to-many join that multiplied rows unnoticed
• Temporal leakage: future information encoded in "current" features
• Near-constant features: variables with 99% the same value provide no signal

Discover What Your Data Actually Says

Documentation lies. Schema definitions are aspirational. The actual data in production has a different story — and EDA is how you read it.

• Distributions are rarely Gaussian — most real-world features are skewed or multimodal
• Outliers are often real events that the model must handle, not noise to discard
• Correlations between features suggest redundancy and multicollinearity risks
• Missing value patterns encode information — missingness is rarely completely random
• Class imbalance ratios determine your evaluation metric choices

Guide Feature Engineering

The best features are usually suggested by the data itself. EDA reveals which transformations, interactions, and aggregations are worth trying.

• Highly skewed continuous features suggest log-transformation
• Bimodal distributions suggest a hidden categorical split
• Strong correlation between two features suggests ratio or interaction features
• Cyclic datetime patterns (hour, day-of-week) suggest sinusoidal encoding
• Groups in scatter plots suggest segment-specific models or interaction terms

Validate Your Assumptions

Every ML project starts with assumptions about the data. EDA is where you test them before committing to a model architecture and feature set.

• "The classes are roughly balanced" — check this; often wrong
• "Feature X is a good predictor of Y" — check correlation; sometimes spurious
• "The data is i.i.d." — check for temporal autocorrelation and batch effects
• "Missing values are random" — visualize missingness patterns; often systematic
• "The test set is representative" — compare distributions between train and test

Numerical Features

Analyze each numerical feature in isolation before considering relationships. Key statistics and plots:

• Histogram: reveals distribution shape — normal, skewed, bimodal, uniform
• Box plot: shows median, IQR, and outliers using Tukey fences (1.5× IQR)
• QQ plot: compare empirical quantiles against theoretical normal distribution
• Skewness: |skew| > 1 is highly skewed; |skew| > 0.5 is moderately skewed
• Kurtosis: high kurtosis (leptokurtic) indicates heavy tails and extreme outlier risk
• Mean vs median: large gap indicates skew; use median for robust central tendency reporting

Categorical Features

Categorical analysis focuses on the count and frequency of each level, plus cardinality assessment.

• Value counts: frequency of each category; sort descending to see dominant levels
• Bar chart: visualize value counts; horizontal orientation for long category names
• Cardinality: number of unique values — high cardinality needs special encoding strategies
• Rare categories: categories with <1% frequency may need grouping into "Other"
• Unexpected values: typos, inconsistent casing ("Male"/"male"/"M"), legacy codes

Datetime & Time Series Features

Temporal data requires its own analysis to reveal trends, seasonality, and gaps.

• Time-series plot: raw values over time; reveals trend and outlier events
• Seasonality decomposition: STL decomposition splits trend + seasonal + residual components
• Autocorrelation (ACF/PACF): detect lag-based dependencies
• Gap detection: missing time periods that could indicate data pipeline failures
• Distribution by time bucket: hour-of-day, day-of-week — reveals cyclic patterns

Missing Value Analysis

Missingness should never be ignored — its pattern is itself informative.

• Missing percentage per feature: sort descending; features >80% missing may need dropping
• Missing completely at random (MCAR): missingness unrelated to any variable
• Missing at random (MAR): missingness related to other observed variables — can impute
• Missing not at random (MNAR): missingness related to the missing value itself — hardest case
• Missingness heatmap: visualize which rows have missing values across which columns simultaneously

Correlation Analysis

Measure pairwise linear and monotonic relationships between numerical features.

• Pearson r: linear correlation; assumes normality; sensitive to outliers
• Spearman ρ: monotonic correlation; rank-based; robust to outliers; preferred for skewed data
• Kendall τ: concordant vs discordant pairs; more robust than Spearman; slower for large datasets
• Strong correlations (>0.8) between features indicate multicollinearity — one may be droppable
• Strong correlation with target indicates high predictive utility

Target Correlation Analysis

For supervised learning, the most important bivariate analysis is feature-vs-target correlation.

• Rank features by absolute Spearman correlation with target
• Low correlation does not mean useless — interactions and nonlinear relations matter
• For classification: plot class-conditional distributions of each feature
• Box plots (feature value | class) reveal separability at a glance
• Mutual information captures nonlinear relationships that correlation misses

Multicollinearity Detection

Correlated features cause instability in linear models and inflate coefficient variances.

• Correlation matrix heatmap: visualize all pairwise correlations at once
• VIF (Variance Inflation Factor): VIF > 5 suggests problematic collinearity; VIF > 10 is severe
• Cluster correlated features together and keep one representative per cluster
• PCA collapses collinear features into orthogonal components

Categorical vs Continuous Relationships

When one variable is categorical and another is continuous, different plots reveal different aspects of the relationship.

• Box plot: median and spread per category; good for outlier comparison
• Violin plot: shows full distribution per category; better than box for multimodal distributions
• Strip plot + swarm plot: all individual data points overlaid per category
• ANOVA / Kruskal-Wallis test: statistical test for mean differences across groups

ydata-profiling

Formerly pandas-profiling, this is the most popular automated EDA tool. One line generates a comprehensive HTML report covering statistics, distributions, correlations, and missing values for every column.

from ydata_profiling import ProfileReport
import pandas as pd

df = pd.read_csv("dataset.csv")
profile = ProfileReport(
    df,
    title="Dataset EDA Report",
    explorative=True,
    correlations={
        "pearson": {"calculate": True},
        "spearman": {"calculate": True},
    }
)
profile.to_file("eda_report.html")

Sweetviz

Sweetviz specializes in train/test comparison — generating side-by-side visualizations that make distribution shift between splits immediately obvious. Essential for validating train/test splits.

import sweetviz as sv
import pandas as pd

train = pd.read_csv("train.csv")
test = pd.read_csv("test.csv")

# Compare train vs test distributions
compare_report = sv.compare(
    [train, "Train"], [test, "Test"],
    target_feat="label"
)
compare_report.show_html("train_test_comparison.html")

# Single dataset analysis
analyze_report = sv.analyze(train, target_feat="label")
analyze_report.show_html("train_analysis.html")

D-Tale

D-Tale provides an interactive web UI for exploring pandas DataFrames. You can filter, sort, chart, and compute statistics interactively — useful for ad-hoc exploration in development environments.

import dtale
import pandas as pd

df = pd.read_csv("dataset.csv")

# Launch interactive UI (opens browser tab)
d = dtale.show(df)
d.open_browser()

# In Jupyter
import dtale.app as dtale_app
dtale_app.USE_COLAB = False  # set True for Colab
dtale.show(df)

Lux

Lux is a Jupyter-native library that automatically recommends visualizations based on what attributes it detects in your DataFrame. It surfaces non-obvious patterns without you needing to specify what to plot.

import lux
import pandas as pd

df = pd.read_csv("dataset.csv")

# Just displaying the DataFrame in Jupyter
# activates Lux's automatic recommendation engine
df  # shows interactive Lux widget

# Specify an intent to focus recommendations
df.intent = ["label"]  # recommendations relative to target
df  # now shows intent-driven recommendations

Transforming Skewed Features

EDA reveals which features need transformation before they can be useful to linear models or distance-based algorithms.

• Log transform: apply to right-skewed features (income, request counts, file sizes); use log(x+1) to handle zeros
• Square root transform: milder than log; useful for count data with occasional large values
• Box-Cox transform: data-driven power transform; requires all positive values
• Yeo-Johnson transform: like Box-Cox but handles zeros and negatives
• Always check skewness before and after; target |skew| < 0.5

Encoding Outliers as Features

Rather than removing outliers, EDA may reveal they are meaningful events worth flagging explicitly.

• Create binary "is_outlier_X" flag for extreme values in feature X
• Winsorize: cap extreme values at 1st/99th percentile instead of dropping
• In fraud detection, outliers are often the signal — never blindly remove them
• Use IQR method (below Q1 - 1.5×IQR or above Q3 + 1.5×IQR) for outlier boundaries

Binning & Interaction Features

EDA-revealed non-linear relationships and bimodal distributions suggest creative feature engineering.

• Binning/bucketing: convert continuous age into "young/middle/senior" buckets if EDA shows non-linear relationship with target
• Interaction features: multiply two correlated features; ratio of two related features (e.g., debt/income)
• Polynomial features: add x² or x³ if scatter plot suggests quadratic relationship with target
• Document every feature engineering decision with the EDA evidence that motivated it

Datetime & Missing Value Features

Temporal features and missingness patterns are two of the most commonly overlooked sources of signal.

• Datetime decomposition: extract hour, day-of-week, month, quarter, is_weekend, is_holiday
• Time since event: days since last purchase, hours since last login
• Missing as signal: create "was_X_missing" binary flag before imputing X
• Missingness count: total number of missing features per row as a feature
• Recency, frequency, monetary (RFM): aggregate time-series data into summary statistics