Define your analytical question and measurement metric before selecting indicators, charts, or models. Practical data analysis iterates through: "Define Question ➔ Ingest Data ➔ Clean ➔ Explore ➔ Model & Infer ➔ Communicate & Verify".
| Core Question | Primary Tool / Metric | Secondary Sanity Checks |
|---|---|---|
| Is the data trustworthy? | Missingness rate, primary key uniqueness, boundary & format validation | Sampling bias, measurement units, time window, joined row counts |
| What is the central tendency & spread? | Mean and standard deviation; median and interquartile range (IQR) | Skewness, outliers, sample size, subgroup divergence |
| What does the distribution look like? | Histogram, box plot, Q-Q plot | Binning granularity, multi-modality, fat tails (do not assume normality) |
| How do categories or time periods differ? | Bar charts, time-series line charts, grouped aggregations | Base denominators, sample composition, seasonality, missing timestamps |
| Are two continuous variables related? | Scatter plot, Pearson / Spearman rank correlation | Non-linear curvature, extreme outliers, confounding strata |
| How do we predict a continuous target? | Linear regression, hold-out set MAE / RMSE | Data leakage, residual heteroscedasticity, out-of-domain extrapolation |
| Did an intervention cause an impact? | Randomized controlled trials (A/B testing) or causal inference with explicit identification | Control group validity, selection bias, confounders (correlation $\neq$ causation) |
Related Tools: Pandas Cheat Sheet · NumPy Cheat Sheet · SQL Commands · Core AI Algorithms
1. The Data Analysis Workflow
| Stage | Key Operations | Minimum Deliverable |
|---|---|---|
| 1. Define Problem | Determine business goal, unit of analysis, target population, and time window | Metric definition, denominator rules, success criteria |
| 2. Ingest Data | Extract data from SQL databases, APIs, or event streams; document sampling rules | Raw data snapshot, data dictionary, schema versions |
| 3. Clean Data | Inspect missing values, deduplicate primary keys, normalize units and constraints | Cleaning pipeline scripts, before/after quality audit |
| 4. Explore Data (EDA) | Visualize distributions, compute summary statistics, spot anomalies across subgroups | Exploratory plots, correlation matrices, testable hypotheses |
| 5. Model & Infer | Select model matching problem structure, validate assumptions and hold-out errors | Effect estimates or predictive scores, confidence intervals |
| 6. Communicate | Synthesize actionable insights, boundaries of validity, and next experiments | Executive presentation, reproducible notebook, monitoring alerts |
Predictive Workloads: Always split training and test sets before performing distribution-dependent imputation, scaling, or feature selection. In cross-validation, re-fit all transformations strictly within each training fold to avoid data leakage.
2. Variable Data Types & Permissible Operations
| Data Type | Definition & Real-World Examples | Common Summary Statistics | Critical Pitfalls |
|---|---|---|---|
| Nominal (Categorical) | Unordered categories (e.g., city, blood type, device model) | Frequency counts, mode, percentage shares | Numeric encoding (1, 2, 3) does not imply distance or mathematical ordering |
| Ordinal | Ordered categories with unequal intervals (e.g., survey satisfaction, low/med/high) | Median rank, cumulative frequency, percentile intervals | Differences between ranks (e.g., 4★ to 5★ vs 1★ to 2★) are not mathematically equal |
| Discrete Numeric | Countable integer values (e.g., page views, ticket counts, days active) | Mean, median, dispersion, Poisson/Negative Binomial counts | Often heavily zero-inflated and right-skewed |
| Continuous Numeric | Measured values along a real continuum (e.g., latency, revenue, temperature) | Mean, variance, standard deviation, quantiles, histograms | Must verify consistent units, calibration precision, and measurement limits |
3. Data Cleaning Checklist
| Anomaly Type | Initial Diagnostic | Recommended Treatment | Boundary Constraints |
|---|---|---|---|
| Missing Values (NaN/Null) | Calculate missing ratio per column and subgroup; identify MNAR vs. MAR patterns | Keep explicit missing indicators; impute via domain logic (e.g., median/MICE) or drop | Missing does not mean zero; naive mean imputation distorts variance and covariances |
| Duplicate Rows | Check business primary keys, transaction UUIDs, and timestamps | Drop duplicate keys based on business precedence; aggregate when valid | Multiple transactions by the same user are not duplicates |
| Type Inconsistencies | Mixed string/numeric columns, inconsistent date formats, currency signs | Cast explicitly into standard types (ISO 8601 timestamps, float64, integer) | Beware of timezone discrepancies and daylight saving offsets |
| Boundary Violations | Negative prices, age $> 150$, percentages outside $[0, 1]$ | Cap via domain logic, flag for audit, or mark as invalid | Log every modification to maintain audit reproducibility |
4. Descriptive Statistics & Spread Metrics
Given an observed sample $x_1, x_2, \dots, x_n$ ($n \ge 2$):
$$\bar{x} = \frac{1}{n}\sum_{i=1}^{n} x_i,\qquad s^2 = \frac{1}{n-1}\sum_{i=1}^{n}(x_i - \bar{x})^2,\qquad s = \sqrt{s^2}$$
| Metric | Mathematical Formula | Sensitivity to Outliers | Best Use Case |
|---|---|---|---|
| Sample Mean ($\bar{x}$) | $\frac{1}{n}\sum x_i$ | High | Symmetric, bell-shaped distributions without extreme tails |
| Median ($Q_2$) | 50th percentile rank | Extremely Robust | Skewed distributions (income, web page visit duration, latency) |
| Standard Deviation ($s$) | $\sqrt{\frac{1}{n-1}\sum(x_i-\bar{x})^2}$ | High | Dispersion metric matching original measurement units |
| Interquartile Range (IQR) | $Q_3 - Q_1$ (75th - 25th percentile) | Extremely Robust | Non-parametric dispersion; basis for Tukey's box plot fences ($1.5 \times \text{IQR}$) |
5. Chart Selection Decision Matrix
| Analytical Objective | Recommended Chart Type | Secondary Alternative | What to Avoid |
|---|---|---|---|
| Univariate Distribution | Histogram (tunable bins) | KDE density plot | Overly coarse binning that hides multimodality |
| Category vs. Metric Comparison | Horizontal Bar Chart | Box Plot (to display variance) | 3D bar charts, truncated baseline scales for non-ratio metrics |
| Continuous Trend Over Time | Time-Series Line Chart | Area Chart (for stacked composition) | Non-chronological ordering, missing timestamp gaps |
| Relationship Between Two Variables | Scatter Plot | 2D Hexbin / Contour Plot (for large $N$) | Drawing correlation lines through non-linear U-shaped scatters |
| Proportional Composition | Stacked 100% Bar Chart | Treemap / Donut Chart | Pie charts with $> 5$ slices; comparing unaligned angles |
6. Correlation Analysis & Pitfalls
The Pearson correlation coefficient $r_{xy} \in [-1, 1]$ measures linear association between two variables:
$$r_{xy} = \frac{\sum_{i=1}^{n}(x_i - \bar{x})(y_i - \bar{y})}{\sqrt{\sum_{i=1}^{n}(x_i - \bar{x})^2 \sum_{i=1}^{n}(y_i - \bar{y})^2}}$$
- Pearson $r$: Assumes linearity and continuous normal variables; sensitive to outliers.
- Spearman $\rho$: Measures monotonic relationships using rank orders; robust to non-linear stretches and outliers.
- Correlation $\neq$ Causation: A correlation coefficient of 0 only indicates the absence of a linear relationship; deterministic non-linear curves (e.g., $y = x^2$) can yield $r \approx 0$.
7. Joint & Marginal Probability Distributions
For two discrete random variables, the joint probability mass function is $p(x, y) = P(X = x, Y = y)$. Summing over the complementary variable yields the marginal probability:
$$\sum_x \sum_y p(x,y) = 1,\qquad p_X(x) = \sum_y p(x, y),\qquad p_Y(y) = \sum_x p(x, y)$$
| $X \backslash Y$ | $Y = 0$ | $Y = 1$ | Marginal Row Total $p_X(x)$ |
|---|---|---|---|
| $X = 0$ | 0.1 | 0.2 | 0.3 |
| $X = 1$ | 0.2 | 0.5 | 0.7 |
| Marginal Col Total $p_Y(y)$ | 0.3 | 0.7 | 1.0 |
- Conditional Probability: $P(Y = y \mid X = x) = \frac{p(x, y)}{p_X(x)}$ where $p_X(x) > 0$. Example: $P(Y = 1 \mid X = 1) = \frac{0.5}{0.7} = \frac{5}{7}$.
- Statistical Independence: $X$ and $Y$ are independent if and only if $p(x, y) = p_X(x)p_Y(y)$ for all pairs $(x, y)$. In the table above, $0.5 \neq 0.7 \times 0.7 = 0.49$, proving $X$ and $Y$ are dependent.
8. Simple Linear Regression Foundations
Modeling a continuous response variable $Y$ as a linear function of predictor $X$ with an error term:
$$Y_i = \beta_0 + \beta_1 x_i + \varepsilon_i,\qquad \hat{y}_i = \hat{\beta}_0 + \hat{\beta}_1 x_i,\qquad e_i = y_i - \hat{y}_i$$
| Parameter / Metric | Interpretation | Important Caveats |
|---|---|---|
| Intercept ($\beta_0$) | Expected value of $Y$ when $X = 0$ | Has no physical meaning if $X=0$ lies far outside observation ranges |
| Slope ($\beta_1$) | Expected unit change in $Y$ per unit increase in $X$ | Units are $\text{unit}(Y) / \text{unit}(X)$; does not prove causal intervention |
| Ordinary Least Squares (OLS) | Minimizes the sum of squared residuals: $\sum e_i^2$ | Sensitive to high-leverage outliers and heteroscedasticity |
| Coefficient of Determination ($R^2$) | Proportion of response variance explained by model: $1 - \frac{SS_{\text{res}}}{SS_{\text{tot}}}$ | Can be negative on test sets; high $R^2$ does not validate causal mechanisms |
Evaluation metrics across $m$ validation observations:
$$\text{MAE} = \frac{1}{m}\sum_{i=1}^{m}|y_i - \hat{y}i|,\qquad \text{RMSE} = \sqrt{\frac{1}{m}\sum{i=1}^{m}(y_i - \hat{y}_i)^2}$$