This comprehensive study guide unifies theoretical inferential testing parameters with practical online experimentation systems, product metric definitions, causal inference rules, and standard statistical test selections.
A/B Testing: A randomized experimentation process wherein two or more versions of a structural variable (e.g., variant A and B) are shown to separate, isolated segments of website users concurrently to extract performance data and determine which setup optimizes core business metrics best.
A/A Testing: An alternative deployment approach where both split cohorts are given the exact same treatment. Used as a structural baseline sanity verification to ensure the tracking pipelines, splitting hashes, and statistical tools match expected error rates before assigning treatments.
Overall Evaluation Criterion (OEC): A balanced, continuous, or composite quantitative metric defining the primary objective parameter of an experiment.
Guardrail Metrics: Secondary business metrics designed to track systemic health or organizational values (e.g., site latency, error pages). Used to trigger diagnostic alerts regarding misleading or broken analysis setups.
Randomization Unit: The discrete entity or user ID randomly mapped to an experimental treatment or control bucket.
Data Leakage (Interference): Systemic pollution occurring when the true behavior of the control pool is actively influenced or shifted by the treatment assigned to the test group.
SUTVA Assumptions: The Stable Unit Treatment Value Assumption. A core pillar in causal inference requiring that the potential treatment outcome of any single experimental unit remains independent of the allocation mechanics of all other units.
Conversion Rates: The specific percentage of targeted users who complete a desired down-funnel tracking event or macro action.
Lift: The proportional percentage shift calculated in a targeted metric between the active treatment group and the baseline control group.
Sample Ratio Mismatch (SRM): A critical mathematical discrepancy manifesting when the empirical ratio of observed user sample sizes across control and variant buckets varies significantly from the design target allocation, pointing to systemic pipeline bias.
2. Experimental Metrics & Operational Thresholds
Population: The theoretical entire universe of data rows, entities, or objects that a researcher desires to run inference against.
Sample: A selected, mathematically small subset of a parent population captured to undergo active tracking.
Sample Mean: The calculated average boundary tracking score of a sample matrix.
Sample Variability: The structural degree to which discrete rows within a sample distribution diverge or spread away from each other.
Discrete Metrics: Metrics bounded to capture only countable integer spaces or finite categorical flags.
Continuous Metrics: Floating-point parameters capable of shifting to any infinite numerical decimal value within an active range.
Null Hypothesis: The default baseline position dictating an absolute absence of a relationship shift or true difference between treatment buckets.
P-value: The exact mathematical probability of observing sample outcomes at least as extreme as the empirical data gathered, given that the null hypothesis holds true.
Confidence Interval: An inferential boundary range structured to encapsulate the true underlying population parameter with a chosen accuracy percentage.
Type I Error: The incorrect rejection of a valid, true null hypothesis (False Positive). Controlled explicitly by the Significance Level.
Type II Error: Failing to reject an invalid, false null hypothesis when a real effect exists (False Negative). Inverse parameter bounds control Statistical Power (the likelihood of correctly identifying a treatment effect).
Effect Size: A standardized, quantitative score capturing the raw structural magnitude of a relationship between features.
Minimum Detectable Effect (MDE): The absolute lowest, most sensitive threshold improvement or effect size an experiment is mathematically powered to capture.
Practical or Substantive Significance: Evaluating whether an isolated statistically significant change is structurally large enough to justify the real-world operational or economic overhead of a project.
Pooled Variance: A parametric optimization approach applied to combine separate sample standard deviations when the parent source parameters are assumed identical.
3. Statistical Engine & Density Transformations
Probability Density Function (PDF): A continuous function defining the relative probability density layer of a continuous random variable tracking near a targeted coordinate point.
Cumulative Density Function (CDF): A function modeling the cumulative probability space that a random variable tracks less than or precisely equal to a targeted value.
Percent Point Function (PPF): The inverse calculation of the standard CDF. Maps an array of targeted probability thresholds back to their coordinate location parameter on a random variable distribution.
4. Parametric vs. Non-Parametric Test Directories
Parametric Tests: Statistical evaluation structures built on explicit structural assumptions regarding the shape and distribution of the source data layer (e.g., tracking a continuous normal shape).
Non-Parametric Tests: Distribution-free algorithmic tests that execute validation comparisons without placing mathematical constraints or assumptions on raw data distributions.
z-Test: A parametric test routine applied to gauge differences across sample means when the parent population variances are completely known.
Student's t-test: An foundational parametric testing framework comparing means across two groups when population variances must be calculated from empirical data. Splits down into a One-sample t-test (matching an individual sample against a constant) and an Independent two-sample t-test (comparing two distinct user cohorts).
Welch's t-test: An optimized parametric version of the two-sample t-test utilized when the compared user segments do not share equal variances.
ANOVA (Analysis of Variance): A parametric framework used to compare mean parameters across three or more population groups simultaneously.
Kruskal-Wallis Test: A distribution-free, non-parametric alternative to ANOVA used to compare the rank distributions across three or more cohorts without checking for normality.
Chi-Square Test for Independence: A non-parametric test used to determine if a significant structural association exists between two categorical variables tracking inside a contingency matrix.
Fisher's Exact Test: A specialized non-parametric routine used to parse associations within discrete categorical vectors when sample sizes drop too low to support Chi-Square approximations.
Kolmogorov-Smirnov Test for Normality: A continuous non-parametric test tracking empirical distribution shapes against a reference distribution to check for general normality violations.
5. Software Engineering Foundations
OOP (Object-Oriented Programming): A fundamental modular code layout paradigm mapping software logic into decoupled object classes that bundle relevant internal variables (data structures) and methods together.
Python Builtins: The foundational array of structural functions, exceptions, and constants instantly exposed in memory by runtime interpreters without demanding external library imports or package declarations.