Module 03 - Probability & Distributions

A structured guide tracking random variable classifications, density transformation functions, parametric sampling distributions, and non-parametric estimators.

Outline and Summary of Key Elements and Terms

1. Random Variables
   • A random variable is a variable whose value is a numerical outcome of a random phenomenon.
   • Random variables can be discrete (e.g., the outcome of a die roll) or continuous (e.g., the height of a person).
   • Understanding random variables is essential for probability and statistical modeling.
2. IID: Independent and Identically Distributed
   • A collection of random variables is independent and identically distributed (IID) if each variable has the same probability distribution as the others and all are mutually independent.
   • IID random variables are commonly assumed in many statistical analyses.
   • The IID assumption simplifies calculations and allows for the use of certain statistical methods.
3. Types of Distributions
   • There are many different types of probability distributions, including normal, uniform, binomial, Bernoulli, Poisson, and others.
   • Each distribution has its own unique characteristics and parameters that define its shape and behavior.
   • Choosing the appropriate distribution is crucial for accurate statistical modeling and inference.
4. Probability Density Function (PDF)
   • The PDF is a function that describes the relative likelihood of a continuous random variable taking on a given value.
   • The area under the PDF curve between two points represents the probability that the variable falls within that range.
   • The total area under the PDF curve is always equal to 1.
5. Cumulative Density Function (CDF)
   • The CDF is a function that gives the probability that a random variable is less than or equal to a given value.
   • The CDF is the mathematical integral of the PDF.
   • The CDF is useful for calculating probabilities of events that fall within a certain range.
6. Percent Point Function (PPF)
   • The PPF is the mathematical inverse of the CDF.
   • It gives the value of the random variable for which the CDF is equal to a given probability.
   • The PPF is useful for finding quantiles and percentiles of a distribution.
7. Kernel Density Estimation (KDE)
   • KDE is a non-parametric method for estimating the PDF of a random variable.
   • It uses a kernel function to smooth the observed data and estimate the underlying distribution.
   • KDE is useful when the data does not fit a known parametric distribution.
8. Normal Distribution
   • The normal distribution is a bell-shaped, symmetrical distribution that is widely used in statistics.
   • It is characterized by its mean (μ) and standard deviation (σ).
   • Many natural phenomena follow a normal distribution.
9. Z Distribution
   • The Z distribution is a standard normal distribution with a mean of 0 and a standard deviation of 1.
   • It is used to standardize normal distributions and calculate probabilities.
   • Z-scores represent the number of standard deviations a data point is from the mean.
10. t Distribution
    • The t-distribution is similar to the normal distribution but has heavier tails.
    • It is used when the sample size is small or the population standard deviation is unknown.
    • The t-distribution approaches the normal distribution as the sample size increases.
11. Uniform Distribution
    • The uniform distribution is a distribution where all values within a given range are equally likely.
    • It is often used to model random events with no clear preference for any particular outcome.
    • The uniform distribution can be discrete or continuous.
12. Binomial Distribution
    • The binomial distribution is a discrete distribution that models the probability of a certain number of successes in a sequence of independent trials.
    • Each trial has only two possible outcomes (success or failure).
    • The binomial distribution is characterized by the number of trials (n) and the probability of success on each trial (p).
13. Bernoulli Distribution
    • The Bernoulli distribution is a special case of the binomial distribution with only one trial.
    • It models the probability of a single event with two possible outcomes (success or failure).
    • The Bernoulli distribution is characterized by the probability of success (p).
14. Multinomial Distribution
    • The multinomial distribution is a generalization of the binomial distribution to more than two possible outcomes.
    • It models the probability of a certain number of occurrences of each outcome in a sequence of independent trials.
    • The multinomial distribution is characterized by the number of trials (n) and the probabilities of each outcome.
15. Poisson Distribution
    • The Poisson distribution is a discrete distribution that models the probability of a certain number of events occurring in a fixed interval of time or space.
    • It is often used to model rare events.
    • The Poisson distribution is characterized by the average rate of events (λ).
16. Chi-Squared Distribution

    A continuous probability distribution representing the sum of squared standard normal deviates.

    • Shape: Right-skewed distribution depending heavily on degrees of freedom (k). As k increases, the distribution becomes more symmetrical.
    • Range: Bounded from 0 to infinity.
    • Uses: Goodness of Fit testing, categorical Contingency Tests of Independence, and constructing confidence intervals for population variance.
    • Example: Checking if a die is fair by rolling it 60 times and executing a goodness-of-fit comparison against uniform expected frequencies (10 per face).
17. Gamma Distribution

    A continuous, flexible probability distribution widely used to model positive, skewed data parameters.

    • Parameters: Controlled by a Shape parameter (α or k) and a Rate parameter (β) or inverse scale parameter (θ = 1/β).
    • Properties: Highly flexible; includes the exponential distribution and Chi-Squared distribution as mathematical special cases.
    • Uses: Reliability analysis (time-to-failure systems), queuing theory (waiting line deltas), meteorological calculations (rainfall metrics), and Bayesian prior assignments.
18. Beta Distribution

    A continuous probability distribution strictly defined and bounded on the interval [0, 1], representing random variables for probabilities or proportions.

    • Parameters: Defined by two shape parameters (α and β) that customize its visual configuration.
    • Conjugate Prior: Serves as the conjugate prior for binomial likelihood distributions in Bayesian statistics, ensuring the calculated mathematical posterior remains an updated Beta form.
    • Uses: Modeling advertisement click-through rates (CTR), task duration modeling within PERT/project management environments, and training nodes inside variational autoencoders.