Probability: Basic Concepts and Definitions

Introduction to Probability

Probability is a mathematical framework for quantifying uncertainty and analyzing random phenomena. It originated in the 16th century as a way to solve gambling problems and has since become fundamental in statistics, science, and many applied fields.

• Random phenomenon: A process whose outcome cannot be predicted with certainty, even if all possible outcomes are known.

• Trial: Each repetition or observation of a random phenomenon (e.g., each attempt to cross a train track).

• Outcome: The result of a single trial (e.g., stopped by a train or not).

• Sample space (S): The set of all possible outcomes of a random experiment.

• Event: Any subset of the sample space, often denoted by capital letters (A, B, ...).

Example: Flipping a coin, rolling a die, drawing a card, or selecting a person at random are all random phenomena.

Deterministic phenomenon: A process with a single, predictable outcome (e.g., today's date, number of fingers).

Types of Probability

Empirical Probability

Empirical probability is based on observed data from repeated trials. It is also called relative frequency probability.

• Calculated as the ratio of the number of times an event occurs to the total number of trials.

Law of Large Numbers (LLN): As the number of trials increases, the empirical probability approaches the true probability.

Theoretical Probability

Theoretical probability is based on mathematical models and assumptions, typically when all outcomes are equally likely.

• Calculated as the ratio of the number of favorable outcomes to the total number of possible outcomes.

Example: The probability of flipping a head with a fair coin is 0.5.

Personal (Subjective) Probability

Personal probability reflects an individual's degree of belief, intuition, or educated guess about the likelihood of an event.

• Not guaranteed to match the true probability in the long run.

• Example: A doctor estimates an 80% chance of recovery for a patient based on experience.

Sample Spaces and Events: Examples

• Three fair coins flipped: Event A = all faces the same. Sample space S = {HHH, HHT, HTH, HTT, THH, THT, TTH, TTT}. Event A = {HHH, TTT}.

• Two fair dice rolled: Event A = sum is 8. Sample space S = 36 outcomes. Event A = {(2,6), (3,5), (4,4), (5,3), (6,2)}.

• Conditional experiment: Toss a die, flip a coin once if even, twice if odd. Event A = die shows 3 and no tails in coin flips.

Law of Large Numbers and Simulation

Law of Large Numbers (LLN)

As the number of independent trials increases, the observed relative frequency of an event approaches its theoretical probability.

• Independence is crucial for LLN to hold.

• LLN is not the same as the 'law of averages' (i.e., past outcomes do not affect future probabilities).

Simulation Example in R: Simulating coin flips for n = 10, 100, 1000 shows that the proportion of heads approaches 0.5 as n increases.

Counting Techniques in Probability

Multiplication Principle

If a process consists of k steps, with ways to do step 1, ways for step 2, ..., ways for step k, then the total number of ways is:

Example: Choosing a rat (F or M) and administering one of three treatments (A, B, placebo) yields outcomes.

Permutations (Order Matters)

Permutations count the number of ways to arrange k objects from n distinct objects, where order matters and there is no replacement.

• n! (n factorial): Product of all positive integers up to n.

• In R: factorial(n)

Combinations (Order Does Not Matter)

Combinations count the number of ways to choose k objects from n distinct objects, where order does not matter.

• In R: choose(n, k)

• Example: Choosing 3 books from 5:

Order vs. No Order

• Combinations: Order does not matter (e.g., team selection).

• Permutations: Order matters (e.g., password).

Additional info: This is the binomial theorem, relating combinations to algebraic expansions.

Probability Axioms and Venn Diagrams

Probability Axioms

• Axiom 1: For any event A,

• Axiom 2: The probability of the sample space is 1:

Venn Diagrams

Venn diagrams visually represent relationships between sets (events) and the sample space. The area of each region reflects the relative probability.

Relationships Between Events

• Complement: or is the set of outcomes in S but not in A.

• Intersection: is the set of outcomes in both A and B.

• Union: is the set of outcomes in A, B, or both.

Disjoint and Independent Events

• Disjoint (Mutually Exclusive): (cannot occur together).

• Independent: Occurrence of A does not affect probability of B and vice versa.

Probability Rules

Complement Rule

The probability that an event does not occur is 1 minus the probability that it does occur.

• "At least one" is the complement of "none".

• "Neither A nor B" is the complement of "A or B".

General Addition Rule

For any two events A and B:

• If A and B are disjoint, .

• For "A or B, but not both":

Example: Titanic Tragedy

Questions: What is P(first class)? What is P(first class or survived)? What is P(first class or survived, but not both)?

Conditional Probability and Independence

Conditional Probability

The probability of event B given that event A has occurred is:

• "Among" or "given" in a problem indicates conditional probability.

• Example: Probability of breast cancer among women:

Example Table: Breast Cancer Prevalence in the UK

Questions: What is the probability for having breast cancer among women? What is the probability of being a woman among all suffering breast cancer?

Properties of Conditional Probability

• If B ⊆ A, then

• Conditional complement:

Independence

• Events A and B are independent if and

• General multiplication rule:

• If independent:

To verify independence: Check if or

Additional info: Disjoint events cannot be independent unless one has probability zero.

Tree Diagrams and Bayes' Rule

Tree Diagrams

Tree diagrams are useful for visualizing sequences of events, marginal, joint, and conditional probabilities.

Example: College graduation rates by type of high school attended and graduation status.

• Calculate overall graduation probability by summing joint probabilities along branches.

Bayes' Rule

Bayes' Rule allows us to reverse conditional probabilities:

For multiple disjoint events :

Example: Given machine defect rates and production shares, find the probability a defective item came from a specific machine.

Summary Table: Key Probability Concepts