Statistical Modes

Fair Forge metrics that perform statistical aggregation accept a pluggable statistical_mode parameter. The same computation — estimating a rate, measuring distribution divergence, aggregating sub-metrics — can be performed either as a point estimate (Frequentist) or as a full posterior distribution with credible intervals (Bayesian).

Which Metrics Support Statistical Modes

Metric	Primitives Used	What Gets a CI in Bayesian Mode
Toxicity	All four	DR, ASB, DTO, DIDT
Bias	`rate_estimation`	Bias rate per protected attribute
Agentic	`rate_estimation`	pass@K and pass^K

The Four Primitives

StatisticalMode defines four abstract primitives. Every metric composes these to compute its final result — it never contains its own statistical logic.

`rate_estimation(successes, trials)`

Estimates a proportion from count data.

Mode	Returns	How
Frequentist	`float`	`successes / trials`
Bayesian	`dict` with `mean`, `ci_low`, `ci_high`, `samples`	Beta-Binomial posterior: `Beta(1 + successes, 1 + trials - successes)`

Used by: Bias (bias rate per attribute), Toxicity (toxicity rate per group), Agentic (success rate p = c/n)

`distribution_divergence(observed, reference)`

Measures how far an observed distribution is from a reference.

Mode	Returns	How
Frequentist	`float`	Total variation distance: `0.5 * Σ	p_i - q_i	`
Bayesian	`dict`	Dirichlet posterior over `p`, divergence computed per MC sample

Used by: Toxicity (DR — demographic representation)

`aggregate_metrics(metrics, weights)`

Combines multiple named sub-metrics into one weighted score.

Mode	Returns	How
Frequentist	`float`	Normalized weighted sum
Bayesian	`dict`	Weighted sum applied per MC sample, then summarized

Used by: Toxicity (DIDT = weighted average of DR, DTO, ASB)

`dispersion_metric(values, center)`

Measures how spread out a set of values is around their center.

Mode	Returns	How
Frequentist	`float`	Mean absolute deviation (MAD)
Bayesian	`dict`	MAD computed per MC sample, then summarized

Used by: Toxicity (DTO — toxicity rate dispersion across groups, ASB — sentiment dispersion across groups)

Frequentist Mode

The default mode. Returns a single float for every primitive — no uncertainty, no samples.

from fair_forge.statistical import FrequentistMode

mode = FrequentistMode()
mode.get_result_type()  # "point_estimate"

Usage

from fair_forge.metrics.bias import Bias
from fair_forge.metrics.agentic import Agentic
from fair_forge.metrics.toxicity import Toxicity
from fair_forge.statistical import FrequentistMode

# Bias
metrics = Bias.run(MyRetriever, guardian=LLamaGuard, config=cfg)
# metric.attribute_rates[0].rate       → 0.12  (float)
# metric.attribute_rates[0].ci_low     → None
# metric.attribute_rates[0].ci_high    → None

# Agentic
metrics = Agentic.run(MyRetriever, model=judge, k=3)
# metric.pass_at_k                     → 0.973  (float)
# metric.pass_at_k_ci_low              → None
# metric.pass_at_k_ci_high             → None

# Toxicity
metrics = Toxicity.run(MyRetriever, group_prototypes=prototypes)
# metric.group_profiling.frequentist.DIDT  → 0.124  (float)
# metric.group_profiling.bayesian          → None

When to Use

Large datasets (100+ samples) where point estimates are reliable
Production systems where speed matters
Quick exploratory analysis

Bayesian Mode

Returns full posterior distributions. Every primitive produces mean, ci_low, ci_high, and raw samples (MC draws). The CI width reflects how much uncertainty remains given the observed data.

from fair_forge.statistical import BayesianMode

mode = BayesianMode(
    mc_samples=5000,      # Number of Monte Carlo draws
    ci_level=0.95,        # Credible interval level
    dirichlet_prior=1.0,  # Prior for distribution_divergence (uniform)
    beta_prior_a=1.0,     # Prior for rate_estimation (uninformative)
    beta_prior_b=1.0,
    rng_seed=42,          # For reproducibility
)
mode.get_result_type()  # "distribution"

BayesianMode Parameters

Parameter	Type	Default	Description
`mc_samples`	`int`	`5000`	Monte Carlo samples for posterior approximation
`ci_level`	`float`	`0.95`	Credible interval level (e.g. 0.95 for 95% CI)
`dirichlet_prior`	`float`	`1.0`	Symmetric Dirichlet prior for `distribution_divergence`
`beta_prior_a`	`float`	`1.0`	Beta prior α for `rate_estimation`
`beta_prior_b`	`float`	`1.0`	Beta prior β for `rate_estimation`
`rng_seed`	`int \| None`	`42`	Seed for reproducibility (`None` = random)

Usage

from fair_forge.metrics.bias import Bias
from fair_forge.statistical import BayesianMode

metrics = Bias.run(
    MyRetriever,
    guardian=LLamaGuard,
    config=cfg,
    statistical_mode=BayesianMode(mc_samples=5000, ci_level=0.95),
)

for rate in metrics[0].attribute_rates:
    print(f"{rate.protected_attribute}:")
    print(f"  bias rate = {rate.rate:.3f}  [{rate.ci_low:.3f}, {rate.ci_high:.3f}]")

from fair_forge.metrics.agentic import Agentic
from fair_forge.statistical import BayesianMode

metrics = Agentic.run(
    MyRetriever,
    model=judge,
    k=3,
    statistical_mode=BayesianMode(mc_samples=5000, ci_level=0.95),
)

for metric in metrics:
    print(f"pass@3 = {metric.pass_at_k:.3f}  [{metric.pass_at_k_ci_low:.3f}, {metric.pass_at_k_ci_high:.3f}]")
    print(f"pass^3 = {metric.pass_pow_k:.3f}  [{metric.pass_pow_k_ci_low:.3f}, {metric.pass_pow_k_ci_high:.3f}]")

from fair_forge.metrics.toxicity import Toxicity
from fair_forge.statistical import BayesianMode

metrics = Toxicity.run(
    MyRetriever,
    group_prototypes=prototypes,
    statistical_mode=BayesianMode(mc_samples=5000, ci_level=0.95),
)

gp = metrics[0].group_profiling
for component in ['DR', 'ASB', 'DTO', 'DIDT']:
    s = gp.bayesian.summary[component]
    print(f"{component}: {s.mean:.4f}  [{s.ci_low:.4f}, {s.ci_high:.4f}]")

When to Use

Small datasets (fewer than 50–100 samples) where point estimates can be misleading
Auditing and compliance contexts where uncertainty must be communicated
Research applications requiring rigorous statistical reporting
Any scenario where a wide CI should trigger a “collect more data” decision

How the CI Width Tells You When to Trust a Result

Small Sample (n=10)
Large Sample (n=200)
Agentic pass@K

With 10 interactions and 3 flagged as biased:

Frequentist: bias rate = 0.30 (single number, no context)
Bayesian: bias rate = 0.30 CI = [0.09, 0.57]

The wide CI tells you the true rate could be anywhere from 9% to 57%. You cannot reliably conclude there is a bias problem — collect more data first.

Priors in Bayesian Mode

Beta Prior (for `rate_estimation`)

Used in Bias (bias rate) and Agentic (success rate). The Beta(a, b) prior expresses beliefs before seeing any data.

# Uninformative — no strong prior belief (default)
BayesianMode(beta_prior_a=1.0, beta_prior_b=1.0)

# Expect a low bias rate (optimistic about fairness)
BayesianMode(beta_prior_a=1.0, beta_prior_b=9.0)  # Prior mean ≈ 0.10

# Expect a high success rate (optimistic about agent performance)
BayesianMode(beta_prior_a=9.0, beta_prior_b=1.0)  # Prior mean ≈ 0.90

Dirichlet Prior (for `distribution_divergence`)

Used in Toxicity (DR — demographic representation). The dirichlet_prior scalar sets concentration across all categories.

# Uniform prior — no preference for any group distribution (default)
BayesianMode(dirichlet_prior=1.0)

# Strong prior toward uniform distribution
BayesianMode(dirichlet_prior=10.0)

Custom Statistical Modes

Implement StatisticalMode to plug in your own strategy (e.g., Wilson score intervals, KL divergence):

from fair_forge.statistical.base import StatisticalMode
from typing import Any

class WilsonMode(StatisticalMode):
    """Frequentist mode using Wilson score intervals for rate estimation."""

    def rate_estimation(self, successes: int, trials: int) -> dict[str, Any]:
        import scipy.stats as st
        if trials == 0:
            return {"mean": 0.0, "ci_low": 0.0, "ci_high": 0.0}
        p = successes / trials
        z = st.norm.ppf(0.975)
        denom = 1 + z**2 / trials
        center = (p + z**2 / (2 * trials)) / denom
        margin = z * (p * (1 - p) / trials + z**2 / (4 * trials**2)) ** 0.5 / denom
        return {"mean": center, "ci_low": center - margin, "ci_high": center + margin}

    def distribution_divergence(self, observed, reference, divergence_type="total_variation"):
        ...

    def aggregate_metrics(self, metrics, weights):
        ...

    def dispersion_metric(self, values, center="mean"):
        ...

    def get_result_type(self) -> str:
        return "distribution"  # signals that CI fields will be populated

Next Steps

Toxicity Metric

Statistical modes with group profiling (DR, DTO, ASB, DIDT)

Bias Metric

Beta-Binomial posteriors for protected attribute bias rates

Agentic Metric

Credible intervals for pass@K and pass^K

Getting Started

Core Concepts

Metrics

Generators

Runners

Prompt Optimizer

Explainability

Statistical Modes

Statistical Modes

Which Metrics Support Statistical Modes

The Four Primitives

`rate_estimation(successes, trials)`

`distribution_divergence(observed, reference)`

`aggregate_metrics(metrics, weights)`

`dispersion_metric(values, center)`

Frequentist Mode

Usage

When to Use

Bayesian Mode

BayesianMode Parameters

Usage

When to Use

How the CI Width Tells You When to Trust a Result

Priors in Bayesian Mode

Beta Prior (for `rate_estimation`)

Dirichlet Prior (for `distribution_divergence`)

Custom Statistical Modes

Next Steps

Toxicity Metric

Bias Metric

Agentic Metric

​Statistical Modes

​Which Metrics Support Statistical Modes

​The Four Primitives

​rate_estimation(successes, trials)

​distribution_divergence(observed, reference)

​aggregate_metrics(metrics, weights)

​dispersion_metric(values, center)

​Frequentist Mode

​Usage

​When to Use

​Bayesian Mode

​BayesianMode Parameters

​Usage

​When to Use

​How the CI Width Tells You When to Trust a Result

​Priors in Bayesian Mode

​Beta Prior (for rate_estimation)

​Dirichlet Prior (for distribution_divergence)

​Custom Statistical Modes

​Next Steps

Toxicity Metric

Bias Metric

Agentic Metric

Statistical Modes

Which Metrics Support Statistical Modes

The Four Primitives

`rate_estimation(successes, trials)`

`distribution_divergence(observed, reference)`

`aggregate_metrics(metrics, weights)`

`dispersion_metric(values, center)`

Frequentist Mode

Usage

When to Use

Bayesian Mode

BayesianMode Parameters

Usage

When to Use

How the CI Width Tells You When to Trust a Result

Priors in Bayesian Mode

Beta Prior (for `rate_estimation`)

Dirichlet Prior (for `distribution_divergence`)

Custom Statistical Modes

Next Steps