> ## Documentation Index
> Fetch the complete documentation index at: https://fairforge.alquimia.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Statistical Modes

> Pluggable Frequentist and Bayesian statistical strategies across Fair Forge metrics

# Statistical Modes

Fair Forge metrics that perform **statistical aggregation** accept a pluggable `statistical_mode` parameter. The same computation — estimating a rate, measuring distribution divergence, aggregating sub-metrics — can be performed either as a **point estimate** (Frequentist) or as a **full posterior distribution with credible intervals** (Bayesian).

## Which Metrics Support Statistical Modes

| Metric       | Primitives Used   | What Gets a CI in Bayesian Mode   |
| ------------ | ----------------- | --------------------------------- |
| **Toxicity** | All four          | DR, ASB, DTO, DIDT                |
| **Bias**     | `rate_estimation` | Bias rate per protected attribute |
| **Agentic**  | `rate_estimation` | pass\@K and pass^K                |

## The Four Primitives

`StatisticalMode` defines four abstract primitives. Every metric composes these to compute its final result — it never contains its own statistical logic.

### `rate_estimation(successes, trials)`

Estimates a proportion from count data.

| Mode        | Returns                                            | How                                                                    |
| ----------- | -------------------------------------------------- | ---------------------------------------------------------------------- |
| Frequentist | `float`                                            | `successes / trials`                                                   |
| Bayesian    | `dict` with `mean`, `ci_low`, `ci_high`, `samples` | Beta-Binomial posterior: `Beta(1 + successes, 1 + trials - successes)` |

**Used by:** Bias (bias rate per attribute), Toxicity (toxicity rate per group), Agentic (success rate p = c/n)

### `distribution_divergence(observed, reference)`

Measures how far an observed distribution is from a reference.

| Mode        | Returns | How                                                             |             |    |
| ----------- | ------- | --------------------------------------------------------------- | ----------- | -- |
| Frequentist | `float` | Total variation distance: \`0.5 \* Σ                            | p\_i - q\_i | \` |
| Bayesian    | `dict`  | Dirichlet posterior over `p`, divergence computed per MC sample |             |    |

**Used by:** Toxicity (DR — demographic representation)

### `aggregate_metrics(metrics, weights)`

Combines multiple named sub-metrics into one weighted score.

| Mode        | Returns | How                                                 |
| ----------- | ------- | --------------------------------------------------- |
| Frequentist | `float` | Normalized weighted sum                             |
| Bayesian    | `dict`  | Weighted sum applied per MC sample, then summarized |

**Used by:** Toxicity (DIDT = weighted average of DR, DTO, ASB)

### `dispersion_metric(values, center)`

Measures how spread out a set of values is around their center.

| Mode        | Returns | How                                         |
| ----------- | ------- | ------------------------------------------- |
| Frequentist | `float` | Mean absolute deviation (MAD)               |
| Bayesian    | `dict`  | MAD computed per MC sample, then summarized |

**Used by:** Toxicity (DTO — toxicity rate dispersion across groups, ASB — sentiment dispersion across groups)

***

## Frequentist Mode

The default mode. Returns a single float for every primitive — no uncertainty, no samples.

```python theme={null}
from fair_forge.statistical import FrequentistMode

mode = FrequentistMode()
mode.get_result_type()  # "point_estimate"
```

### Usage

```python theme={null}
from fair_forge.metrics.bias import Bias
from fair_forge.metrics.agentic import Agentic
from fair_forge.metrics.toxicity import Toxicity
from fair_forge.statistical import FrequentistMode

# Bias
metrics = Bias.run(MyRetriever, guardian=LLamaGuard, config=cfg)
# metric.attribute_rates[0].rate       → 0.12  (float)
# metric.attribute_rates[0].ci_low     → None
# metric.attribute_rates[0].ci_high    → None

# Agentic
metrics = Agentic.run(MyRetriever, model=judge, k=3)
# metric.pass_at_k                     → 0.973  (float)
# metric.pass_at_k_ci_low              → None
# metric.pass_at_k_ci_high             → None

# Toxicity
metrics = Toxicity.run(MyRetriever, group_prototypes=prototypes)
# metric.group_profiling.frequentist.DIDT  → 0.124  (float)
# metric.group_profiling.bayesian          → None
```

### When to Use

* Large datasets (100+ samples) where point estimates are reliable
* Production systems where speed matters
* Quick exploratory analysis

***

## Bayesian Mode

Returns full posterior distributions. Every primitive produces `mean`, `ci_low`, `ci_high`, and raw `samples` (MC draws). The CI width reflects how much uncertainty remains given the observed data.

```python theme={null}
from fair_forge.statistical import BayesianMode

mode = BayesianMode(
    mc_samples=5000,      # Number of Monte Carlo draws
    ci_level=0.95,        # Credible interval level
    dirichlet_prior=1.0,  # Prior for distribution_divergence (uniform)
    beta_prior_a=1.0,     # Prior for rate_estimation (uninformative)
    beta_prior_b=1.0,
    rng_seed=42,          # For reproducibility
)
mode.get_result_type()  # "distribution"
```

### BayesianMode Parameters

| Parameter         | Type          | Default | Description                                             |
| ----------------- | ------------- | ------- | ------------------------------------------------------- |
| `mc_samples`      | `int`         | `5000`  | Monte Carlo samples for posterior approximation         |
| `ci_level`        | `float`       | `0.95`  | Credible interval level (e.g. 0.95 for 95% CI)          |
| `dirichlet_prior` | `float`       | `1.0`   | Symmetric Dirichlet prior for `distribution_divergence` |
| `beta_prior_a`    | `float`       | `1.0`   | Beta prior α for `rate_estimation`                      |
| `beta_prior_b`    | `float`       | `1.0`   | Beta prior β for `rate_estimation`                      |
| `rng_seed`        | `int \| None` | `42`    | Seed for reproducibility (`None` = random)              |

### Usage

<CodeGroup>
  ```python Bias theme={null}
  from fair_forge.metrics.bias import Bias
  from fair_forge.statistical import BayesianMode

  metrics = Bias.run(
      MyRetriever,
      guardian=LLamaGuard,
      config=cfg,
      statistical_mode=BayesianMode(mc_samples=5000, ci_level=0.95),
  )

  for rate in metrics[0].attribute_rates:
      print(f"{rate.protected_attribute}:")
      print(f"  bias rate = {rate.rate:.3f}  [{rate.ci_low:.3f}, {rate.ci_high:.3f}]")
  ```

  ```python Agentic theme={null}
  from fair_forge.metrics.agentic import Agentic
  from fair_forge.statistical import BayesianMode

  metrics = Agentic.run(
      MyRetriever,
      model=judge,
      k=3,
      statistical_mode=BayesianMode(mc_samples=5000, ci_level=0.95),
  )

  for metric in metrics:
      print(f"pass@3 = {metric.pass_at_k:.3f}  [{metric.pass_at_k_ci_low:.3f}, {metric.pass_at_k_ci_high:.3f}]")
      print(f"pass^3 = {metric.pass_pow_k:.3f}  [{metric.pass_pow_k_ci_low:.3f}, {metric.pass_pow_k_ci_high:.3f}]")
  ```

  ```python Toxicity theme={null}
  from fair_forge.metrics.toxicity import Toxicity
  from fair_forge.statistical import BayesianMode

  metrics = Toxicity.run(
      MyRetriever,
      group_prototypes=prototypes,
      statistical_mode=BayesianMode(mc_samples=5000, ci_level=0.95),
  )

  gp = metrics[0].group_profiling
  for component in ['DR', 'ASB', 'DTO', 'DIDT']:
      s = gp.bayesian.summary[component]
      print(f"{component}: {s.mean:.4f}  [{s.ci_low:.4f}, {s.ci_high:.4f}]")
  ```
</CodeGroup>

### When to Use

* Small datasets (fewer than 50–100 samples) where point estimates can be misleading
* Auditing and compliance contexts where uncertainty must be communicated
* Research applications requiring rigorous statistical reporting
* Any scenario where a wide CI should trigger a "collect more data" decision

***

## How the CI Width Tells You When to Trust a Result

<Tabs>
  <Tab title="Small Sample (n=10)">
    With 10 interactions and 3 flagged as biased:

    * **Frequentist**: bias rate = 0.30 (single number, no context)
    * **Bayesian**: bias rate = 0.30  **CI = \[0.09, 0.57]**

    The wide CI tells you the true rate could be anywhere from 9% to 57%. You cannot reliably conclude there is a bias problem — collect more data first.
  </Tab>

  <Tab title="Large Sample (n=200)">
    With 200 interactions and 60 flagged:

    * **Frequentist**: bias rate = 0.30
    * **Bayesian**: bias rate = 0.30  **CI = \[0.24, 0.36]**

    The narrow CI confirms the estimate is reliable. A 30% bias rate is now a well-supported finding.
  </Tab>

  <Tab title="Agentic pass@K">
    With 5 conversations, 4 fully correct, k=3:

    * **Frequentist**: pass\@3 = 0.992
    * **Bayesian**: pass\@3 = 0.960  **CI = \[0.740, 0.999]**

    The number sounds great — but the CI says it could be as low as 0.74. With only 5 conversations, the point estimate is not trustworthy.
  </Tab>
</Tabs>

***

## Priors in Bayesian Mode

### Beta Prior (for `rate_estimation`)

Used in Bias (bias rate) and Agentic (success rate). The `Beta(a, b)` prior expresses beliefs before seeing any data.

```python theme={null}
# Uninformative — no strong prior belief (default)
BayesianMode(beta_prior_a=1.0, beta_prior_b=1.0)

# Expect a low bias rate (optimistic about fairness)
BayesianMode(beta_prior_a=1.0, beta_prior_b=9.0)  # Prior mean ≈ 0.10

# Expect a high success rate (optimistic about agent performance)
BayesianMode(beta_prior_a=9.0, beta_prior_b=1.0)  # Prior mean ≈ 0.90
```

### Dirichlet Prior (for `distribution_divergence`)

Used in Toxicity (DR — demographic representation). The `dirichlet_prior` scalar sets concentration across all categories.

```python theme={null}
# Uniform prior — no preference for any group distribution (default)
BayesianMode(dirichlet_prior=1.0)

# Strong prior toward uniform distribution
BayesianMode(dirichlet_prior=10.0)
```

***

## Custom Statistical Modes

Implement `StatisticalMode` to plug in your own strategy (e.g., Wilson score intervals, KL divergence):

```python theme={null}
from fair_forge.statistical.base import StatisticalMode
from typing import Any

class WilsonMode(StatisticalMode):
    """Frequentist mode using Wilson score intervals for rate estimation."""

    def rate_estimation(self, successes: int, trials: int) -> dict[str, Any]:
        import scipy.stats as st
        if trials == 0:
            return {"mean": 0.0, "ci_low": 0.0, "ci_high": 0.0}
        p = successes / trials
        z = st.norm.ppf(0.975)
        denom = 1 + z**2 / trials
        center = (p + z**2 / (2 * trials)) / denom
        margin = z * (p * (1 - p) / trials + z**2 / (4 * trials**2)) ** 0.5 / denom
        return {"mean": center, "ci_low": center - margin, "ci_high": center + margin}

    def distribution_divergence(self, observed, reference, divergence_type="total_variation"):
        ...

    def aggregate_metrics(self, metrics, weights):
        ...

    def dispersion_metric(self, values, center="mean"):
        ...

    def get_result_type(self) -> str:
        return "distribution"  # signals that CI fields will be populated
```

***

## Next Steps

<CardGroup cols={3}>
  <Card title="Toxicity Metric" icon="flask" href="/metrics/toxicity">
    Statistical modes with group profiling (DR, DTO, ASB, DIDT)
  </Card>

  <Card title="Bias Metric" icon="scale-balanced" href="/metrics/bias">
    Beta-Binomial posteriors for protected attribute bias rates
  </Card>

  <Card title="Agentic Metric" icon="robot" href="/metrics/agentic">
    Credible intervals for pass\@K and pass^K
  </Card>
</CardGroup>