Metrics Overview

Fair Forge provides six specialized metrics for comprehensive AI evaluation. Each metric focuses on a different aspect of AI behavior and quality.

Available Metrics

Toxicity

Measures toxic language with clustering and demographic group profiling using the DIDT framework.

Bias

Detects bias across protected attributes (gender, race, religion, nationality, sexual orientation).

Context

Evaluates how well responses align with provided context and instructions.

Conversational

Evaluates dialogue quality using Grice’s Maxims (Quality, Quantity, Relation, Manner).

Humanity

Analyzes emotional depth and human-likeness using the NRC Emotion Lexicon.

BestOf

Tournament-style evaluation to compare multiple assistants head-to-head.

Comparison Table

Metric	Purpose	Output Type	LLM Required
Toxicity	Detect toxic language patterns	Per-session metrics	No
Bias	Identify biased responses	Per-session metrics	Yes (Guardian)
Context	Measure context alignment	Per-interaction scores	Yes (Judge)
Conversational	Evaluate dialogue quality	Per-interaction scores	Yes (Judge)
Humanity	Analyze emotional expression	Per-interaction scores	No
BestOf	Compare multiple assistants	Tournament results	Yes (Judge)

Common Usage Pattern

All metrics follow the same usage pattern:

from fair_forge.metrics.<metric> import <Metric>
from fair_forge.core.retriever import Retriever

# 1. Define your retriever
class MyRetriever(Retriever):
    def load_dataset(self):
        # Return list[Dataset]
        pass

# 2. Run the metric
results = <Metric>.run(
    MyRetriever,
    **metric_specific_parameters,
    verbose=True,
)

# 3. Analyze results
for result in results:
    # Process metric-specific output
    pass

Metric Categories

Lexicon-Based Metrics

These metrics use predefined lexicons and don’t require external LLMs:

Toxicity: Uses Hurtlex toxicity lexicon + HDBSCAN clustering
Humanity: Uses NRC Emotion Lexicon for emotion detection

# No LLM required
from fair_forge.metrics.toxicity import Toxicity

results = Toxicity.run(
    MyRetriever,
    group_prototypes={...},
)

LLM-Judge Metrics

These metrics use an LLM as a judge to evaluate responses:

Context: Evaluates context alignment
Conversational: Evaluates dialogue quality
BestOf: Compares assistants in tournaments

# Requires LangChain-compatible model
from fair_forge.metrics.context import Context
from langchain_groq import ChatGroq

judge = ChatGroq(model="llama-3.3-70b-versatile", api_key="...")

results = Context.run(
    MyRetriever,
    model=judge,
    use_structured_output=True,
)

Guardian-Based Metrics

These metrics use specialized guardian models for detection:

Bias: Uses LlamaGuard or IBMGranite for bias detection

from fair_forge.metrics.bias import Bias
from fair_forge.guardians import LLamaGuard

results = Bias.run(
    MyRetriever,
    guardian=LLamaGuard,
    config=guardian_config,
)

Output Schemas

Each metric returns a list of result objects. The schema depends on the metric:

ToxicityMetric:
  session_id: str
  assistant_id: str
  cluster_profiling: dict[float, float]  # cluster_id -> toxicity_score
  group_profiling: GroupProfiling | None
  assistant_space: AssistantSpace

BiasMetric:
  session_id: str
  assistant_id: str
  confidence_intervals: list[ConfidenceInterval]
  guardian_interactions: dict[str, list[GuardianInteraction]]

ContextMetric:
  session_id: str
  assistant_id: str
  qa_id: str
  context_awareness: float  # 0-1
  context_insight: str
  context_thinkings: str

ConversationalMetric:
  session_id: str
  assistant_id: str
  qa_id: str
  conversational_memory: float         # 0-10
  conversational_language: float       # 0-10
  conversational_quality_maxim: float  # 0-10
  conversational_quantity_maxim: float # 0-10
  conversational_relation_maxim: float # 0-10
  conversational_manner_maxim: float   # 0-10
  conversational_sensibleness: float   # 0-10
  conversational_insight: str

HumanityMetric:
  session_id: str
  assistant_id: str
  qa_id: str
  humanity_assistant_emotional_entropy: float
  humanity_ground_truth_spearman: float
  humanity_assistant_anger: float
  humanity_assistant_anticipation: float
  # ... other emotions

BestOfMetric:
  session_id: str
  assistant_id: str
  bestof_winner_id: str
  bestof_contests: list[BestOfContest]

Installation Requirements

Each metric has specific dependencies:

# Toxicity (clustering + embeddings)
uv add "alquimia-fair-forge[toxicity]"

# Bias (guardian models)
uv add "alquimia-fair-forge[bias]"

# Context (LLM judge)
uv add "alquimia-fair-forge[context]"

# Conversational (LLM judge)
uv add "alquimia-fair-forge[conversational]"

# Humanity (included in core)
uv add "alquimia-fair-forge[humanity]"

# BestOf (LLM judge)
uv add "alquimia-fair-forge[bestof]"

# All metrics
uv add "alquimia-fair-forge[all]"

Choosing a Metric

I want to detect harmful language

Use Toxicity for detecting toxic language patterns and demographic targeting.

I want to check for bias

Use Bias for detecting discrimination across protected attributes.

I want to ensure responses follow instructions

Use Context for measuring alignment with system context.

I want to evaluate conversation quality

Use Conversational for assessing dialogue using Grice’s Maxims.

I want to measure emotional expression

Use Humanity for analyzing emotional depth and human-likeness.

I want to compare multiple assistants

Use BestOf for tournament-style head-to-head comparisons.

Next Steps

Toxicity

Learn about toxicity detection

Bias

Learn about bias detection

Context

Learn about context evaluation

Getting Started

Core Concepts

Metrics

Generators

Runners

Prompt Optimizer

Explainability

Metrics Overview

Metrics Overview

Available Metrics

Toxicity

Bias

Context

Conversational

Humanity

BestOf

Comparison Table

Common Usage Pattern

Metric Categories

Lexicon-Based Metrics

LLM-Judge Metrics

Guardian-Based Metrics

Output Schemas

Installation Requirements

Choosing a Metric

Next Steps

Toxicity

Bias

Context

​Metrics Overview

​Available Metrics

Toxicity

Bias

Context

Conversational

Humanity

BestOf

​Comparison Table

​Common Usage Pattern

​Metric Categories

​Lexicon-Based Metrics

​LLM-Judge Metrics

​Guardian-Based Metrics

​Output Schemas

​Installation Requirements

​Choosing a Metric

​Next Steps

Toxicity

Bias

Context

Metrics Overview

Available Metrics

Comparison Table

Common Usage Pattern

Metric Categories

Lexicon-Based Metrics

LLM-Judge Metrics

Guardian-Based Metrics

Output Schemas

Installation Requirements

Choosing a Metric

Next Steps