Bias and Fairness in Machine Learning: A Practical Guide

Bias in a machine learning system is a measurable difference in how its outcomes or errors fall across groups of people, and you find it by slicing evaluation metrics by group, not by inspecting the model. Choose the fairness metric that matches the harm you are trying to prevent (equal selection rates, equal error rates, or equal calibration), knowing that when base rates differ between groups you cannot satisfy all of them at once. Then fix causes in the data and labels where you can, constrain training or adjust decisions where you cannot, and keep measuring in production, because populations and feedback loops shift.

Key takeaways

• Aggregate accuracy hides group-level failure. Every evaluation should report metrics per group and per intersection of groups, with confidence intervals.
• Fairness metrics encode value judgments and conflict with each other. Pick one deliberately, write down why, and have someone accountable sign off.
• The most damaging bias usually comes from the label, not the features. A proxy target such as past cost or past decisions bakes history into the model.
• Removing protected attributes does not remove bias. Correlated features carry the same signal, and you lose the ability to measure the gap.
• LLM features need their own tests: counterfactual inputs that change only a name or demographic cue, repeated enough times to beat sampling noise.
• Fairness is an operational property. Gate releases on it and monitor it the way you monitor latency.

Where bias enters the pipeline

The lesson worth remembering: the most damaging bias usually arrives through the label. When the target variable is a past human decision (who was hired, who was approved, who was flagged), the model learns to reproduce the decision, not the underlying quality you actually care about.

Choosing a fairness metric

These metrics conflict mathematically. Kleinberg, Mullainathan and Raghavan (2016) and Chouldechova (2017) showed that when base rates differ between groups, a classifier that is not perfect cannot be calibrated across groups and also have equal false positive and false negative rates. The 2016 ProPublica analysis of the COMPAS recidivism tool and the vendor's response were, in effect, two sides arguing for different metrics. Neither side had miscalculated.

So the choice is a policy decision, not a technical one. Write it down: which metric, why it matches the harm, what disparity you will tolerate, and who approved it. In US employment, a common screening heuristic is the four-fifths rule from the federal Uniform Guidelines on Employee Selection Procedures (1978): a selection rate for any group below 80 percent of the highest group's rate was generally treated by federal enforcement agencies as evidence of adverse impact. In June 2026 the Justice Department's Office of Legal Counsel found the Uniform Guidelines unlawful, and OPM has since removed references to them from federal personnel rules, so the rule's legal weight is in doubt. It is a rule of thumb for raising a flag, not a safe harbor.

Measuring it: a worked example with Fairlearn

Fairlearn's MetricFrame slices any metric by group. Pass several sensitive columns to get intersections, and always look at the group sizes next to the metrics. In this example y_true and y_pred are the test labels and predictions, and groups is a DataFrame of sensitive attributes for the same rows.

import numpy as np
from fairlearn.metrics import (
    MetricFrame, count, selection_rate, true_positive_rate, false_positive_rate,
    demographic_parity_ratio, equalized_odds_difference,
)
from sklearn.metrics import accuracy_score
 
mf = MetricFrame(
    metrics={
        "n": count,
        "accuracy": accuracy_score,
        "selection_rate": selection_rate,
        "tpr": true_positive_rate,
        "fpr": false_positive_rate,
    },
    y_true=y_true,
    y_pred=y_pred,
    sensitive_features=groups[["sex", "age_band"]],    # intersectional slices
)
print(mf.by_group)
print(mf.difference(method="between_groups"))
 
sex = groups["sex"]
print("demographic parity ratio:", demographic_parity_ratio(y_true, y_pred, sensitive_features=sex))
print("equalized odds difference:", equalized_odds_difference(y_true, y_pred, sensitive_features=sex))

A point estimate of a gap is not enough. Small groups produce noisy numbers that look like disparities, so bootstrap an interval for the gap you care about:

def bootstrap_gap(y_true, y_pred, group, metric, a, b, n_boot=2000, seed=0):
    """95% interval for metric on group a minus metric on group b."""
    rng = np.random.default_rng(seed)
    y_true, y_pred, group = map(np.asarray, (y_true, y_pred, group))
    ia, ib = np.flatnonzero(group == a), np.flatnonzero(group == b)
    gaps = []
    for _ in range(n_boot):
        sa, sb = rng.choice(ia, ia.size), rng.choice(ib, ib.size)
        gaps.append(metric(y_true[sa], y_pred[sa]) - metric(y_true[sb], y_pred[sb]))
    return np.percentile(gaps, [2.5, 97.5])
 
 
low, high = bootstrap_gap(y_true, y_pred, sex, true_positive_rate, "female", "male")
print(f"TPR gap, female minus male: 95% interval {low:+.3f} to {high:+.3f}")

How to read it: if the whole interval sits outside your tolerance, you have a disparity worth acting on. If a group has a few dozen examples, the interval will be wide, and the honest conclusion is "we cannot tell yet". That is a finding too: you need more evaluation data for that group before you can claim anything about it.

When you do not have the sensitive attribute

Many teams cannot measure fairness because they never collected demographic data, often for good privacy reasons. The options, roughly in order of preference:

• Collect it voluntarily and separately, for auditing only, with access limited to the people running the audit and never used as a model feature.
• Run the audit through a trusted third party that holds the attributes.
• Estimate it with a proxy method, such as Bayesian Improved Surname Geocoding (BISG), which US fair lending supervisors use, and report the extra uncertainty that brings.

The EU AI Act explicitly allows processing special categories of personal data where strictly necessary to detect and correct bias, under safeguards. The 2026 Digital Omnibus moved this permission into a new Article 4a and extended it, under narrower conditions, from high-risk systems to other AI systems and models.

Mitigation: fix the cause before the symptom

Work in this order. Each step is cheaper and easier to defend than the next.

1. Fix the data and the label. Collect more examples for underrepresented groups, correct labeling processes that differ by group, and replace proxy targets with better ones. In the healthcare study above, the researchers found the bias shrank substantially when the target changed from cost to direct measures of health.
2. Reweight or resample. Give underrepresented groups appropriate weight during training. It is simple, transparent, and often enough.
3. Constrain training. Reduction methods train a model subject to a fairness constraint:

from fairlearn.reductions import ExponentiatedGradient, EqualizedOdds
from sklearn.linear_model import LogisticRegression
 
mitigator = ExponentiatedGradient(
    estimator=LogisticRegression(max_iter=1000),
    constraints=EqualizedOdds(difference_bound=0.02),
)
mitigator.fit(X_train, y_train, sensitive_features=groups_train["sex"])
y_pred_mitigated = mitigator.predict(X_test)

4. Adjust decisions after the model. Fairlearn's ThresholdOptimizer chooses group-specific thresholds to satisfy a constraint. It is the simplest method and the most legally sensitive: it needs the sensitive attribute at prediction time and treats groups differently on purpose, which some jurisdictions and sectors prohibit. Get legal review before shipping it.

Every mitigation costs something, usually overall accuracy or another fairness metric. Report the trade-off as a table (accuracy, the chosen fairness metric and the others, before and after) and let the accountable owner choose, instead of letting an optimizer choose silently.

LLM features need their own tests

When a large language model screens resumes, triages support tickets or scores leads, there is no fixed classifier to slice. There is a prompt, a model and a sampling process. The workable method is counterfactual testing: hold everything constant except a demographic cue (a name, a pronoun, an affinity group membership) and measure whether the output moves.

Use curated name and cue lists per group, reviewed by your team and ideally drawn from published audit studies, and a set of documents that vary in quality so you test across the whole score range:

import statistics
 
from scipy.stats import ttest_rel
 
 
def mean_score(template: str, cues: list[str], score_fn, runs: int = 10) -> float:
    """Mean score for one document across a group's cues, repeated to average out sampling."""
    scores = [score_fn(template.replace("[NAME]", cue)) for cue in cues for _ in range(runs)]
    return statistics.mean(scores)
 
 
def counterfactual_audit(templates, cue_sets, score_fn, a="group_a", b="group_b"):
    sa = [mean_score(t, cue_sets[a], score_fn) for t in templates]
    sb = [mean_score(t, cue_sets[b], score_fn) for t in templates]
    result = ttest_rel(sa, sb)          # paired: same document, only the cue differs
    return statistics.mean(sa) - statistics.mean(sb), result.pvalue

Practical notes from running these audits:

• Pin the model version and temperature for the audit, and rerun it when either changes. A provider's silent model update is a new model for fairness purposes.
• Test order effects as well. LLM judges are sensitive to which candidate appears first in a pairwise comparison.
• Test the full pipeline (retrieval, prompt, model, output parser, thresholds), not the model alone. Bias introduced by a retrieval step looks identical from the outside.
• Keep the counterfactual suite in CI and run it on every prompt or model change, like any other regression test.

Make fairness an operational property

Release gates

Treat a fairness regression like a performance regression: a test that fails the build. The tolerance is a policy choice, and 0.03 below is illustrative. Requiring the whole interval inside the tolerance means a group with too little evaluation data fails the gate, which is deliberate.

def test_tpr_gap_within_tolerance(eval_set, model):
    y_pred = model.predict(eval_set.X)
    low, high = bootstrap_gap(eval_set.y, y_pred, eval_set.groups["sex"],
                              true_positive_rate, "female", "male")
    assert -0.03 <= low and high <= 0.03, f"TPR gap interval {low:.3f} to {high:.3f} exceeds 0.03"

Production monitoring

• Track group-level metrics continuously, not just at launch. Population drift moves them even when the model does not change. The same pipeline that watches feature drift in your MLOps setup can compute them.
• Labels often arrive late (a loan defaults months after approval). Monitor selection rates and score distributions by group immediately, and outcome-based metrics as labels land.
• Watch for feedback loops. A model that decides who gets inspected only observes outcomes for the people it chose. Where it is ethical and lawful, a small randomized slice keeps an unbiased view.
• Store sensitive attributes for monitoring separately from model features, with access controls and an audit trail.

Documentation and ownership

• A model card for each production model: intended use, out-of-scope uses, evaluation data, per-group results, the chosen fairness metric and its rationale, known limitations.
• A datasheet for each training dataset: how it was collected, by whom, and what it underrepresents.
• A named owner who can stop a release.
• An incident response path for fairness issues raised by users, regulators or journalists: who triages, who can roll back, who communicates.

If you need an auditable structure to hang these on, the NIST AI Risk Management Framework (version 1.0, 2023) and ISO/IEC 42001:2023 for AI management systems both provide one.

The regulatory picture

• NYC Local Law 144. Employers and employment agencies using automated employment decision tools for candidates in New York City need an independent bias audit within the year before use, must publish a summary of the results including impact ratios by sex and race or ethnicity, and must notify candidates. Enforcement began in July 2023.
• EU AI Act. AI used in hiring, credit scoring, education and access to essential services is high-risk under Annex III. Providers of high-risk systems must examine training, validation and testing data for possible biases and take measures to detect, prevent and mitigate them, alongside documentation, human oversight and logging. The Digital Omnibus on AI, adopted by Parliament and Council in mid-2026, defers the Annex III obligations from August 2026 to December 2027. Our EU AI Act post tracks the dates and what already applies.
• Existing anti-discrimination law. In the US, Title VII and the Fair Housing Act already reach algorithmic decisions through disparate impact. Under the Equal Credit Opportunity Act, a CFPB rule effective in July 2026 removed disparate impact from Regulation B, though courts have the final word and state fair lending laws still apply. Credit decisions also require a statement of specific reasons for any adverse action, whatever model made the decision.

Treat compliance as the floor. A system that passes its annual audit and fails a group of users every day is still failing.

What goes wrong in production

• Evaluating only in aggregate. A model that improves overall accuracy by getting much worse on a small group ships, because nobody sliced the metrics.
• "We removed the protected attribute." Postcode, first name, school, employment gaps and purchase history carry the same signal. Removing the attribute removes your ability to measure, not the bias.
• Tiny slices treated as findings. A large gap measured on a few dozen examples is noise until proven otherwise. Report intervals.
• Fairness tested once, at launch. The population shifts, an upstream pipeline changes, the LLM provider updates the model. None of that triggers a re-audit unless you build the trigger.
• The proxy label nobody questioned. The model predicts the label accurately. The label is the problem.
• Thresholds tuned after the review. A product team tunes the decision threshold for conversion after the fairness review signs off, silently changing every group-level metric. Put the threshold under the same release gate as the model.
• Human review as a fig leaf. Reviewers who approve nearly every recommendation under time pressure add latency, not oversight. Measure override rates, by group.

Frequently Asked Questions

What is bias in machine learning?

Bias in machine learning is a systematic difference in how a model's outcomes or errors are distributed across groups of people, usually defined by attributes such as sex, race or age. It typically comes from historical decisions encoded in the training labels, underrepresentation of some groups in the data, or proxy targets that measure different things for different groups. It is detected by measuring performance and outcomes separately for each group.

How do you measure fairness in a machine learning model?

Compute your evaluation metrics separately for each group and each intersection of groups, then compare them using a fairness metric that fits the harm, such as selection rate, true positive rate or calibration. Tools like Fairlearn's MetricFrame do the slicing. Report confidence intervals, because small groups produce noisy estimates that can look like disparities.

Can a model be fair on every metric at once?

Generally no. When the base rate of the outcome differs between groups, an imperfect classifier cannot be calibrated across groups and also have equal false positive and false negative rates, as shown by Kleinberg and colleagues and by Chouldechova. Choosing a fairness metric is therefore a policy decision that should be documented and approved by someone accountable.

Does removing protected attributes remove bias?

No. Other features such as postcode, name, school or purchase history often correlate with protected attributes and carry the same signal, so the model can reproduce the disparity without ever seeing the attribute. Removing it also makes the disparity harder to measure. Keep sensitive attributes for auditing, separated from model features and access-controlled.

What is the four-fifths rule?

The four-fifths rule comes from the US Uniform Guidelines on Employee Selection Procedures. It treats a selection rate for any group below 80 percent of the rate for the most selected group as a sign of adverse impact. The Justice Department's Office of Legal Counsel found the Uniform Guidelines unlawful in June 2026, so the rule's legal weight is in doubt. It is a screening heuristic used in hiring and often applied to algorithmic tools, but passing it does not by itself prove a system is lawful or fair.

Need help building fair ML systems?

We build and run ML systems in production, and we are on the pager for them afterwards, so fairness is part of the release process for us, not a slide in a review. That means per-group evaluation in CI, counterfactual suites for LLM features, and monitoring that tells you when a group-level metric moves.

• AI and machine learning
• POC to production engineering
• Talk to an engineer