What Is MLOps? A Practical Introduction for Engineers

MLOps is the engineering discipline that makes machine learning models reproducible, releasable and observable in production. It extends DevOps to cover two things DevOps never had to manage: data that changes independently of code, and models whose correctness is statistical rather than pass or fail. In practice that means versioned data and models, automated training and evaluation pipelines with release gates, controlled rollouts, and monitoring for drift and delayed labels.

This introduction explains what each of those pieces does, shows a minimal setup that a small team can run from one Git repository, and lists the failures that show up once a model is live.

Key takeaways

• ML has three things that change: code, data and the model. DevOps versions and tests only the first. MLOps adds the other two.
• ML failures are silent. A broken model still returns HTTP 200 with a well-formed answer. Without evaluation gates and monitoring, nobody notices.
• Start with practices, not a platform. Versioned data, tracked runs, a CI evaluation gate and basic monitoring cover most of the value for the first models.
• Automate retraining before you automate promotion. A model should never reach users without passing the same gate a human-built candidate would.
• Every production model needs a named owner. Someone who is paged when it degrades and who decides when it retires.

Why DevOps alone is not enough

Software teams already have version control, CI/CD, automated tests and rollbacks. ML uses all of that and still breaks in new ways, because it has more than one source of change.

The second difference is how failure looks. A bug in ordinary code usually throws, times out or fails a test. A model trained on a corrupted batch, or fed a feature computed differently from training, keeps answering, and its answers keep looking plausible. Testing therefore moves from "does this output equal that value" to "does this model perform at least as well as the current one, on data it has not seen, across the segments we care about".

The MLOps maturity levels

Google Cloud's widely cited MLOps architecture guide describes three levels. They are a useful map of where a team is and what the next step costs.

Most teams should aim for level 1 on any model that matters to the business, and move to level 2 only when many models or frequent pipeline changes make manual pipeline releases the bottleneck.

The core components

You do not need all of these on day one. The feature store in particular is a scaling tool, not a starting requirement.

A minimal MLOps setup that works

A practical starting point for a small team uses three things it may already have: Git, DVC for data and pipeline versioning, and a CI workflow that retrains and gates every change. MLflow can be added for richer experiment tracking, but this setup works without it.

churn-model/
  data/
    raw.dvc              DVC pointer to the raw snapshot in object storage
  src/
    prepare.py
    train.py
    evaluate.py          writes metrics.json
  ci/
    gate.py
  dvc.yaml               pipeline definition
  dvc.lock               exact hashes of every input and output
  params.yaml
  metrics.json           committed, so main always holds the champion's numbers
  .github/workflows/model-ci.yml

The pipeline lives in dvc.yaml. DVC hashes every dependency and reruns only the stages whose inputs changed, so dvc repro on a clean checkout rebuilds exactly the model that dvc.lock describes.

stages:
  prepare:
    cmd: python src/prepare.py
    deps:
      - src/prepare.py
      - data/raw
    outs:
      - data/prepared
  train:
    cmd: python src/train.py
    deps:
      - src/train.py
      - data/prepared
    params:
      - train.learning_rate
      - train.max_iter
    outs:
      - models/model.pkl
  evaluate:
    cmd: python src/evaluate.py
    deps:
      - src/evaluate.py
      - models/model.pkl
      - data/prepared
    metrics:
      - metrics.json:
          cache: false

cache: false keeps metrics.json in Git rather than in the DVC cache, which is what makes the gate below possible: the version on main is always the champion's result.

The day-to-day commands are few:

dvc remote add -d storage s3://acme-ml-artifacts/churn
dvc repro                 # rebuild whatever changed
dvc push                  # upload data and model outputs to the remote
dvc metrics diff main     # compare this branch's metrics with main

The gate is a short script that fails CI when the candidate regresses:

"""ci/gate.py: fail the build if the candidate is worse than main."""
import json
import sys
 
TOLERANCE = 0.005
WATCHED = ["pr_auc", "recall_at_precision_90", "pr_auc_new_customers"]
 
candidate = json.load(open(sys.argv[1]))
baseline = json.load(open(sys.argv[2]))
 
failures = [
    f"{name}: {candidate[name]:.4f} vs main {baseline[name]:.4f}"
    for name in WATCHED
    if candidate[name] < baseline[name] - TOLERANCE
]
if failures:
    print("Evaluation gate failed:\n  " + "\n  ".join(failures))
    sys.exit(1)
print("Evaluation gate passed")

Note the slice metric (pr_auc_new_customers). Overall metrics hide regressions in small segments, and small segments are often where complaints and compliance questions come from.

The workflow ties it together. It authenticates to the artifact bucket with OIDC rather than stored keys, pulls the pinned data, rebuilds, and gates.

name: model-ci
on:
  pull_request:
    paths: ["src/**", "params.yaml", "dvc.yaml", "dvc.lock", "data/*.dvc"]
 
permissions:
  id-token: write
  contents: read
 
jobs:
  train-and-gate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
        with:
          fetch-depth: 0
      - uses: actions/setup-python@v7
        with:
          python-version: "3.12"
      - uses: aws-actions/configure-aws-credentials@v6
        with:
          role-to-assume: arn:aws:iam::123456789012:role/ml-ci
          aws-region: us-east-1
      - run: pip install -r requirements.lock "dvc[s3]"
      - run: dvc pull
      - run: dvc repro
      - run: dvc metrics diff origin/main --md >> "$GITHUB_STEP_SUMMARY"
      - run: git show origin/main:metrics.json > baseline.json
      - run: python ci/gate.py metrics.json baseline.json

That is a level 1 setup in a few files. For the CI side in more depth, including caching, environments and deployment approvals, see our guide to CI/CD with GitHub Actions. For the training and serving code that sits inside src/, see building production AI pipelines in Python.

Releasing models safely

Passing the gate means the model is better on historical data. It does not prove it behaves well on live traffic. Release in stages.

Whatever the strategy, keep the previous version deployable and make rollback one configuration change. If rolling back requires retraining, you do not have a rollback.

Monitoring in four layers

Monitor what would make someone act, at the layer where the problem first becomes visible.

1. Service health. Latency, error rate, throughput, resource use. The same SLOs as any service.
2. Data quality. Validation failures, null rates, freshness and volume of the inputs. A feed that stopped updating is a classic "model problem" that is not a model problem.
3. Drift. Live feature and score distributions compared with the training data, for example with the population stability index. This is the earliest model-level signal you get.
4. Outcome quality. Precision, recall or error against true labels, which often arrive days or weeks later.

The fourth layer needs a pipeline of its own: log every prediction with its model version, then join it to outcomes as they settle.

SELECT
  p.model_version,
  date_trunc('week', p.scored_at) AS week,
  count(*) FILTER (WHERE p.flagged AND o.is_fraud)         AS true_positives,
  count(*) FILTER (WHERE p.flagged AND NOT o.is_fraud)     AS false_positives,
  count(*) FILTER (WHERE NOT p.flagged AND o.is_fraud)     AS false_negatives
FROM predictions p
JOIN outcomes o ON o.transaction_id = p.transaction_id
WHERE o.label_settled_at IS NOT NULL
GROUP BY 1, 2
ORDER BY 2, 1;

Grouping by model version is what lets you say, after a release, whether the new model actually did better on real outcomes, which is the question the evaluation gate could only answer about the past.

Retraining: scheduled, triggered, never unattended

Retraining is how a model keeps up with the world. There are two common triggers: a schedule (weekly, monthly) and a monitoring signal (sustained drift, a drop in outcome quality). Start with a schedule; it is predictable and easy to reason about. Add triggers once you trust the monitoring that fires them.

Automating retraining is safe. Automating promotion is not, unless the retrained model passes the same evaluation gate and staged rollout as any other candidate. For decisions with legal or financial weight, keep a human approval on promotion and record who approved what.

Who owns what

Small teams collapse several roles into one person. That is fine. Leaving the last row empty is not.

Common mistakes

• Building a platform before shipping a model. Months of infrastructure with no model in production teaches the team nothing about which manual step actually hurts.
• Treating the notebook as the source of truth. Notebooks are for exploration. The model that ships comes from versioned code run by a pipeline.
• Evaluating on a random split of time-ordered data. Offline metrics look excellent because the training set contained the future. Split by time.
• Monitoring only the service. Green latency and error dashboards say nothing about whether predictions are still right.
• Promoting automatically on a successful retrain. One corrupted batch becomes a production incident.
• Ignoring feedback loops. A fraud model's blocked transactions never get labels; a recommender shapes the clicks it later trains on. Hold out a small random slice from the model's influence, or your labels drift toward the model's own opinion.
• No lineage. When an auditor or a customer asks why a decision was made, you need the model version, its training data version and the features it saw. Log them from the start.

Frequently Asked Questions

What is MLOps in simple terms?

MLOps is the set of engineering practices that lets a team ship machine learning models the way it ships software: reproducibly, through automated checks, with a way to roll back and with monitoring once they are live. It covers versioning data and models, automating training and evaluation, controlling releases and watching for drift. The goal is that a model in production is as boring to operate as any other service.

What is the difference between MLOps and DevOps?

DevOps manages code and infrastructure, where a passing test suite is strong evidence the release works. MLOps adds data and trained models, which change independently of code and whose correctness is statistical, so a model can return valid responses that are quietly wrong. MLOps therefore adds data versioning, evaluation gates on held-out data, and monitoring for drift and delayed labels on top of normal DevOps practice.

What tools are used in MLOps?

A common open-source stack is DVC or lakeFS for data versioning, pandera or Great Expectations for validation, MLflow for experiment tracking and the model registry, Airflow, Prefect or Dagster for pipelines, and FastAPI, BentoML or KServe for serving, with Evidently or NannyML for monitoring. The cloud platforms bundle equivalents in SageMaker, Vertex AI and Azure Machine Learning. Choose tools after you know which manual step hurts most, not before.

Does a small team need MLOps?

A small team needs the practices, not a platform. One model in production already justifies versioned training data, tracked experiments, an evaluation check before release, and basic monitoring, all of which fit in a Git repository and a CI workflow. Dedicated platforms, feature stores and automated retraining become worth their cost once several models or frequent releases make the manual steps the bottleneck.

What is model drift and how do you detect it?

Model drift is the decline in a model's usefulness because the world it predicts has changed, either in its inputs (data drift) or in the relationship between inputs and outcomes (concept drift). Data drift can be detected immediately by comparing live feature and score distributions with the training data, for example with the population stability index. Concept drift usually shows up only when true labels arrive, so you also need a pipeline that joins predictions with outcomes and tracks quality over time.

What is the difference between MLOps and LLMOps?

LLMOps applies the same discipline to applications built on large language models, where you usually do not train the model yourself. The versioned artifacts become prompts, retrieval configuration and the provider's model version, and the evaluation gate runs on a curated set of task examples instead of a labeled holdout table. Monitoring adds cost per call, latency, and output quality checks such as groundedness.

Need MLOps that someone actually operates?

We build ML systems for teams whose model works in a notebook and now has to work every day: pipelines, gates, rollouts and monitoring, sized to the number of models you actually have. If we built it, we are on the pager for it, which keeps the design honest.

• AI & Machine Learning practice
• POC-to-Production Engineering
• Talk to an engineer