Enterprise Generative AI Strategy: A Practical Framework

An enterprise generative AI strategy is a short list of use cases tied to business metrics, a shared platform every team builds on (model access through a gateway, permission-aware access to company data, evaluation and observability), a governance policy that says which data may go to which models, and a cost model finance can audit. Start with two or three internal use cases where a person reviews the output and a wrong answer is cheap. Measure them against a baseline, scale what worked, and stop what did not.

Key takeaways

• Pick use cases by value, the cost of a wrong output and who catches it, not by how good the demo looked.
• Most enterprise value comes from building on model APIs with your own data and workflows. Buy for commodity tasks; train or self-host only with a specific reason.
• Build the platform once: gateway, data access with permissions, evaluation, observability. Every later use case gets cheaper.
• Model cost is driven by context length and calls per request. Model it before launch, with per-team budgets.
• Governance should make the safe path the easy path. A policy that only says no produces shadow AI.

What a GenAI strategy actually contains

A useful strategy fits on a few pages and every part has an owner.

If a section of your strategy has no owner and no artifact, it will not happen.

Choosing use cases

Score candidates on five criteria

Start where review already exists

The single most useful lens is: what happens when the output is wrong, and who notices?

Common first wins sit in the top-left: internal knowledge search over policies and documentation, drafting replies for support agents, summarizing long documents or calls for a person who then acts, extracting fields from documents into a system with a review queue, and coding assistance.

Resist launching ten pilots. Two or three, resourced properly, with go or no-go dates, will teach you more than ten with a part-time owner each.

Build, buy, or build on APIs

Buying includes the AI features appearing inside software you already pay for: office suites, CRMs, IDEs, help desks. They are often the fastest route for commodity tasks. Read the data processing terms before switching them on.

Building on APIs is where most differentiated value sits: your data, your workflow, your controls. Fine-tuning and self-hosting are for specific reasons such as high volume on a narrow task, latency, cost at scale or data residency, not a default. Our general build vs buy framework applies here with one twist: the AI market moves fast enough that today's build may be next year's commodity feature, so build thin and keep the expensive parts portable.

The platform layer: build it once

The second use case should be much cheaper than the first. That only happens if the first one leaves shared infrastructure behind.

• Model gateway. One entry point for every model call: credentials, routing and fallback between providers, rate limits, per-team budgets, logging and redaction. It also makes switching models a configuration change instead of a code change. Treat it as critical infrastructure: it holds every provider key and sees every prompt. The LiteLLM incidents of 2026 show what that means in practice.
• Permission-aware retrieval. A shared service that indexes company content and returns only what the asking user may read, using the source systems' existing permissions. Our RAG architecture guide covers how to build it.
• Evaluation harness. A golden set of real inputs and expected outcomes per use case, run on every prompt or model change. Without it, every model upgrade is a guess.
• Observability. Traces per request, latency, errors, cost per use case and team.
• Versioned prompts and configuration, deployed through the same review process as code.

Cost modeling

GenAI costs behave differently from per-seat software. They scale with usage, and the biggest driver is how much text you send per call, multiplied by how many calls each user request makes.

A model you can actually reason with:

from dataclasses import dataclass
 
 
@dataclass
class UseCase:
    name: str
    requests_per_day: int
    input_tokens: int          # system prompt + retrieved context + history
    output_tokens: int
    calls_per_request: int = 1  # agents and multi-step chains make several calls
    cached_share: float = 0.0   # share of input tokens served from a prompt cache
 
 
def monthly_model_cost(uc: UseCase, price_in: float, price_out: float,
                       cached_discount: float = 0.9, days: int = 30) -> float:
    """Prices are per million tokens. Read them from your provider's current price list."""
    calls = uc.requests_per_day * uc.calls_per_request * days
    effective_in = uc.input_tokens * (1 - uc.cached_share * cached_discount)
    return calls * (effective_in * price_in + uc.output_tokens * price_out) / 1_000_000
 
 
PRICE_IN, PRICE_OUT = 1.00, 5.00  # placeholders for illustration, not any vendor's prices
 
for uc in [
    UseCase("support assist, lean context", 10_000, 3_000, 300),
    UseCase("support assist, stuffed context", 10_000, 12_000, 300),
    UseCase("same, with prompt caching", 10_000, 12_000, 300, cached_share=0.7),
    UseCase("agent, 6 calls per request", 10_000, 8_000, 300, calls_per_request=6),
]:
    print(f"{uc.name:34} ${monthly_model_cost(uc, PRICE_IN, PRICE_OUT):>10,.0f} / month")

With those placeholder prices it prints:

support assist, lean context       $     1,350 / month
support assist, stuffed context    $     4,050 / month
same, with prompt caching          $     1,782 / month
agent, 6 calls per request         $    17,100 / month

The absolute numbers depend entirely on your prices and volumes. The ratios are the point: the same use case costs three times as much when retrieval stuffs the prompt, and an agent that makes six calls per request costs an order of magnitude more than a single call. Set cached_discount from your provider's cache pricing.

Levers, roughly in order of effort:

Model cost is rarely the whole bill. Embeddings, vector storage, the gateway and observability stack, and above all the engineering time for evaluation and maintenance belong in the model. So do per-team budgets with alerts, set before launch.

Governance that enables

Good governance makes the safe path the easy one.

Map data classes to approved models

The rest of the policy

• A sanctioned tool that people actually want to use. If the approved assistant is worse than the free one on someone's phone, the free one wins. Shadow AI is a product problem as much as a policy problem.
• An inventory of AI applications: owner, models, data classes touched, risk tier, evaluation status, last review date.
• Human review proportional to risk. Automated checks for low-risk internal content; human approval before customer-facing or regulated outputs.
• Regulatory mapping. Transparency duties for chatbots and AI-generated content have applied in the EU since 2 August 2026, apart from a grace period to 2 December 2026 for machine-readable marking by generative systems placed on the market before 2 August 2026; our post on the EU AI Act changes of 2 August 2026 covers what engineering teams must ship. Sector rules (finance, health, employment) add their own.
• AI incidents in the incident process. Data leaked through a prompt, a harmful output, an agent that took a wrong action, a prompt injection. Define severity levels and who gets paged.
• A review cadence. Models, vendors and rules change every quarter. So should the policy.

Security belongs in the same conversation: prompt injection through documents and emails, over-permissioned connectors, and agents with write access to production systems are the new attack surface, and they should go through the same threat modeling as any other integration.

Measuring ROI

Most GenAI pilots fail at measurement, not technology.

• Baseline first. Measure the current process before launch: handle time, throughput, error rate, cycle time, cost per unit. If you cannot measure today, you cannot prove tomorrow.
• Use a holdout. Roll out by team or region and keep a comparable group without the tool for a period. Compare the same metrics.
• Measure quality and rework, not just speed. Drafts that need heavy editing, or answers that generate follow-up tickets, can erase the time saved.
• Distrust self-reported time savings. People overestimate them. Use system data where possible.
• Count the full cost: tokens, platform, engineering, licenses, training and change management.

Report in business terms. A leadership team cares about cost per resolved ticket, not evaluation scores. Set kill criteria in advance too: a pilot that misses its target by its go or no-go date stops, and its lessons go into the next one.

A practical rollout plan

Days 0 to 30: foundations. Choose two or three use cases with named owners and measured baselines. Stand up the gateway with logging and budgets. Publish governance version one: data classes, approved tools, review rules. Build a golden set for each use case.

Days 30 to 60: pilots. Build on the shared platform. Run evaluations in CI. Release to a limited group with a holdout. Review metrics weekly with the business owner.

Days 60 to 90: decide. Go or no-go per use case against the baseline. Scale the winners. Write down the platform patterns that worked so the next team starts from them. Pick the next wave.

After that: operate. Re-evaluate models when new ones ship, using the golden sets. Review cost per use case monthly. Retire what does not pay. Grow internal skills through the teams that shipped, not through slide decks.

Who builds and who runs it

A small central platform team owns the gateway, retrieval service, evaluation tooling and governance tooling. Product and business teams own their use cases end to end, including the metrics and the on-call. Avoid a central "center of excellence" that must approve and build everything: it becomes the bottleneck, and the teams closest to the work stop proposing ideas.

What goes wrong in production

• Pilot purgatory. No owner, no metric, no decision date. The pilot runs forever and never ships or stops.
• Demo-driven selection. The use case that looked best in a ten-minute demo is rarely the one with measurable value and tolerable errors.
• Governance that only says no. People use personal accounts instead, with company data and no logging.
• Cost surprises. Context grows, an agent loops, a batch job runs on the most expensive model. No per-team budget, no alert, a large invoice.
• Lock-in through convenience. Prompts, data and evaluations built around one vendor's proprietary features make switching expensive. Keep them portable behind the gateway.
• Model deprecations. A provider retires the model your prompts were tuned for. Without golden sets you cannot tell whether the replacement is better or worse.
• Tools nobody opens. A separate AI portal sees initial curiosity and then silence. Put capabilities inside the tools people already work in.
• No production owner. A GenAI application is a production system. It needs monitoring, runbooks and someone on call.

Frequently Asked Questions

What should an enterprise generative AI strategy include?

It should name a small set of use cases with owners and baseline metrics, a shared platform that every team builds on, a governance policy that maps data sensitivity to approved models and tools, and a cost model finance can audit. It also needs a measurement plan with go or no-go dates for each pilot. A strategy that is only a list of tools or a vision statement is not a strategy.

How do you choose generative AI use cases?

Score candidates on measurable value, the cost of a wrong output and who would catch it, data readiness, fit with tools people already use, and whether a business owner will run it. Start where a person already reviews the output and mistakes are cheap, such as drafting, summarization, internal search and agent assist. Leave autonomous, customer-facing decisions in regulated areas until you have evaluation and controls in place.

Should an enterprise build or buy generative AI?

Buy when an existing product already covers the use case well and the vendor's data terms are acceptable. Build on model APIs when you need your own data, workflow integration or control over the experience, which is where most enterprise value sits. Fine-tune or self-host models only when there is a clear data, cost, latency or residency reason that the APIs cannot meet.

How do you measure the ROI of generative AI?

Measure the current process before launch, then compare users with the tool against a holdout group on the same operational metrics, such as handle time, throughput, error rate and rework. Count the full cost, including tokens, platform, people and change management, not just the model bill. Self-reported time savings overstate results, so prefer measured outcomes.

How much does generative AI cost to run in production?

Model cost is roughly requests times calls per request times tokens per call times the price per token, plus embeddings, vector storage and infrastructure. Context length and the number of calls per request dominate, which is why agents and stuffed prompts are expensive and why retrieval quality, caching and routing to smaller models matter. Engineering time for evaluation and maintenance is often the larger line item and should be in the model too.

How do you govern employee use of generative AI tools?

Classify data, map each class to approved tools and models, and give people a sanctioned tool that is good enough to use, because banning AI without an alternative creates shadow AI. Keep an inventory of AI applications with owners and risk tiers, require human review in proportion to risk, and log usage through a central gateway. Review the policy on a regular cadence as tools and regulations change.

Need help turning GenAI strategy into running systems?

Strategy is the easy part to write and the hard part to ship. Our senior engineers build the gateway, retrieval, evaluation and governance plumbing, take pilots to production against measured baselines, and stay on the pager for what we build. If we built it, we're on the pager.

• AI and machine learning services
• POC-to-production engineering
• Talk to an engineer