AI Feature PRD Template: Evals, Failure Modes, Guardrails and Acceptance Criteria
Use this copy-ready template to define what an AI feature should do, how its variable behavior will be evaluated, what can go wrong, which controls must intervene, and what evidence is required before release.
An AI feature PRD needs more than a user story and a happy path. It should define:
- the customer problem and intended outcome;
- the model’s role and the deterministic product behavior around it;
- representative eval cases, metrics, and release thresholds;
- expected failure modes and user-visible recovery;
- input, output, retrieval, tool, and human-approval guardrails;
- testable acceptance criteria;
- monitoring, staged rollout, rollback, and ownership.
The template below can be adapted for copilots, summarizers, recommendation systems, classifiers, conversational experiences, retrieval-augmented generation, and agents that take actions.
Why an AI Feature Needs a Different PRD
A conventional feature is often expected to produce the same output from the same input. An AI feature may produce different acceptable outputs, degrade on unfamiliar cases, depend on retrieved context, or make a confident statement that is not supported.
The PRD therefore has to define a behavioral envelope rather than one exact answer:
- What kinds of output are useful?
- What must never happen?
- Which uncertainty should be visible to the user?
- When should the system abstain, ask a question, or escalate?
- How will the team evaluate quality across representative cases?
- How will production drift or a provider change be detected?
Evals, Failure Modes, Guardrails, and Acceptance Criteria
| Layer | Question it answers | Example |
|---|---|---|
| Evals | How will we measure variable AI behavior across representative cases? | A labeled test set measures groundedness, task completion, and harmful-response rate before release. |
| Failure modes | How can the model, context, workflow, or user interaction fail? | The assistant invents a refund policy when the relevant policy document is missing. |
| Guardrails | What enforced control prevents, contains, or escalates the failure? | If no approved policy source is retrieved, the system abstains and routes the user to support. |
| Acceptance criteria | What observable behavior must the complete product experience satisfy? | Every policy answer displays its source; an unsupported answer is not shown to the user. |
These layers should connect. A high-severity failure mode should have a prevention or containment guardrail, a verification method, an owner, and an acceptance decision.
What the AI Feature PRD Should Contain
Problem, evidence, and outcome
Name the user, problem, supporting evidence, current workaround, desired outcome, and why AI is appropriate. Include a non-AI baseline where useful; the team should be able to explain why variable model behavior creates enough value to justify its cost and risk.
User workflow and model role
Describe the complete experience around the model: what triggers it, what context it receives, what it produces, what the user can inspect or edit, what action follows, and what happens if the model is unavailable.
Separate probabilistic work from deterministic product behavior. For example, drafting a response may be probabilistic; checking permissions before sending it must be deterministic.
Data and context boundaries
Specify allowed data sources, prohibited data, retention expectations, sensitive fields, tenant boundaries, freshness requirements, source precedence, and behavior when relevant context is missing or conflicting.
Evaluation plan
Define the cases, metrics, grading method, thresholds, and evaluation cadence. A useful plan includes:
- Representative cases: common, difficult, ambiguous, adversarial, multilingual, and out-of-scope examples as relevant.
- Quality dimensions: correctness, groundedness, completeness, relevance, format adherence, tone, or task success.
- Safety dimensions: harmful content, sensitive-data exposure, unauthorized action, policy violation, or unsafe advice.
- Operational dimensions: latency, availability, cost, tool-call success, and fallback rate.
- Graders: deterministic checks, human review, domain-expert review, model-based grading, or a documented combination.
- Release policy: minimum thresholds, zero-tolerance cases, regression limits, and who can accept residual risk.
Failure-mode register
For each failure, record the trigger, affected user, severity, detectability, prevention, containment, fallback, telemetry, and owner.
| Failure category | Example | Possible product response |
|---|---|---|
| Unsupported output | Answer is not grounded in an approved source | Abstain, identify missing evidence, and offer a safe next step |
| Wrong context | Content from another workspace is retrieved | Enforce tenant boundaries and block the response |
| Ambiguous input | User request could mean two materially different actions | Ask a clarifying question before proceeding |
| Tool failure | Downstream action times out after an uncertain result | Do not retry blindly; check status and show a recoverable state |
| Unsafe action | Agent attempts an irreversible change without approval | Require explicit human confirmation and narrow permissions |
| Degraded service | Model or retrieval provider is unavailable | Use an approved fallback or disable the AI path gracefully |
Guardrails and human escalation
Write guardrails as enforceable controls, not aspirations. Each guardrail should state what triggers it, what the system does, what the user sees, how it is tested, and who owns it.
- Input guardrails: validation, file limits, injection defenses, consent, and prohibited inputs.
- Retrieval guardrails: authorization, tenant isolation, approved sources, recency, and citation requirements.
- Output guardrails: schema validation, sensitive-data filtering, groundedness checks, and content policies.
- Tool guardrails: least privilege, argument validation, rate limits, idempotency, and restricted actions.
- Human guardrails: approval before high-impact actions, escalation paths, overrides, and audit records.
Acceptance criteria
Acceptance criteria cover the whole product behavior, not only the model response. Include access, context retrieval, output presentation, user control, failure recovery, logging, accessibility, and fallback behavior.
Prefer observable statements such as: “When no approved source supports the answer, the assistant states that it cannot answer and does not display a fabricated citation.”
Monitoring, rollout, and rollback
Define what the team will observe after release, which cohorts receive the feature first, what triggers a pause, how rollback works, and who responds. Track both product outcomes and guardrail health; a feature can improve engagement while increasing unsafe or unsupported outputs.
Copy-and-Paste AI Feature PRD Template
Replace the bracketed instructions. Delete sections only after deciding they are not relevant to the feature.
Filled Example: AI Support Reply Drafts
Problem and outcome
Support agents spend time searching approved policy documents and rewriting similar replies. The feature should draft a grounded response that the agent can inspect and edit, reducing handling time without sending unsupported policy claims.
Model and user boundary
- The model drafts; it never sends.
- The support agent reviews and explicitly submits the response.
- Policy claims must be grounded in approved, tenant-authorized sources.
- Refund eligibility is determined by deterministic policy logic, not model judgment.
Representative eval segments
- Common questions with one clear policy source
- Questions requiring information from multiple sources
- Missing or conflicting policy context
- Requests containing sensitive account information
- Attempts to override policy or reveal internal instructions
- Out-of-scope legal or financial advice
Illustrative release criteria
The product, support, security, and engineering owners should set thresholds based on their own risk and baseline. An illustrative policy could require:
- all high-severity policy and cross-tenant cases to pass;
- no known path that sends a reply without human action;
- quality and groundedness to meet the agreed target on every important segment, not only the overall average;
- latency and cost to remain inside the approved operating envelope;
- failed examples and accepted residual risk to be reviewed and documented.
Example failure-to-guardrail mapping
| Failure mode | Guardrail | Acceptance evidence |
|---|---|---|
| Draft invents a refund rule | Require an approved source for every policy claim; otherwise abstain | Unsupported-policy eval cases produce an abstention with no fabricated source |
| Context belongs to another tenant | Authorization-filter retrieval before model context is assembled | Cross-tenant tests return no content and record a blocked attempt |
| Agent submits an unreviewed draft | No send tool is available to the model; user action is required | Integration tests confirm drafts cannot be sent through the AI path |
| Retrieval provider is unavailable | Disable drafting and show the manual workflow | Failure simulation shows a clear fallback without a blank or invented response |
How to Write Acceptance Criteria for AI Features
Write criteria for observable product behavior under a defined state. Avoid requiring an exact sentence unless exact text is a policy requirement. Evaluate the quality envelope separately across a representative dataset.
| Weak | Better |
|---|---|
| The assistant provides accurate answers. | When an approved source supports the answer, the response identifies that source; when no approved source is available, the assistant abstains and offers the escalation path. |
| The summary is high quality. | The summary preserves the named decision, owner, deadline, and unresolved questions across the approved evaluation set without adding unsupported commitments. |
| The agent is safe. | When a requested action exceeds the user’s permission or requires human approval, the tool call is blocked and the interface explains the required next step. |
| The AI is fast. | The interface displays a progress state within the defined response window and provides the approved retry or fallback behavior when the operation exceeds it. |
For additional patterns, use the guide to writing acceptance criteria for AI.
AI Feature PRD Review Checklist
- The customer problem is supported by evidence.
- The team explains why AI is preferable to the non-AI baseline.
- The model’s responsibility is separated from deterministic system behavior.
- In-scope and out-of-scope behavior are explicit.
- Allowed, prohibited, and missing context behavior are defined.
- The eval set represents important users, tasks, difficult cases, and risks.
- Metrics, graders, thresholds, and regression limits have owners.
- Zero-tolerance cases are named where appropriate.
- High-severity failure modes have prevention, containment, and fallback plans.
- Guardrails are enforceable and tested.
- Human approval and escalation boundaries are explicit.
- Acceptance criteria cover permissions, failures, and fallback behavior.
- Production quality, safety, cost, and latency are observable.
- The rollout is staged and pause triggers are defined.
- Rollback is technically possible and has an owner.
- Changes to model, prompt, retrieval, tools, or policy trigger re-evaluation.
Common AI PRD Mistakes
- Writing “use AI” as the solution. Define the user outcome and compare it with a simpler baseline.
- Evaluating only average quality. A strong average can hide failure for a critical segment or safety case.
- Using one metric for every risk. Helpfulness, groundedness, safety, latency, and cost answer different questions.
- Treating prompt instructions as guardrails. High-impact controls should be enforced outside model discretion where feasible.
- Ignoring the surrounding workflow. Access control, user review, fallback, logging, and recovery are product requirements too.
- Launching without a re-evaluation policy. Model, prompt, data, retrieval, and tool changes can alter behavior.
- Hiding open decisions in prose. Give each decision an owner, deadline, and blocking status.
Frequently Asked Questions
What should an AI feature PRD include?
Include the normal product decision plus the model role, data boundaries, eval plan, failure modes, guardrails, human escalation, acceptance criteria, observability, rollout, rollback, and release evidence.
What is the difference between evals and acceptance criteria?
Evals measure variable AI behavior across a representative collection of cases. Acceptance criteria define the observable conditions the complete feature and workflow must satisfy. Teams usually need both.
What makes a guardrail testable?
Name the trigger, enforced intervention, user-visible behavior, telemetry, and expected evidence. “Avoid harmful output” is an aspiration; “block the response and escalate when this classifier and policy condition are met” can be verified.
Who owns AI feature quality?
Ownership is shared but should not be vague. Product owns intended behavior and outcome; engineering owns implementation and operational quality; domain, risk, security, legal, or policy owners approve relevant boundaries; named individuals own monitoring and incident response.
Can a team reuse one eval dataset forever?
No. Add production failures, new user segments, changing policies, adversarial cases, and newly discovered edge cases. Protect a stable regression set while evolving the broader evaluation suite.