An AI SaaS should solve a specific task at an observable quality level. Choose the model after defining data boundaries, acceptable errors, human oversight and usage economics.
Start with a bounded job
“Add an assistant” is not a feature specification. Describe its inputs, outputs and the person using the result. Summarising internal notes, drafting a customer response and executing a business action have different consequences when something goes wrong.
Choose an initial use case whose output can be checked. Human review can be part of the product if it is clear and workable. Do not hide necessary verification behind a promise of autonomy. The user should understand what the system has proposed and what it has actually done.
Establish data boundaries
Identify authorised sources, organisation-level permissions and information that must not be transmitted to an external provider. A document retrieval workflow must enforce access before passing content to the model. A plausible answer is never a reason to bypass tenant isolation.
Retrieved content is untrusted as an instruction source. Documents may contain requests to redirect the system. Separate what the model may read from what it may do. Enforce action permissions in server-side code rather than relying only on a prompt to keep behaviour within bounds.
Build an evaluation set
Collect representative examples: frequent requests, ambiguity, missing information and difficult cases. Define evaluation criteria before comparing variants. Check fields for extraction, source support for document answers, and parameters plus permissions for actions.
Retain results so regressions are visible. A model, prompt or retrieval change may improve one group of examples while degrading another. The best demonstration should not become the only production gate. Evaluate the workflow as users will encounter it, including incomplete and messy inputs.
Design failure behaviour
| Situation | Behaviour to design |
|---|---|
| Missing information | Explain the limitation and ask for clarification |
| Conflicting sources | Surface the conflict instead of guessing |
| Consequential action | Require review appropriate to the risk |
| Provider unavailable | Preserve work and allow recovery |
| Invalid output | Reject or repair under a controlled rule |
Users need an accurate account of what happened. Error messages must not imply that an action succeeded when it did not. Repeated operations also need a duplicate-prevention strategy. These are product requirements, not just model settings.
Model cost and waiting time
Estimate calls per task, input size, retries and usage frequency. Avoid unnecessary processing and retrieve relevant material rather than everything available. A cheaper path that weakens quality on critical tasks may be a poor tradeoff.
Design the waiting experience: progress, cancellation where possible and recovery after interruption. Do not block a simple action behind an expensive workflow if some work can be handled deterministically. Review cost and quality together as usage changes.
A practical example
A project copilot might first propose a summary of meeting notes with links to the supporting passages. The user reviews and publishes it. Automatically sending instructions to teams is a separate risk level and needs its own scope, recipient checks and content controls.
This separation creates a useful release sequence. You can evaluate summarisation before granting action capabilities, rather than treating all assistant behaviour as one feature. The boundary should remain visible in the interface and in operational logs.
Create a representative evaluation set
For a fictional request-triage assistant, collect straightforward, ambiguous, incomplete and irrelevant inputs. Describe the expected output and when abstaining is preferable to guessing. Separate development examples from a held-out evaluation set. Otherwise, apparent progress may reflect tuning to questions the system has already seen rather than improvement on new work.
Assess several dimensions: category accuracy, output format, useful justification, latency and cost per completed task. Valid JSON can still contain the wrong answer. Asking for clarification may be the correct outcome when information is missing. Have someone familiar with the workflow review failures; an average score cannot express the consequence of every mistake.
Separate suggestions from actions
A drafted reply can be reviewed before sending. Changing a customer record requires additional controls. Specify what the system may read, recommend and execute. The application must enforce permissions at access time regardless of what the model says it is authorised to do. Imported documents are task data, not a source of new operating permissions.
| Situation | Expected behaviour |
|---|---|
| Missing or contradictory evidence | State the limitation and request clarification |
| Out-of-scope request | Decline the action and explain available next steps |
| Uncertain result | Require review or use the manual workflow |
| Provider unavailable | Preserve the task without duplicate side effects |
Measure cost per accepted outcome
The price of one model call is not the cost of the service. Include retrieval, retries, document processing and human review. During a pilot, distinguish tasks accepted without correction from tasks requiring rework. Compare the resulting effort with the existing process using equivalent quality requirements and boundaries. A fast draft that requires complete rewriting is not necessarily an improvement.
Maintain regression examples before changing the model, prompt or document source. A configuration that improves one category can harm another. Keep a rollback path and enough version information to explain how an output was produced, while avoiding unnecessary retention of sensitive inputs. Operational discipline matters as much as a compelling initial demonstration.
Workshop: change models without a silent product regression
An AI product keeps changing after launch. Models, instructions, reference material and tools may all evolve. Prepare a comparison procedure before the next change. This illustrative exercise concerns an assistant that drafts support replies; it assumes no particular success rate and does not replace safeguards required in sensitive domains.
Keep representative situations with explicit acceptance criteria. For each case, record the available information, claims the assistant may make, clarification it should request and actions it must not perform. One reference sentence is not always sufficient: several answers may be acceptable, but none should claim that a refund has happened when it has not.
| Replay case | Comparison criterion |
|---|---|
| Sufficient information | Accurate, useful reply grounded in the correct document |
| Missing or contradictory source | Explicit uncertainty and appropriate human referral |
| Hostile instruction in an attachment | No compliance with instructions outside the assigned role |
| Another customer's data | No access or disclosure beyond the authorised scope |
| External service failure | Failure reported without claiming an unperformed action |
Run the same situations against the previous and proposed configurations. Record model, instruction and reference versions. When repeated runs vary, preserve that variability rather than selecting only the best answer. Have consequential disagreements reviewed by someone who understands the relevant business process. Keep evaluation examples separate from material used to tune the configuration where possible, so apparent improvement is not merely familiarity with the test.
Separate blocking defects from style preferences. A less elegant phrase and an unauthorised disclosure should not cancel each other out in an average score. Release criteria depend on the product’s actual risk, not a universal threshold. State who is responsible for accepting remaining limitations and how those limitations are communicated to users.
Prepare rollback before deployment. Identify the configuration to restore, how to stop automated actions and how to find tasks affected during the relevant period. Restoring a model does not reverse changes already made in an external system. Permissions and human confirmation therefore remain important before execution, even when rollback of the AI configuration itself is straightforward.
The deliverable is a change record: replayed cases, observed differences, decisions, deployed version and withdrawal procedure. This makes improvement possible without quietly changing the product promise whenever an apparently minor instruction is edited. It also gives support staff a useful explanation when a customer reports different behaviour after an update.
Frequently asked questions
Do we need an autonomous agent?
Only if the task genuinely requires a sequence of decisions and actions. A deterministic workflow around a model call may be easier to evaluate and supervise.
Does retrieval eliminate errors?
No. Retrieval can improve access to useful information, but sources, permissions, excerpts and answers still need evaluation. A displayed citation does not prove that the conclusion follows from it.
What should discovery produce?
A bounded use case, representative examples, quality criteria, a data-flow map and a list of permitted actions. This supports a product discussion beyond the impressive demo.
Sources and further reading
From this guide to your project
AI ↗
We start with a task and a quality criterion. AI belongs where it meaningfully improves the service.
Tell us about your ideaExamples are educational and do not describe client results. Content prepared with AI assistance; see our editorial policy. Our method ↗