Problem-Solution Page
Your AI Prototype Is Impressive. It Is Not Production-Ready Yet.
A prototype that works in a demo is not the same as a system that works under real load, with real user data, real failure modes, and real recovery paths. The demo runs on your machine with clean test data. Production runs at 2am with a user on slow mobile who entered something you never expected.
Prototype vs Production
8 Things That Are Different in Production
Real users enter unexpected data
Your test cases had sensible inputs. Real users paste URLs into name fields, submit empty forms, enter 10,000-character strings, and use the product in ways that were never tested.
The AI model returns something unexpected
A prototype assumes the model gives the right output format. In production, the model occasionally refuses, truncates, returns null, or uses a different structure. The code must handle every case.
Latency is unpredictable under load
A single API call on your machine takes 1.2 seconds. Under real load, with 50 concurrent users and rate limits, the same call takes 8 seconds or times out. The UI needs to handle this.
API costs are not controlled
A demo makes 5 API calls per session. Production with 200 users per day makes 1,000. Without token limits, caching, and cost monitoring, the API bill grows linearly with users and can exceed revenue on launch day.
Secrets are not managed properly
In a prototype, API keys are sometimes hardcoded or in files that get committed. In production, every key must be in an environment variable, rotated on any suspected exposure, and never visible in client-side code.
No evaluation layer exists
A prototype is judged by whether the demo looked good. A production AI feature needs test cases with known expected outputs, run on every deployment, so you know when a change to the prompt or model breaks something.
User permissions are not enforced at the feature level
A prototype usually has one test user. Production has free users, paid users, admins, and users who downgraded. Every AI feature must check what the current user is allowed to access before running.
There is no plan for when it fails
A prototype either works or shows an error. A production system needs a graceful degradation path: if the AI call fails, show the best available fallback result and log the failure so it can be investigated.
5-Point Review
What a CTO Checks Before Calling a Prototype Ready
Output validation
Every AI output that feeds into a user action or a data write must be validated before use. Check for null, check for the expected format, check for minimum length where relevant. An unvalidated AI output that reaches a database or a user-facing display is a production risk.
Every AI output has a validation step before it writes or displays. Null and error cases have defined fallback behaviour.
The code assumes the model always returns the right format. The first time it does not, the page breaks or writes corrupt data.
Evaluation test cases
Write 10 to 20 test inputs with known expected outputs for the core AI feature. Run them before each deployment. If the pass rate drops below the threshold you set, the deployment does not go to production. Without this, prompt changes, model upgrades, or data changes can silently degrade the product.
You have a set of test cases and a pass threshold. You run them before deployment and check the results.
Quality is judged by "it felt good in the demo." No test cases. No threshold. No comparison between before and after.
Cost and rate-limit controls
Set a per-user token budget. Set a per-day API cost ceiling. Add a rate limit on AI-heavy endpoints. Check whether your current usage would still be affordable at 10x the current user count. If the answer is no, the cost structure needs work before more users arrive.
You can estimate the API cost per active user per month and confirm it stays below your unit economics target at scale.
You do not know the cost per user. No rate limits. No ceiling. The first traffic spike will cause a surprise bill.
Fallback and graceful degradation
For each AI feature, define what the product does when the AI call fails or times out. A summary feature that returns an empty screen on failure is worse than a feature that shows the raw data with a note that the summary is temporarily unavailable. Define the fallback before launch.
Each AI feature has a documented fallback. The fallback has been tested deliberately and works.
The fallback is "show an error." That is not a fallback. That is an unhandled failure.
Monitoring and alert design
Set up error rate monitoring on AI endpoints. Alert when the error rate exceeds 5% in a rolling 15-minute window. Track average latency and alert when it exceeds your defined threshold. Without monitoring, production failures stay invisible until users complain publicly or churn.
You have error rate and latency alerts. The alerts fire on a Slack channel or email. You tested the alert by intentionally breaking an endpoint.
No monitoring. You find out about failures when a user emails or when you manually check the logs.
Diagnosis
Signs Your AI Product Is Not Production-Ready Yet
| What you see | What it usually means | What to do next |
|---|---|---|
| Works on your machine but fails in production | Environment variables missing or production config different from local | Compare local and production environment variables line by line |
| AI output is wrong for edge-case inputs | No input validation or prompt does not handle unexpected formats | Add input sanitisation and test at least 5 edge-case inputs |
| Page breaks when AI returns null or empty | No output validation before rendering | Add null checks and fallback display for every AI output path |
| API costs doubled after launch | No per-user token budget or caching layer | Add response caching for repeatable queries and set per-user limits |
| AI feature works 80% of the time | No fallback for the 20% failure case | Define the fallback, build it, test it deliberately |
| Nobody noticed the AI was returning errors for 2 days | No monitoring or alerting on AI endpoints | Add error rate tracking and a Slack or email alert within 24 hours |
FAQ
Questions Founders Ask Before Going to Production
How do I know when a prototype is ready for production?
When it passes the 5-point review in this page. Validated outputs, evaluation test cases, cost controls, a defined fallback, and monitoring in place. That is the minimum bar.
Do I need evals before launch?
Yes. 10 to 20 test cases with expected outputs is not a large investment. It takes a few hours to write and can save weeks of debugging after a model update or prompt change breaks something silently.
What model should I use in production?
The one that passes your eval test cases at the lowest cost. Not the newest, not the most capable. The one that reliably produces the right output format for your specific use case.
Is this different from a standard software production review?
Partly. The standard checks (secrets, environment, error handling, monitoring) apply. AI adds output validation, evals, fallback design, and cost controls as additional requirements.
Can we add evals and monitoring after launch?
You can, but the risk window is the time between launch and when those are in place. Add monitoring first. It costs least and immediately reduces the failure-detection window from days to minutes.
How is this different from the AI-built app safety check?
The app safety check covers auth, payments, data isolation, and general production readiness. This page focuses specifically on the AI feature layer: validation, evals, cost, fallbacks, and monitoring for the AI components.
Related Problems
Other Problems Founders Check Around This
AI Product Engineering
The prototype proved it works. Now build a version that holds up.
If the 5-point review found open questions on validation, evals, cost, fallbacks, or monitoring, the AI feature needs production engineering work before real users depend on it.
Start With a Technical Audit