How to Tell If an AI Use Case Is Actually Worth Building

Most AI projects don’t fail in production. They fail in the meeting six months earlier where nobody asked the right questions, and a use case that was never going to work got funded anyway. The demo looked great. The demo always looks great.

After building AI into several of my own products and helping teams do the same, I’ve found that four questions predict success better than any amount of model benchmarking. You can answer them before writing a line of code.

Question 1: Does it tolerate a wrong answer?

The single best predictor of whether an AI use case will work is how much it can be wrong without causing harm.

Language models are probabilistic. They will be confidently wrong some percentage of the time, and you don’t get to pick which time. So the question isn’t “can the model be accurate?” — it’s “what happens on the inputs where it isn’t?”

RedNudge scores Reddit threads for relevance from 1 to 10. If it scores a thread a 7 when a human would say 6, nothing breaks — you skim one extra thread. That’s a use case built on a forgiving foundation. Now imagine the same model deciding whether to approve a loan. Same technology, completely different tolerance for error. One is a product; the other is a lawsuit.

Build where being wrong is cheap. Be very careful where it isn’t.

Question 2: Is a human in the loop — and should they be?

The safest AI features put a human between the model and the consequence. The model drafts; the person decides.

Propelr writes social posts in your voice, but you hit publish. The model does the hard part — the blank page — and the human does the part that carries reputational risk. That division is deliberate. An AI that drafts is a productivity tool. An AI that publishes unsupervised is a liability generator waiting for its worst output to go public.

The amount of oversight should scale with the blast radius:

Action	Cost of a wrong answer	Human in the loop?
Draft an email	Low — reader catches it	Light
Summarize a document	Low to medium	Light
Auto-tag a support ticket	Medium — mis-routed	Spot checks
Approve a refund	High — real money	Required
Send a legal or medical notice	Severe	Required, always

If your use case needs to run fully unsupervised and a wrong answer is expensive, you don’t have an AI use case yet. You have a research problem.

Question 3: Do you already have the data?

AI features are only as good as the context you can feed them. The question that quietly kills more projects than any other is: where does the model get what it needs to be right here?

If the answer is “from data we already have, cleanly,” you’re in good shape. If the answer is “from a system we’d have to build, integrate, and clean first,” then the AI feature isn’t the project — the data plumbing is, and it’s ten times the work. Be honest about which project you’re actually signing up for.

The teams that ship fast are usually the ones who picked a use case sitting on top of data they already had. The teams that stall picked the impressive use case and discovered the data for it didn’t exist in usable form.

Question 4: Is a wrong answer recoverable?

Finally: when the model is wrong — and it will be — can you undo it?

Recoverable failures are drafts, suggestions, and rankings. A bad suggestion is ignored. A bad ranking is re-sorted. Nobody gets hurt.

Unrecoverable failures are actions that can’t be walked back: a sent message, a processed payment, a deleted record, a public statement. The moment an AI feature takes an irreversible action without a human checkpoint, its rare confident mistakes become your problem in the worst possible way.

Design so that the model’s failures are recoverable, and you can ship something imperfect today and improve it. Design so its failures are permanent, and you’ve built a system that’s one bad output away from a crisis.

Put it together

The good use cases share a shape: they tolerate error, keep a human in the loop, run on data you already have, and fail recoverably. The ones that die share the opposite shape, and they die after you’ve spent the budget — unless you ask these four questions first, in the cheap meeting, before anyone falls in love with the demo.

Once a use case clears these four questions, the work shifts from should we to how do we not break it in production — evals, fallbacks, cost ceilings and human checkpoints. That’s the next checklist.

This evaluation is most of what I do when a team asks me to help them adopt AI. If you’d rather run your use cases past someone who’s shipped a few, that’s exactly what we do.

Frequently asked questions

How do you evaluate whether an AI use case is worth building?

Score it on four axes: tolerance for imperfect answers, whether a human stays in the loop, whether you already have the data, and whether the cost of a wrong answer is recoverable. Use cases that tolerate error, keep a human reviewing, use data you already have, and fail safely are the ones worth building. Use cases that need perfect answers, run unsupervised, depend on data you don't have, and fail expensively are the ones that die in a demo.

Why do so many AI projects fail after a good demo?

Because demos are curated and production is not. A demo shows the model's best case on hand-picked inputs; production is the long tail of weird, adversarial, and ambiguous inputs the demo never showed. Projects fail when nobody asked what happens on the 5% of inputs where the model is confidently wrong, and there was no human or fallback to catch it.

Should a human always be in the loop for AI features?

Not always, but the amount of human oversight should match the cost of a wrong answer. Drafting an email a person will read before sending needs little oversight. Auto-approving a refund, sending a legal notice, or changing a medical record needs a human checkpoint. Match the guardrail to the blast radius, not to how impressive the automation looks.