Most AI features that launched in 2024 are already dead. They did not die because the models got worse — they died because nobody used them.
There is a simple test. Turn it off for a week. If nobody complains, it was not useful.
The features that survive share three properties: they replace a step users already resent, they fail visibly (not silently), and they get faster the more you use them.
The switch-off test
Turn the feature off for a week and tell nobody. If no one asks for it back, it was decoration. This single test replaces months of debate, because usage under silence is the only honest signal you have.
If support tickets spike, you have found a feature worth investing in — and you now know exactly which job it does.
Three properties useful AI features share
- It removes a step people already resent — retyping, re-formatting, hunting for a record, writing the first draft.
- It fails visibly. Users can see when the model was unsure and can correct it in one action, instead of silently inheriting a wrong answer.
- It gets faster with use, because it learns your data, your templates and your tone rather than starting cold each time.
What to measure instead of "engagement"
- Task completion time before and after the feature existed.
- Correction rate — how often a human edits the output before accepting it.
- Abandonment — how many people start the AI flow and finish manually.
- Cost per accepted output, including tokens, review time and support load.
Where AI genuinely pays for itself
In the businesses we work with across Sri Lanka, the wins are rarely glamorous: quotation drafting, inbox triage, WhatsApp reply suggestions, document extraction, and search over internal knowledge. These are high-volume, low-judgement tasks with clear right answers, which is exactly where current models are reliable.
The failures are the opposite: open-ended chat with no job to do, and anything where a wrong answer costs more than a missing one. If you want a second opinion on your own site, our free website analyzer reviews it in a couple of minutes.
