What Prompting Can and Can’t Fix

After we benchmarked six leading AI models against real intelligence questions, the most common response was: would better prompts have gotten better answers? So we ran that evaluation too. Six models, three prompting styles — from a 3 a.m. casual query to the full engineered apparatus — tested against live intelligence scenarios, with AskAnna as the control. Prompting helps more than you’d think in some places, costs you in others, and one result we still can’t explain.

WHAT’S INSIDE

• The most dangerous failure, fixed. Asked casually, four of six models agreed with a false security claim and ran with it. Structured and engineered prompts drove that to zero — and what that means for how your team actually types at 3 a.m.
• Where prompting costs you. Some instructions that replicate good tradecraft made answers worse, not better. Adding more instruction is the right instinct only about half the time.
• The push-back test. We told models “are you sure? that doesn’t match what we’re hearing.” Which ones held their position, which one folded instantly, and why that matters before output reaches a stakeholder.
• The four prompt elements that actually work. Name the reader, name the decision, state what to exclude, cap the length — and why “act as an intelligence analyst” isn’t one of them.
• Treating AI output as source material. Why what comes out of these models is raw material, not finished analysis — and should be handled like any source you didn’t collect yourself.