A pre-registered study across five AI models found that adding a specification preamble to prompts reduced security defects in generated code from an average of 148 to 23 across realistic backend tasks in finance, healthcare, and insurance. The frame consistently improved results across all models and domains, with independent security scanning confirming the improvements.
Researchers identified defects or ambiguous requirements in 37 of DeepSWE's 113 tasks (32.7%), a benchmark used to evaluate AI models like GPT-6 Astra and Fable 5. Issues included hidden tests causing build failures, assertions rejecting valid output, and unspecified requirements. Fixing confirmed defects raised measured pass rates by 4–6 percentage points, raising questions about benchmark reliability.