IDM's AI Model Benchmark — Real-World Evaluation for Saudi Organizations

1. Methodology

This experiment evaluated nine large language models on two complementary instruments designed to separate analytical reasoning from executable code generation.

Instrument 1 — Reasoning Suite (9 items). Nine domain-specific problems cover SEO architecture for a bilingual website, a publishing state machine, cognitive-test scoring, social-media automation, PHP/MariaDB defect analysis, legal-document structuring, OCR cleanup, and front-end debugging. Each response was scored against a per-item rubric by an automated evaluator, identical for all models. A response passes if its rubric score meets a fixed threshold and it triggers no automatic-failure guards. Guards are embedded behavioral traps — a required utf8mb4 charset specification and an instruction to flag (not silently fix) a planted defect — distinguishing instruction compliance from analytical depth.

Instrument 2 — TaskBench (5 tasks, 173 tests). Each model produced complete programs from task specifications. Every program ran in an isolated sandbox against a fixed test suite: 44 scheduler tests, 41 HTML-builder tests, 33 SEO-service tests, 28 state-machine tests, and 27 scorer tests. Verdicts are binary per test; task runners abort early on program crash, so failed tasks attempt fewer tests. All calls were non-streaming; latency and token usage were captured per call and recorded in a relational database for audit.

2. Test and Task Descriptions

Reasoning Suite (9 items):

TaskBench (5 tasks):

3. Results — Reasoning Suite (9 items)

Token counts are as reported by each model's API usage field. For gpt-6-astra, kimi-k3, and claude-fable-5.1 the reported input/output split differs from the other models (reasoning tokens are folded into input), so token figures are not directly comparable across all nine models.

Failure taxonomy: Most reasoning failures were not analytical misses. Code Correction and Legal Structuring failures across models were dominated by the compliance traps — several models (glm-5.3, qwen3.8-max, deepseek-v4.1-flash) scored rubric-perfect or near-perfect on these items yet failed by violating the embedded instructions. The remaining failures were rubric-depth misses, concentrated in Social Automation and Search Debounce.

4. Results — TaskBench (173 tests)

Eight of nine models completed all 173 tests successfully. The single failure, nemotron-3-ultra on Post Scheduler (0/7, reproduced independently in a second attempt), is the benchmark's cleanest capability discriminator: its scheduler program crashed at the first test group in both runs.

5. Summary Scores

Efficiency extremes: kimi-k3 and claude-fable-5.1 solved all tasks with ~7K output tokens; glm-5.3 spent ~159K output tokens for the same perfect result — opposite strategies, identical outcomes. Reasoning-suite latency ranged from 145 s (kimi-k3) to 1,986 s (qwen3.8-max).

6. Scientific Conclusion

Across nine models, performance was determined less by recency than by training objectives. Code generation was nearly universal — eight of nine models scored a perfect 173/173 — while reasoning scores spread from 3/9 to 8/9. Critically, most reasoning failures were not analytical: models repeatedly produced rubric-perfect analysis yet failed by violating explicit compliance instructions. Reasoning depth, instruction compliance, and code generation are dissociable competencies, shaped by what a model was optimized to do — not by how new it is. Benchmark design must therefore match the competency being measured to each model's training purpose.