Domain 4 · 16% of the exam · 10 of 63 questions
Evaluation, Testing & Optimization
Proving the system works and keeping it working. Defining metrics that are both meaningful and observable, building evaluation sets that include the cases you would rather not think about, running experiments whose results can be attributed to one cause, diagnosing failures to the right layer, and optimising cost and latency after measuring instead of before.
every one of the 32 pool items in this domain · explanations as you go · free
The 6 objectives in this domain
A real score report grades each of these separately, so a weakness here is addressable at this level rather than by re-reading the whole domain. Each one has its own drill.
- 4.1Define evaluation metrics (accuracy, latency, cost, safety, security)6 questions
- 4.2Design evaluation datasets and test frameworks using mixed methodologies5 questions
- 4.3Conduct A/B testing and iterative improvements5 questions
- 4.4Diagnose system issues (prompt failure, hallucinations, model mismatch)5 questions
- 4.5Optimize token usage, latency, and cost-performance trade-offs5 questions
- 4.6Monitor system performance using logging and observability tools6 questions
The other 6 domains
Drilling one domain tells you about one domain. A full-length timed mock in blueprint proportions is what produces a score you can read.
Sit a timed mock →