Objective 4.4 · Domain 4 · 16% of the exam
Diagnose system issues (prompt failure, hallucinations, model mismatch)
Diagnosis: telling a prompt failure from a hallucination from a model mismatch from a retrieval miss. The discriminating question is what the model was actually given. A confident wrong answer whose supporting document was never retrieved is a retrieval bug, and no amount of prompt work will fix it. Expect stems that describe a symptom and reward the answer isolating the layer before changing anything.
5 questions · answers and explanations shown as you go · free, no sign-up
The rest of domain 4: Evaluation, Testing & Optimization
This domain is 16% of a scored form, about 10 of the 63 questions, spread across 6 objectives. Drill the whole domain or pick another objective below.
- 4.1Define evaluation metrics (accuracy, latency, cost, safety, security)(6)
- 4.2Design evaluation datasets and test frameworks using mixed methodologies(5)
- 4.3Conduct A/B testing and iterative improvements(5)
- 4.5Optimize token usage, latency, and cost-performance trade-offs(5)
- 4.6Monitor system performance using logging and observability tools(6)
One objective is a narrow slice. A full-length timed mock is what tells you whether the whole thing holds together.
Sit a timed mock →