Objective 4.3 · Domain 4 · 16% of the exam
Conduct A/B testing and iterative improvements
A/B testing on a non-deterministic system, which means holding everything but one variable steady, running long enough to separate signal from sampling noise, and resisting the urge to change the prompt and the model in the same experiment. Items often describe an experiment whose result cannot be attributed to anything and ask what was wrong with the design.
5 questions · answers and explanations shown as you go · free, no sign-up
The rest of domain 4: Evaluation, Testing & Optimization
This domain is 16% of a scored form, about 10 of the 63 questions, spread across 6 objectives. Drill the whole domain or pick another objective below.
- 4.1Define evaluation metrics (accuracy, latency, cost, safety, security)(6)
- 4.2Design evaluation datasets and test frameworks using mixed methodologies(5)
- 4.4Diagnose system issues (prompt failure, hallucinations, model mismatch)(5)
- 4.5Optimize token usage, latency, and cost-performance trade-offs(5)
- 4.6Monitor system performance using logging and observability tools(6)
One objective is a narrow slice. A full-length timed mock is what tells you whether the whole thing holds together.
Sit a timed mock →