Books › AI Evaluation Engineer roadmap
Best books for AI Evaluation Engineers
This list is for people who will decide whether an AI system is good enough to ship. It starts with the measurement habits the field inherits from statistics and machine learning, because a score is only useful if you can say how far to trust it. The middle books cover how LLM systems are built and where they fail, so you know what each part of the system needs to be tested for. Retrieval, prompts and agents each fail in different ways, and the reading covers all of them. The lead books are about impact. A system can pass every benchmark and still cause harm to particular groups, and the reading helps you argue for measuring that before a launch.
11 books in 3 stages, about 3,180 pages (at least 80 hours of reading). The notes are ours, with no ratings or sales claims.
See the full AI Evaluation Engineer roadmap →Best courses for AI Evaluation Engineers →
How to choose your books
A short method, so you read the right book for where you are instead of the longest list.
Find your stage
Getting started · 3 books
Measurement and statistics: how to read a score, decide what it proves, and avoid the comparisons that mislead.
Jump to this stage ↓Doing the job · 6 books
How LLM systems are built and what they get wrong: models, prompts, retrieval and agents, each with the failures you need to test.
Jump to this stage ↓Senior and leading · 2 books
Impact and accountability: what an evaluation should measure beyond accuracy, and how to argue for it inside a product team.
Jump to this stage ↓
If you are unsure, start one stage lower than you think. Books in a later stage assume the working knowledge the earlier ones give you.
Begin with one book
Our starting pick
AI Snake Oil
by Arvind Narayanan, Sayash Kapoor · 2024
It shows how to test a claim about an AI system before you believe it. The questions it teaches are the ones every evaluation has to answer.
See the book ↓Then pick by your situation
Choose the next book by the problem in front of you, not by its place in the list.
If you need to build a test set and read the results without fooling yourself
read Practical Statistics for Data Scientists (getting started)
If you are testing a retrieval or prompt-based system and need the failure modes first
read Prompt Engineering for LLMs (doing the job)
If you are deciding which errors matter most for the people the system affects
read Weapons of Math Destruction (senior and leading)
Set a time budget
| Stage | Books | Pages | Reading | At 3 hrs a week |
|---|---|---|---|---|
| Getting started | 3 | 1,160 | 29 hrs | 10 wks |
| Doing the job | 6 | ≥ 1,390 | ≥ 35 hrs | ≥ 12 wks |
| Senior and leading | 2 | 630 | 16 hrs | 5 wks |
| All 3 stages | 11 | ≥ 3,180 | ≥ 80 hrs | ≥ 27 wks |
Treat this as a ceiling, not a target. Read one book at a time, apply it, then move on; read reference-style books in sections. Totals are approximate.
Judge any book before you buy it
Age
This list runs from 2013 (Data Science for Business) to 2025 (AI Engineering). Older books earn their place by teaching principles that do not change. For tools and platforms, buy the latest edition.
Level
Check the stage before the title. A book from a later stage will make sense only once you have faced the problems it solves.
Format
Some books suit reading start to finish; others work as references. Read the reference-style ones in sections, when a problem calls for them.
Fit
Ask what you will do differently the week after you finish. If you cannot answer, the book is for a different moment. Come back to it when the situation arrives.
Getting started
Measurement and statistics: how to read a score, decide what it proves, and avoid the comparisons that mislead.
- Start here
AI Snake Oil
by Arvind Narayanan, Sayash Kapoor · 2024
~380 pages~10 hrs to read
How to test an AI claim before believing it: what a benchmark measures, what it misses, and why a good demo is not evidence. It frames the questions an evaluation has to answer.
View book → 
Data Science for Business
by Foster Provost, Tom Fawcett · 2013
~410 pages~10 hrs to read
How to frame a model evaluation around the decision it supports: expected value, cost of errors and how to compare models on the outcomes that matter.
View book →
Practical Statistics for Data Scientists
by Peter Bruce, Andrew Bruce, Peter Gedeck · 2020
~370 pages~9 hrs to read
The statistics you need to read results honestly: sampling, confidence intervals, hypothesis tests and the errors that come from multiple comparisons.
View book →
Doing the job
How LLM systems are built and what they get wrong: models, prompts, retrieval and agents, each with the failures you need to test.

Designing Machine Learning Systems
by Chip Huyen · 2022
~390 pages~10 hrs to read
How models are evaluated, monitored and retrained once they serve real traffic. It connects offline metrics to the production decisions that depend on them.
View book →
AI Engineering
by Chip Huyen · 2025
~350 pages~9 hrs to read
Building applications on foundation models, with a substantial section on evaluation methods, including AI judges and how to design them.
View book →
Natural Language Processing with Transformers
by Lewis Tunstall, Leandro von Werra, Thomas Wolf · 2022
A hands-on tour of the transformer models that most evaluations test, with worked examples of fine-tuning and measuring their output.
View book →
Prompt Engineering for LLMs
by John Berryman, Albert Ziegler · 2024
~250 pages~6 hrs to read
How prompts shape model behavior and how to structure them so their output can be tested. It is the input side of most evaluation suites.
View book →
Hands-On Large Language Models
by Jay Alammar, Maarten Grootendorst · 2024
~400 pages~10 hrs to read
Visual, code-first explanations of how LLM systems work, including retrieval and generation pipelines that need evaluating one part at a time.
View book →
Build a Large Language Model (From Scratch)
by Sebastian Raschka · 2024
Builds a small GPT-style model step by step. Doing this shows what the model can and cannot learn, which makes later evaluation decisions easier to justify.
View book →
Senior and leading
Impact and accountability: what an evaluation should measure beyond accuracy, and how to argue for it inside a product team.

The Alignment Problem
by Brian Christian · 2020
~360 pages~9 hrs to read
The history of how machine learning systems come to optimize the wrong thing. It is the background for deciding what an evaluation should measure beyond accuracy.
View book →
Weapons of Math Destruction
by Cathy O'Neil · 2016
~270 pages~7 hrs to read
How opaque scoring systems cause harm when nobody audits their outcomes. It gives a lead the case for evaluating impact across groups, not only average scores.
View book →
Book links go to Amazon and are affiliate links. As an Amazon Associate I earn from qualifying purchases.
Frequently asked questions
Which book should a AI Evaluation Engineer read first?
AI Snake Oil by Arvind Narayanan, Sayash Kapoor. It shows how to test a claim about an AI system before you believe it. The questions it teaches are the ones every evaluation has to answer.
Do I need to read all 11 books?
No. The list is a menu, not a syllabus. Find your stage (getting started: 3, doing the job: 6, senior and leading: 2), begin with one book (we suggest AI Snake Oil), then pick the next book by the problem you are facing at work. Come back to the others when the situation they cover arrives.
How long does it take to read these books?
The 9 books with a known page count add up to about 3,180 pages, roughly 80 hours at 1.5 minutes a page. At three hours a week that is about 27 weeks; at six hours a week, about 13. The list is meant to be read over time, one book at a time, not in one run.
Are these books up to date?
They range from 2013 (Data Science for Business) to 2025 (AI Engineering). Older titles are included because they teach principles that have not changed; for tools and platforms, we favor recent editions. Check the edition before you buy.
Related reading
Some links are affiliate links: we earn a small commission if you buy, at no cost to you. It does not change what we list.