Books › AI Evaluation Engineer roadmap

Best books for AI Evaluation Engineers

This list is for people who will decide whether an AI system is good enough to ship. It starts with the measurement habits the field inherits from statistics and machine learning, because a score is only useful if you can say how far to trust it. The middle books cover how LLM systems are built and where they fail, so you know what each part of the system needs to be tested for. Retrieval, prompts and agents each fail in different ways, and the reading covers all of them. The lead books are about impact. A system can pass every benchmark and still cause harm to particular groups, and the reading helps you argue for measuring that before a launch.

11 books in 3 stages, about 3,180 pages (at least 80 hours of reading). The notes are ours, with no ratings or sales claims.

See the full AI Evaluation Engineer roadmap →Best courses for AI Evaluation Engineers →

How to choose your books

A short method, so you read the right book for where you are instead of the longest list.

Find your stage

  • Getting started · 3 books

    Measurement and statistics: how to read a score, decide what it proves, and avoid the comparisons that mislead.

    Jump to this stage ↓
  • Doing the job · 6 books

    How LLM systems are built and what they get wrong: models, prompts, retrieval and agents, each with the failures you need to test.

    Jump to this stage ↓
  • Senior and leading · 2 books

    Impact and accountability: what an evaluation should measure beyond accuracy, and how to argue for it inside a product team.

    Jump to this stage ↓

If you are unsure, start one stage lower than you think. Books in a later stage assume the working knowledge the earlier ones give you.

Begin with one book

Our starting pick

AI Snake Oil

by Arvind Narayanan, Sayash Kapoor · 2024

It shows how to test a claim about an AI system before you believe it. The questions it teaches are the ones every evaluation has to answer.

See the book ↓

Then pick by your situation

Choose the next book by the problem in front of you, not by its place in the list.

Set a time budget

StageBooksPagesReadingAt 3 hrs a week
Getting started31,16029 hrs10 wks
Doing the job6≥ 1,390≥ 35 hrs≥ 12 wks
Senior and leading263016 hrs5 wks
All 3 stages11≥ 3,180≥ 80 hrs≥ 27 wks

Treat this as a ceiling, not a target. Read one book at a time, apply it, then move on; read reference-style books in sections. Totals are approximate.

Judge any book before you buy it

  • Age

    This list runs from 2013 (Data Science for Business) to 2025 (AI Engineering). Older books earn their place by teaching principles that do not change. For tools and platforms, buy the latest edition.

  • Level

    Check the stage before the title. A book from a later stage will make sense only once you have faced the problems it solves.

  • Format

    Some books suit reading start to finish; others work as references. Read the reference-style ones in sections, when a problem calls for them.

  • Fit

    Ask what you will do differently the week after you finish. If you cannot answer, the book is for a different moment. Come back to it when the situation arrives.

Getting started

Measurement and statistics: how to read a score, decide what it proves, and avoid the comparisons that mislead.

  • Start here

    AI Snake Oil

    by Arvind Narayanan, Sayash Kapoor · 2024

    ~380 pages~10 hrs to read

    How to test an AI claim before believing it: what a benchmark measures, what it misses, and why a good demo is not evidence. It frames the questions an evaluation has to answer.

    View book →
  • Cover of Data Science for Business

    Data Science for Business

    by Foster Provost, Tom Fawcett · 2013

    ~410 pages~10 hrs to read

    How to frame a model evaluation around the decision it supports: expected value, cost of errors and how to compare models on the outcomes that matter.

    View book →
  • Cover of Practical Statistics for Data Scientists

    Practical Statistics for Data Scientists

    by Peter Bruce, Andrew Bruce, Peter Gedeck · 2020

    ~370 pages~9 hrs to read

    The statistics you need to read results honestly: sampling, confidence intervals, hypothesis tests and the errors that come from multiple comparisons.

    View book →

Doing the job

How LLM systems are built and what they get wrong: models, prompts, retrieval and agents, each with the failures you need to test.

  • Cover of Designing Machine Learning Systems

    Designing Machine Learning Systems

    by Chip Huyen · 2022

    ~390 pages~10 hrs to read

    How models are evaluated, monitored and retrained once they serve real traffic. It connects offline metrics to the production decisions that depend on them.

    View book →
  • Cover of AI Engineering

    AI Engineering

    by Chip Huyen · 2025

    ~350 pages~9 hrs to read

    Building applications on foundation models, with a substantial section on evaluation methods, including AI judges and how to design them.

    View book →
  • Cover of Natural Language Processing with Transformers

    Natural Language Processing with Transformers

    by Lewis Tunstall, Leandro von Werra, Thomas Wolf · 2022

    A hands-on tour of the transformer models that most evaluations test, with worked examples of fine-tuning and measuring their output.

    View book →
  • Cover of Prompt Engineering for LLMs

    Prompt Engineering for LLMs

    by John Berryman, Albert Ziegler · 2024

    ~250 pages~6 hrs to read

    How prompts shape model behavior and how to structure them so their output can be tested. It is the input side of most evaluation suites.

    View book →
  • Cover of Hands-On Large Language Models

    Hands-On Large Language Models

    by Jay Alammar, Maarten Grootendorst · 2024

    ~400 pages~10 hrs to read

    Visual, code-first explanations of how LLM systems work, including retrieval and generation pipelines that need evaluating one part at a time.

    View book →
  • Cover of Build a Large Language Model (From Scratch)

    Build a Large Language Model (From Scratch)

    by Sebastian Raschka · 2024

    Builds a small GPT-style model step by step. Doing this shows what the model can and cannot learn, which makes later evaluation decisions easier to justify.

    View book →

Senior and leading

Impact and accountability: what an evaluation should measure beyond accuracy, and how to argue for it inside a product team.

  • Cover of The Alignment Problem

    The Alignment Problem

    by Brian Christian · 2020

    ~360 pages~9 hrs to read

    The history of how machine learning systems come to optimize the wrong thing. It is the background for deciding what an evaluation should measure beyond accuracy.

    View book →
  • Cover of Weapons of Math Destruction

    Weapons of Math Destruction

    by Cathy O'Neil · 2016

    ~270 pages~7 hrs to read

    How opaque scoring systems cause harm when nobody audits their outcomes. It gives a lead the case for evaluating impact across groups, not only average scores.

    View book →

Book links go to Amazon and are affiliate links. As an Amazon Associate I earn from qualifying purchases.

Frequently asked questions

Which book should a AI Evaluation Engineer read first?

AI Snake Oil by Arvind Narayanan, Sayash Kapoor. It shows how to test a claim about an AI system before you believe it. The questions it teaches are the ones every evaluation has to answer.

Do I need to read all 11 books?

No. The list is a menu, not a syllabus. Find your stage (getting started: 3, doing the job: 6, senior and leading: 2), begin with one book (we suggest AI Snake Oil), then pick the next book by the problem you are facing at work. Come back to the others when the situation they cover arrives.

How long does it take to read these books?

The 9 books with a known page count add up to about 3,180 pages, roughly 80 hours at 1.5 minutes a page. At three hours a week that is about 27 weeks; at six hours a week, about 13. The list is meant to be read over time, one book at a time, not in one run.

Are these books up to date?

They range from 2013 (Data Science for Business) to 2025 (AI Engineering). Older titles are included because they teach principles that have not changed; for tools and platforms, we favor recent editions. Check the edition before you buy.

Some links are affiliate links: we earn a small commission if you buy, at no cost to you. It does not change what we list.