AI Evaluation Engineer roadmap › Courses
Best courses for AI Evaluation Engineers
This list is for people who design the tests an AI system is judged by. It starts with the measurement basics from machine learning, since a metric is only useful when you know what it can and cannot show.
The build courses cover the working tools: a model-side course, a short hands-on course on tracking runs, and the walkthroughs from two eval harnesses that you can run in continuous integration.
Most of the material is free. There is no certificate that replaces a working eval suite, so the portfolio item to build is a test set, a rubric and a report on where the system fails.
- 6 items in 3 stages: 6 courses and tutorials, 0 certifications.
- We opened every page on November 18, 2026 and show only what the provider states.
- No prices, ratings or enrollment counts.
See the full AI Evaluation Engineer roadmap →Best books for AI Evaluation Engineers →
How to choose your courses
A short method, so you spend your hours on the right thing for where you are instead of collecting certificates.
Find your stage
Getting started · 1 items
Machine learning measurement: the metrics and dataset chapters that every evaluation depends on.
Jump to this stage ↓Doing the job · 4 items
Working tools: the LLM course for the model side, a run-tracking course, and eval harness walkthroughs from OpenAI and Promptfoo.
Jump to this stage ↓Senior and leading · 1 items
Retrieval and generation metrics: the Ragas documentation, for teams that evaluate a full RAG system rather than a single prompt.
Jump to this stage ↓
If you are unsure, start one stage lower than you think. Later items assume what the earlier ones teach.
Begin with one course
Our starting pick
Machine Learning Crash Course
Google for Developers
It covers the classification metrics and dataset problems that every evaluation depends on. It is free, and you can read the chapters in the order you need them.
See the course ↓Then pick by your situation
Choose the next course by the problem in front of you, not by its place in the list.
If you want a first hands-on eval harness that runs from the command line
take Getting started (doing the job)
If you need to track experiments and compare versions of an LLM system
take Evaluating and Debugging Generative AI (doing the job)
If you are evaluating retrieval-augmented generation and need repeatable metrics
take Ragas documentation (senior and leading)
Free path or paid path?
6 of the 6 courses and tutorials can be read or taken without paying, so you can build most of the skill at no cost. Check each certification page for exam fees.
- Machine Learning Crash Course
- LLM Course
- Evaluating and Debugging Generative AI
- Evals guide
- Getting started
- Ragas documentation
Paid courses earn their place when you need structure, deadlines or feedback. Check what the provider says is included before you commit.
Judge any course before you commit
Who runs it
Prefer courses from the people who build or govern the tool. They tend to stay current, and an exam from a tool’s own maker is the clearest credential for that tool.
Hands-on or watch-only
In this field you learn by running things. Choose material where you build a pipeline, a cluster or a dashboard over material you only watch.
How current it is
Tools change fast. Look for a version or last-updated date on the provider’s page, and check anything several years old against the current docs.
What you will be able to show
Ask what you will have at the end: a working repository, a deployed service or a passed exam. If it is only a certificate of completion, pair it with a project of your own.
Getting started
Machine learning measurement: the metrics and dataset chapters that every evaluation depends on.
What you will learn
- Classification metrics: thresholds, confusion matrices, precision, recall and AUC
- Datasets, generalization and overfitting
- Intro to large language models
- ML fairness and auditing for bias
Good for
The metrics and dataset chapters give you the measurement basics you need before designing a test set for any model.
Also stated by the provider
- Interactive lessons, as the course index describes
- Includes a newer module on large language models
Doing the job
Working tools: the LLM course for the model side, a run-tracking course, and eval harness walkthroughs from OpenAI and Promptfoo.
What you will learn
- Transformer models, pipelines and tokenizers
- Fine-tuning and curating high-quality datasets
- Building and reasoning with advanced LLM techniques
Good for
The model-side background for evaluation: how the systems you test are trained, tokenized and fine-tuned.
Also stated by the provider
- Free and without ads, as the page states
- Expects about 6 to 8 hours of work per chapter
- Materials are Apache 2 licensed
What you will learn
- Tracking experiments and versioning data with Weights & Biases
- Tracing and evaluating LLM calls across complex interactions
- Evaluating diffusion models
- Fine-tuning a language model
Good for
A short hands-on course on tracking and comparing model runs, useful when you need a repeatable workflow across versions.
Also stated by the provider
- Intermediate level, about 1 hour of video
- Access stated as free for a limited time during the platform beta
- Uses Weights & Biases tooling
What you will learn
- Creating an eval for a task
- Testing a prompt with uploaded test data
- Creating an eval run and analyzing its results
Good for
The platform’s own walkthrough of building an eval, running it on a prompt and reading the results, so you see one major provider’s workflow.
Also stated by the provider
- Part of the OpenAI API documentation
- Covers uploading test data and creating eval runs
What you will learn
- Writing prompt test cases with assertions
- Comparing models and prompts in a web viewer
- Running evals from the command line
Good for
Quick hands-on practice with an open-source eval harness that you can later run in continuous integration.
Also stated by the provider
- Command-line tool with a web viewer
- Examples for prompt, model, RAG and agent evaluation
Senior and leading
Retrieval and generation metrics: the Ragas documentation, for teams that evaluate a full RAG system rather than a single prompt.
What you will learn
- Experiment-based evaluation loops
- Built-in metrics for LLM applications
- Dataset management and result tracking
Good for
Useful once you evaluate retrieval and generation together and need documented metrics you can repeat across experiments.
Also stated by the provider
- Open-source framework, documented on Read the Docs
- Quickstart, core concepts and how-to guides
Details such as hours, format and whether material is free are as the providers state them on November 18, 2026, and can change. Confirm on the provider’s page before you commit. See our disclosure.
Frequently asked questions
Which course should a AI Evaluation Engineer take first?
Machine Learning Crash Course, from Google for Developers. It covers the classification metrics and dataset problems that every evaluation depends on. It is free, and you can read the chapters in the order you need them.
Can a AI Evaluation Engineer learn this for free?
Largely, yes. 6 of the 6 courses and tutorials here can be read or taken without paying (Machine Learning Crash Course, LLM Course, Evaluating and Debugging Generative AI, Evals guide and more). Check each certification page for exam fees: not every page states one.
How was this list put together?
We opened each course, tutorial and certification page on November 18, 2026 and wrote down only what the page states: who runs it, what it covers, hours and exam format where given, and whether the material is free to read. We do not show prices, ratings or enrollment figures. Details can change, so confirm on the provider’s page before you commit.