Books › GPU Performance Engineer roadmap

Best books for GPU Performance Engineers

This list starts with GPU programming and moves through profiling and deep learning kernels to the system-level performance work of a lead. The books cover the architecture the code runs on, so the optimizations have a reason behind them.

7 books in 3 stages, about 3,500 pages (at least 88 hours of reading). The notes are ours, with no ratings or sales claims.

See the full GPU Performance Engineer roadmap →Best courses for GPU Performance Engineers →

How to choose your books

A short method, so you read the right book for where you are instead of the longest list.

Find your stage

  • Getting started · 2 books

    GPU programming and the parallel model: the textbook and first CUDA book.

    Jump to this stage ↓
  • Doing the job · 4 books

    Optimization and deep learning workloads: optimization references, Python access to the GPU and the hardware behind neural networks.

    Jump to this stage ↓
  • Senior and leading · 1 books

    System-wide performance: architecture reference and measurement methods across a whole system.

    Jump to this stage ↓

If you are unsure, start one stage lower than you think. Books in a later stage assume the working knowledge the earlier ones give you.

Begin with one book

Our starting pick

Programming Massively Parallel Processors

by Wen-mei W. Hwu, David B. Kirk, Izzat El Hajj · 2022

It is the standard textbook on the GPU execution model and memory hierarchy. Every later kernel optimization assumes the ideas in it.

See the book ↓

Then pick by your situation

Choose the next book by the problem in front of you, not by its place in the list.

Set a time budget

StageBooksPagesReadingAt 3 hrs a week
Getting started289022 hrs7 wks
Doing the job4≥ 1,680≥ 42 hrs≥ 14 wks
Senior and leading193023 hrs8 wks
All 3 stages7≥ 3,500≥ 88 hrs≥ 29 wks

Treat this as a ceiling, not a target. Read one book at a time, apply it, then move on; read reference-style books in sections. Totals are approximate.

Judge any book before you buy it

  • Age

    This list runs from 2010 (CUDA by Example) to 2022 (Programming Massively Parallel Processors). Older books earn their place by teaching principles that do not change. For tools and platforms, buy the latest edition.

  • Level

    Check the stage before the title. A book from a later stage will make sense only once you have faced the problems it solves.

  • Format

    The longest book here, Computer Architecture, runs about 940 pages. Read very long books as references: pick the chapter that matches the problem you have.

  • Fit

    Ask what you will do differently the week after you finish. If you cannot answer, the book is for a different moment. Come back to it when the situation arrives.

Getting started

GPU programming and the parallel model: the textbook and first CUDA book.

  • Cover of Programming Massively Parallel Processors
    Start here

    Programming Massively Parallel Processors

    by Wen-mei W. Hwu, David B. Kirk, Izzat El Hajj · 2022

    ~580 pages~15 hrs to read

    The standard textbook on GPU programming: the execution model, memory hierarchy, and the parallel patterns that performance work depends on. It is the foundation for the CUDA step.

    View book →
  • CUDA by Example

    by Jason Sanders, Edward Kandrot · 2010

    ~310 pages~8 hrs to read

    A hands-on introduction to CUDA with complete programs for common parallel problems. The examples are short enough to modify and profile yourself.

    View book →

Doing the job

Optimization and deep learning workloads: optimization references, Python access to the GPU and the hardware behind neural networks.

  • Professional CUDA C Programming

    by John Cheng, Max Grossman, Ty McKercher · 2014

    CUDA optimization in depth: memory access, streams, multi-GPU work and profiling. It is the book to keep open while tuning a kernel.

    View book →
  • Hands-On GPU Programming with Python and CUDA

    by Dr. Brian Tuomanen · 2018

    ~310 pages~8 hrs to read

    GPU programming from Python, including PyCUDA and the libraries that wrap it. It is useful for teams whose research code is written in Python but whose bottlenecks sit on the GPU.

    View book →
  • Efficient Processing of Deep Neural Networks

    by Vivienne Sze, Yu-Hsin Chen, Tien-Ju Yang, Joel S. Emer · 2020

    ~430 pages~11 hrs to read

    How deep learning workloads map onto hardware: data movement, dataflow and the energy cost of each access. It explains why many kernel optimizations work.

    View book →
  • Cover of Computer Architecture

    Computer Architecture: A Quantitative Approach

    by John L. Hennessy, David A. Patterson · 2017

    ~940 pages~24 hrs to read

    Memory hierarchies, pipelining and parallel processors, with quantitative tools for judging performance. It gives the architectural vocabulary the rest of the role uses.

    View book →

Senior and leading

System-wide performance: architecture reference and measurement methods across a whole system.

  • Cover of Systems Performance

    Systems Performance

    by Brendan Gregg · 2020

    ~930 pages~23 hrs to read

    Measurement and analysis of whole systems, from CPU and memory to networks and storage. Leads use it to set performance methods across teams, not only inside one kernel.

    View book →

Book links go to Amazon and are affiliate links. As an Amazon Associate I earn from qualifying purchases.

Frequently asked questions

Which book should a GPU Performance Engineer read first?

Programming Massively Parallel Processors by Wen-mei W. Hwu, David B. Kirk, Izzat El Hajj. It is the standard textbook on the GPU execution model and memory hierarchy. Every later kernel optimization assumes the ideas in it.

Do I need to read all 7 books?

No. The list is a menu, not a syllabus. Find your stage (getting started: 2, doing the job: 4, senior and leading: 1), begin with one book (we suggest Programming Massively Parallel Processors), then pick the next book by the problem you are facing at work. Come back to the others when the situation they cover arrives.

How long does it take to read these books?

The 6 books with a known page count add up to about 3,500 pages, roughly 88 hours at 1.5 minutes a page. At three hours a week that is about 29 weeks; at six hours a week, about 15. The list is meant to be read over time, one book at a time, not in one run.

Are these books up to date?

They range from 2010 (CUDA by Example) to 2022 (Programming Massively Parallel Processors). Older titles are included because they teach principles that have not changed; for tools and platforms, we favor recent editions. Check the edition before you buy.

Some links are affiliate links: we earn a small commission if you buy, at no cost to you. It does not change what we list.