Reinforcement Learning Books

Reinforcement Learning Books

Despite being a relatively young field of study, reinforcement learning has a good body of literature. Personally, I find reading books to be both the fastest and the most pleasant way to consume information, and thus tend to buy as many books as I can about topics I want to learn more about.

Having researched reinforcement learning for the last three years, I’ve assembled quite a few books on the topic (see the figure above). The first book I read was the first edition of Sutton & Barto’s book (the worn out book to the left in the figure). This is a book I can highly recommend for getting started with reinforcement learning. It is written in a very accessible way and covers nearly all of the key principles of reinforcement learning. What this book lacks, however, is some of the latest advances in reinforcement learning (particularly policy gradient methods). Luckily, a new edition of this book have been released which adds chapters about policy gradient methods, and so that was the second book that I bhought. While these two books cover everything you need to get started with reinforcement learning, they do not cover the mathematical foundations that the field rests on, which are vast. A more rigorous treatment of reinforcement learning is available in the books by Bertsekas and Tsitsiklis. The books by Bertsekas and Tsitsiklis also differ from Sutton & Barto in that they describe reinforcement learning from a control-theoretic perspective rather than a computer science perspective. Yet another perspective of reinforcement learning is provided by the books of Powell, which describes the field from the perspective of operations research. Finally, there is also a wide range of books dedicated to specific formalisms that reinforcement learning builds upon, e.g. Markov decision processes, dynamic programming, and stochastic approximations. My personal favorite books on these topics are the books by Puterman, Krishnamurthy, Borkar, and Bellman.