Rethinking Reinforcement Learning

Abstract:

Reinforcement learning (RL) has a tremendous potential for many real-world applications. Current RL algorithms, however, face challenges due to their sample inefficiency and computational cost. In this talk, I present my research program aimed at overcoming these limitations. My approach, at the high-level, is to rethink the fundamental algorithms in RL, which have been largely unchanged for decades. I focus on addressing two critical questions:

Q1: How should we learn a model in model-based reinforcement learning?

Model-based RL offers a promising approach to design sample efficient RL agents. But the conventional model learning methods may not be the right approach to learn a good model. I introduce the decision-aware model learning framework as an alternative.

Q2: How to learn the value function faster?

The value function is a cornerstone of many RL algorithms, and the Value Iteration (VI) algorithm is the basis of many RL algorithms to compute the value function. The VI algorithm, however, is slow for problems with long planning horizon. I introduce ideas that significantly accelerate VI by borrowing techniques from control theory and numerical linear algebra.

Biography:

Amir-massoud Farahmand is an assistant professor at the Department of Computer Science, University of Toronto since 2019. He was a research scientist and CIFAR AI Chair at the Vector Institute in Toronto between 2018–2024, and principal research scientist at Mitsubishi Electric Research Laboratories in Cambridge, USA between 2014-2018. He received his PhD from the University of Alberta in 2011, followed by postdoctoral fellowships at McGill University (2011–2014) and Carnegie Mellon University (CMU) (2014).

Amir-massoud’s research vision is to understand the computational and statistical mechanisms required to design efficient AI agents that interact with their environment and adaptively improve their long-term performance. He also has experience in developing RL and ML methods to solve industrially-motivated problems.

Amir-massoud has extensively published in top and selective venues in machine learning. He has served as a member of editorial board of Machine Learning Journal and Transactions on Machine Learning Research, senior program committee of flagship conferences in ML, and has won many best reviewer awards.