April 13, 2017: Invited Speaker Seminar: Practicable Robust Markov Decision Processes

Dr. Huan Xu
Georgia Tech

Thursday, April 13, 2017 at 2:00 pm
Room EV003.309

Abstract

Markov decision processes (MDP) is a standard modeling tool for sequential decision making in a dynamic and stochastic environment. When the model parameters are subject to uncertainty, the "optimal strategy" obtained from MDP can significantly under-perform than the model's prediction. To address this, robust MDP has been developed which is based on worst-case analysis. However, several restrictions of the robust MDP model prevent it from practical success, which I will address in this talk. The first restriction of standard robust MDP is that the modeling of uncertainty is not flexible and can lead to conservative solution. In particular, it requires that the uncertainty set is "rectangular" - i.e., it is a Cartesian product of uncertainty sets of each state. To lift this assumption, we propose an uncertainty model which we call “k-rectangular" that generalizes the concept of rectangularity, and we show that this can be solved efficiently via state augmentation. The second restriction is that it does not take into account the learning issue - i.e., how to adapt the model in an efficient way to reduce the uncertainty. To address this, we devise an algorithm inspired by reinforcement learning that, without knowing the true uncertainty model, is able to adapt its level of protection to uncertainty, and in the long run performs as good as the minimax policy as if the true uncertainty model is known. Indeed, the algorithm achieves similar regret bounds as standard MDP where no parameter is adversarial, which shows that with virtually no extra cost we can adapt robust learning to handle uncertainty in MDPs.

Biography

Dr. Huan Xu is an assistant professor at the Stewart School of Industrial & Systems Engineering at Georgia Tech. His current research interest focuses on data, learning, and decision making. Specifically, he is interested in machine learning, high-dimensional statistics, robust and stochastic optimization, sequential decision making, and application to large-scale systems. He is currently an associate editor of both IEEE Transactions on Pattern Analysis and Machine Intelligence and Computational Management Science. He received his B. Eng. from Shanghai Jiaotong University in 1997, his M. Eng from the National University of Singapore in 2003, and his Ph.D. from McGill University in 2009. Before joining Georgia Tech, he was an assistant professor at the National University of Singapore, after a postdoc stint at UT-Austin.