Alisa Devlic: Superhuman racing AI through deep reinforcement learning

Many potential applications of artificial intelligence involve making real-time decisions in physical systems while interacting with humans. Automobile racing represents an extreme example of these conditions; drivers must execute complex tactical maneuvers to pass or block opponents while operating their vehicles at their traction limits. 

Racing simulations, such as the PlayStation game Gran Turismo, faithfully reproduce the nonlinear control challenges of real race cars while also encapsulating the complex multi-agent interactions. Here we describe how we trained agents for Gran Turismo that can compete with the world’s best e-sports drivers. We combine state-of-the-art model-free deep reinforcement learning algorithms with mixed scenario training to learn an integrated control policy that combines exceptional speed with impressive tactics. In addition, we construct a reward function that enables the agent to be competitive while adhering to racing’s important, but under-specified, sportsmanship rules. 

We demonstrate the capabilities of our agent, Gran Turismo Sophy, by winning a head-to-head competition against four of the world’s best Gran Turismo drivers. By describing how we trained championship-level racers, we illuminate the possibilities and challenges of using these techniques to control complex dynamical systems in domains where agents must respect imprecisely defined human norms.

Alisa Devlic is a Senior Research Scientist at Sony AI, working on applying and advancing the state-of-the-art in deep reinforcement learning in the gaming domain. She obtained her Ph.D. in Communication Systems from KTH, Stockholm, Sweden and did her postdoctoral work at IMT Atlantique, Rennes, France. Before joining Sony AI, she worked both in industry labs (Ericsson Research and Huawei Technologies) and academia (KTH, IMT Atlantique, and University of Zagreb). Alisa (co)authored more than 30 papers and obtained best paper awards at IEEE WoWMoM 2015, ICC 2017 and MMSP 2018. She was part of the Sony AI team who developed a GT Sophy, the first racing agent that can outperform the world’s best e-sports drivers and published results in the Nature journal. She started working in the field of Reinforcement Learning in 2020, when she joined Sony AI.