Photorealism in Grand Theft Auto V

GTA V, greener and more real. © Richter et. al./Intel ISL/Rockstar Games.

Researchers Stephan Richter, Hassan Abu AlHaija, and Vladlen Koltun at Intel Labs have developed a technique based on machine-learning that adds an astonishing level of photorealism to the game Grand Theft Auto V. The method overlays more natural colors and improves reflections, shadows, textures and other visual subtleties. “Our method modifies images from the game to look more realistic. It is a convolutional network which produces images frame by frame and can be run at interactive rates,” the researchers say in the video, adding that the dataset, called Cityscapes, contains “mostly images of German cities” taken from vehicles. This Germanic set has the effect of greening the normally sun-dried California hills in the game.

However, you won’t be installing this mod anytime soon. While the scientists believe their approach is the most robust and consistent of all AI-based systems, its photorealism is still limited by the availability of samples. It would take many, many more images to improve all aspects of the game and interior scenes in particular. In addition, the operation gobbles up computing power: even with a powerful GeForce RTX 3090 graphics card, the unoptimized code introduces half-a-second of inference. Besides, do people really want photorealistic games?

⇨ YouTube, “Enhancing Photorealism Enhancement.”

⇨ YouTube, “Vladlen Koltun: Further Towards Photorealism (May 2021).”

⇨ Github, Stephan Richter, Hassan Abu AlHaija & Vladlen Koltun, “Enhancing Photorealism Enhancement.”

⇨ Engadget, Jon Fingas, “‘Grand Theft Auto V’ mod adds uncanny photorealism through AI.”

2021-05-13