Artificial intelligence models still need to resolve certain details when it comes to generating images, but no one can deny that they have advanced by leaps and bounds, and they made a lot of noise when they crossed the video frontier. However, the folks at Google Research decided to explore another potential for them: their use as a graphics engine. The result is GameNGen, derived from version 1.4 of Stable Diffusion… and the chosen game for the first simulation was Doom. Rip and tear, with neural models.
Using algorithms to develop video games is nothing new. For example, procedural generation has been with us for decades (from Elite in 1984 to Astroneer in 2019, to name two cases), and beyond its inevitable limitations, it can provide good results under the right conditions.
Obviously, with the generative storm over our heads, not a few are looking to adopt that technology to create objects and/or textures, but a team composed of members from Google Research, Google DeepMind, and the Tel Aviv University decided to follow another path: transforming a generative model into a graphics engine that creates real-time environments. What's its name? GameNGen.
GameNGen: Diffusion Model as a Real-Time Game Engine, ft. Doom
The official page explains that GameNGen is the first graphics engine powered by a neural model, which enables real-time interaction with complex spaces and maintains a quality relatively constant throughout the session. In fact, GameNGen can simulate the classic version of Doom at 20 FPS using just one TPU. In other words, this graphics engine combines the processing of previous frames with player commands to generate new frames without losing cohesion. The whole level is built around it.
The first step was to train an agent to learn to play Doom, with a system of penalties (accumulated damage and deaths) and rewards (collecting items, killing enemies, and finishing the level). However, the most notable thing is that they reused Stable Diffusion 1.4 for the generation process. Stable Diffusion is not particularly robust at maintaining consistency between frames, and tends to suffer from drifting, but they solved that problem with contextual frames contaminated with Gaussian noise, giving rise to a kind of "self-correction system".
GameNGen is far from perfect, and a trained eye will quickly spot various artifacts (the eyebrows and eyes of the Doomguy are a good starting point), but as a simulation of Doom, it remains impressive.
Official site and paper: Click here. Source: Tom's Hardware.
