Researcher teaches GPT-4 to play DOOM and kill everything in sight

We all know about the 1993 shooter DOOM and at that time, any hardware was tested with the question: Will it play DOOM? The Register spotted a student project that proves GPT-4’s capability to play the old game.

Adrian de Wynter, a principal applied scientist at Microsoft and a researcher at the University of York in England, developed a method to test if GPT-4’s reasoning and planning capabilities extend to the 1993 first-person shooter Doom. As it turns out, yes, the bot can run and play it, but you could beat it easily.

How can GPT-4 run DOOM?

In fact, it’s more complicated because the GPT-4 LLM can’t run DOOM’s code directly, but its variant, GPT-4V, can also work with images and play the game without any special training.

So, de Winter used GPT-4V to get the screenshots from the game, together with structure descriptions. Then, an Agent model can make decisions based on the visual input and the previous history. The commands to the game are sent back and translated into keystrokes.

That makes the model pretty slow and sluggish in playing the game, but it has other problems as well:

For example, it would be very common for the model to see a zombie on the screen, and start firing at it until it hit it (or died). What’s the issue here? Well, first that the zombie goes out of view. Worse, it is still alive and will whack you at some point. So you gotta go after it, right? After all, in Doom, it’s whack or be whacked. It turns out that GPT-4 forgets about the zombie and just keeps going. Note: the prompt explicitly tells the model what to do if it is taking damage and it can’t see an enemy. Better yet, it just goes off on its merry way, gets stuck in a corner, and dies. It did turn around a couple of times, but in nearly 50-60 runs, I observed it… twice, I wanna say.

Adrian de Wynter

So, GPT-4V just forgets about the enemies if they get out of sight, which could be a hilarious problem even for the most inexperienced human player.

But the problem here is not the bot’s poor skills in killing monsters in a game but the ethical ramifications of the programming:

On the ethics department, it is quite worrisome how easy it was for (a) me to build code to get the model to shoot something; and (b) for the model to accurately shoot something without actually second-guessing the instructions,

Adrian de Wynter

In other words, the AI was more than happy to follow the instructions for shooting targets without asking any questions. Although it can be fun to see an AI playing a game and making its own decisions to complete its programming goals, the researcher’s conclusion is that the bot has no idea what it’s doing.

After all, this programming could be used in real scenarios for automated drones that could produce real damage.

What do you think about GPT-4 playing DOOM? Let’s talk about that in the comments section below.