ChatGPT Lost a Chess Game to an Atari 2600!
ChatGPT might be great for responding to emails or quick-drafting a document, but it's not quite ready to take on the world's chess prodigies—or indeed, chess-playing consoles from 50 years ago. In a unique experiment, an engineer pitted the latest ChatGPT 4o model against his Atari 2600's chess engine on the beginner difficulty level, and ChatGPT got handily defeated, eventually conceding.
Since IBM's Deep Blue supercomputer defeated the then-world chess champion and legendary player, Gary Kasparov, in 1997, chess engines have held a commanding lead over their human counterparts. The latest Stockfish models have an estimated ELO (chess rating) of over 3,600, while the best chess players in the world have only ever crested 2800. ChatGPT and the Atari 2600 are both well below either of these ratings, but the matchup is intriguing nonetheless.
In it, Citrix Engineer Robert Caruso said that after chatting with ChatGPT about the history of chess, he wanted to find out "how quickly" it would defeat a chess computer that can only think one or two moves ahead. So he pulled out an emulation of Atari's 1979 Video Chess cartridge and had ChatGPT analyze board positions based on images to decide its next moves. He expected it to be a cakewalk for the cutting-edge large language model, which cost tens of millions of dollars in training alone.
It turned out to be anything but.
"ChatGPT got absolutely wrecked on the beginner level," Caruso said in his LinkedIn post. "Despite being given a baseline board layout to identify pieces, ChatGPT confused rooks for bishops, missed pawn forks, and repeatedly lost track of where pieces were—first blaming the Atari icons as too abstract to recognize, then faring no better even after switching to standard chess notation."
While the emulated Atari console's software wasn't exactly masterminding its own moves, they were enough to prove too much for ChatGPT. After an hour and a half, and even with Caruso helping ChatGPT from making some of its most catastrophically poor moves, it conceded. But not before asking if it could "start over" for another go.
It's fair to point out that ChatGPT isn't a chess computer. But it is often described in incredibly lofty terms by its creators at OpenAI, and AI evangelists who claim we're on the precipice of wider general AI development, and in untold job losses and societal upheaval.
While ChatGPT doesn't need to beat an old chess computer to do any of that, it might stand a greater chance of success if it didn't fall flat when asked to do things outside its wheelhouse.
Perhaps it's better we keep using it for more mundane tasks for now.
