Chatbots are genuinely impressive when you watch them do things they’re good at, like writing a basic email or creating weird, futuristic-looking images. But ask generative AI to solve one of those puzzles in the back of a newspaper, and things can quickly go off the rails.

That’s what researchers at the University of Colorado at Boulder found when they challenged large language models to solve sudoku. And not even the standard 9×9 puzzles. An easier 6×6 puzzle was often beyond the capabilities of an LLM without outside help (in this case, specific puzzle-solving tools).

A more important finding came when the models were asked to show their work. For the most part, they couldn’t. Sometimes they lied. Sometimes they explained things in ways that made no sense. Sometimes they hallucinated and started talking about the weather.

If gen AI tools can’t explain their decisions accurately or transparently, that should cause us to be cautious as we give these things more control over our lives and decisions, said Ashutosh Trivedi, a computer science professor at the University of Colorado at Boulder and one of the authors of the paper published in July in the Findings of the Association for Computational Linguistics.

«We would really like those explanations to be transparent and be reflective of why AI made that decision, and not AI trying to manipulate the human by providing an explanation that a human might like,» Trivedi said.

Don’t miss any of our unbiased tech content and lab-based reviews. Add CNET as a preferred Google source.

The paper is part of a growing body of research into the behavior of large language models. Other recent studies have found, for example, that models hallucinate in part because their training procedures incentivize them to produce results a user will like, rather than what is accurate, or that people who use LLMs to help them write essays are less likely to remember what they wrote. As gen AI becomes more and more a part of our daily lives, the implications of how this technology works and how we behave when using it become hugely important.

When you make a decision, you can try to justify it, or at least explain how you arrived at it. An AI model may not be able to accurately or transparently do the same. Would you trust it?

Why LLMs struggle with sudoku

We’ve seen AI models fail at basic games and puzzles before. OpenAI’s ChatGPT (among others) has been totally crushed at chess by the computer opponent in a 1979 Atari game. A recent research paper from Apple found that models can struggle with other puzzles, like the Tower of Hanoi.

It has to do with the way LLMs work and fill in gaps in information. These models try to complete those gaps based on what happens in similar cases in their training data or other things they’ve seen in the past. With a sudoku, the question is one of logic. The AI might try to fill each gap in order, based on what seems like a reasonable answer, but to solve it properly, it instead has to look at the entire picture and find a logical order that changes from puzzle to puzzle. 

Read more: 29 Ways You Can Make Gen AI Work for You, According to Our Experts

Chatbots are bad at chess for a similar reason. They find logical next moves but don’t necessarily think three, four or five moves ahead — the fundamental skill needed to play chess well. Chatbots also sometimes tend to move chess pieces in ways that don’t really follow the rules or put pieces in meaningless jeopardy. 

You might expect LLMs to be able to solve sudoku because they’re computers and the puzzle consists of numbers, but the puzzles themselves are not really mathematical; they’re symbolic. «Sudoku is famous for being a puzzle with numbers that could be done with anything that is not numbers,» said Fabio Somenzi, a professor at CU and one of the research paper’s authors.

I used a sample prompt from the researchers’ paper and gave it to ChatGPT. The tool showed its work, and repeatedly told me it had the answer before showing a puzzle that didn’t work, then going back and correcting it. It was like the bot was turning in a presentation that kept getting last-second edits: This is the final answer. No, actually, never mind, this is the final answer. It got the answer eventually, through trial and error. But trial and error isn’t a practical way for a person to solve a sudoku in the newspaper. That’s way too much erasing and ruins the fun.

AI struggles to show its work

The Colorado researchers didn’t just want to see if the bots could solve puzzles. They asked for explanations of how the bots worked through them. Things did not go well.

Testing OpenAI’s o1-preview reasoning model, the researchers saw that the explanations — even for correctly solved puzzles — didn’t accurately explain or justify their moves and got basic terms wrong. 

«One thing they’re good at is providing explanations that seem reasonable,» said Maria Pacheco, an assistant professor of computer science at CU. «They align to humans, so they learn to speak like we like it, but whether they’re faithful to what the actual steps need to be to solve the thing is where we’re struggling a little bit.»

Sometimes, the explanations were completely irrelevant. Since the paper’s work was finished, the researchers have continued to test new models released. Somenzi said that when he and Trivedi were running OpenAI’s o4 reasoning model through the same tests, at one point, it seemed to give up entirely. 

«The next question that we asked, the answer was the weather forecast for Denver,» he said.

(Disclosure: Ziff Davis, CNET’s parent company, in April filed a lawsuit against OpenAI, alleging it infringed Ziff Davis copyrights in training and operating its AI systems.)

Explaining yourself is an important skill

When you solve a puzzle, you’re almost certainly able to walk someone else through your thinking. The fact that these LLMs failed so spectacularly at that basic job isn’t a trivial problem. With AI companies constantly talking about «AI agents» that can take actions on your behalf, being able to explain yourself is essential.

Consider the types of jobs being given to AI now, or planned for in the near future: driving, doing taxes, deciding business strategies and translating important documents. Imagine what would happen if you, a person, did one of those things and something went wrong.

«When humans have to put their face in front of their decisions, they better be able to explain what led to that decision,» Somenzi said.

It isn’t just a matter of getting a reasonable-sounding answer. It needs to be accurate. One day, an AI’s explanation of itself might have to hold up in court, but how can its testimony be taken seriously if it’s known to lie? You wouldn’t trust a person who failed to explain themselves, and you also wouldn’t trust someone you found was saying what you wanted to hear instead of the truth. 

«Having an explanation is very close to manipulation if it is done for the wrong reason,» Trivedi said. «We have to be very careful with respect to the transparency of these explanations.»

While most earbud options fit tightly into your ears to help shut out any background noise, they aren’t for everyone. They can be uncomfortable, so picking up a pair of open earbuds is the go-to alternative for many.

These clip-on Baseus Bowie MC1 earbuds are a great option, and their usual $70 asking price is already a bit of a bargain. And they’re even better now that you can pick them up for just $47. You even get to choose from black and white colors when placing your order, too.

The Baseus Bowie MC1 earbuds are our pick for the best affordable option when it comes to open-ear earbuds. Our editors love them even at full price, so a chance to grab them for even less is an amazing value. They feature a unique clip-on design that wraps around your ears, making them secure and comfortable to wear.

Hey, did you know? CNET Deals texts are free, easy and save you money.

The buds offer up to 9 hours of battery life or up to 40 with the charging case, as well as an IP57 certification, which makes them splash- and dust-proof. Plus, they’re pretty good when it comes to phone calls.

Why this deal matters

Baseus’ Bowie MC1 earbuds are already our pick for the best affordable open-ear earbuds. So, when you can grab them at their lowest-ever price, you’ll get even more value out of them. Plus, ongoing tariff concerns mean these earbuds could be getting more expensive in the coming months, so grab yourself a pair while they’re still extremely affordable.

There are plenty of earbud options out there, but finding the right ones for you can feel nearly impossible as a result. Thankfully, if you’re after some of the best noise-canceling wireless earbuds, and you’d like to save some money, then we might have the answser for you.

The excellent Anker Soundcore Liberty 4 NC earbuds are currently on sale with 43% off, which means they’re down to $57. This is the lowest we’ve seen them, but this is also a limited-time deal, which means it’s not likely to last for long.

These earbuds have 11mm drivers, as well as hi-res wireless, LDAC technology and adaptive noise canceling that should reduce noise from your environment by up to 98.5%, according to Anker. That means you can stay immersed in the music (or other content you love) regardless of what’s going on around you.

Plus, because they’re equipped with six mics, they offer good call quality, regardless of your environment. We found them comfortable, too, and loved the bass they put out.

Hey, did you know? CNET Deals texts are free, easy and save you money.  

The Liberty 4 NC earbuds have an impressive battery life as well, offering up to 10 hours of playback per charge, with an additional 50 hours available with the charging case. And just 10 minutes of fast charging can get you up to four hours of playback. They’re also rated IPX4 water-resistant, so any inclement weather while you’re out shouldn’t cause these buds any harm.

Another great feature these earbuds offer is multipoint connection, which lets you connect to your phone and your computer at once, which is super convenient. That’s a lot of features for earbuds this affordable.

If you’re not totally sold on this model, be sure to check out our roundup of all the best headphone and earbuds deals happening now. 

Why this deal matters

These excellent earbuds are discounted fairly close to their record-low price right now. During Black Friday we saw these Liberty 4 NC buds hit a new low of $60, and right now, they’re $3 cheaper still. Amazon has labeled this as a limited-time deal, so we expect it to expire soon.