HN Reader

To be fair to the OpenAI team, if read in context the situation is at worst ambiguous.

The deleted tweet that the article is about said "GPT-5 just found solutions to 10 (!) previously unsolved Erdös problems, and made progress on 11 others. These have all been open for decades." If it had been posted stand-alone then I would certainly agree that it was misleading, but it was not.

It was a quote-tweet of this: https://x.com/MarkSellke/status/1979226538059931886?t=OigN6t..., where the author is saying he's "pushing further on this".

The "this" in question is what this second tweet is in turn quote-tweeting: https://x.com/SebastienBubeck/status/1977181716457701775?t=T... -- where the author says "gpt5-pro is superhuman at literature search: [...] it just solved Erdos Problem #339 (listed as open in the official database erdosproblems.com/forum/thread/3…) by realizing that it had actually been solved 20 years ago"

So, reading the thread in order, you get

  * SebastienBubeck: "GPT-5 is really good at literature search, it 'solved' an apparently-open problem by finding an existing solution"
  * MarkSellke: "Now it's done ten more"
  * kevinweil: "Look at this cool stuff we've done!"

I think the problem here is the way quote-tweets work -- you only see the quoted post and not anything that it in turn is quoting. Kevin Weil had the two previous quotes in his context when he did his post and didn't consider the fact that readers would only see the first level, so wouldn't have Sebastien Bubek's post in mind when they read his.

That seems like an easy mistake to entirely honestly make, and I think the pile-on is a little unfair.

“Mathematician Thomas Bloom, who runs erdosproblems.com, pushed back right away. He called the statements "a dramatic misinterpretation," clarifying that "open" on his site just means he personally doesn't know the solution - not that the problem is actually unsolved.”

What mathematician uses this as the definition for “open”? I don’t go around saying that most problems in this textbook are open questions, just because I don’t know how to do them.

> GPT-5 is proving useful as a literature review assistant

No, it does not. It only produces a highly convincing counterfeit. I am honestly happy for people who are satisfied with its output: life is way easier for them than for me. Obviously, the machine discriminates me personally. When I spend hours in the library looking for some engineering-related math made in the 70s-80s, as a last resort measure, I can try to play this gambling with chat, hoping for any tiny clue to answer my question. And then for the following hours, I am trying to understand what is wrong with the chat output. Most often, I experience the "it simply can't be" feeling, and I know I am not the only one having it.

Yann LeCun's "Hoisted by their own GPTards" is fantastic.

After the circular financing schemes involving hundreds of billions of dollars were uncovered, nothing I read about the AI business and its artificial hype machine surprises me anymore.

Thanks for calling that out. You're right to be upset.

The sad truth about this incident is that it reveals that OpenAI does not have a serious effort to actually work on unsolved math problems.

Wouldn't be surprised if OpenAI employees are being asked to phrase (market) things this way. This is not the first time they claimed GPT-5 "solved" something [1]

[1] https://x.com/SebastienBubeck/status/1970875019803910478

edit: full text

It's becoming increasingly clear that gpt5 can solve MINOR open math problems, those that would require a day/few days of a good PhD student. Ofc it's not a 100% guarantee, eg below gpt5 solves 3/5 optimization conjectures. Imo full impact of this has yet to be internalized...

“AGI achieved internally”

Another case of culture flowing from the top I guess.

You are telling me a language model trained on Reddit can't solve novel problems? Shocking.

Edit: we are in peak damage control phase of the hype cycle.

This happening the same week as DeepMind’s seemingly legitimate AI-assisted cancer treatment breakthrough is a startlingly bad look for OpenAI.

My boss always used to say “our only policy is, don’t be the reason we need to create a new policy”. I suspect OpenAI is going to have some new public communication policies going forward.

I try not to lose sight of the first time that I heard (some years back) that people were using this new LLM thing for DM'ing ("dungeon mastering", leading) a game of Dungeons and Dragons. I thought, this must be bullshit or some kind of witchcraft.

Definitely not anti-AI here. I think I have been disappointed though, since then, to slowly learn that they're (still) little beyond that.

Still amazing though. And better than a Google search (IMHO).

Was that "researcher" just a hallucinating LLM?

You would think Open AI employees have a pretty good grasp of their model capabilities, but even if you don’t, you probably always want to be on the cautious side for every claim you see on the internet.

This just seems to be the Open AI culture, which for better or worse has helped foster the AI hype environment we are currently in.

Whatever happened to "don't get high on your own supply"?

Imagine having your retirement fund manager investing in a couple million more based on the original tweet. Very cool.

This honestly doesn’t surprise me. We have reached a point where it’s becoming clearer and clearer that AGI is nowhere to be seen, whereas advances in LLM ability to ‘reason’ have slowed down to (almost?) a halt.

> Summary (from the article)

* OpenAI researchers claimed or suggested that GPT-5 had solved unsolved math problems, but in reality, the model only found known results that were unfamiliar to the operator of erdosproblems.com.

* Mathematician Thomas Bloom and Deepmind CEO Demis Hassabis criticized the announcement as misleading, leading the researchers to retract or amend their original claims.

* According to mathematician Terence Tao, AI models like GPT-5 are currently most helpful for speeding up basic research tasks such as literature review, rather than independently solving complex mathematical problems.

I make mistakes all the time. This seems like a genuine mistake, not malice.

Imagine if you were talking about your own work online, you make an honest mistake, then the whole industry roasts you for it.

I’m so tired of hearing everyone take stabs at people at OpenAI just because they don’t personally like sama or something.

How fing obvious was it that AI slop did not do anything other than scarpe some websites.

Making such a claim should at the very least require proof that the information wasn’t in the training data.

The original tweet was clearly misunderstood...

https://x.com/SebastienBubeck/status/1977181716457701775:

> gpt5-pro is superhuman at literature search:

> it just solved Erdos Problem #339 (listed as open in the official database https://erdosproblems.com/forum/thread/339) by realizing that it had actually been solved 20 years ago

https://x.com/MarkSellke/status/1979226538059931886:

> Update: Mehtaab and I pushed further on this. Using thousands of GPT5 queries, we found solutions to 10 Erdős problems that were listed as open: 223, 339, 494, 515, 621, 822, 883 (part 2/2), 903, 1043, 1079.

It's clearly talking about finding existing solutions to "open" problems.

The main mistake is by Kevin Weil, OpenAI CTO, who misunderstood the tweet:

https://x.com/kevinweil/status/1979270343941591525:

> you are totally right—I actually misunderstood @MarkSellke's original post, embarrassingly enough. Still very cool, but not the right words. Will delete this since I can't edit it any longer I think.

Obviously embarassing, but completely overblown reaction. Just another way for people to dunk on OpenAI :)

Humans hallucinating about AI.

This entire thing has been pretty disingenuous on both sides of the fence. All the anti-AI (or anti OpenAI) people are doing victory laps, but what GPT-5 Pro did is still very valuable.

1) What good is your open problem set if really its a trivial "google search" away from being solved. Why are they not catching any blame here?

2) These answers still weren't perfectly laid out for the most part. GPT-5 was still doing some cognitive lifting to piece it together.

If a human would have done this by hand it would have made news and instead the narrative would have been inverted to ask serious questions about the validity of some these style problem sets and/or ask the question how many other solutions are out there that just need pieced together from pre-existing research.

But, you know, AI Bad.

Another instance of openAI manipulating results to prolong their unsustainable circular hype bubble.

The inevitable collapse could be even more devastating than the 2008 financial crisis.

All while so vast resources are being wasted on non-verifiable gen AI slob, while real approaches (neuro-symbolic like DeepMind's AlphaFold) are mostly ignored financially because they don't generate the quick stock market increases that hype does.

This is just tit-for-tat clickbait. The researcher’s wording was a bit unclear for sure, but far from incorrect.