points by yewenjie 4 days ago | 0 comments
> The training data will degrade in quality if more people rely on LLMs so I'd expect models to eventually start getting worse even if they currently enjoy apparent improvements in output.
RL is not dependent on training data. The mathocalypse didn't happen because they found better training data somewhere.
[–] DamnInteresting 4 days ago
[–] Shaanveer 4 days ago
sorry but
given how post-training actually works, there's essentially no direct way in which literal PII-scrubbed data made its way into a production oai datamix for RL warmup purposes in 2026 to the extent this data has influence, it's indirect/at the level of process decisions.
what i am trying to say more specifically is, the folk model some people seem to still have about ai companies "directly training on user transcripts" in this naive fashion is firstly wrong, and secondarily, would be counterproductive as a strategy for continued progress
"transcript shows human trying to do an exotic thing evals don't currently cover -> build synth constructions that broadly cover its genre of problem class -> RL on the verifier set designed to cover the failure"
that's the loop...
the """flywheel""" is... diagnostics! wow!
and so this kind of process improvement driven use of data has an obvious diffuse causal influence on what happens during training, right?
but ~none of that process influence looks like "specific mathematician's specific intuitions leak into the training data".
[–] grey-area 4 days ago
It’s perfectly plausible that unscrupulous people at OpenAI (and that’s the culture right from the top) searched user transcripts for relevant approaches to problems they were working on, or searched them for problems to look at, and nudged models based on that.
I agree it’s unlikely this happened by accident, or entered training data (though not impossible), but it could certainly be within the context agents were using when tackling problems.
We have no evidence either way, but they had all the data they would need to do this.
[–] RugnirViking 4 days ago
saw an interview from the mathematician involved, Tristan Buckmaster, on the numberphile yt channel, and he said he thinks it was more a case of people at anthropic leaking it in person to openai. Apparently with a hint about the area they were looking at it could have been easy (ish) to get the same approach, and indeed the paper openai released for Navier–Stokes uses their approach
[–] greysphere 4 days ago
Rl works on one of two things - (1) a literal set of rules for a system or (2) training data. The former matches perfectly to things like (most modern) math and games that have perfect sets of rules. Even things like physics and robotic where our rules/models are extraordinarily good, rl barely adds value. Move one step farther to chemistry or a complicated physics scenario like weather, and you are back almost entirely dependant on (2) data. Now there might be a few more domains with rigorous rulesets that haven't been explored fully (id bet with some effort programming might make some strides here, maybe something like circuit layout, maybe even law if folks decided to make a lean-like law scheme...) but the vast majority of problems AI is targeting are dependant on data.
[–] rsfern 4 days ago
I’m not sure that’s a good argument that models will get good at novel brainstorming and research ideation. Where do you get scalable and high signal verifiable rewards for creativity and insight?
There are reasons to suspect that OpenAI may have intentionally or unintentionally trained on chat logs from mathematicians--mathematicians who were actively using OpenAI to try to to solve the same problems themselves.[1] If those mathematicians were on the right track, and getting close to a solution, that could be classified as finding "better training data somewhere."
[1] https://cybernews.com/ai-news/openai-accused-stealing-anothe...