points by WithinReason 1 day ago | 0 comments
> Let's also imagine for the purpose of conversation that one of the things that can be gotten at an extremely high skill level through increased intelligence is the power of persuasion.
Definitely not true as you can't use reinforcement learning to increase that capability. Only LLM capabilities that you can improve in an RL loop grow substantially.
[–] dTal 12 hours ago
Uh this seems exactly backwards - reinforcement learning with human feedback teaches only the power of persuasion. The only way to get rewarded is to persuade a human to reward you. Why do you think the models are all so damn sycophantic?
[–] WithinReason 9 hours ago
You can't put RLHF in an automated recursive improvement loop because you need humans for it.
[–] c1ccccc1 23 hours ago
> Definitely not true as you can't use reinforcement learning to increase that capability.
What? Citation needed.
[–] brabel 22 hours ago
Well don’t you need a bunch of humans to test your persuasion capabilities? How would you run improvement tests in a loop??
[–] swid 22 hours ago
You’d launch something like ChatGPT and get millions of people to use it everyday.
[–] pigpop 17 hours ago
What is the measurement? I don't think most people come back to a conversation and say "gee you really convinced me to do X, here's the proof that I did it" and even if they did that data would still be very messy and require verification.
I think this is very unlikely, but world changing if true.
Additionally, what does super intelligence even mean and why would we expect it to be obtainable from the data that we have?
I'd be delighted if someone can provide some definitions for these terms.