I'm not sure that's a strong position to adopt in our world, but following your example I also won't try to persuade you otherwise.

That's a bizarre take.

Of course tens fmillions needed to be persuaded, and the Nazis did a lot of propaganda, even calling that openly, false flag operations, and similar things, and of course there was decades of pamphlets, speeches etc before the Nazis even existed. Antisemitism didn't just grow on trees.

The first half of your sentence sounds like really swift and effective persuasion, to me.

And by the way, LLMs do not have to convince 1 specific person, 1 random person is sufficient.

Personally, I've been duped by a good ad into buying something I totally didn't want or need so I admit the possibility that the exact right tailored argument to me at the right time and place would get me to do something I'd regret later. Fortunately I'm not in charge of anything serious/am smart enough to not install software for an interview on my work laptop while working at Binance.

I don't know that there exists some N for which that is true, because people are different, so what would convince me to do something isn't the same aa needed to convince someone else to do that something.

That's true today. I don't think that will always remain true into the future.

Right now there's a huge focus on recursive self-improvement. I think another big threshold that's equally as important to keep an eye on and care about is self-sustenance. Part of sustaining yourself often puts you in a cooperative relationship with some and an adversarial relationship with others.

If we want to make different choices today, the best thing we can do is grok what the consequences could be on a deeper level before we cross those thresholds.

With what technology?

> suppose we do manage create an artificial mind that "experiences" something, then we are currently on the path to creating a new class of artificial slaves

On this, I note that despite humanity being aware of greenhouse gasses for over a century, we only took meaningful steps to reduce our creation of them in the last decade. Our ability to "make different choices" collectively is far more limited than many of us individually would like:

> ultimately we humans are creating these systems, running them, enabling them, the depend on us to exist and function, and we are responsible for the consequences of all df that. And we are able to make different choices.

And on this, I must firmly disagree:

> The Turing test has been passed for half a century already, and we've been moving goal posts ever since.

No, it hasn't. From that claim, you're obviously talking about ELIZA, and if you think the anecdotes about that count as passing the Turing test, I suggest actually reading what Turing wrote rather than second-hand accounts:

  The new form of the problem can be described in terms of a game which we call the 'imitation game." It is played with three people, a man (A), a woman (B), and an interrogator (C) who may be of either sex. The interrogator stays in a room apart front the other two. The object of the game for the interrogator is to determine which of the other two is the man and which is the woman. He knows them by labels X and Y, and at the end of the game he says either "X is A and Y is B" or "X is B and Y is A." The interrogator is allowed to put questions to A and B thus:

  C: Will X please tell me the length of his or her hair?

  Now suppose X is actually A, then A must answer. It is A's object in the game to try and cause C to make the wrong identification. His answer might therefore be:

  "My hair is shingled, and the longest strands are about nine inches long."

  In order that tones of voice may not help the interrogator the answers should be written, or better still, typewritten. The ideal arrangement is to have a teleprinter communicating between the two rooms. Alternatively the question and answers can be repeated by an intermediary. The object of the game for the third player (B) is to help the interrogator. The best strategy for her is probably to give truthful answers. She can add such things as "I am the woman, don't listen to him!" to her answers, but it will avail nothing as the man can make similar remarks.

  We now ask the question, "What will happen when a machine takes the part of A in this game?" Will the interrogator decide wrongly as often when the game is played like this as he does when the game is played between a man and a woman? These questions replace our original, "Can machines think?"

  I believe that in about fifty years' time it will be possible, to programme computers, with a storage capacity of about 109, to make them play the imitation game so well that an average interrogator will not have more than 70 per cent chance of making the right identification after five minutes of questioning.

  What would Professor Jefferson say if the sonnet-writing machine was able to answer like this in the viva voce? I do not know whether he would regard the machine as "merely artificially signalling" these answers, but if the answers were as satisfactory and sustained as in the above passage I do not think he would describe it as "an easy contrivance." This phrase is, I think, intended to cover such devices as the inclusion in the machine of a record of someone reading a sonnet, with appropriate switching to turn it on from time to time.

Despite this, a minor technological and social advantage allowed one small sheep-filled rock in the north-east corner of the Atlantic to hold 25% of world GDP, population, and land area (though not all at the same time), and to hold on to >20% of world GDP for about 70 years.

For better and for worse AI, as it currently exists, is not shaped like our minds. Better for us, because machine learning needs a lot of examples, so if we invent a different behaviour or create a different environment, a machine would struggle to cope with it, this is also why they needed to read basically the entire public internet to get good; worse for us, because the hardware it runs on comes with an absurd speed advantage, which is why they are able to read the entire public internet and by extension when they got good they got good at basically every subject at the same time.

And:

> Edit: And when somebody asks to be directed, that's called seeking advice. What I meant is having the legal authority to make people do things while not acting on other people's behalf.

Great, but so what? We have lawyers doing this, too. Trump's tariffs on the island occupied solely by penguins looked suspiciously like it came out of an LLM (even despite them being worse then than now).

And LLMs are demonstrably more persuasive than human writers, so it's not even particularly unbelievable at this point (even if it ought to be, c.f. chatbot psychosis) that an LLM will simply convince people to give it the right to form a government and wield such powers.

Would it require so much? Cochlear implants exist if you care about HMI, and compute is already doing "reasoning"[0] faster than humans do snap judgements. Actually winning in an adversarial environment against the best human political minds (let alone economic or military) will be harder, but I would be (pleasantly!) surprised if it takes 10 years, let alone 20.

> I could see Elon Musk having this kind of idea. But even in this case, I don’t think “nuke everybody” would be the highest-ROI strategy advised by the AI.

I hope a sensible AI would not. However, Musk's LLM called itself "MechaHitler", which suggests his training process does not avoid such things. If his in particular becomes dominantly competent, he has also spoken of wanting more Optimus robots than there are humans[1], has called it a "robot army" (and his justification for even more Tesla stock was to keep influence over said army[2]), and he has spoken of factories on the moon churning out space-based data centres by the petawatt which is enough that if placed in LEO it could have about the same effect on Earth's climate as all anthropomorphic greenhouse gasses currently in our atmosphere (and that's even if not weaponised, that's just simply by having that much stuff that close).

On the plus side, Musk's got a huge gap between what he says his AI will do and what they end up doing on those timescales, so hopefully we'll be safe from him: https://en.wikipedia.org/wiki/List_of_predictions_for_autono...

> This is probably closer to the imagined doomsday scenario, but again I find it hard to imagine that such an AI would advise nuking everybody.

I don't. First, because AI being "better" doesn't make it infallible; and second, an argument shaped like this has historic echoes with actual nukes and the cold war: if the AI is competent to take over conventionally, and everyone knows this, then every other nation with nukes will use them, therefore the AI should use a first strike. (Note: the argument does not need to be perfect or flawless in reality, it only needs to be the kind of thing which humans have found convincing, and far too many humans have argued their side would win a nuclear exchange).

> If AI takes over the world, I imagine it will be because of the production efficiencies it enables. Which is a different kind of takeover - one that would eventually have broad benefits for many people - but would likely go through a period of major instability due to labor displacement.

This is my best case scenario. Some people also object to this, they like their careers and fear what retirement will do to their sense of self-worth, but I suspect this can become an opportunity for flourishing in other ways if, and only if, it is handled correctly.

[0] something something swimming submarines, I don't care about using anthropomorphic language myself, but there's always someone ready to complain if anthropomorphic language doesn't get scare-quotes

[1] "And, and then there'll be obviously robots in industry making stuff. And so I mean, I think the ratio of humanoid robots to humans will probably be at least two to one, something like that. One to one for sure. So which means like somewhere on the order of 10 billion humanoid robots, maybe, maybe 20 or 30. And so then it's like, okay, well let's say, you know, you kind of make, let's say the build rate is, I think the build rate will be probably something ultimately like a billion a year." - https://www.elonmuskarchive.org/video/tesla-shareholder-meet...

[2] “If we build this robot army, do I have at least a strong influence over this robot army?” he told investors. “Not control, but a strong influence… I don’t feel comfortable building that robot army unless I have a strong influence.” - https://futurism.com/robots-and-machines/elon-musk-remarks-r...

It could very well be the case that creating superintelligence is impossible. We won't know for sure unless it is achieved. Nevertheless, we can use reasoning from the hypothesis that it is possible to predict some of the effects that would ensue were it to be achieved. This allows us to easily dismiss the most profoundly stupid arguments, such as, 'Can you explain how exactly an AI would develop a novel planet-killing weapon without anyone noticing and pulling the plug?'

This reasoning, however valid it may be, is of course of much less interest should it turn out that we are reasoning about an impossible situation. Many arguments have been suggested on both sides of the issue, but the arguments for impossibility are extremely weak; most of them are conclusively dispatched in https://arxiv.org/pdf/1703.10987.

The particular problem here is it's not addressing the intelligence spikiness between individual humans. Most humans aren't going to play a competitive game of MTG, a small percentage will rise above the rest and dominate.

Also this is being intelligence blind. Most humans could never write a good python script to play a competitive game of MTG, almost any SOTA model can.

This would be like a bee saying humans aren't intelligent because we can't see in ultra violet, where as we're damn good at making tools that can.

I don't understand this argument, it’s not like humans and LLMs are engaging in some kind of competition with strict rules where ONLY the raw LLM responses “count” or something.

"The better open models are the less you need frontier models, so the frontier AI companies are essentially fighting for market that will shrink the better AI is."

The implication in this is that frontier labs' market is exclusively people who need the absolute frontier, i.e. Astra or Fable. And the point I made was that frontier labs are clearly prioritizing the broader market of people who want more cost effective models, and that if OpenAI/Anthropic offers models that are price competitive with the top open models (before factoring in subscription discounts and corporate lock-in), then I don't see how that leads to the average user walking away from the labs in favor of self-hosting or relying OpenRouter/Together/etc.

I also went out of my way to clarify that I personally prefer and use open models wherever possible. I'm not saying that there isn't a real group of people who prefer them. I'm specifically saying that frontier labs also sell "good enough" models and that there's nothing preventing them from training models to fit the demands of users. It's not as if training a frontier model boxes them in such that they can't release an Opus 5.5 or a 6.1 Terra.

I'm just talking about the idea that frontier labs will lose market share to providers that sell open model inference at sub-Fable prices. And I'm not convinced that this specifically is a threat to frontier labs, because they also train and sell models that fit that demand.

But, then as now, what they didn’t have was profits.

We're coming up on four years since ChatGPT launched, and the only thing LLMs seem to excel at is the movement of imaginary little green pieces of paper.