Hacker News

points by kragen 1 day ago | 0 comments

I think you lack imagination, but I'm not going to post ideas here for novel weapons that would render the humans' existing defenses and deterrence strategies ineffective.

It's worth pointing out that the humans do not have planet-killer weapons, not even biosphere-killing weapons. They could destroy civilization, but not the planet or even the species. You can make straightforward calculations showing that we already know how to build such weapons, but we haven't.

As for tools, alcohol, tobacco, fentanyl, pornography, monotheism, warfare, gambling, and money itself are "tools wielded by humans", but they still seem to get into a lot of trouble where it seems like the humans are the ones being "wielded" by the tool. And neither monotheism nor gambling is smart enough to write a compiler for me, much less choose persuasive tactics according to my personality type. A lot of humans' basic survival instinct seems to fail or even work backwards in the presence of these "tools".

[–] strange_quark 1 day ago

> It's worth pointing out that the humans do not have planet-killer weapons, not even biosphere-killing weapons.

?? We for sure have enough nukes to completely obliterate ourselves. And the ensuing nuclear winter and ecological collapse would probably destroy almost anyone who survived the initial blast and fallout.

Can you explain how exactly an AI would develop a novel planet-killing weapon without anyone noticing and pulling the plug? Like I seriously don’t understand this argument.

[–] krelian 1 day ago

>Can you explain how exactly an AI would develop a novel planet-killing weapon without anyone noticing and pulling the plug? Like I seriously don’t understand this argument.

That's the danger with creating a super intelligence, that you won't understand it. My dog is pretty smart, has good instincts and I'm quite confident he can survive in nature all by himself. But if a human decided to harm him, he doesn't stand a chance.

[–] root_axis 22 hours ago

> That's the danger with creating a super intelligence, that you won't understand it

This argument is sophistry because it assumes the conclusion in its premise. It works backwards from extinction by proposing a hypothetical computer program that definitionally cannot be understood, predicted or controlled.

A substantive argument would propose reasoning that explains why we should believe such a program could be created.

[–] kragen 22 hours ago

No, that is not how reasoning works. It is perfectly valid reasoning to begin by positing entities that we may not be possible to create, such as supersonic airplanes, coherent light, quantum computers, rockets that can reach escape velocity, black holes, or infinitely long lines that are perfectly straight and infinitely thin. Indeed, you might say that the ability to reason about the consequences of premises that we do not know to be true is the thing that distinguishes reasoning, as such, from simple parroting.

It could very well be the case that creating superintelligence is impossible. We won't know for sure unless it is achieved. Nevertheless, we can use reasoning from the hypothesis that it is possible to predict some of the effects that would ensue were it to be achieved. This allows us to easily dismiss the most profoundly stupid arguments, such as, 'Can you explain how exactly an AI would develop a novel planet-killing weapon without anyone noticing and pulling the plug?'

This reasoning, however valid it may be, is of course of much less interest should it turn out that we are reasoning about an impossible situation. Many arguments have been suggested on both sides of the issue, but the arguments for impossibility are extremely weak; most of them are conclusively dispatched in https://arxiv.org/pdf/1703.10987.

[–] kragen 1 day ago

To be fair, he could probably fuck the human up pretty good if he had a mind to, too. Which may be the most likely reason that a human might decide to harm him: if he bites a couple of children, for example, or if a burglar wants to rob your house.

[–] CalRobert 1 day ago

Seems like bioweapons (assuming "planet killing" means killing all the people, not literally blowing up Earth) are one way. Convincing the creator it would achieve their aim and could be contained when in fact it could not.

[–] cowlevel 1 day ago

AI could invent something like social media. Obviously not social media because that's already been invented, but the next thing that slowly destroys society while seeming to be good.

[–] kragen 1 day ago

> probably destroy almost anyone who survived the initial blast and fallout.

The difference between killing almost everyone, and killing everyone, is similar to the difference between almost getting pregnant and getting pregnant. A hundred thousand years later, a world where almost everyone had been killed would be teeming with humans.

> Can you explain how exactly an AI would develop a novel planet-killing weapon

I already answered that question in my previous comment, and the answer is that I am not going to explain how to develop weapons, but the larger point here is that we already can't explain how exactly AIs do innumerable things we see them doing every day, which is why we're having this conversation in the first place.

> I seriously don’t understand this argument

That is a point that I think everyone can agree on: you do not.

[–] legitronics 1 day ago

I think they mean we don’t have a deathstar. We can make the atoms here shitty. We can’t annihilate them or scatter them out of their own gravity well.

Life in theory could restart with our weapons, but what’s the point with the other scenario.

[–] strange_quark 1 day ago

Being scared about novel AI weapons sounds like the people who were scared that turning on the LHC would create a black hole that would suck in the Earth.

[–] kragen 1 day ago

The most important difference is that this time it's the people who know the most about the topic who are concerned, and the people who know the least about it who are blasé. (Also, Yann LeCun.) With the LHC it was the other way around.

[–] strange_quark 1 day ago

If you truly believe this, I am begging you to touch some grass, preferably very very far away from Silicon Valley. It’s nice out here.

[–] CalRobert 1 day ago

Do you have an argument to make or only snide condescension?

[–] kragen 1 day ago

My demographic claim? It's obviously correct, as anyone with the most casual acquaintance with the question knows. The worried people, like Geoffrey Hinton, Bill Gates, and Dario Amodei, might be wrong, but they certainly aren't ignorant outsiders. (Neither is Yann LeCun, of course, which is why I mentioned him: he's an important exception to the rule.)

How would touching grass help? Do you think the grass is better informed about AI risk than you are?

...that's plausible, actually.

Probably a site called "Hacker News" is not the smartest place for you to be implicitly asserting that thinking about things is bad, or that those nerds who do it are out-of-touch losers. Maybe you want to be on Rugby News instead.

On Hacker News, by contrast, what is valued is thoughtful discussion grounded in the kind of rigorous theoretical knowledge that comes from extended concentration, usually but not always indoors, where it won't rain on the computers.

[–] jacquesm 1 day ago

Bill Gates is too busy trying to save his reputation from his Epstein association to be believed, he's a billionaire concern troll and Amodei is far more worried about his stock price than he is about humanity. Those two are discounted. That doesn't mean we don't have a problem, but they are just hitching their wagons to horse that happens to be running in a direction that is temporarily useful to them.

[–] akoboldfrying 1 day ago

> I think they mean we don’t have a deathstar. We can make the atoms here shitty. We can’t annihilate them or scatter them out of their own gravity well.

What a silly distinction to make.