u/2Punx2Furious

Off topic.

It's powerful specifically at finding software vulnerabilities, that's why it's not releasing to the public (it's a perfectly good reason).

Off topic, removed.

If you think this will be limited to white collar, you're in for a surprise. But yes, they'll probably be the first.

He put it in a not very precise way, but yes, more or less.

We won't "control" an ASI, no chance.

At best it will be aligned in a way that it cares for our well-being, which is a desirable future, but it's not a given.

I actually wrote a probability calculator two years ago for this:

https://www.reddit.com/r/ControlProblem/comments/18ajtpv/i_wrote_a_probability_calculator_and_added_a/

But I haven't updated those numbers.

"Solved" should probably be close to 0%, instead there should be another option of "apparently aligned" or something similar, but I guess it should be somewhat higher.

No current AI is even apparently aligned in a way that satisfies me at the moment.

If any of the current LLMs (alignment-wise) were to become an ASI, the future would be bleak.

In the past I thought it was a solvable problem.

After having reasoned about it for a long time, I now think it's unsolvable, the best we can probably get is for an AI to be "apparently aligned", but we'll never be able to know for sure if it is robustly and permanently aligned.

There is also no way to control an ASI, unless you're an even more powerful ASI.

Also stopping is not an option.

So, yes, I guess being scared is the only rational thing left.

No, sorry.

The only scenario where it keeps us alive long-term is when it's terminally aligned to care about us being alive.

If it's an instrumental reason, there are probably better ways to achieve the relative terminal goal, even if you can't think of any, a superintelligent AI probably can, and it would be unwise to put all our hopes in an ASI not being able to think of better ways to do something, since it will probably be very good at that.

Since not everyone is always clear on the definitions:

• Goals can be either terminal or instrumental.
• Terminal goals are goals that an agent (human, animal, AI) wants to achieve just for the goal's sake, not necessarily for any other reason.
• Instrumental goals are goals an agent decides to pursue in order to achieve some other goal (terminal, or instrumental).
• Both types can have instrumental sub-goals (some goals can be instrumental and terminal at the same time).

Example:

• You have a terminal goal of surviving (which is also an instrumental goal to achieve anything else).
• You are hungry, so eating becomes an instrumental goal to satisfy the terminal goal of surviving (eating tasty things can also be a terminal goal by itself, because you'd want to do it even if it wasn't instrumental to something else).
• You need to go to the store to get something to eat, this is another instrumental sub-goal, it's not terminal, because you wouldn't care about going to the store unless you needed to get something there.
• You need to get dressed to go out, this is also instrumental, and so on until you satisfy your terminal objective...

Not surprised, that subreddit has become ideological garbage.

He realized this just now?

He's very behind in alignment theory.

That alignment is the only solution to co-exist with ASI was obvious for a long time.

Not inevitable, but very likely.

But you probably wouldn't like the alternative.

Yeah, I doubt we hit ASI proper with just scaling, but we still get massive societal shift.

But I also very much doubt we'll only get scaling and no other breakthrough, seems very unlikely.

That's a big assumption, but ok. Even if we got stuck at current AI, and only scaled compute for some reason, we still have a lot of low-hanging fruits, and things would change forever automation-wise.

You might not ever get ASI proper, and a lot of related scenarios, but still a radical disruption in the entire world economy, with relative second and nth-order effects.

But yes, we don't know, we can only guess, and I'm guessing it will become superhuman at everything soon.

Current systems are already far smarter than humans on several metrics, just not all of them.

Sure, you can say "we don't know", but do you really think it's likely that it won't happen? Seems more like cope, given the trends.

It's not impossible, I guess, but I don't see it as a likely scenario, because it kind of requires solving alignment, which we haven't, and if we do, there are better and more likely scenarios in that branch.

There are many "objections" to AI risks, at different levels of the path, some call them "stops" of the "doom train".

You're stopping pretty early at "superintelligence might not even be possible."

I think it's fairly obvious that it is possible to be smarter than humans. Why would human level be the pinnacle of intelligence? That's an absurd position.

One might more reasonably argue that we're not very close to it (and I'd disagree), but it's pretty obvious that it's possible, in theory.

Unfortunately, I think people like you are (that being, most people).

Yes, I can think of everything.

Ah, ok, should have realized I was talking to a frog in a well.

Good luck with your fantasy land where people do exactly the right thing and are infallible.

You should take your own advice and actually think.

Are we currently keeping any AI in a "secure" facility?

Are we currently implementing any form of security that isn't laughably insufficient?

What makes you think that as soon as we get an AI that is capable of real damage, we'll suddenly figure it out, and keep it "constrained" (or that we'd even know we got there)?

Why would a superintelligent AI even give you any hint that it is not completely benevolent, and prompt you to act in such a way to begin with? Why would we not give it access to everything, if it's so useful?

And even if by some miracle we put it in some "secure" facility, with all the best practices you can think of, you really think you can outsmart a superintelligence that wants to get out?
You think we can think of everything?
How are you going to check its outputs, when they become incomprehensible to humans?
Are we just going to stop using it?
Why even make it in the first place?

Maybe you should stop living in fantasy land.

Alignment and control are not the same thing.

Correct.

Absolutely delusional to think you can make a facility "secure" against something that is far more intelligent than you.

Like an ant placing a few grains of sand in front of the door of your house.

People in these comments are beyond gone. There is nothing left of their mind.

What other amenities do you need if you have infinite virtual worlds where you can do anything?

There's a small chance that it matters to be nice to current AIs. Still, probably doesn't hurt.

But yes, do try to enjoy life as long as you can.