“Don't build unaligned AGI” is an excuse to give a narrow elite exclusive control of what AI is produced under the pretext of preventing anyone from building unaligned AGI; all actionable policy under that banner fits that description.
Whether or not that elite group produces AGI, much less, “unaligned AGI”, is largely immaterial to the practical impacts (also, from the perspective of anyone outside the controlling elite, what the controlling elite would view as aligned, whether or not it is a general intelligence, is unaligned; alignment is not an objective property.)
False. There are people working on frontier AI who have co-opted some of the safety terminology in the interests of discrediting it, and discussions like this suggest that that strategy is working.
> all actionable policy under that banner fits that description
Actionable policy: "Do not do any further frontier AI capability research. Do not build any models larger or more capable than the current state of the art. Stop anyone who does as you would stop someone refining fissile materials, with no exceptions."
> (also, from the perspective of anyone outside the controlling elite, what the controlling elite would view as aligned, whether or not it is a general intelligence, is unaligned; alignment is not an objective property.)
You are mistaking "alignment" for things like "politics", rather than "not killing everyone".
“Do not” doesn't serve the goal, unless you have absolute universal buy in, active prevention (which means some entity evaluating and deciding on threats); that's why the people serious about this have argued that those who pursue it need to be willing to actively destroy computing infrastructure of those who do not submit to the restriction regime.
Also, "alignment" doesn't mean "not killing everyone", it means "functioning according to (some particular set of) human's preferred set of values and goals". "Killing everyone" is a consequence some have inferred if unaligned AI is produced (redefining "alignment" to mean "not killing everyone" makes the whole argument circular.)
The AI alignment problem has, at its root, the notion of being capable of being aligned. Long, long before you get to following any particular instructions, there are problems like "humans are made of atoms, if you repurpose the atoms for other things the humans die, don't do that". We don't know how to do that or things on par with that, let alone anything more precise than that.
The darkly amusing shorthand for this: if the AGI tiles the universe with tiny flags, it really doesn't matter whose flag it is. Any notion of "whose values" really can't happen if you can't align at all.
I'm not disagreeing with you that "AI alignment" is more complex than "don't kill everyone"; the point I'm making is that anyone saying "but whose values are you aligning with" is fundamentally confused about the scale of the problem here. Anyone at any point on any reasonable human values spectrum should be able to agree that "don't kill everyone" is an essential human value, and we're not even there yet.
Whether or not that elite group produces AGI, much less, “unaligned AGI”, is largely immaterial to the practical impacts (also, from the perspective of anyone outside the controlling elite, what the controlling elite would view as aligned, whether or not it is a general intelligence, is unaligned; alignment is not an objective property.)