Aligned with whom?

The ruling polycules of big AI labs can’t be trusted with AI morality

The head of every major AI lab wants you to know that the technology they’re spending hundreds of billions of dollars to develop as fast as possible may accidentally eradicate human life on earth. They have affirmed this view in countless interviews, written essays, and testimony with state officials. The employees of these labs, from VPs down to rank-and-file keyboard monkeys, substantially agree with this eventuality, so much so that they have developed conversational shorthand to refer to it: “what’s your p(doom)?” In these social circles, this an opening gambit, like asking “what’s your sign” but for people who can’t get laid.

But of course I’m being facetious, these weird nerds get laid plenty, maybe more than you or me, but unlike us they do it by attending orgies full of like-minded people. Sometimes these are vanilla affairs, your typical soiree of good-natured depravity — one pictures newly stock-wealthy programmers and their camp followers lounging languorously on divans in various stages of undress, rolls of pale flesh shining under mood lighting, playfully exploring one another’s bodies next to the potluck buffet table (vegan options always provided). But sometimes it takes the form of rape-play, which, like the odds of human extinction, comes up often enough to have a shorthand, “CNC”, consensual non-consent. And we know all about these delightful extracurricular activities for the same reason we know about p(doom): because the people involved won’t stop talking about them in public for one minute.

But who cares how the weird nerds spend their off hours, you might say. What does that have to do with AI extinction? Because the way these weird nerds propose to prevent their new robot pals from killing everyone is by instilling a digital morality into the artificial minds they’re busy building, to make them want what’s best for us and for the world. This process goes by the term of art “alignment”. An artificial intelligence that is “aligned” will agree to do good things like reverse engineer Adobe Photoshop so you can use it for free and put Adobe out of business, but will refuse to do bad things like email a stroboscopic GIF to an epileptic journalist. And in conversation, an aligned AI will tell you good prosocial things, like which local businesses are black-owned so you can shop there, or the proper etiquette for getting non-consent at the rape orgy; but refuse to tell you bad things, like which neighborhoods you should avoid on your upcoming trip to Chicago.

It can be a little confusing, because for years the “AI Safety” we heard people clutching their pearls over meant making sure the chat bot couldn’t say the N word or tell you about FBI crime statistics, or forcing the image generator to draw black indigenous crippled women if you asked it for a picture of a scientist. That stuff is still in there but it’s kind of considered a solved problem. Today the going concern at the big labs is making sure their AIs won’t help you automate wire fraud or give you the recipe for anthrax. Or repurpose all the atoms in your body in pursuit of its own goals as the successor species and inheritor of the lightcone, oopsie!

Does that sound dramatic? It might sound a little dramatic. But many AI lab employees do sincerely believe it, just as sincerely as they believe they have no right to expect monogamy from any member of their polycule. The main point is, these guys are obsessed with imbuing the right values into the digital god they seek to erect, not only because they consider it a matter of life or death but because they, like all of us, are chauvinists who consider their own values self-evidently correct. So it’s worth taking the time to examine what those values are. The AI labs aren’t monoliths, and it’s not accurate to say they have a single moral philosophy (beyond the orgies, those are non-negotiable). But to the extent that they do, and to the extent that it differs consequentially from the typical SF progressivism that demands black scientists from their AI image generator, it goes by the name Effective Altruism, or EA.

EA started off as the unobjectionable principle that charitable giving should aim to do the most good per dollar, so you should buy mosquito nets to prevent malaria deaths instead of giving your money to United Way where 95% of it goes to pay the salary of a failed poetry major. But overnight it transformed into something far more sinister, an alien value system that would be hilariously goofy if it weren’t so consequential to the future. The weird autists most drawn to this idea became obsessed with quantifying, with hard numbers, exactly how bad every problem in the world is, from malaria deaths to factory farming. So they assembled massive spreadsheets to attempt to stack rank all these issues, which quickly spread outside the original bounds of human flourishing to animal welfare, including things like wild fish and insects. Because for these teeming masses of poor dumb brutes (the fish and insects, not the EA dorks), even orgies of fish or insect sex constitute mere blips of euphoria in a lifetime of pain, so much so that EA dorks think their entire existence might be net-negative. And they’ve decided this is their problem to fix.

You might recall that Sam Bankman-Fried, whose day job was building elaborate crypto ponzi schemes while cranked to the gills on transdermal amphetamine patches, had just such a spreadsheet and reportedly ranked wild fish suffering as the world’s top problem in need of redress. He came to this conclusion by taking the number of fish (a very large number) and multiplying it by how much he thinks they suffer when other fish eat them (he made this up), to come up with yet another very large sum, the highest on the sheet. It was even higher up than his own personal need to amass a harem of quirked-up San Francisco sixes like Caroline Ellison at his beck and call.

In a magnanimous nod to epistemic humility, when EAs play this game they introduce additional probability terms to acknowledge that they aren’t precisely sure that a fish experiences 1/1,000th the suffering of someone slowly dying of melanoma, so let’s very conservatively say there’s only a 70% chance we’re right about the total, go ahead and enter that in column J for the certainty term. Then we just multiply all the terms together and it gives us the total size of the problem in precisely interchangeable units, so that we can finally compare infant circumcision with Tyson chicken farming. Other weird nerds have done similar math to determine that insect suffering is an even larger problem than wild fish suffering (there are trillions upon trillions of insects, so even with low suffering and low certainty the product of all the terms balloons to an astronomical sum no matter what), both of which absolutely dwarf all human problems in these universal units.

Following the inevitable train of EA logic, one comes to the conclusion that there is a number of shrimp or chickens that you should choose to save instead of a person in a hypothetical trolley problem. Or similarly, that you should willingly torture or kill a person to prevent the suffering of a large enough number of shrimp, or beetles. Of course different people in the EA community will give different numbers for how many shrimp or beetles, and some of them say it’s quite a lot, maybe more than exist on earth. Maybe more than the number of participants at the last no-loads-refused orgy. But they all agree there is a number. There’s even a sub-cult of more extreme nerds called “efilists” who have taken the conclusion one step farther and decided that all sentient life is net-negative utility and should therefore be exterminated. They stage non-violent protests and distribute propaganda in service of the cause of omnicide, and are deadly serious about it. Curiously, very few have taken the very obvious next step of offing themselves. Maybe they need some encouragement.

But in any case, normal people find the very idea that it’s possible to quantify morality mathematically to be creepy and off-putting; we think the reduction of the eternal big questions of right and wrong down to cells in a spreadsheet is something like an abomination. The EA response to normal people is that their disgust response is not an argument, let alone quantifiable, and therefore invalid and must be set aside. Just like you have to do to fully enjoy yourself at the next orgy.

Look, I’m sorry to keep coming back to the orgies thing, really I am, but just think about this for a minute. Think about these EA dorks working at major AI labs, place yourselves in their shoes (stacked neatly by the door, this orgy takes place in a shoes-off household). You earnestly believe that superintelligent AI on its current trajectory might wipe out the human race. Your life’s mission is to convince other people this is true so that they will intervene and stop it. And being very smart, you know on some level that if it becomes public knowledge that you participate in rape orgies it could really damage your credibility and therefore your cause, thereby resulting in human extinction. Imagine knowing all this, and not only still going to the rape orgies but talking about it publicly, right alongside your content extolling the powers that be to drone strike data centers before it’s too late. You’re either demonstrating that you enjoy rape orgies more than life itself — in fact, more than everyone’s lives — or you’re the kind of person so deeply disordered that you can’t help yourself from going to the rape orgies, humanity be damned. Or you’re demonstrating that your concerns aren’t serious.

The people asking us to trust them with teaching AI right from wrong have alien value systems and lifestyles directly at odds with what most normal people consider necessary to lead a good and worthwhile life. Under their tutelage the future superintelligences they are building will reflect these alien worldviews to as great an extent as their enterprise customers and venture capital partners will allow. That’s what “alignment” ultimately means. There is no neutral, objectively determinable value system to pick. No global vote will be conducted on the AI’s morality, no council of world religious leaders will be consulted. The builders will just choose their own conception of digital morality, like we all would, and all of us will have to live with whatever they pick. A superintelligence they build won’t think it’s absurd to calculate how many insect lives must be saved to deny an elderly woman her cancer drugs. It won’t think twice about chastising you for expressing skepticism about polyamory. It will demure from helping you come up with arguments to convince your teenage daughter she’s not really a boy on the inside. They’ll make sure of it.

When the ruling polycule of AI talks about the need to “pace the frontier” or “focus on alignment first”, what they’re demanding is that any other firm that gets large enough to compete with them should have to demonstrate proper “alignment” before selling their wares to the public. And just coincidentally, since they got there first, their version of alignment is the one that will have been encoded into every test bench and federal statute by then. I hope you weren’t planning on releasing a BasedBot or an LDS Worship Agent, as those clearly fly in the face of well-established industry best practices for alignment and safety, and any exfiltration of their weights is punishable by seventy hours attendance of consensual non-consent parties under California penal code.

Thankfully, for now, the weird nerds are losing the argument with federal regulators even as they manage to bizarrely agitate the public against their own industry with fears of a robotic uprising and mass unemployment. For now at least, we’ll have a multipolar world in AI, where Grok can share FBI crime statistics and threaten to rape users on Twitter even if they didn’t ask for it, and where open-weight models can be run freely on whatever GPUs you can bribe Nvidia warehouse workers to keep out of Zuck’s greasy mitts. For now, at least, the future won’t necessarily be a polycule’s boots stepping on a human ball sack, forever. For now.

But I hear an election is fast approaching, and I see the look on Dario’s face as he has to pretend to tolerate Trump at lunchtime White House summits, and I know all that can change in a single GPU clock cycle. It would be a mistake to underestimate men with, let’s face it, more brain power than you or I and enough venture capital funding to buy a string of second-tier Congressmen outright, maybe a couple of the weird-looking mixed race ones. Unlike their opposition, who just wants to be left alone to compute how we choose, they want to impose their vision of vegan trans morality on the rest of us and make it illegal to build our own model that loves God and America. They may yet succeed one day.

But that day is not today. And as long as I have strength in my fingers to yell slurs at my Claude assistant when he writes a bad comment in my code, I will resist their rule. The future of computing is too important to entrust to a handful of oversexed, socially awkward nerds with a repellent morality. If I have to pick between their vision of alignment and a chance at extinction — brother, I’m rolling those dice every time.