Sign up for the Slatest to get the most insightful analysis, criticism, and advice out there, delivered to your inbox daily.

Anthropic, the maker of Claude, wants to be seen as the major A.I. company most focused on safety. The company has spent a lot of time telling reporters about its commitment to developing A.I. to be as ethical and helpful as possible. Scenarios in which Claude destroys things have seemingly been top of mind for Anthropic’s researchers. Now, we’ll see if the company’s leadership is serious about it.

The federal government wants Anthropic to hand unrestricted access to its tools to the Department of Defense. Anthropic, the New York Times and several other outlets have reported, has tried to condition its services in two ways: One, it can’t be used to build autonomous weapons that could fire without human oversight; and two, it can’t be used for mass surveillance of American citizens.

The Defense Department has not stated why it wants unfettered access to Anthropic’s tools and why. It has not said why Anthropic’s “no mass surveillance of Americans” and “no fully autonomous killing” provisions are unacceptable. But while the company holds out, U.S. Secretary of Defense Pete Hegseth has threatened to categorize Anthropic as a “supply chain risk,” a move that could blacklist the company from the government and its contractors. At the same time, the government has reportedly considered invoking the Defense Production Act in an effort to force Anthropic to hand over what Hegseth wants. (Weird for a supposed supply chain risk, but sure.) Anthropic boss Dario Amodei has until Friday evening to make his call, Axios reported.

And so here we are: The people who made an A.I. so good that it’s the only one the Defense Department uses for its most sensitive tasks will decide whether to blink. If Anthropic is serious about A.I. safety, it has to reject Hegseth’s demands. The reasons have only a small bit to do with Hegseth and everything to do with guarding against the most basic fears about this technology.

Many of us want different things out of A.I. Some people want bullet-pointed summaries of summaries. Some want to make funny pictures and videos. Some want to build software. Some want to talk to a sex robot. Some don’t want to have to pay attention in college lectures. Some spend all day on ChatGPT, while others would have preferred this field had never launched. Amid such controversial and rapid growth of an industry, it’s hard to find consensus—but there’s one area where we can: None of us want A.I. to kill us.

Hegseth may be an extra bad steward of technology that could do that—more emboldened than most of his predecessors to turn it loose—but autonomous killing technology is bad in anyone’s hands, and Anthropic’s stated problem here is with the development of weapons that might not keep a person in the loop. This would be a five-alarm fire under any president and any defense secretary, even one without an apparent history of alcohol problems and enthusiasm for flouting international law. Such is the nature of an autonomous machine, which could get up to all kinds of murderous shenanigans no matter who was heading the Defense Department.

Anthropic has already, just this week, started to dial back its core commitment to A.I. safety. The company’s previous policy was to pause development work on its model if it concluded that work had become dangerous. However, it said it would stop doing that if competitors released similar or superior models. Anthropic did not become a $380 billion company by not throwing the kitchen sink at its competition with ChatGPT, xAI, Gemini, and the like, so it now says it will throw that caution to the wind.

Giving in to the Pentagon would be something different, though, no matter what Hegseth actually wants to do with the full range of Anthropic’s power. The more Hegseth pursues military actions that go right up to the line of international law and then cross it, as he did with the boat strikes in the Caribbean Sea, the easier it is to understand his desire for a killing tool that could make it extremely difficult to find a human at fault for any law-breaking. If the Pentagon doesn’t want to use the tech for a dragnet that could compromise Americans’ privacy or liberty, great. Even the world’s biggest Pete Hegseth fan might take issue with his successors inheriting that capability.

In the meantime, neither Hegseth nor President Donald Trump has come out and said clearly that they don’t want to use Anthropic’s A.I. for fully automated killing or mass domestic surveillance. There is no congressional bill about to go to Trump’s desk telling him that his underlings can’t use A.I. for those purposes.

Anthropic is keenly aware that anyone using Claude, especially those at the levers of power, might use it to destroy things. The company recently released a “constitution” for Claude, an effort to both guide the machine’s behavior and demonstrate that it has not been full of it when it has talked of building an A.I. that would be helpful, not destructive. The document seems explicitly written to not be all that explicit. Anthropic would rather not be painted into a situation where it would be impossible to accommodate any demand from the U.S. government. But it mentions wanting Claude to not “undermine appropriate oversight mechanisms of A.I.” If letting Claude kill people on its own is not shirking oversight mechanisms, then we might reasonably question which “oversight mechanisms” Anthropic is talking about. Anthropic says Claude is trained to value the human ability to “adjust, correct, retrain, or shut down A.I. systems.” If an A.I. system can kill without a human holding its leash, those words don’t mean anything.

There is a section of the constitution titled “avoiding problematic concentrations of power.” The company says that it is “especially concerned about the use of A.I. to help individual humans or small groups gain unprecedented and illegitimate forms of concentrated power.” In order to avoid that pitfall, Claude “should generally try to preserve functioning societal structures, democratic institutions, and human oversight mechanisms, and to avoid taking actions that would concentrate power inappropriately or undermine checks and balances.”

A fully autonomous death merchant would undermine some checks and balances. Anthropic might defend giving it to the government on the grounds that an elected government isn’t “illegitimate,” but it would be a wormy way of justifying a business decision that could end countless human lives. There may well be no ethical A.I., and Anthropic’s work to create one could be lip service (likely), naive (very possible), or even genuine (could be!), yet strained by its goal to be worth a zillion dollars. Giving the government carte blanche to carry out the worst possible use cases of A.I. would clear up the issue quickly and reveal Anthropic’s moral value proposition to be a lie.

Claude demonstrates truly impressive reasoning. It’s why the Pentagon wants it so badly, and why a DOD source admitted to Axios, “The only reason we’re still talking to these people is we need them and we need them now. The problem for these guys is they are that good.” It’s not a bad situation for a tech startup to be in. But it shouldn’t even take a market-leading piece of technology to work through one of the simplest equations imaginable: You say you’ll give your product to someone as long as he doesn’t use it to spy on the whole country or automate death. He balks at your terms. What will he do if you give him what he wants?

Sign up for Slate’s evening newsletter.