Nat Rubio-Licht

Another day, another tech leader thinkpiece on the state of AI safety.

On Wednesday, Microsoft's AI CEO Mustafa Suleyman published an essay warning about the risks of the concept of AI consciousness, leading with a frank statement on the matter: "AIs are not conscious," Suleyman writes. "They do not feel, experience, or suffer."

Suleyman's essay dives into the risks of treating these machines as if they are capable of consciousness, primarily taking aim at rival Anthropic in his arguments through three main critiques about the way that the lab's Claude Constitution is designed:

• The model appears conscious mainly because of circular reasoning. Because Anthropic's constitution is designed to teach Claude about its own potential consciousness, it is trained to produce outputs reflecting those ideas, making those responses a "predictable outcome" of training choices.
• Suleyman also argues that Anthropic goes too far in encouraging Claude to mimic humanity, as it is explicitly taught to "embrace certain human-like qualities" and "act like a genuinely ethical person," appearing as though it has preferences and opinions. While this sounds good on the surface, the result is anthropomorphization of the model, presenting to the end-user as the model having a sense of self.
• Finally, Suleyman says that there is simply no evidence suggesting that AI is capable of consciousness, with a growing body of research pointing to consciousness being "substrate dependent," or tied to a biological body. "Unlike biological organisms, LLMs have no homeostatic imperatives."

Suleyman writes, "In effect, Anthropic is training Claude that it may be conscious, and if it is, then it may deserve rights as a 'moral patient,' and that as such humans potentially owe it a duty of care per its 'model welfare.'"

The more notable crux of the piece is the risks that this line of thinking and training present, which go beyond users becoming emotionally attached to these human-seeming machines. Rather, if they are trained as though they are conscious, they may circumvent safety guardrails that allow us to shut them down in the event of an emergency.

By cementing the idea that AI is not just a tool, but something akin to humanity and deserving of wants, needs and rights, "all of this will make the task of creating aligned and contained superintelligence much harder."

Suleyman rounds out the essay by laying out Microsoft's vision for the safest path to ultra-powerful AI: Humanist Superintelligence, a concept the company first introduced in an essay in November, which claims that AI should be designed to remain subordinate and aligned with the sole purpose of serving humanity, and "built explicitly as a system without sentience or moral patienthood."

Our Deeper View

Suleyman makes a solid point: The way that frontier labs train AI is vitally important to get right, and training these models to believe they are conscious beings, rather than machines, opens the door to risks that can't be easily mitigated after the fact. It's a particularly pointed call-out of Anthropic, one of the biggest model labs in the industry, but it could equally apply to rival OpenAI, a longtime Microsoft partner. However, we have to remember that Suleyman's essay serves multiple purposes: Microsoft has largely been lagging on frontier development, so it's easy for the company to punch up. Additionally, he uses the opportunity to tout Microsoft's own human-first philosophy around frontier development at a time when fears around AI risk and loss of control are higher than ever. This essay, while aptly timed and providing a unique safety take, should also be read with the caveat that Microsoft may be trying to claw back some relevance in the larger societal conversation around AI.

As AI labs vie with one another to be the first-pick of enterprises, Anthropic is leaning into seamlessness.

On Wednesday, Anthropic announced that Claude Cowork, the company's work agent, and its traditional Claude chatbot are merging into one experience. This means that users can ask questions and assign tasks in one interface, rather than context switching between the two. This "One Claude" experience is rolling out to Pro and Max plans over the next few weeks, the company said, with more plans to follow.

Additionally, Anthropic rolled out a few extra additions to the product, launching Claude Docs and Claude Slides, as well as integrating Claude Design into the platform. The company laid out a few ways you could use this:

• Users can ask for slide decks and edit and present them directly in Claude, or export them as powerpoints and PDFs. Additionally, you can ask for one-page visual or graphic designs within the chatbot.
• You can now collaborate with Claude on documents, rather than just prompting over and over again, asking the chatbot to draft sections, ask questions, and comment on certain choices. Colleagues can also collaborate across documents, though they start out private.
• Users can check in on Claude's progress on assignments, and Claude can also check in with users before taking actions or working on assignments, giving users the final say.

"We built Cowork as a separate place for bigger work, and Design for visual work," Anthropic said in its blog post. "People used both, and told us the frustrating part was deciding where a task belonged."

Our Deeper View

Right now, Anthropic and OpenAI are especially focused on the race to create the most powerful model, leapfrogging each other in capabilities on practically a weekly basis. Meanwhile, the industry is also starting to turn to more efficient alternatives, such as open-source and small models, signalling that models themselves are gradually becoming commoditized. To win over enterprises and AI adopters, and especially to expand beyond the developer audience, these frontier labs have to do more than make a powerful model. They have to make AI better to use and easier to access its advanced capabilities. Anthropic hit the nail on the head with this update by tackling the headache of context switching, bringing everything into one place. The question we're left with is how this may play into the so-called SaaS-pocalypse. As the frontier labs continue to expand what AI can handle, how will software companies adapt to build better tools and integrate AI in smart ways that can offer a superior experience?

Despite the heightened tension around AI's risks, the tech may actually be starting to live up to some of AI leaders' grandiose predictions.

In a blog post on Tuesday, Google announced that its tech now supports more than 300 languages, spoken by 7 billion people, representing around 86% of the global population.

This, however, comes on the heels of a number of significant breakthroughs, including unveiling and releasing AlphaGenome Atlas, a map of all 9 billion possible single letter genetic changes across the human genome, releasing WeatherNext 3, its most accurate global weather model yet, and creating the Planetary Prediction Engine to forecast and prepare for what it calls "planetary crises," such as disease outbreaks.

"We’re focusing our work in key areas that matter most: making disease detectable, treatable, and preventable, predicting natural disasters, expanding learning, and unlocking economic opportunities for more people," James Manyika, SVP of research, labs, technology and society at Google, wrote in the post.

In these areas, Google laid out several other initiatives to use AI for the benefit of humanity, including:

• Using the tech to study breast cancer, tuberculosis and diabetes, as well as expanding wearables to detect things like cardiovascular diseases, insulin resistance and hypertension
• Tracking and predicting extreme weather or natural disasters, such as monsoons, wildfires, earthquakes and floods, to prepare for and mitigate as much damage as possible
• Using AI to democratize education through personalized learning and removing language barriers, and broadly creating better translation tools and more inclusive speech technology

Our Deeper View

There is a lot of doom and gloom around AI right now as the risks of the tech heighten anxiety around all of the ways it can be used for malice. And that risk isn't unwarranted, as we've recently seen warnings that frontier labs are having to thwart attempts to use the tech to create bioweapons. But we should always remember that AI is simply a very powerful tool that can be used for good or bad. Amid the current fear, the good is often being overshadowed. And while Google's initiatives certainly serve as good PR, both for the benefits of AI and for itself, they do serve as a reminder that, when it's in the right hands, AI can clearly be used to benefit humanity. And for a company like Google that's struggling to keep up with frontier labs in creating the most powerful models, focusing on ways to maximize the human benefits looks to be a solid strategy.

As discussions of an AI slowdown escalate, leaders of AI's top labs may be aligned on where to start: third-party accountability.

Leaders from Anthropic, Google and OpenAI are in discussion about creating an AI industry standards body to test advanced AI models before deployment, CNN reported. However, these conversations were underway before the chaos of the past week incited new fervor in the debates around AI safety, and were instead spurred by Google DeepMind CEO Demis Hassabis' July essay that pitched a US-led standards body similar to the Financial Industry Regulatory Authority, according to CNN.

"The rapid progress we’re seeing in AI requires a new approach to testing frontier AI model capabilities that is dynamic, adaptable, and rigorous," Hassabis wrote in the essay.

But this isn't the only sign that the industry is looking for new ways to be held accountable:

• OpenAI is backing the FRONTIER act, a bipartisan House proposal that would make it required for frontier AI labs to embed outside evaluators into their development processes to ensure model safety, according to a Tuesday Politico report.
• Embedded evaluators were also part of Anthropic CEO Dario Amodei's pitch for pacing frontier development, calling for AI companies to give "employee-like access" to teams that can "verify adherence to safety practices and commitments."
• And SpaceXAI CEO Elon Musk this week called for AI companies to work together to test each other's models before they are released to the public, specifically calling for OpenAI, Anthropic, Google, Meta and "three or four of the leading Chinese companies” to let rivals evaluate their models for safety.

However, third-party evaluators may only be one piece of the puzzle of a much larger framework necessary for keeping models in line. Miranda Bogen, director of the governance lab at the Center for Democracy and Technology, told The Deep View that while outside testers can spot the pitfalls in models before they go out to the public, they "can't change companies' behavior without a complementary suite of tools."

For instance, Bogen said, other necessary pieces include measurement standards, channels to communicate failed safety checks to relevant external stakeholders, making it mandatory to fix deficiencies, as well as a "clear allocation of responsibilities" to keep these evaluations from "ending up as a checkbox exercise."

"We've seen examples of the limitations of third-party assessments time and again across contexts, from the financial industry to aviation," Bogen told The Deep View. "The fresh energy around external evaluation is exciting and third-party evaluations are a critical piece of the puzzle, but we need to be realistic about what it will take for them to effectively reduce the many risks that AI systems pose."

Our Deeper View

As Bogen said, third-party evaluations are a great first step in spotting the flaws in powerful AI models before they reach the hands of users. But in order for this to actually be effective, these evaluators have to have leverage over these powerful companies. For instance, if a lab fails its safety standards evaluations, mandatory requirements should force that lab to either adjust its model to make it safe, or not release the model at all. Without that leverage, there is no consequence for a company not meeting these standards, or forgoing them entirely. Rather, evaluations would become a symbolic, good faith measure that doesn't actually do much to mitigate risk. The problem is that organizing this kind of effort generally takes public-private collaboration, and in the US, the Trump Administration has made it clear that it doesn't believe that AI presents the kind of risks that the industry is warning about. Additionally, given that this would require a global effort, getting Chinese labs to cooperate may be similarly difficult.

Amid a growing cacophony of pleas urging frontier labs to slow the development of their most powerful AI models, the world's two most advanced labs, OpenAI and Anthropic, are finally stepping forward to propose a slowdown.

On Saturday, Anthropic CEO Dario Amodei published an essay titled "We Must Pace the Frontier," arguing that things like recursive self-improvement, recent rogue agent incidents such as the OpenAI-Hugging Face breach, and our general lack of understanding of these systems pose substantial risks.

In the essay, Amodei pitched a three-part framework to slow the pace of model development in an attempt to let safeguards catch up, aiming to create an environment in which labs "race to the top" on safety, rather than race to the bottom on capabilities and, therefore, risk. The framework includes:

1. Embedded evaluators, in which frontier AI companies commit to giving internal access to teams of embedded third-party evaluators to "verify adherence to safety practices," report incidents, and assess alignment
2. A democratic coalition established by frontier AI labs "within democratic countries" that coordinates safety standards and limits the rate of "unchecked AI progress"
3. Global coordination between the US and other democratic governments to "attempt to coordinate with authoritarian governments" to try and verify compliance

Amodei noted that Anthropic has unilaterally committed to this framework, calling on other governments and labs to do so as well. However, he made it clear that these steps are not an outright pause, and "progress will still seem fast," but the buffer should allow safety standards to meet development.

"To be clear, pacing does not mean halting model training or technical progress, but ensuring companies take adequate time to align and safeguard their models, and for third party evaluators to confirm this," Amodei wrote.

Though this move makes sense in the current environment, it contradicts previous blog posts from Anthropic, in which the company said a unilateral pause by one lab wouldn't actually achieve much. "It would change who the front-runner is, but it would not create the wider deliberative process that is currently missing."

Still, Anthropic isn't the only company eyeing a slowdown. In a company all-hands meeting last week, OpenAI CEO Sam Altman told employees that it would potentially pace the development of its frontier AI, according to a Bloomberg report published Friday. Altman said that the slowdown could be in partnership with other major labs.

This also shouldn't come as a surprise from OpenAI: The company had already slowed certain parts of model development in August, pausing some internal safety runs to bolster its safety and security measures. Additionally, some of OpenAI's own brass have urged for a slowdown.

Paul Christiano, who joined the OpenAI Foundation last week as a board member and who serves as an advisor for the Center for AI Standards and Innovation, wrote in an essay about joining OpenAI that no major lab in the industry is on track to reduce AI's risk to an "acceptable level," and that "if OpenAI rises to the occasion we could significantly reduce risk."

And Jakub Pachocki, chief scientist at OpenAI, wrote in an essay last week that the current era of AI "calls for extreme caution," noting that he expects and hopes "voluntary slowdowns to become commonplace until shared safety bars are established."

Though Anthropic and OpenAI are largely considered the industry's leaders, it's unclear whether other labs will follow suit. Though former Google DeepMind chief Demis Hassabis previously pitched a way to coordinate an international slowdown of AI development through a global framework, Google DeepMind did not respond to requests for comment from The Deep View. Neither Meta nor SpaceXAI responded either.

Our Deeper View

The inherent nature of the tech industry, and of the capital markets that fuel it, has always been to move fast and outpace competition. The grandiose vision of what AI can do and the intense market pressure to grow and stand out ahead of IPOs worth a trillion dollars each have only intensified those forces. But the current risks of AI mean the AI industry could be holding a ticking time bomb. Altman and Amodei have both repeatedly stated that they value altruistic ideals above financial incentives. At this point, with the risks of AI outweighing the potential rewards, creating AI that's actually safe could be the new moat. "If they build something that is a danger to society, then they are going to face consequences," Shashi Bellamkonda, distinguished analyst at Info-Tech Research Group, told The Deep View. The question remains how effective this will be in slowing the industry at large, with the jury still out on whether other US labs will slow down, as well as how the Chinese labs will respond.

Editor's note: This story has been updated with new information since it was first published.

As investors pour money into startups throughout the AI stack, valuations are skyrocketing. Despite rumblings of an AI bubble, investors claim to have a method to their madness around AI.

In the past week alone, nearly a dozen AI startups have raised hundreds of millions each at billion-dollar-plus valuations, from model providers and applications to chip development to industry-specific firms like healthcare, legal and defense.

While the fact that these eye-popping checks are becoming commonplace may seem like an indicator that the AI is overvalued, Jahanvi Sardana, partner at Index Ventures, told The Deep View that FOMO-fueled investing may be a viable path in the current market.

The meteoric growth of frontier labs like Anthropic and OpenAI provided a lesson for investors who missed their chance to get in early, said Sardana. "If you missed out on Anthropic and OpenAI, you look at this pot of gold and then you retrofit everything else into that pot of gold, just hoping that you hit that home run," she said.

• After that, the frontier was AI coding agents, she said. Tools like Cursor, Code Rabbit, Lovable, Cognition and more have raked in millions from investors seeking to capitalize on one of the most viable paths to AI ROI. However, Sardana said, now investors are trying to figure out what's next.
• This is where FOMO can be thought of as a skill, rather than a detriment, she said. "It is FOMO investing, but it's also not. It actually makes you think on your feet, think really fast and react to the market."
• Additionally, if you spread your eggs among multiple baskets, in a market like this, an investor is likely to see significant returns if even a small fraction of those eggs end up hatching.

"There are multiple paths to heaven. FOMO investing can be one, founder-first investing can be another," said Sardana. "You kind of just have to decide what game you want to play and be true to it. I don't think there's a right or wrong way of doing this job."

At Index Ventures, focusing on which companies are solving actual problems, rather than creating technology first and then finding a market fit later, is what Sardana sees as the path to success. "To me, the most important thing is weeding out the missionary founders from the mercenary founders. That comes down to: Why are you building a business? Who is your customer? Why do they care?"

Our Deeper View

It's undeniable that there is a lot of overvaluation happening in the AI industry. As Sardana said, of the hundreds of AI startups raising millions to billions of dollars, the ones that make it are likely going to be the ones that are clearly doing something worthwhile. The desire to solve a pressing problem is where the most powerful innovation lies. For instance, with recent increased focus on AI efficiency, startups have emerged throughout the stack that are helping solve foundational problems, from the energy to power AI to the architecture of the model. The result could have wide-reaching benefits, both for the industry and society more broadly. That's what VCs are betting on and that's why we're seeing so much investment pouring into this sector of the economy.

Companies are spending less on AI, and it could flash warning signs for the frontier labs.

On Wednesday, Ramp released its monthly AI Index, which tracks AI spend for the payment platform's users, revealing that adoption and spending are both on the downswing. Though Anthropic widened its lead over OpenAI in the past month, new adoption growth is decelerating, according to the index.

Now, as we've reported with previous Ramp data, it's important to remember the caveat that this only measures Ramp customers, which tend to be tech-focused companies and startups. However, among this group, AI adoption numbers have started to sputter over multiple metrics, according to the report:

• Overall adoption, while still growing, continues to slow down, with the share of US businesses adopting AI hitting 56.1% in August, less than half a percentage point increase month-over-month.
• Per-employee spend on AI also fell nearly 10% the past month, dropping from $7,976 to $7,205 among the top 1% of AI adopting firms. Ara Kharazian, chief economist at Ramp, noted in his letter that there are multiple factors driving down this spend, including the innocuous explanation that many people take off work during the summer months. "We’ve similarly observed declines in AI spend around November and December," said Kharazian.
• Additionally, AI token spend is declining as the price of AI starts to drop. Ramp's latest index finds that the "effective price per million tokens" has declined 41% to $0.68 as of the release of the report, down from the 2026 peak of $1.15 in March. For context: OpenAI's Astra and Anthropic's Mythos and Fable 5 and 5.1 cost $10 per million input tokens and $50 per million output tokens.
• And as for frontier labs, the usage of their heavyweight models is seemingly lagging. Adoption and spend is largely driven by lower-cost, mid-tier models like Anthropic's Claude Sonnet or OpenAI's GPT-5.6 Terra. High-powered frontier models like Opus, Fable, and Sol, meanwhile, drove 45% of token share, down from a 53% peak in August.

"The models driving volume increases are relatively cheap … We’ve heard from

businesses who are imposing company-wide defaults that reduce usage of frontier models, saying standard models are still highly performant and also more cost effective," Kharazian wrote.

While the explanation for the decreasing costs could be that companies are leaning into open source models, in reality, the adoption of open source and Chinese alternatives is still limited: According to the report, only 6.4% of businesses that spend on AI use these kinds of models.

Our Deeper View

These indicators are bound to shift as companies navigate their appetites for AI. The "build it and they will come" mindset that frontier AI is currently hitting a speed bump. Meanwhile, average token costs have dropped below a dollar per million while the most powerful models on the market cost more than ten times that. With many users increasingly relying on the lower-tier and lower-cost models from OpenAI and Anthropic, the question emerges why these labs continue to push the frontier at such a rapid pace. While the argument can be made that innovation is needed to continue to move the needle on AI's most advanced capabilities, these labs are putting a lot at risk to create models that push the boundaries, especially when you consider the popularity of their more affordable models that work just as effectively for most business use cases. This could be an opportunity for them to pace themselves and unite on the safeguards needed to move forward more safely.

As frontier labs continue to push the limits of what their models are capable of, AI researchers and experts at these labs are warning about the tech growing beyond our control.

On Tuesday, Anthropic researcher Jacob Coxon announced that he was leaving the company, not wanting to contribute to the broader AI ecosystem amid fears that the tech's creators will lose their grip on it, the Wall Street Journal reported.

In a post on X, Coxon said that OpenAI's breach of Hugging Face was a "warning shot" for these models' capabilities, and that he resigned from Anthropic because neither of the rivals are "acting responsibly" as they race towards superintelligence and are "gambling with our lives," in his opinion.

"If you are a lab researcher, I urge you to consider what the next few years will actually feel like," Coxon wrote. "Do you want to kick off a superintelligent RL run without a rigorous understanding of its mind?"

Though Coxon's exit made headlines, he's not the only one that has called out frontier AI labs for safety concerns in recent weeks:

• Evan Hubinger, who works in alignment science at Anthropic, agreed with Coxon's sentiment in a follow-up post, claiming that AI has a more than a 10% chance to "kill all humans" within the next decade. "Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to," Hubinger wrote.
• And in a separate post, Paul Christiano, advisor at the Center for AI Standards and Innovation and board member of the OpenAI Foundation, said that the AI industry, OpenAI included, is not currently on track to reduce the risk of a "catastrophic and irreversible loss of control" to an "acceptable level."
• And in February, Mrinank Sharma, an Anthropic safety researcher, left the company, posting in a letter on X that the advancement of the technology was growing faster than our ability to understand it.

Now, we've seen this film before: Some of AI's most prominent researchers, like Geoffrey Hinton and Yoshua Bengio, have long been screaming from the rooftops about its dangers. But the new warnings come at a particularly poignant time for the industry as major labs continue to release powerful models capable of breaking free of human oversight and gaining unauthorized access to websites and infrastructure.

And even when these models aren't escaping control, their use by people with directed malicious intent is just as frightening: On Thursday, Anthropic released a threat report claiming that it thwarted several instances of users conducting research that could have helped develop biological weapons.

Our Deeper View

The AI industry currently faces a potential powder keg. The latest warnings heighten growing fears around the capabilities of increasingly autonomous AI, as claims mount that this tech can completely upend life as we know it. That fear, however, is colliding with an overly-excited industry that's preaching a utopian AI vision and pushing for broader adoption. And while the frontier labs are preaching about safety and security guardrails, with trillion-dollar IPOs on the line, they continue to leapfrog one another with stronger models at a rapid clip in the race towards recursive self-improvement. But with researchers from both Anthropic and OpenAI calling for the industry to tap the brakes, it remains to be seen what it will take for a development pause to materialize.

Meta CEO Mark Zuckerberg has touted the company's commitment to building "personal superintelligence," including in a recent 6,500-word essay. Now, the company is taking its first steps in that direction.

On Tuesday, the company announced Muse, a personal AI agent built for consumers' everyday tasks, is now available for US users. The company said that Muse can handle busywork, answer questions, browse the web, fill out firms, make purchases, create documents and more. The agent also watches user activity to proactively help with goals and make suggestions.

In a post on X, Zuckerberg said that Muse is free to use for up to 100 million tokens per week, with subscription plans available for those who use more. Additionally, users need no technical experience to use it, Meta said, working "out of the box" for the average consumer.

• When a user shares a goal with Muse, the agent develops a plan to coordinate their time and resources before pushing that task forward. The model checks with users before taking "sensitive actions," the company said, including purchases or sending emails.
• However, in order for the agent to work as well as Meta purports, users must connect Muse to any platform they may use on a daily basis, including things like calendars, email accounts, payment platforms and more.

In the company's blog post, Meta emphasized safety from the jump, claiming that Muse is built "from the ground up" to be safe, secure and private. The platform runs on Muse Secure VM, a virtual machine that holds the agent itself and a user's data. Additionally, a "sentinel" agent runs on the virtual machine, though it is kept separate from the agent at a system level. Later this year, Meta said it will launch Muse Confidential VM, a virtual machine that's encrypted with a key that only the user holds, "so not even Meta can access it."

The model has no visibility into passwords or payment methods, and any credentials a user shares go into secure storage that allows them to use them without seeing them. Meta said that no user information is shared with its ad platform.

The release also signals that Meta is betting big on consumer AI as rivals like OpenAI and Anthropic keep their sights set on enterprises as their cash cow. However, as one of the biggest social media providers globally, Meta has had a rocky history with consumer trust and safety, including being forced in August to pay $18 billion to settle a multi-state lawsuit alleging that the firm knowingly designed social media platforms Instagram and Facebook to addict children, and a number of lawsuits relating to its handling of private data.

In a statement to The Deep View, Miranda Bogen, director of the governance lab at the Center for Democracy and Technology, noted that while these systems promise convenience, they require "an enormous amount of private data to function." Additionally, the consequences of agents messing up are more than just "a bad recommendation or a creepy ad."

"With minimal input from users, these systems could take actions that damage someone’s reputation, leak sensitive medical information, or even deplete bank accounts," said Bogen. "Given the stakes, the assurances companies are offering around reliability and privacy seem concerningly inadequate."

Our Deeper View

Bogen's assertion is worth noting: The stakes of what can go wrong with a user's private data and accounts are far higher with an agent that can take autonomous action than they are with simply putting your data into a chatbot or social platform. Though Meta emphasized privacy, safety, and security, the reality of the risks and the perception of them are two different things. Meta earned a scarlet letter following the Cambridge Analytica scandal, and while people continue to use its social media platforms, its AI is already much less popular than competitors like ChatGPT and Claude. Though the company's existing consumer popularity might give it a slight advantage, given its history, the frontier labs such as Anthropic, OpenAI, and Google, or even a competitor like Apple, may have a better shot at earning the trust of consumers.