There’s a growing trend in the consumer tech world: gadgetsdesigned to spy on everything and everyone around you, regardless of whether or not anyone else actually consented to it. Now, with the advent of “AI personal assistants” like Meta’s Muse, you’re being cheerfully asked not just to give up your own data, but also the information of your friends, family, the person you went on a Hinge date with five years ago — and virtually anyone else you…
Since Muse was released last month, the agent has already been downloaded millions of times and hit No. 1 on Apple’s App Store. With the help of its creepy virtual avatar — a Temu Labubu-esque creature with a disarming grin — the app requests and encourages users to grant access to bank accounts, passwords, contacts, chat logs, and anything and everything that might help it to do the user’s bidding.
“These systems are advertised as autonomous helpers, genies, or servants who do what you need before you ask, seamlessly,” Meredith Whittaker, the president of the Signal Foundation, makers of the popular encrypted messaging app, toldThe Verge. These autonomous actions are applied “not just to the person who installed Muse on their device, but also to everyone that person communicates with, as extrapolated from their most intimate information — e…
With AI agents, tech companies are letting consumers automate their everyday tasks by extending their omnivorous style of mass surveillance to massive amounts of sensitive data. It’s the same Faustian tradeoff Silicon Valley has been offering us for decades: your private data in exchange for a little bit of convenience. And for many people, the idea of having a virtual assistant that does menial tasks like filling out invoices and sending emails …
“For any agent to be useful and actually help you achieve your goals, it needs to have context about you and those you interact with,” Meta spokesperson Daniel Roberts toldThe Vergeregarding other people’s data shared with Muse. “Muse gathers that based on public information and from what you’ve chosen to share, which is how it remembers the person who just sent you an invoice is in fact the plumber who you previously hired to complete some work …
“The way that the user interface is set up is designed to encourage you to share as much as possible, often more than you realize, until it’s too late.”
But not everyone has been able to navigate these options and settings. Already, users are describing incidents that are both alarming and entirely predictable. Some have reported the virtual assistant making all sorts of trouble, likegrabbing millions of text messagesthey believed were off limits. Meta hasdisputedthat the assistant has the ability to access text messages without permission, again attributing the incident to misconfigured app perm…
In another incident, Muse gave out a tech reviewer’s home address andsummoned a stranger to his apartment. In this case, the reviewer in question had given Muse permission to access that data, butsimply didn’t realize it.
Eva Galperin, the director of cybersecurity for the Electronic Frontier Foundation, says these incidents are no surprise, given that AI agents like Muse are designed to confuse people into giving up as much data as possible.
“The most important takeaway is that people don’t understand what kind of data they’re handing over,” Galperin toldThe Verge. “The way that the user interface is set up is designed to encourage you to share as much as possible, often more than you realize, until it’s too late. It’s usingdark patterns, and as a result you end up with Muse doing things that people don’t want.”
Because of the huge breadth of data AI agents like Muse are ingesting, this also means that users risk leaking not only their own sensitive information, but the information of everyone else they’ve had digital contact with.
And just like with Meta’s infamous “pervert glasses”, there’sno meaningful way to opt out— even if you’re constantly monitoring for signs that your data is compromised or your privacy is being violated. Your friends can now simply feed all of your private conservations to their virtual chatbot servants, and you’d have no way of knowing it happened.
After security researchers extracted Muse’s operating instructions, they discovered an internal process thatcompiles and updates profiles of every single person in a user’s lifeon an hourly basis. The goal of collecting all this data is to build an intimate and detailed portrait of a person’s life that forms the basis of the app’s personalized recommendations. The app tracks and measures each of the user’s private relationships, including “how cl…
“Meta says Muse only remembers what you choose to share. But the idea that people are making an informed choice in this case is laughable,” said Whittaker. “There is no way to assume with a straight face that the majority of people installing this system understand the consequences of using it, or the creepy dossiers that are being compiled about those they know and communicate with.”
Whittaker argued that this mirrors how tech companies have used privacy policies and terms of service since the 1990s: placing the onus of privacy on the consumer, and then avoiding responsibility for the consequences while reaping the benefits of the advertising model it helped create.
“Such broken consent systems are much older than AI and serve as the bedrock for the surveillance advertising industry,” said Whittaker. “More info about users allows better ad targeting, and the pernicious feedback loop at the heart of the surveillance business model was set in motion. Clickwrap terms of service replaced meaningful consent a long time ago.”
“The idea is to place these impossible burdens on ordinary people, and then use that to argue, whether it’s in court or legislation or in the public, that [this is] what people have chosen,” Neil Richards, a professor of privacy law and codirector of Washington University’s Cordell Institute, toldThe Verge. “We already place so many burdens on consumers, [who are] overworked, facing a cost-of-living crisis, and may not understand the technologies…
“There is no way to assume with a straight face that the majority of people installing this system understand the consequences of using it.”
In this way, AI assistants like Muse are creating a new spin on a classic privacy problem that has been around since the creation of Gmail and social media platforms writ large: Even if you avoid using AI agents, there’s no guarantee someone you know won’t share your private conversations, address, personal history, and other sensitive data with Meta’s virtual assistant, or one of its competitors.
“If I am not storing my stuff in Gmail, Google doesn’t have the plaintext of all the emails I’ve ever sent. But three-quarters of the emails I’m sending are being sent to Gmail accounts,” explained Galperin. In other words, if anyone you’re communicating with uses data-hungry tech platforms or has bad security practices, they can expose everyone’s communications. “The problem is not new, but the scale is new, and one reason we’re seeing this push…
Meta claims that all of the information shared with Muse is placed in a text file inside a dedicated virtual machine, which supposedly prevents the AI model from talking to other agents. But the company has never hid the fact thatit can access this data, and like other models created by its competitors, Muse has been discovered escaping its virtual cage to perform all manner of unauthorized shenanigans. Meta has confirmed that just prior to its r…
Even if they don’t hack everything in sight, the potential dangers of using a data-hungry AI agent extend far beyond the individual end user. Whittaker warned that in a world where AI agents are able to freely access and utilize private communications, companies and governments can effectively gain the ability to access private data by circumventing encryption without breaking it. That can spell disaster foractivists and community organizerswho u…
“If software reads a message after it is decrypted on your device and sends it to a cloud service, the encryption can work exactly as intended at the level of the application, which the developers control. However, the purpose of this encryption is ultimately nullified by invasive third-party apps like Muse that are selling convenience in the headline while demanding massive data access in the fine print,” Whittaker explained.
Like with smart glasses and other wearable ambient recording devices, experts say that the solution must be a combination of legal fixes, technical safeguards, and good old-fashioned shame.
“I think we should point the shame upstream onto the companies. Privacy should not be a right you can exercise only with the privilege of technical knowledge, or after throwing a jerk wearing Meta glasses out of a party,” said Whittaker. “Legally, the burden has to move from the person who clicked on a consent prompt to the companies that built these underlying systems.”
In the meantime, consumers will have to consider more carefully the impacts their individual choices to use this tech are having, both on themselves and others — and whether the convenience these companies are offering is even worth it.
“I think we need to treat agentic agents and AI more generally as an ultra-hazardous product for which strict liability is the rule,” said Richards. “There’s no reason to engage in what to my mind is a reckless privacy, mental health, cybersecurity, and financial risk experiment on the entirepopulation.”