If your company has deployed AI agents in the last eighteen months, your privacy posture almost certainly changed — and probably without your board, your CPO, or your customers knowing. Agents are not chatbots. They ingest email, call APIs, query databases, browse the web, and move personal data across trust boundaries your existing privacy framework was never designed to govern. In June 2025, researchers disclosed EchoLeak (CVE-2025-32711), a zero-click vulnerability in Microsoft 365 Copilot (CVSS 9.3) that exfiltrated internal customer data from a single crafted email with no user interaction. [1] Most agentic deployments were architected before that threat model existed.

EchoLeak (CVE-2025-32711) allowed attackers to pull internal data out of Microsoft 365 Copilot with a single crafted email — no user interaction required. CVSS 9.3. Source: Aim Labs, Microsoft, arXiv 2509.10540 (2025) [1]

The Scale of the Blind Spot

IBM’s 2025 Cost of a Data Breach Report found that 13 percent of organizations surveyed reported breaches of AI models or applications, and 97 percent of those lacked proper AI access controls. [2] The global average cost of a data breach was $4.44 million. [2] Meanwhile, Open Worldwide Application Security Project (OWASP), a nonprofit foundation that provides free, open-source resources, tools, and guidelines to help people and companies build secure software, published its first Top 10 for Agentic Applications in December 2025, led by three new failure modes that most CISOs have never tested against: memory poisoning, tool misuse, and privilege compromise. [3]

Of the organizations reporting a breach of an AI model or application in 2025, 97 percent said they did not have proper AI access controls in place. Source: IBM Security, Cost of a Data Breach Report 2025 [2]

Where Your Data Perimeter Actually Breaks

Every chat window is a privacy frontier most companies don’t actively govern. Customers routinely paste SSNs, account credentials, health details, and confidential contract language into agent conversations — inputs your privacy notice likely never told them would travel through a third-party foundation model. Employees do the same when they use an agent to summarize or reason over sensitive documents. Once inside the agent, that data can be used to fine-tune the underlying model without an enterprise no-training agreement, cached in vector stores controlled by the model provider, surfaced in another tenant’s context via poisoned memory, or passed to a chain of downstream tools the agent is authorized to call.

The September 2025 “ForcedLeak” exploit in Salesforce Agentforce (CVSS 9.4) demonstrated this end-to-end: attackers slipped instructions into a web form and the agent autonomously exfiltrated CRM records. [9] NeuralTrust and Noma Security report that roughly 80 percent of organizations observe their agents taking actions exceeding the access their owners believed they had. [10]

80 percent of organizations surveyed in 2025 reported their AI agents had already taken unauthorized actions — including accessing or sharing sensitive information beyond their intended scope. Source: NeuralTrust / Noma Security, agent risk survey data, 2025 [10]

Real-World Incidents Already on the Books

Samsung (2023–2024). Three Samsung engineers pasted proprietary source code and meeting transcripts into ChatGPT in a 20-day window; Samsung Electronics banned public generative AI tools company-wide in response. [4]

Air Canada (Moffatt, 2024). A Canadian tribunal held Air Canada liable when its public chatbot gave incorrect bereavement-fare guidance — rejecting the “separate legal entity” defense. [5]

Emotional Companion Chatbot (2025). Italy’s DPA fined a U.S. emotional-companion chatbot €5 million for GDPR violations in how the agent collected personal data. [6]

Salesloft Drift (August 2025). Threat actor UNC6395 used stolen OAuth tokens from Drift’s Salesforce integration to access customer environments across more than 700 organizations — a textbook privilege-compromise attack against an agentic integration. [7]

CIPA class actions (ongoing). U.S. courts have allowed class actions to proceed against companies whose AI chat features record customer conversations for model improvement without explicit consent — under California’s Invasion of Privacy Act. [8]

What "Agent-Ready Privacy" Looks Like

  1. Govern the agent estate under one board. A cross-functional review board — privacy, security, legal, product — should inventory every agent, every tool call, every system of record it touches, and every category of personal data that can cross each boundary, and must approve every new agent and new tool.
  2. Control what customers and staff can feed the model. Filter inbound prompts for SSNs, PHI, PCI, and credentials; require enterprise no-training terms with every foundation-model provider; disable data retention and vector-store persistence wherever possible.
  3. Rewrite privacy notices to reflect agents. If an agent can act on a customer’s behalf, your consumer-facing notice must say so plainly. Regulators in the EU, California, and Colorado will read your notice against what your agent actually does.
  4. Least privilege plus red-teaming at the tool level. Agents get their own service accounts, minimum-necessary scopes on every tool, short-lived tokens, and auditable logs. Regular red-team exercises should target indirect prompt injection, memory poisoning, and data exfiltration — the OWASP agentic top risks.
  5. Keep humans in the loop for sensitive outcomes. For sensitive customer actions (account changes, data disclosures, financial commitments, health information), require a human approval step until your governance is proven.
Connect with us for a free privacy risk assessment to get started.

‍

SOURCES

1. Rehberger et al., “EchoLeak: The First Real-World Zero-Click Prompt Injection Exploit in a Production LLM System,” arXiv 2509.10540; CVE-2025-32711. https://arxiv.org/abs/2509.10540

2. IBM Security, 2025 Cost of a Data Breach Report. https://www.ibm.com/reports/data-breach

3. OWASP Gen AI Security Project, Top 10 for Agentic Applications (December 2025). https://genai.owasp.org/2025/12/09/owasp-top-10-for-agentic-applications-the-benchmark-for-agentic-security-in-the-age-of-autonomous-ai/

4. Bloomberg, “Samsung Bans Generative AI Use by Staff After ChatGPT Data Leak” (May 2023); AI Incident Database entry 768. https://incidentdatabase.ai/cite/768/

5. Moffatt v. Air Canada, 2024 BCCRT 149 (British Columbia Civil Resolution Tribunal, Feb. 2024). https://www.americanbar.org/groups/business_law/resources/business-law-today/2024-february/bc-tribunal-confirms-companies-remain-liable-information-provided-ai-chatbot/

6. Garante per la protezione dei dati personali (Italy) enforcement action, May 2025, summarized by Buchanan Ingersoll & Rooney. https://www.bipc.com/european-authority-fined-emotional-ai-company-for-privacy-violations

7. Reco AI, “AI & Cloud Security Breaches: 2025 Year in Review” (Salesloft Drift / UNC6395). https://www.reco.ai/blog/ai-and-cloud-security-breaches-2025

8. WilmerHale, “Year in Review: 2024 Generative AI Litigation Trends” (CIPA wiretapping class actions). https://www.wilmerhale.com/en/insights/blogs/wilmerhale-privacy-and-cybersecurity-law/20250307-year-in-review-2024-generative-ai-litigation-trends

9. Noma Security, “ForcedLeak: AI Agent risks exposed in Salesforce Agentforce” (CVE assigned, CVSS 9.4, Sep. 2025). https://noma.security/blog/forcedleak-agent-risks-exposed-in-salesforce-agentforce/

10. NeuralTrust, “Why Your AI Model Might Be Leaking Sensitive Data”; Help Net Security coverage of agentic AI security surveys (2025). https://neuraltrust.ai/blog/ai-model-data-leakage-prevention

‍