What happens when your AI chatbot leaks a customer's email address

It starts small. A customer asks your AI assistant about their order. The assistant, trying to be helpful, pulls the customer's email address from context and repeats it back in the response. No one notices. The log entry sits there.

Then someone asks: "Can you show me all my bookings?" The assistant says: "Sure, I found three bookings under thomas@example.com." Now that email address has travelled through your LLM provider's servers — probably in the US — and landed in a chat transcript stored on your backend. You've just given your LLM provider access to personal data you didn't need to share, and you may have created a GDPR cross-border transfer problem.

Why LLMs are so good at leaking PII

Language models are trained to be helpful and complete. When personal data is in the prompt, the model naturally includes it in the response because that's what makes the response feel relevant and accurate. There's no built-in concept of "this information is sensitive and should stay implicit."

The model doesn't know that Thomas's email address is a legally protected piece of personal data under Article 4(1) of GDPR. It just knows it was in the context and that including it makes the answer more useful.

The three moments where leakage happens

• Inbound: your user sends a message containing PII that gets forwarded verbatim to the LLM provider
• Outbound: the LLM response includes PII from its context window or from patterns it has learned to generate
• Logs: your application stores the full prompt-response pair in a database or log stream, locking in the exposure

What GDPR actually requires here

Article 25 of GDPR requires data minimisation and privacy by design. That means you should not send personal data to a third party (like an LLM provider) if the task can be done without it — or you should scrub it first.

Article 46 requires appropriate safeguards for transfers outside the EU. If your LLM provider processes data in the US and you're sending raw customer emails or names in the prompt, you need either an adequacy decision, standard contractual clauses, or a compliant alternative.

Most SaaS companies building on LLMs have neither in place. They're assuming the LLM provider's DPA covers it. It doesn't — not if the prompt contains personal data that didn't need to be there.

Scrubbing before the forward

The pattern that actually works: before forwarding any prompt to your LLM, run it through a PII detector. Replace names, emails, phone numbers, and other identifiers with generic placeholders. The model still understands the request — it just doesn't see the specific personal details.

"Thomas asked about his order" becomes "[PERSON] asked about their order." The model answers correctly. Thomas gets his answer. You never sent Thomas's name to a third-party server.

The same applies on the way back out. If the model still manages to generate something that looks like an email address or a Dutch BSN number in the response, you catch and redact it before it reaches the user.

This is what Guarrix does

We built Guarrix as a proxy that sits between your application and your LLM provider. Every message goes through our inbound pipeline: PII detection via Microsoft Presidio, injection and jailbreak detection via PromptGuard 2. Then we forward the cleaned prompt. Then we scan the response on the way back out.

We never see the content after scrubbing. Our audit logs record only metadata — token counts, which entity types were detected, latency, action taken. No message content, no names, no emails.

If you're building an AI product that handles EU customer data, you need this layer. Your LLM provider's terms of service are not a substitute for data minimisation.