Physical Address
304 North Cardinal St.
Dorchester Center, MA 02124
PII discovery is the unglamorous problem at the heart of data privacy. Find every place in your data estate that contains personally identifiable information, classify it correctly, and apply the right controls. Every governance vendor pitches this as solved. It isn’t. Worth understanding why, because the gap between “we have a PII discovery tool” and “we know where our PII is” is wide enough to drive a regulatory fine through.
What discovery has to do
Sounds simple. Look at every column in every table in every database. Determine whether the values are PII. Classify the type. Apply policy.
Each step is harder than it looks.
“Every column” assumes you know what tables exist. Most organisations have shadow data – spreadsheets, exports, dev databases, backup snapshots, S3 buckets with informal data. The catalogued estate is a fraction of the actual estate. Discovery tools that scan only the catalogued estate miss most of the problem.
“Determine whether the values are PII” is harder than pattern matching. Email addresses have a recognisable pattern; phone numbers mostly do; names don’t. A column called customer_name is obviously PII. A column called note with free-text customer feedback might or might not be, depending on whether the customer mentioned their address in the note. The hardest cases are free-text fields where PII appears unpredictably.
“Classify the type” matters because different PII types have different rules. Health data is regulated differently from contact data. Financial identifiers differ again. A discovery tool that says “this is PII” without saying “this is PHI under HIPAA” or “this is a payment card identifier under PCI” gives you less than half the story.
Why pattern matching falls short
First-generation PII discovery is regex-based. Find values matching the email pattern; classify the column as email. This catches the obvious cases and misses everything subtle.
- Email addresses with unusual formats. Plus-addressing, international TLDs, encoded characters.
- Phone numbers in free-text fields, mixed with other text, in various international formats.
- Names. There’s no regex for “is this a person’s name.” ML classifiers exist but their precision/recall is far from perfect.
- Addresses, which span multiple cultural formats.
- Pseudo-identifiers that are PII in combination. Date of birth alone might be fine; date of birth + postcode is identifying.
- Free-text PII. A support ticket transcript might contain a credit card number a customer pasted in.
Why ML-based discovery is also incomplete
The newer tools (Microsoft Presidio, Google Cloud DLP, AWS Macie, Collibra, BigID, OneTrust) use ML classifiers trained on labelled PII data. They work better than regex but they’re still probabilistic. The numbers vendors quote are typically “90%+ accuracy on the benchmark corpus.” The benchmark corpus is not your corpus. Field-of-use accuracy is usually 70-85% on real estate data, which sounds fine until you realise that 15-30% missed PII over a 10,000-column estate is 1,500-3,000 columns of unclassified personal data.
And the false positives matter too. A discovery tool that flags 5% of columns as PII when they aren’t generates audit work that the team eventually starts ignoring. The signal-to-noise tradeoff is real.
What actually helps
The mature approach combines several techniques.
Schema-aware classification. Use the column name and type as strong priors. A column named email typed VARCHAR is almost certainly email. Combine the metadata signal with content sampling. Most modern tools do this.
Shift-left on producers. Application teams declare PII fields at the point of data production, in version-controlled schema or data contracts. The discovery tool verifies rather than discovers. This is the only path I’ve seen scale, because it stops trying to discover what was never made discoverable in the first place.
Free-text scrubbing. Treat free-text fields as untrusted by default. Run them through an LLM-based PII redactor at ingestion. Modern models do this surprisingly well; presidio plus an LLM is a good combination. Don’t rely on after-the-fact discovery for unstructured data.
Periodic full-estate sweeps. Run a discovery sweep every few months that genuinely covers everything, including the shadow estate. Be willing to fail audits to expose gaps. The teams that pretend they have full coverage when they don’t are the ones that get caught later.
Human review of flagged columns. A discovery tool is a starting point, not an authority. Every classified column needs at least a one-time human review for accuracy and policy assignment.
The regulatory subtext
The reason this matters in 2026 is that the regulators are getting more sophisticated about what they expect. “We bought a PII discovery tool” isn’t a sufficient answer to GDPR Article 30 or to a state attorney general’s investigation. The expectation is increasingly that you can show what you classified, why, with what confidence, and what you missed. Discovery output as evidence, not just as input.
And the AI angle adds a new urgency. LLMs trained on your data inherit whatever PII is in the data. RAG systems that retrieve from un-scrubbed corpora can regurgitate PII in answers. The downstream consequences of unclassified PII have multiplied. Discovery is no longer a checkbox; it’s a prerequisite for letting any AI touch your data.
Next week – GDPR’s right to erasure meets the data lake. The most expensive sentence in European data law.
