Benchmark
October 2026
How Well Do Today's Tools Remove Personal Details From Clinical Notes?
11 de-identification tools plus Health Universe, 4 collections, 5,866 documents
Before clinical notes can be shared for research, every name, date, phone number, address and other personal detail has to be removed. This is called de-identification, and most teams now use software to do it. We tested 11 dedicated de-identification tools people use today (cloud services and free software), plus our own tool, Health Universe, on the same 5,866 documents to see how well each one finds those details.
0.956
Health Universe F2, #1 of 12
0.985
Core recall (names, dates, IDs, contact)
0.917
Precision
5,866
Documents in 4 collections
The documents come from four collections. Two are real clinical notes shared with researchers under data-use agreements (Stanford SHIELD and PhysioNet Deidentified Medical Text). Two are synthetic: realistic notes written for testing, with no real patients (ASQ-PHI, Technetium-I). The real notes went only to software running on our own machines or to cloud services that don't keep the data, and we publish only the totals for them.
Two Numbers: Recall and Precision
“Maria Lopez was seen at Mercy Hospital on March 3 by Dr. Ken Ito. Follow up in 2 weeks.”
It has 4 personal details (underlined), 39 characters in all. We count characters rather than whole items, so a long name counts for more than a short date. Here's what two tools might remove:
“Maria Lopez was seen at Mercy Hospital on March 3 by Dr. Ken Ito. Follow up in 2 weeks.”
Recall
100%
Precision
85%
F2
0.97
Recall: it removed all 39 personal characters.
Precision: it removed 46 characters and 39 of them were personal. The 7 in “2 weeks” weren't, so a little useful information was lost.
“Maria Lopez was seen at Mercy Hospital on March 3 by Dr. Ken Ito. Follow up in 2 weeks.”
Recall
46%
Precision
100%
F2
0.52
Recall: it removed only 18 of the 39 personal characters, so the hospital (14) and the doctor (7) are still visible.
Precision: all 18 characters it removed were personal.
The personal details are “Ken Ito” and “555-0142”, 15 characters including the space (underlined). The tool removed “Dr. Ken” and “0142”, 11 characters in all.
Recall answers “did it find all the personal details?” A missed detail is a privacy leak, so recall matters most. Precision answers “did it remove only personal details?” Low precision means useful clinical information gets blacked out. Because we count characters, partly catching a detail earns partial credit: removing only “Lopez” from “Maria Lopez” covers 5 of its 11 characters, and “Maria” is still a leak. A recall of 0.990 means 1% of the personal information was missed.
F2 rolls recall and precision into one score, counting recall twice as much as precision, since a missed detail is the worse mistake. The table is sorted by F2. A tool scores high on F2 only if it finds nearly everything without blacking out too much else. In the example, Tool A scores 0.97 and Tool B 0.52. Core recall counts only names, dates, IDs and contact details. Tools follow different rules for ages, places and organisations, so core recall compares them on the identifiers every policy removes.
Results
| Overall (mean of 4 datasets) | Recall by dataset | ||||||||
|---|---|---|---|---|---|---|---|---|---|
# | System | Recall | Core recall | Precision | F2 | SHIELD real notes | PhysioNet real notes | ASQ-PHI synthetic | Technetium-I synthetic |
| 1 | Health Universe Ours | 0.968 | 0.985 | 0.917 | 0.956 | 0.950 | 0.981 | 0.978 | 0.964 |
| 2 | obi deid_roberta_i2b2 runs locally | 0.893 | 0.904 | 0.888 | 0.891 | 0.933 | 0.798 | 0.944 | 0.897 |
| 3 | Amazon Comprehend Medical (DetectPHI) AWS | 0.916 | 0.944 | 0.809 | 0.890 | 0.892 | 0.826 | 0.951 | 0.994 |
| 4 | OpenMed runs locally | 0.820 | 0.883 | 0.825 | 0.814 | 0.871 | 0.708 | 0.732 | 0.968 |
| 5 | Cloud Healthcare API de-identify Google Cloud | 0.834 | 0.867 | 0.719 | 0.794 | 0.807 | 0.635 | 0.899 | 0.994 |
| 6 | Sensitive Data Protection (DLP) Google Cloud | 0.803 | 0.829 | 0.687 | 0.754 | 0.779 | 0.640 | 0.855 | 0.939 |
| 7 | Stanford de-identifier runs locally | 0.720 | 0.670 | 0.893 | 0.748 | 0.654 | 0.582 | 0.876 | 0.768 |
| 8 | GLiNER2 runs locally | 0.888 | 0.884 | 0.515 | 0.702 | 0.791 | 0.880 | 0.986 | 0.896 |
| 9 | Philter runs locally | 0.834 | 0.885 | 0.656 | 0.678 | 0.819 | 0.879 | 0.776 | 0.862 |
| 10 | NLM Scrubber runs locally | 0.773 | 0.916 | 0.608 | 0.649 | 0.780 | 0.672 | 0.685 | 0.953 |
| 11 | Microsoft Presidio runs locally | 0.718 | 0.876 | 0.532 | 0.607 | 0.679 | 0.677 | 0.657 | 0.860 |
| 12 | OpenBioNER-deid runs locally | 0.304 | 0.265 | 0.616 | 0.323 | 0.334 | 0.306 | 0.396 | 0.180 |
Overall scores are the plain average over the four collections. Core recall counts only names, dates, IDs and contact details, the identifiers every policy removes. Health Universe follows HIPAA Safe Harbor, so by design it keeps ages under 90, states, countries and organisation names, which these answer keys count as personal; its overall recall understates it, and core recall compares like for like. Chips mark the best value in each column. We left out NVIDIA Nemotron-PII because its answer key covers general personal data (company names, places, passwords) across many industries rather than clinical PHI. Technetium-I's answer key does not count clinicians' names as personal, so on that collection only, every tool keeps “Dr. Name” text. The test code is open source (MIT license): github.com/Health-Universe/deid-benchmark.
Get Access
Want the full results, or to run Health Universe on your own test set? Tell us a little about your work and we'll send you access.