Deduplicating candidate records in your ATS

Deduplicating candidate records means finding every profile in your ATS that belongs to the same person and merging them into one, without losing a note, a stage change or a consent flag along the way. The reliable approach works in confidence tiers: merge automatically on exact email, almost automatically on normalized phone numbers, and only with human review on fuzzy name-plus-history matches. The merge itself should union everything and overwrite nothing, and every point of entry needs a duplicate check, or the database starts refilling the day you finish cleaning it.

Why do databases drift to 20-30% duplicates?

Nobody enters the same candidate twice on purpose. Duplicates accumulate because the same person keeps arriving through different doors, looking slightly different each time:

• Multiple email addresses. The developer who applied with a personal Gmail in 2024 applies again in 2026 with a new address. If your system keys identity purely on email, you now have two people.
• Job board re-imports. Each board exports its own version of a candidate. Apply through Indeed in March and LinkedIn in September, and most systems create two records, each with a separately parsed CV.
• Manual entry alongside automatic entry. A recruiter quick-adds someone after a call; the formal application lands through the careers page a day later.
• Migrations and merged databases. Switching systems, or combining two agencies’ books, imports overlap in bulk. This is the largest single duplication event most teams ever experience, which is why dedup belongs in the plan for migrating an ATS without losing data, not in the cleanup afterwards.

Run a few of these paths for a few years and the numbers get ugly: audits of mature recruitment databases commonly find duplicate rates in the 20-30 percent range. The cost is not untidiness. Split records mean split history, so the note about a strong interview sits on one profile while outreach goes out from the other. Two recruiters contact the same person in the same week. And the worst case is legal rather than embarrassing: a candidate who asked to be forgotten on one record keeps receiving email through their surviving twin.

Which matching signals can you actually trust?

Matching works when you process signals in confidence order and normalize before you compare. Lowercase and trim email addresses. Strip formatting and apply country codes to phone numbers. Then work down the table:

The fuzzy tier deserves care. A name by itself is worthless: any sizable database holds several unrelated people with the same common name. But a similar name combined with the same employer over the same years, or the same degree from the same school, is strong. Use string similarity rather than exact comparison so Liz matches Elizabeth and transliterated spellings match each other, set the threshold high, and accept that some pairs genuinely need a human eye. A reviewer clears a queued match in seconds; an automated false merge can take an hour to untangle.

One honest trade-off: tighten thresholds and you leave real duplicates in place; loosen them and you weld strangers together. The false merge is the worse failure, because it puts one person’s notes and consent history on another person’s record. When in doubt, leave the pair split and queue it.

What merge rules keep notes and history safe?

Matching finds the pairs; merging is where data actually gets destroyed, and most horror stories trace back to a merge that kept one record and threw the other away. The rules that prevent that:

• Pick a survivor, union the rest. Keep the more complete or more recently active record as primary, then copy in everything the other holds. Two phone numbers means the person has two phone numbers, not that one is wrong.
• Notes are append-only. Every recruiter note from both records survives, stamped with author, date and which record it came from. A merge should never ask anyone to choose the better note.
• Applications stay separate. One person who applied to two jobs in different years is one profile with two applications. Collapse the person, never the applications: that combined timeline is exactly what makes rediscovering past applicants work later.
• Consent merges to the strictest setting. An opt-out on either record wins. The shorter retention window wins. A deletion request covers the whole person. Anything looser lets duplicates quietly defeat your suppression lists, the failure mode behind honoring candidate opt-outs across channels.
• Make it reversible. Store the pre-merge state and offer an unmerge. The first week of running a new matcher will surface its mistakes; you want them to be edits, not losses.

How do you prevent duplicates at the point of entry?

Cleaning the database without fixing intake is mopping with the tap running. Prevention is one rule applied everywhere: no write path may create a candidate without checking for an existing match first.

That means every path. The careers site form, the manual add button, each job board import, the API, and the CV parser: when a parser turns a CV into a profile, the next step should be a match lookup, not an insert. The same goes for anything new you bolt on top. AI screening pipelines that write enriched profiles back are notorious for minting near-duplicates, a failure mode we unpack in adding LLM screening without duplicate records.

Two behaviors make prevention humane rather than annoying. First, surface the match instead of deciding silently: “this looks like the Sarah Veld who applied in March” with an attach-or-create choice takes a recruiter two seconds and beats any silent rule. Second, treat reapplication as signal, not noise. Someone returning after eighteen months is interest you did not pay for; attaching their new application to their existing profile preserves the context that makes the second conversation better than the first.

Where does Recruitifly fit?

Recruitifly treats identity hygiene as a compliance matter, not housekeeping. The platform is built in the EU with GDPR tooling throughout: retention windows, consent tracking, deletion requests and suppression lists, all of which only mean something when each person exists exactly once. Fly, the assistant that works across the whole platform, parses CVs into structured profiles and searches the talent pool on request, so “do we already know this person” is a question you can ask before anyone gets added. And because every write in Recruitifly is propose-then-confirm, changes to candidate records arrive prepared, shown to you, and applied only when you approve, which is exactly the posture an operation as hard to undo as a merge deserves. We consider that correct design, not caution.

We are in private beta. If duplicate records are quietly splitting your candidate history or leaking past your suppression lists, talk to us; there is a 7-day free trial on paid tiers when you are ready to test the cleanup against your own data.