CRM deduplication rules that work

Every CRM collects duplicates over time. Built-in dedup catches exact matches; the rest needs normalization, scoring and careful merge rules.

CRM deduplication rules that work

Give a CRM a year or two of web forms, imports, chat widgets and integrations, and duplicates pile up. The same customer exists as three contacts, the same company as two accounts with different owners. Reports double-count, segments misfire, and two reps call the same person in one afternoon.

The built-in deduplication tools in most CRMs catch exact matches. The real duplicates are subtler, and catching them takes rules.

How duplicates form

  • Phone number variants: +1-555-0134, (555) 0134, 5550134.
  • Email case and whitespace: John@example.com vs john@example.com .
  • Name variations: "Robert Smith", "Bob Smith", "R. Smith".
  • Company suffixes: "ACME Corp", "ACME Corp.", "ACME Corporation".
  • Imports stacking on top of existing records instead of updating them.
  • Web forms, chat widgets and integrations creating new records without checking for a match.

Rules that work

  • Normalize on save. Lowercase and trim emails, convert phones to E.164, strip company suffixes and punctuation. Store the normalized values in separate fields used for matching, and keep the originals for display.
  • Fuzzy-match names. Levenshtein or Jaro-Winkler distance catches typos ("Smith" vs "Smyth"); a nickname table catches "Bob" vs "Robert".
  • Score across several fields. Never dedupe on a single field. Combine email, phone, name and company into a score: high means auto-merge, medium goes to a review queue, low is ignored.
  • Never auto-merge without a backup. Keep the original records and a merge log for a while after each merge, so a wrong merge can be undone.

In code, the normalization step is usually small:

email_norm   = lower(trim(email))
phone_norm   = to_e164(phone, default_country)
company_norm = strip_suffixes(lower(trim(company)))  -- "ACME Corp." -> "acme"

Merge rules: which value wins

Finding duplicates is half the job. The other half is deciding what the merged record looks like. Pick a survivor record (for example, the one with open deals or the oldest one) and then decide field by field: the most recent phone number, the earliest "created" date, the owner from the record with active deals. Activities, deals, notes and attachments all move to the survivor. Write these rules down once, or every merge becomes a judgment call.

Accounts need their own rules. Company duplicates are harder than contact duplicates: subsidiaries, branches and rebranded companies look alike but aren't always the same customer. Match on tax or registration number and website domain first, name second. Keep parent and child relationships instead of merging branches into one account, and when in doubt, send the pair to review rather than merging it automatically.

Stop new duplicates at the door

  • Forms and integrations should look up an existing record by normalized email or phone and update it rather than create a new one.
  • Imports should run through the same matching rules as everything else, with a preview of what will be created and what will be updated.
  • Store external IDs from integrated systems on the record, so repeated syncs hit the same contact.
  • Show reps a "similar record exists" hint when they create a contact by hand.

Tools

Built-in deduplication in the major CRMs handles exact matches and some fuzzy ones; marketplace add-ons go further. Features change, so check what your CRM offers today before buying anything. For a custom CRM, or a messy export you're about to migrate, scripts over the exported data give you full control over the rules and a clear log of every decision.

Checklist

  • Normalized matching fields for email, phone and company.
  • Scoring thresholds for auto-merge and manual review.
  • Written survivor and field-level merge rules.
  • Merge log with the ability to undo.
  • Matching on create for every form, import and integration.
  • A scheduled duplicate report, reviewed regularly.

Clean data is also what makes segmentation work; see CRM segmentation rules. If you're building or reworking a CRM, put deduplication into the data model from day one. Retrofitting it is always more expensive.

Fighting duplicates in your CRM? Reach us on Telegram or MAX.

Custom CRM development
Related service — let's discuss your project.

Got a project on this topic? Tell us the task — we'll come back with an estimate and a schedule.

Message on Telegram → Message on MAX