The 60-Line Python Script I Replaced With One Command

This is the script I wrote to deduplicate a customer CSV. It worked. I used it for months.

import csv
from collections import defaultdict
from difflib import SequenceMatcher
import re

def normalize_phone(phone):
    """Strip everything except digits."""
    if not phone:
        return ""
    return re.sub(r'[^\d]', '', phone)

def normalize_name(name):
    """Lowercase, strip whitespace, remove titles."""
    if not name:
        return ""
    name = name.lower().strip()
    for title in ['mr.', 'mrs.', 'ms.', 'dr.', 'jr.', 'sr.']:
        name = name.replace(title, '')
    return ' '.join(name.split())

def similarity(a, b):
    """SequenceMatcher ratio between two strings."""
    if not a or not b:
        return 0.0
    return SequenceMatcher(None, a, b).ratio()

def find_duplicates(filepath, threshold=0.85):
    """Find duplicate records in a CSV file."""
    # Read data
    records = []
    with open(filepath) as f:
        reader = csv.DictReader(f)
        for row in reader:
            records.append(dict(row))

    # Normalize
    for r in records:
        r['_name_norm'] = normalize_name(r.get('name', ''))
        r['_phone_norm'] = normalize_phone(r.get('phone', ''))
        r['_email_norm'] = (r.get('email', '') or '').lower().strip()

    # Block by first 3 chars of normalized name
    blocks = defaultdict(list)
    for i, r in enumerate(records):
        key = r['_name_norm'][:3] if r['_name_norm'] else 'UNK'
        blocks[key].append(i)

    # Compare within blocks
    pairs = []
    for block_ids in blocks.values():
        for i in range(len(block_ids)):
            for j in range(i + 1, len(block_ids)):
                a, b = records[block_ids[i]], records[block_ids[j]]
                name_sim = similarity(a['_name_norm'], b['_name_norm'])
                email_sim = 1.0 if a['_email_norm'] == b['_email_norm'] else 0.0
                phone_sim = 1.0 if a['_phone_norm'] == b['_phone_norm'] else 0.0
                score = name_sim * 0.5 + email_sim * 0.3 + phone_sim * 0.2
                if score >= threshold:
                    pairs.append((block_ids[i], block_ids[j], score))

    # Cluster with union-find
    parent = list(range(len(records)))
    def find(x):
        while parent[x] != x:
            parent[x] = parent[parent[x]]
            x = parent[x]
        return x
    def union(x, y):
        px, py = find(x), find(y)
        if px != py:
            parent[px] = py

    for i, j, _ in pairs:
        union(i, j)

    # Write output
    clusters = defaultdict(list)
    for i in range(len(records)):
        clusters[find(i)].append(i)

    with open(filepath.replace('.csv', '_deduped.csv'), 'w', newline='') as f:
        writer = csv.writer(f)
        fields = list(records[0].keys())
        fields = [k for k in fields if not k.startswith('_')]
        writer.writerow(['cluster_id'] + fields)
        for cid, (root, members) in enumerate(clusters.items()):
            for idx in members:
                row = [cid] + [records[idx].get(k, '') for k in fields]
                writer.writerow(row)

    dup_clusters = {k: v for k, v in clusters.items() if len(v) > 1}
    print(f"Found {len(dup_clusters)} duplicate clusters")
    print(f"Total duplicate records: {sum(len(v) for v in dup_clusters.values())}")
    return clusters

if __name__ == '__main__':
    import sys
    find_duplicates(sys.argv[1])

60 lines. It reads the CSV, normalizes names and phones, blocks by name prefix, scores with SequenceMatcher, clusters with union-find, and writes output.

And it has problems.

What's Wrong With It

Scoring is primitive. SequenceMatcher is a decent general-purpose similarity function. It's not a good name similarity function. "Jon Smith" vs "John Smith" gets 0.82 — close to my 0.85 threshold. Change the threshold slightly and I lose real matches. Jaro-Winkler handles name-specific patterns much better.

Blocking is naive. First 3 characters of the normalized name. "Smith, John" and "John Smith" land in different blocks ("smi" vs "joh"). Missed entirely. Adaptive blocking strategies handle this.

Weights are guesswork. I picked 0.5/0.3/0.2 for name/email/phone because it felt right. No data-driven basis. No way to know if 0.4/0.35/0.25 would be better.

No golden records. It tells me records 42 and 87 are duplicates. It doesn't merge them into one clean row. I still need post-processing.

No anomaly detection. If someone's email is "test@test.com" or their phone is "000-000-0000," those are placeholder values that shouldn't count as matches. My script treats them as real data.

Doesn't scale. The nested loop in blocking is O(n²) within each block. At 100K records with hot blocks, it crawls.

No evaluation. I have no idea what the F1 score is. I think it works because spot-checking looks right. That's not measurement.

The Replacement

pip install goldenmatch
goldenmatch dedupe customers.csv

One command. Here's what it does differently:

Side by Side Output

My script:

cluster_id,name,email,phone
0,John Smith,john.smith@gmail.com,(555) 012-3456
0,Jon Smith,jsmith@gmail.com,555-012-3456
0,Jonathan Smith,john.smith@gmail.com,5550123456
1,Sarah Johnson,s.johnson@yahoo.com,555-045-6789
1,Sara Johnson,sarah.j@yahoo.com,(555) 045-6789

Useful. But which "John Smith" is the canonical one? I still have three rows.

GoldenMatch clusters output:

cluster_id,name,email,phone,confidence
1,John Smith,john.smith@gmail.com,(555) 012-3456,0.94
1,Jon Smith,jsmith@gmail.com,555-012-3456,0.94
1,Jonathan Smith,john.smith@gmail.com,5550123456,0.94
2,Sarah Johnson,s.johnson@yahoo.com,555-045-6789,0.91
2,Sara Johnson,sarah.j@yahoo.com,(555) 045-6789,0.91

GoldenMatch golden records output:

golden_id,name,email,phone
G001,Jonathan Smith,john.smith@gmail.com,555-012-3456
G002,Sarah Johnson,s.johnson@yahoo.com,555-045-6789

One row per person. "Jonathan Smith" wins because it's the most complete name. "john.smith@gmail.com" wins because it appears in 2 of 3 records. Phone is normalized.

What I Didn't Realize I Needed

After switching, I discovered features I didn't know I was missing:

Data profiling. goldenmatch profile customers.csv — shows column types, unique counts, null rates, and flags anomalies before I match. Catches data quality issues I used to find after debugging bad clusters.

The TUI. goldenmatch interactive customers.csv — browse clusters, adjust the threshold in real time with arrow keys, inspect golden records. I used to change the threshold in my script, re-run, and eyeball the output. The TUI makes this instant.

Incremental matching. goldenmatch incremental new_records.csv --against existing.csv — match only new records against an existing base. My script re-processed everything from scratch every time.

Rollback. goldenmatch rollback — undo a merge. When my script made a bad cluster, I re-ran from scratch.

The Lesson

My 60-line script was fine for a prototype. It solved the immediate problem. But it was frozen at "prototype quality" — hardcoded weights, primitive scoring, no evaluation, no golden records.

The gap between a working script and a production tool is exactly the features you don't think to build: anomaly detection, incremental matching, rollback, profiling, configurable merge strategies.

I could have added those features to my script. It would have grown to 500 lines, then 1,000, then it would need tests, then a config file, then a CLI framework.

Or I could type:

goldenmatch dedupe customers.csv

I chose the second option.

# Install
pip install goldenmatch

# Replace your script
goldenmatch dedupe your_data.csv

# See what you've been missing
goldenmatch profile your_data.csv

# Golden records (merged canonical rows)
goldenmatch dedupe your_data.csv --output-golden

GitHub: github.com/benzsevern/goldenmatch License: MIT

60 lines of Python taught me what I needed. One command gave me everything I was missing.