Over-combined, under-combined.

Entity resolution has exactly two ways to fail, and they pull in opposite directions.

Over-combination is merging rows that are not the same thing. It is the expensive one: two records go in, one comes out, and the addresses, phone numbers and history of the loser are gone or wrong. Under-combination is failing to merge rows that are the same thing. It is the embarrassing one: your dataset says a company appears four times, and anyone who knows the company can see it.

Every knob you can turn trades one for the other. Loosen the matcher and you over-combine; tighten it and you under-combine. The only real way out is to stop treating it as one dial — which is most of what this post is about.

Here is where OnlyData actually stood yesterday morning, on 32,900 active business rows:

No entity layer at all. A company_domains table where every row was an unlabelled string. And a UNIQUE (domain) constraint which — I will get to this — made it structurally impossible to store the only web presence half the dataset had.

The trap gallery

These are not hypotheticals. Every one is a real collision found by asking the database a single question: which URL identities are claimed by more than one business? That question returned 187 groups. Most were real duplicates. The interesting ones were not.

The one idea that fixed most of it

Every trap above is the same mistake wearing a different hat: treating a URL as an identity when it is really a statement about ownership.

So URLs get sorted onto a ladder. Not "is this a domain" but who controls this page — which then determines how much the URL is allowed to prove.

And the correction that made it right

My first version of this said: only owned domains count as match evidence. Clean, defensible, and wrong in a way that would have thrown away half the dataset. The pushback was immediate and correct:

"kinda — BUT a facebook pathed url or yelp page or instagram profile etc need to be saved correctly and matchable"

The distinction I had flattened is the whole thing. It is never "is this a social URL". It is what exactly is shared:

That last pair is the one a URL cannot settle on its own — the same string is a front door or a dead end depending on whose row it is. The company's name settles it, deterministically, with no model call: if the domain's label is the company's name, they are the operator.

Then the rules ran out

With the ladder in place, the clear cases were clear. 140 identities were shared by more than one business, and after the classifier fixes most were genuine duplicates. What was left was a pile of cases where no rule could be right — and this was the second correction I needed:

"we SHOULD have an 'inference' layer to settle unclear merges and look at all evidence to decide if its a merge/hierarchy/keep separate situation and store that decision etc — not all merges will be clear enough for simple rules"

Correct, and I had been avoiding it. Look at what was actually in the leftover pile:

No threshold separates those. "Different addresses" splits the supermarkets but also splits the two parks, which should be split — and merges nothing useful about SendGrid. They need what a human would do: read all the evidence at once and decide.

So there is now an adjudication layer. The important part is not the inference — it is that the verdict is stored.

And it is allowed to say it does not know. insufficient is a first-class verdict, because a model forced to choose will choose, and a forced merge is the expensive kind of wrong. 20 of 141 verdicts came back insufficient — including twelve that named a real employer the pool simply does not hold yet, which is now a to-create list rather than a silent miss.

What it actually caught

The headline cases are the best illustration, because my regex got four of them confidently wrong first and the reasoning is checkable:

That last row is the one I want to defend, because it is the one that looks like a miss. David Hewitt does work at Pydantic. The headline does not say that, and a pipeline that reads "member of" as "employee of" will be wrong about the open-source contributors it meets far more often than it is right. The honest answer is to not assert it from this evidence.

A merge is not the only answer

The deeper fix is that "are these the same?" was the wrong question to only have one answer to. Three things were being forced through one decision:

• Identity — are these rows the same real-world thing? Stored as entities with dated membership, so un-grouping is recoverable rather than destructive.
• Relationship — are they different things that are connected? Stored as typed, directional, evidenced edges. A shared domain with disagreeing names is now an edge, not a merge.
• Grouping — do I want to look at them as one? Stored as projections, which change no data at all.

Thirty-six Starbucks in Charlotte made this concrete. They are 36 businesses with 36 addresses. "Starbucks in Charlotte" is a genuinely useful thing to look at and is not a thing that exists — it has no address, no phone, and its revenue is not any member's revenue. Collapsing the rows to get the view would delete 35 real locations; refusing the view to protect the rows makes the dataset harder to use than it needs to be. As a projection it costs nothing: every-location shows 36, brands-as-one shows 1, and neither had to destroy the other's answer.

Where the techniques actually are

Entity resolution is a 60-year-old field and it is worth saying plainly which parts of the literature we are and are not using.

What we adopted

• Blocking on a computed key, not a raw field. Comparing 32,900 rows pairwise is 541 million comparisons. Blocking on identity_key — the classified URL identity rather than the raw domain string — is what makes the pass cheap, and it is also what makes it correct, because the classification happens before the blocking rather than after.
• Evidence with weights, not boolean rules. A shared owned domain scores 0.95; a shared Yelp listing 0.8; a shared platform host 0. This is the Fellegi–Sunter instinct — accumulate weighted agreement rather than pass/fail a rule — without yet fitting the weights properly, which is the honest caveat below.
• The explicit negative. not_related is a real stored fact, and there are 190 of them now. Without it an automated process re-examines the same pair forever and eventually merges it by accident. We have the scar: Idaho Central Credit Union and CapEd had to be split back apart after exactly that.
• Identity separate from the row, with dated membership. Hub-and-satellite in the data-vault sense, or a Palantir-style ontology object: the entity is a stable id, and a row's membership in it has a start and an end. "Was a member, no longer" is a date, not a deletion.
• The public suffix list, including its private section. This is the difference between abetterautorepair.wixsite.com (a tenant) and aws.amazon.com (infrastructure), and between chccs.k12.nc.us (a district) and k12.nc.us (every school in North Carolina).

What we deliberately have not done

• Transitive closure on match edges. If A matches B and B matches C, connected-components clustering says A, B and C are one entity. Matching is not transitive, and this is the single most common way a dataset silently develops a 400-row mega-entity. Correlation clustering — which can place a negative edge inside a cluster and pay for it — is the right model and we are not there yet. Right now groups are small enough that the ledger and the weak-group holdback stand in for it.
• Trained match weights. The 0.95/0.8 numbers are judgments, not estimates. A tool like Splink would fit them with expectation-maximisation from the data itself and give calibrated probabilities instead of my guesses. That is the clearest upgrade available and it needs labelled pairs — which the decision ledger is now quietly accumulating.
• Embedding-based candidate generation. Cheap recall for the name-only rows, where 17,293 rows with no domain live. Also the easiest way to over-combine at scale, so it needs the ledger and calibrated weights first.
• Bitemporal identity. Membership has added_at/ended_at, so we know when the record changed. We do not separately track when the world changed. Integromat becoming Make is one entity across time; today that is an edge, not a timeline.

The honest grade

Asked for a real one, so: this is where OnlyData's entity pipelines stand as of this evening, after a day of work on them. The "before" column is yesterday morning.