Capabilities & Build Guide

How OnlyData works end to end — the shared entity graph, how custom datasets get built and stay alive, what's platform vs. per-dataset, and how we co-develop it. Written as the working reference for the team.

1. Platform capabilities at a glance

2. One shared graph, many dataset views (the "co-op" model)

"Co-op" means two related things here, and keeping them straight matters:

2a. The shared canonical graph — the architectural co-op

Datasets are filtered views, not ownership. No dataset owns its entities. Every company any dataset touches becomes (or updates) a canonical od_businesses row; every person becomes (or links to) a canonical od_profiles row. Dataset membership is a separate many-to-many table:

• od_business_lists — (business_id, list_slug, sublist, source, confidence). One company can sit in many lists (Anthropic is in both vibe_coder and ai_economy). Curated taxonomy lives here precisely so the industry classifier can never clobber human curation (migration 030 exists because it once silently did, ~1,169 rows).
• od_custom_rows.profile_slug — the person-side bridge (migration 040). Identity, verified contact, claim state, and signals live on the canonical profile; the custom row keeps only dataset-specific attributes plus freshness metadata (last_verified_at, verification_method, stale_after_days).

The rule we follow (and should harden into the platform): wherever the custom dataset's entity type allows it — companies with a real domain, people with a resolvable identity — rows are created/merged into the canonical graph and the dataset holds only a membership pointer plus its custom attributes. Only attributes that are genuinely dataset-specific (e.g. a realtor's current_listings) stay in the dataset's JSONB. Shared attributes (description, industry, employee estimate, AR score) live once, on the canonical row.

2b. The literal co-op feed — opt-in cooperative sharing

GET /api/coop/profiles returns only profiles where the owner opted in (coop_enabled), licensed cooperatively (attribution required, no resale) to partner tools like Rally. Curated person rows (e.g. Boise agents) enter this feed only when the person claims their profile and opts in — which is exactly the product hook: build the dataset → agents claim → claimed profiles join the co-op → the co-op powers every partner surface.

3. Deep dive: AI Economy — anatomy of a custom dataset build

The AI Economy list (live page) is a curated company dataset: ~26 sectors, companies stored directly in the shared graph with list_slug='ai_economy' memberships. It is the prototype for the repeatable dataset-automation template. Here is exactly what is standard platform and what is custom.

3a. Storage & identity shared

No custom tables at all. Companies are canonical od_businesses rows (source='ai_economy_curated', tier 1); sector membership is od_business_lists rows. A company can carry multiple sublists (unique on business + list + sublist + source). The public page renders straight off the membership query — one shared endpoint, GET /api/lists/:slug/companies, used by every curated list.

3b. Matching & entity resolution shared custom policy

• Platform matcher (shared): the scored matchRow() accumulator described in §1.
• Ingest policy (custom, deliberate): the AI Economy ingest does not fuzzy-match. It does strict exact-domain lookup only. Rationale: on a curated daily ingest, fuzzy matching is how a real company gets swallowed by an unrelated row — sierra.ai was once merged into an outdoor store in Meridian, Idaho. Curated ingest trades recall for zero false merges.
• Merges (shared): soft-merge with reversible audit. Batch dedup clusters by domain / name+city / address+fuzzy-name; a separate detector finds same-company-different-domain pairs but never auto-merges — it reports for human review.
• Membership-rot repair (custom clone of a shared pattern): background dedup strands list memberships on merged "loser" rows. A recursive-CTE repair walks merged_into_id chains, re-points memberships at the canonical survivor, and deletes memberships on dead rows. Runs every day, verified to zero. The 2026-08-16 run found the actual rot engine: 52 half-finished merges where an active loser still held the domain while the canonical had domain NULL — so ingest re-added memberships to losers every morning and repair deleted them every afternoon. Fixed by moving domains onto canonicals and deactivating losers.

3c. Attributes & enrichment shared custom shortcut

Every new company gets the full shared 10-model queue: canonical URL → verification → industry (→ NAICS) → MX/SPF/DKIM profile → B2B/B2C → Agent Readiness → description → AI-native → agent ecosystem role → employee estimate. One custom shortcut: sector membership deterministically implies the agent-ecosystem role (a shared sublist→role map covers five curated lists), so the LLM classifier only runs on non-curated companies (and in shadow mode on curated ones, for comparison).

3d. Cleanup processes shared custom taxonomy

• Taxonomy file (custom): data/taxonomy/ai-economy.json — 26 canonical sectors, a synonym map for drift, forbidden values, per-company overrides (89 domains), casing rules, and an enforcement contract (at ingest / at enrichment / at audit / on merge). This file is the dataset definition for hygiene purposes.
• Category + sublist normalizers (custom scripts, shared pattern): synonym-map fixes auto-apply; forbidden/ambiguous values are surfaced for human triage, never guessed. Sublist patches are per-list so they can never break other datasets' pages.
• Drift auditor (shared): one generic script reads any taxonomy file and exits non-zero on drift.
• Domain hygiene (shared): brand-rollup de-childing enforces one-domain-one-entity; a companion script nulls now-meaningless crawl outputs on de-childed rows. Mega-domains (google, linkedin, github…) are blocked at ingest.

3e. Metrics gap

Honest status: AI Economy has no daily metrics snapshot. Because it isn't an od_custom_datasets row, the generic snapshot and integrity jobs skip it; its only per-run metrics are the JSONL discovery log and the Telegram summary. The Boise pipeline (§4) shipped a metrics + auto-heal contract that AI Economy should adopt. This is recommendation #3 in §7.

3f. What's shared vs. custom — file inventory

4. Deep dive: Boise Real Estate — people + companies, custom levels

Built 2026-08-15→16 for the JV with Andrew ("AI tools for local real estate agents"): concept to live, daily-refreshed data in ~24 hours, precisely because ~80% of it is the shared platform plus the AI Economy template. The dataset being built to demo the service is the service — it's also the JV's own prospect pipeline.

4a. Two coupled halves, deliberately not one table

~135 agents · ~23 companies at time of writing.

4b. Custom levels & hierarchy — how realtor / broker / team / franchise are modeled

• "Broker" is not a separate entity. It's a value of the agent-level role enum (Agent | Broker | Team Lead | Team Co-Owner). This keeps the person half flat and honest — a broker is a person with a license class, not a different kind of record.
• Teams and brokerages are one company tier. A named high-volume team (e.g. Lysi Bishop Real Estate) is a first-class company row in the "High-Volume Real Estate Teams" sector, right next to brokerages. No bespoke hierarchy table.
• National franchises are deliberately NOT entities. KW / Compass / RE/MAX / Sotheby's survive only as parenthetical text in the agent's brokerage string, and their domains are blocked at ingest exactly like mega-platforms. A local office without its own site gets domain = NULL, never the parent's domain. (Future home for this: an entity_type='brand' + parent_business_id — see §7.)

4c. Entity resolution — loose at the data layer, strict at the graph layer

agent (od_custom_rows.row_data)
  └─ profile_slug ──► od_profiles (canonical person)
        └─ od_team_links (person ↔ company, is_primary, relationship_type)
              └─ od_businesses = the TEAM or BROKERAGE (one tier)
                    └─ od_business_lists (sublist = canonical sector)

• row_data.brokerage is a deliberately loose string ("King & Edge Real Estate (Compass)") — human-readable, carries the franchise context.
• row_data.brokerage_domain is the strict join key: the profile seeder resolves it against active canonical companies and only then writes a verified od_team_links edge (relationship type inferred from role: co-owner → cofounder, lead → founder, else employee).
• Dedup is three-layered: pre-insert normalized-name check → shared person-dataset dedup (profile slug → LinkedIn → GitHub → name+company, gap-fill merge, never overwrite non-blank) → idempotent profile seeding on (normalized name, source). Plus the same daily membership-rot repair as AI Economy (cloned SQL).
• Every domain must be attested by the entity's own site before it enters the CSV — never guessed. Known trap: franchise-hosted vanity sites that 301 elsewhere.

4d. Custom attributes (the part that's genuinely per-dataset)

All 14 agent fields live in the dataset's JSONB and are documented in a field dictionary with types, enums, quality rules, and coverage targets: name, brokerage, brokerage_domain, role, specialty, service_area, years_active, website, profile_url, evidence_url, why_in_db, status, email, current_listings. Standout rules: years_active from first-party statements only (never estimated); email from first-party contact pages only (never directories); why_in_db must cite a public artifact; current_listings is the freshness signal ("N (source-url)"). No scraped socials — linkedin_url is intentionally null across the dataset. Everything shared (description, AR score, sector, domain hygiene) lives on the canonical company rows, not here.

4e. Metrics & the auto-heal contract now the template

One daily snapshot covers both halves (agents total/by-status/by-role, email + listings fill %, companies by sector, domain fill %, top brokerages by agent count, 24h deltas), written idempotently into the shared snapshot table. The contract: --check exits clean if today's snapshot exists, otherwise writes it (self-heal); the wrapper gives one retry; on final failure the script itself alerts with method + table + response body. Success also appends a one-line metrics record to the JV agenda doc so the partner sees the numbers daily.

4f. Shared vs. custom, and the co-op angle

• Reused with no fork: person-dataset dedup, embeddings backfill, dataset history/fields/snapshots, drift auditor, domain de-childing, the whole 10-model enrichment queue, the generic dataset REST/MCP surface (no Boise-specific endpoints exist).
• Custom (10 files): two ingests (initial upsert + additions-only), agent batch-adder, profile seeder (adapted from the AI Builders one), metrics script, hygiene clones, taxonomy, field dictionary, two pages, the discovery skill.
• Co-op: Boise agents are seeded unclaimed, so they are not yet in the cooperative feed — they enter when they claim and opt in. Claim-driven co-op growth is the JV's product motion, not a data-engineering step.

5. Daily update processes

Both datasets refresh via the same pattern: a scheduled morning job runs an agent end-to-end through the dataset's discovery skill — the skill markdown is the program; the Python/SQL scripts are its deterministic sub-steps.

6. Co-developing equally (without sharing a personal machine)

Context for Jody: the local enrichment worker and the morning discovery jobs currently run on Cam's personal Mac Studio ("Stu"). That machine also holds Cam's personal context, messages, and credentials — so machine access is not the collaboration surface, and can't be. The good news: the architecture already separates "what runs on Stu" from "what Stu runs on," so equal co-development doesn't require the box.

6a. What's fully shared today

• GitHub (producthackerai/onlydata) — every pipeline, skill, taxonomy, prompt, and page is in the repo. PRs are the co-dev surface for all of it.
• Supabase — the data layer. Jody gets his own dashboard login (org member) and his own credentials for scripts; all curated pipelines talk PostgREST, nothing requires Cam's machine.
• Railway — deploys, logs, and the Railway-side enrichment worker.
• MCP + admin API — queue stats, enrichment status, classification review, attribute edits: all reachable without any machine access.

6b. The pattern: the queue is the API, the repo is the machine

Stu's only privileged role is being a poller: it claims tasks from od_enrichment_queue and runs local models against them. That means Jody can do classifications and model tweaks at full parity like this:

1. Model/prompt changes live in the repo. The local worker's prompts, model choices, required-input contracts, and the sublist→role map are all files. Jody edits → PR → merge. Add a small "worker pulls latest main before each batch" step (recommendation below) and merged changes take effect on the next cycle with zero human-on-Stu involvement.
2. Work is dispatched through the queue. Jody enqueues re-classification jobs (by model, by segment, by dataset) via SQL or an admin endpoint — Stu picks them up on its own schedule. He never needs a shell.
3. Shadow mode for evaluation. The agent-role classifier already runs in shadow on curated rows. Extend that pattern: any model tweak can run shadow-first, results land in a comparison table, and promotion to live is a reviewed PR.
4. Ollama is commodity. For interactive iteration, Jody runs the same worker script against his own Ollama (same models) with his own Supabase creds and worker='jody' — the queue's worker column already supports routing. Nothing about local inference is Stu-specific.

6c. Boundaries (explicit, so nothing is weird later)

• No shell/SSH/screen-share access to Stu — it's a personal machine that happens to donate compute. Same rule as any personal laptop.
• Secrets are per-person: Jody gets his own Supabase key, Railway seat, and (if useful) Telegram bot for pipeline alerts — no shared personal tokens.
• Scheduled agent runs (the 6am discovery jobs) migrate over time to machine-independent scheduling (cloud routines) so "the pipeline ran" never depends on whose desk the Mac is on. The morning-planner routine already made this jump; discovery should follow (§7).

7. Enhancements: the "set-and-forget dataset" strategy

The strategy: customers (or we) define a custom dataset once — entity types, sources, taxonomy, custom fields, cadence — then the platform rips: creates it, finds new records, enriches, auto-heals, and updates daily, using as much shared data and shared code as possible, and pumping every eligible entity into the shared graph. Ranked recommendations:

1. Dataset definition as config. AI Economy and Boise RE are ~90% identical pipelines built by cloning ~10 files each. Collapse the template into one declarative file per dataset (datasets/<slug>.json): entity types, source classes, taxonomy ref, custom field dictionary, ingest policy (additions-only default), discovery cadence, exclusion rules. The harness generates the skill, schedules the job, and wires hygiene. This is the literal "set and forget" enabler — a new dataset becomes a config PR, not an engineering project.
2. One parameterized hygiene suite. The repair SQL, category-merge, and sublist-normalize scripts are per-dataset clones with a hardcoded slug; the drift auditor is already generic. Parameterize the rest by list_slug + taxonomy file so every dataset gets identical healing for free and fixes land once.
3. Metrics parity + registry. Generalize the Boise metrics/auto-heal contract into dataset-metrics.py --dataset <slug> and register list-backed datasets (like AI Economy) in the dataset registry so snapshot, integrity, history, and anomaly machinery cover everything. Today AI Economy is invisible to all four.
4. Kill the half-merge bug class. The membership-rot engine was 52 half-finished merges (active loser holds domain, canonical holds NULL). Make merge atomic — move domain + deactivate loser in one transaction — and add an axiom: no active row's domain may belong to a row that is the merge-target of another. Then rot can't be created, not just repaired.
5. "Promote to graph" as a platform step. Formalize the rule from §2: at custom-dataset creation and on every row add, auto-resolve rows against the canonical graph where entity type allows (company-by-domain, person-by-identity), create canonical entities for misses, and store pointers — so co-op growth is automatic, not a per-dataset convention. Custom JSONB keeps only genuinely dataset-specific fields.
6. Brand/franchise entities. Add entity_type='brand' + parent_business_id so KW/Compass/RE-MAX (and, on the AI side, umbrella companies vs. products) become linkable parents instead of blocked strings — unlocking roll-up queries ("all agents under Compass-affiliated teams") without breaking one-domain-one-entity.
7. Machine-independent scheduling + worker auto-update. Move the two 6am discovery agents to cloud-scheduled routines; have the local worker git pull before each batch and report its running commit into queue telemetry. Together with §6 this makes co-dev truly equal and removes the single-machine dependency.
8. Coverage denominators per dataset. Extend anchor sets (external universes with known totals) to new datasets — for Boise: licensed-agent counts from the state board; for AI Economy: tracked-funding universes — so "how complete are we" is a real metric, which is the honest core of the set-and-forget pitch.

8. Open issues & good first digs

Real, current, and each one teaches a subsystem:

Deeper references: /developers (API + MCP tools) · /datasets (how a domain becomes a full profile) · /stats (axiom + dataset trends) · the repo's skills directory for each dataset's operational spec.