AI Accuracy Benchmark — Latest Run

Run mode: live
Headline provider: openai (gpt-5.4-mini)
Generated: 2026-06-12T05:24:57.355Z

Phase B scope (+ A/C follow-ups)

• Two ERP pairs: Salesforce Account -> NetSuite Customer and Salesforce Account -> Business Central Customer.
• Two base providers: OpenAI (default, gpt-5.4-mini) and Anthropic (claude-haiku-4-5) via --provider; --matrix runs the full provider x pair cross-product (4 cells) in one invocation under a single cost cap.
• Two opt-in providers: OpenRouter (pinned :free model, $0, requires OPENROUTER_API_KEY) and LM Studio (local server, $0, model discovered from /v1/models) via --include-provider (matrix) or --provider (single run). Never part of the default matrix or the canonical headline.
• Metrics: top-1 accuracy with a Wilson 95% CI per cell, plus hallucination count. Self-consistency is still out of scope.
• Remaining cuts: no nightly CI smoke (manual invocation only); population-level absolute-% claims still require broader fixtures than these hand-labeled sets (the CI quantifies sampling noise on the fixture, not fixture representativeness).

Provider x pair matrix

Total estimated cost across runs: $0.1142 (cap $5.00 for the whole invocation).

Headline run (canonical cell: openai x sfdc-to-ns-customers)

The top-level JSON fields (and the README ai_accuracy metric) mirror this cell; the matrix table above carries the rest.

Data-leakage guard

Every fixture is REQUIRED to exclude every (sourceField, targetField) pair that appears in src/services/ai/prompts/FieldMappingPrompts.ts COMMON_MAPPING_EXAMPLES. The few-shot examples ship in production prompts, so allowing them in the benchmark would let the model "cheat" against patterns it was just taught. The prompt may include broad target schema context, but it is REQUIRED to include substantial distractors and not embed only the fixture target answer-set as an allowed vocabulary. Per-pair posture:

• SFDC -> NetSuite (sfdc-to-ns-customers): broad NetSuite Customer schema (195 fields) as candidates, with a >=75 absolute-distractor floor relative to the fixture answer-set.
• SFDC -> Business Central (sfdc-to-bc-customers): the COMPLETE real OData customers schema (19 fields — every scalar Property of src/connectors/fixtures/bc/metadata/customers.xml, parity-tested against the XML) as candidates. Because the real schema is small, the absolute floor is replaced by a proportional rule: the fixture labels at most floor(schema/2) distinct targets, guaranteeing >=50% distractors.

Both exclusions and the per-pair posture are enforced by tests/unit/scripts/run-ai-accuracy-benchmark.dataLeakage.test.ts.

Runbook (operator)

# Rehearsal -- deterministic mock provider, $0 cost. Used by CI / drift tests.
npm run benchmark:ai -- --dry-run --matrix

# The script does NOT load .env -- export the keys into the shell first:
set -a; source .env; set +a

# Live full matrix -- requires BOTH OPENAI_API_KEY and ANTHROPIC_API_KEY.
# MAX_BENCHMARK_COST_USD caps the budget for the WHOLE invocation (default $5).
# Runner refuses to start if the pessimistic worst-case estimate summed over all
# cells exceeds the cap AND aborts mid-run on the iteration whose actual
# cumulative spend (across cells) would cross it.
MAX_BENCHMARK_COST_USD=5 npm run benchmark:ai -- --matrix

# Opt-in $0 providers (never part of the default matrix or the canonical headline):
#   openrouter -> pinned :free model, requires OPENROUTER_API_KEY
#   lmstudio   -> local server; model discovered from <base>/v1/models; base URL from
#                 LMSTUDIO_BASE_URL (unset under WSL resolves the Windows-host gateway)
npm run benchmark:ai -- --matrix --include-provider openrouter --include-provider lmstudio

# Single-cell variants (incompatible with --matrix):
npm run benchmark:ai                                                              # openai x SFDC->NS
npm run benchmark:ai -- --provider anthropic                                      # claude-haiku-4-5
npm run benchmark:ai -- --provider lmstudio                                       # local, $0
npm run benchmark:ai -- --fixture scripts/golden/fixtures/sfdc-to-bc-customers.yaml  # SFDC->BC

# After a live run, re-stamp the README templated value:
npm run metrics:generate && npm run metrics:sync-tokens
git add docs/review/ai-accuracy-benchmark.json docs/review/ai-accuracy-benchmark.md metrics.json README.md

Known limitations

• The benchmark measures top-1 accuracy on small hand-labeled sets. The Wilson 95% CI quantifies sampling noise at that fixture size (small-n intervals are wide by construction), but it is NOT a population-level claim: the fixtures are curated, not sampled from production traffic.
• The "hallucination" metric is conservatively defined as a suggestion whose sourceField does not appear in the test case's labeled source fields. Within-source-but-wrong-target counts as a top-1 miss, not a hallucination.
• The dry-run mock is deterministic by construction (oracle returns the labeled answer). It validates the harness wiring, not the model. Trust the live-run accuracy numbers.