How We Test Dog Breed Detection

What the benchmark measures

The benchmark measures visual breed recognition from reviewed photographs. It records top-1 and top-3 accuracy, response-contract compliance, repeated-run stability, confidence calibration, latency, and estimated inference cost. Subject classification and mixed-dog behavior use separate gates because they answer different questions. The public model comparison uses 100 fixed photos, three repetitions, and one analysis pass per photo in each repetition. See Dog Breed Bench.

It does not measure genetic ancestry. A model can correctly identify a dog's strongest visible resemblance without knowing the dog's family tree.

Dataset construction and review

• 100 canonical breeds with five accepted photographs per breed.
• Three development photos, one regression photo, and one holdout photo per breed; the holdout split has been evaluated before.
• Photographs sourced from Wikimedia Commons with recorded author, license, URL, and SHA-256.
• Each accepted photo must show one meaningful dog subject with visible breed morphology.
• Codex visual review sought to exclude near duplicates, wrong breeds, multiple dogs, weak visibility, and non-photographic media.
• Fifteen confusion packs group visually similar breeds for targeted error analysis.

Breed names and aliases are aligned against the AKC directory and FCI nomenclature. Review decisions remain recorded rather than silently discarded. The accepted photos were reviewed by Codex using contact sheets. The review record does not establish independent verification by breed experts.

Development photos support prompt iteration, and regression photos support routine comparisons. The holdout split was also evaluated during earlier model work, so it is no longer an unseen final test. It remains useful comparison evidence; a new, independently reviewed set is needed before claiming an untouched final evaluation.

Look-alike breed families

The fifteen confusion packs tag families such as the upright-ear shepherds, the northern spitz breeds, the sighthounds, the Welsh corgis, the mastiff types, the black retrievers, and the schnauzers. 52 of the 100 breeds belong to a pack; the remaining breeds have not been assigned to a pack. Before a production decision, every error is inspected by confusion pair and tag rather than treated as one anonymous miss.

Each pack, the visible tells that separate its members, and the state of per-group measurement are published on our look-alike dog breed groups. Per-pack accuracy is added to that page only once it has been measured on a benchmark run.

Historical production release measurements

Release decisions use a strict 30-breed clear-breed suite: one licensed reference photo for each of 30 breeds, run three times each, so that accuracy, the response contract, latency, cost, and repeated-run stability all have enforced floors. Stability means the top answer for a photo did not change between repetitions. The production release evaluated on August 19, 2026 used the same prompt and response schema across those repeated clear-breed cases. The selected vision stage produced the historical results below. Five source photos also appear in the larger 100-breed benchmark, so the two suites are not fully independent.

• 94.4% top-1 accuracy across 90 repeated clear-breed calls.
• 100% top-3 accuracy and 100% response-contract compliance.
• 96.7% repeated-run stability.
• 2.645 second median latency and 7.692 second p95 latency.

A separate 108-call subject-classifier gate reached 100% dog recall and 100% non-dog specificity on its reviewed fixtures. A 60-call mixed-dog robustness gate reached 100% dog detection and response-contract compliance. Its evaluator reported no ancestry claims, but the wording check could miss percentage claims. That check was corrected on September 28, 2026; the mixed-dog benchmark has not been rerun with the corrected evaluator.

These figures were measured on August 19, 2026. They do not report accuracy for newer candidate models or the public model comparison linked above, and do not promise that 94.4% of real-world uploads will be correct. Phone photos, puppies, unusual grooming, occlusion, lighting, and breeds outside the evaluation set can be harder.

Confidence is not ancestry

The detector returns confidence-ranked visual matches. A 70% result means the first candidate was the strongest visual match in that response; it does not mean that a dog is genetically 70% that breed.

For a mixed dog, compare multiple photos and treat shared traits across the shortlist as clues. Use a laboratory DNA test when ancestry evidence matters.

Known limitations

The release suite uses reference photographs. Coverage of age, lighting, framing, and grooming is not consistently labeled, so the measurements above do not establish accuracy for the conditions below. Independently sourced, expert-reviewed cases are needed to measure these limits more reliably.

• Lighting. Backlighting, heavy shadow, and indoor low light hide coat color and texture. A wet or backlit coat can look longer or darker than it is, which matters most for the black retrievers and the white companion dogs.
• Angle and framing. Close crops hide body proportions, head-on shots hide ear carriage and topline, and wide-angle phone lenses distort the muzzle. Size is invisible without something else in the frame, and size is the main difference inside the sighthound, schnauzer, and collie-type families.
• Puppies. Puppy coats, proportions, and ear set differ from the adult reference morphology the detector compares against. A Yorkshire Terrier puppy is black and tan with a short coat, and guardian breed puppies of several breeds look alike.
• Senior dogs. Grey muzzles, thinner coats, changed posture, and weight gain or loss move an older dog away from breed-standard photographs.
• Unusual coat colors. Rare, non-standard, or heavily faded colors push a dog toward a look-alike breed that usually wears that color. Grooming has the same effect: a pet clip removes the coat texture that separates several small companion breeds.
• Multiple dogs and occlusion. The benchmark rejects photos with more than one dog or a partly hidden subject, so behavior on those uploads is not measured. When the subject is ambiguous the detector can describe the wrong animal.
• Look-alike pairs. Closely related breeds can share structure, coat, and expression. The fifteen confusion packs exist to measure this. Production safeguards cover uncertain results in the upright-ear working-breed and pinscher look-alike families.
• Mixed dogs. Visible resemblance cannot establish genetic ancestry or predict individual behavior. The mixed-dog gate checks detection, the three-match response, and wording for possible ancestry claims. The wording check is a heuristic and still needs review; it does not score parent breeds against DNA.
• Provider drift. Provider behavior, models, and pricing can change after a release is measured, which is why the suite is rerun before any change to the production model.

Dataset composition, version 2026.1

The table lists every breed in benchmark v2 with its benchmark id, how many of its five reviewed photos sit in each split, the confusion pack it belongs to, and the Creative Commons licenses under which its photos were published. It is generated from the benchmark manifests by an open script, so it changes only when the dataset changes.

Totals: 500 photos across 100 breeds, split 300 development, 100 regression, and 100 holdout. Licenses in use: CC BY 2.0 (166), CC BY-SA 4.0 (135), CC BY-SA 3.0 (108), CC0 (35), CC BY-SA 2.0 (28), CC BY 3.0 (12), CC BY 4.0 (11), CC BY-SA 2.5 (4), CC BY-SA 3.0 de (1). Every photo also records its author and source URL in the benchmark manifest.

How to read it: the three count columns always total five, because every breed has exactly three development photos, one regression photo, and one holdout photo. A breed with no confusion pack is scored on its own; a breed with a pack is also scored inside that family, and the pack name links to its entry on the research page. The licenses column shows which Creative Commons terms cover that breed's photos, so anyone reusing the composition can see at a glance whether attribution or share-alike terms apply. The CSV has the same columns with one header row and no comments, so it opens cleanly in a spreadsheet or a script.

To reproduce it, run the composition script in the repository against the benchmark manifests. The output is deterministic and carries no timestamp, so a diff between two versions shows exactly which breeds, splits, packs, or licenses changed and nothing else.

• Download the dataset composition (CSV)
• Read the confused-breeds research page for what each pack contains and why.

Release and correction policy

Production candidates are compared on the same reviewed fixtures. Prompt changes use development and regression cases; the previously evaluated holdout provides additional comparison evidence. A release must pass accuracy, response, stability, and latency gates before replacement.

The gate floors are written into the evaluator so they cannot drift between runs: at least 90% top-1, at least 29 of 30 top-3, 100% response-contract compliance, at least 80% repeated-run stability, a p95 latency no higher than 9 seconds, and an estimated cost no higher than $1.50 per 1,000 scans for the vision stage. A candidate that misses any floor is not released, however well it does on the others. Candidates that clear every floor are compared on a Pareto frontier of quality against cost, and the least expensive one that clears the agreed product floors is chosen.

A winner is never picked from a small smoke run. The comparison uses three full repetitions, every error is read by confusion pair and tag, and provider prices are rechecked on the day of the decision because the costs embedded in the evaluator are dated.

Report a reproducible problem through support. Methodology and results are updated when the model, prompt, benchmark, or release decision changes.

Change log

The page version and dates record content changes separately from the dataset composition version. A measured number changes only when a new run has been completed on the reviewed fixtures.

• 2026-08-19. First publication. Benchmark v2 dataset description, the production release results measured that day, the confidence-versus-ancestry explanation, and the release policy.
• 2026-09-02. Version 2026.1. Added the per-breed composition table, the downloadable composition CSV, an expanded known-limitations list, and the link to the confused-breeds research page. No measured number changed.
• 2026-09-28. Page version 2026.2. Clarified Codex review provenance, prior holdout use, overlapping source photos, and the historical mixed-dog evaluator limitation. Corrected the safeguard description and linked the public model comparison. Dataset composition remains version 2026.1; no measured number changed.