How We Test Dog Breed Detection

What the benchmark measures

The benchmark measures visual breed recognition from reviewed photographs. It records top-1 and top-3 accuracy, response-contract compliance, repeated-run stability, confidence calibration, latency, and estimated inference cost. Subject classification and mixed-dog behavior use separate gates because they answer different questions.

It does not measure genetic ancestry. A model can correctly identify a dog's strongest visible resemblance without knowing the dog's family tree.

Dataset construction and review

• 100 canonical breeds with five accepted photographs per breed.
• Three development photos, one regression photo, and one untouched holdout photo per breed.
• Photographs sourced from Wikimedia Commons with recorded author, license, URL, and SHA-256.
• Each accepted photo must show one meaningful dog subject with visible breed morphology.
• Near duplicates, wrong breeds, multiple dogs, weak visibility, and non-photographic media are rejected.
• Fifteen confusion packs group visually similar breeds for targeted error analysis.

Breed names and aliases are aligned against the AKC directory and FCI nomenclature. Review decisions remain recorded rather than silently discarded.

The three splits have different jobs. Development photos are the only ones used while iterating on prompts. The regression photo for each breed is the routine check that a model or prompt change has not broken something that used to work. The holdout photo is never looked at during tuning and is reserved for the final decision on a release, so a candidate cannot be shaped to fit it.

Look-alike breed families

The fifteen confusion packs tag families such as the upright-ear shepherds, the northern spitz breeds, the sighthounds, the Welsh corgis, the mastiff types, the black retrievers, and the schnauzers. 52 of the 100 breeds belong to a pack; the rest have no close twin in the set. Before a production decision, every error is inspected by confusion pair and tag rather than treated as one anonymous miss.

Each pack, the visible tells that separate its members, and the state of per-group measurement are published on our look-alike dog breed groups. Per-pack accuracy is added to that page only once it has been measured on a benchmark run.

Measured production release

Release decisions use a strict 30-breed clear-breed suite that is separate from the 100-breed benchmark: one independently licensed reference photo for each of 30 breeds, run three times each, so that accuracy, the response contract, latency, cost, and repeated-run stability all have enforced floors. Stability means the top answer for a photo did not change between repetitions. The production release evaluated on August 19, 2026 used the same prompt and response schema across those repeated clear-breed cases. The selected vision stage produced:

• 94.4% top-1 accuracy across 90 repeated clear-breed calls.
• 100% top-3 accuracy and 100% response-contract compliance.
• 96.7% repeated-run stability.
• 2.645 second median latency and 7.692 second p95 latency.

A separate 108-call subject-classifier gate reached 100% dog recall and 100% non-dog specificity on its reviewed fixtures. A 60-call mixed-dog robustness gate reached 100% dog detection and response-contract compliance without ancestry claims.

These are benchmark measurements, not a promise that 94.4% of every real-world upload will be correct. Phone photos, puppies, unusual grooming, occlusion, lighting, and breeds outside the evaluation set can be harder.

Confidence is not ancestry

The detector returns confidence-ranked visual matches. A 70% result means the first candidate was the strongest visual match in that response; it does not mean that a dog is genetically 70% that breed.

For a mixed dog, compare multiple photos and treat shared traits across the shortlist as clues. Use a laboratory DNA test when ancestry evidence matters.

Known limitations

The release suite is built from clear, well-lit reference photographs of adult dogs. The conditions below are not yet represented in it, so the measured accuracy above does not cover them. They are listed here because they are the situations in which a real upload is most likely to be wrong, and because independently sourced, expert-reviewed cases of each kind are the next additions to the benchmark.

• Lighting. Backlighting, heavy shadow, and indoor low light hide coat color and texture. A wet or backlit coat can look longer or darker than it is, which matters most for the black retrievers and the white companion dogs.
• Angle and framing. Close crops hide body proportions, head-on shots hide ear carriage and topline, and wide-angle phone lenses distort the muzzle. Size is invisible without something else in the frame, and size is the main difference inside the sighthound, schnauzer, and collie-type families.
• Puppies. Puppy coats, proportions, and ear set differ from the adult reference morphology the detector compares against. A Yorkshire Terrier puppy is black and tan with a short coat, and guardian breed puppies of several breeds look alike.
• Senior dogs. Grey muzzles, thinner coats, changed posture, and weight gain or loss move an older dog away from breed-standard photographs.
• Unusual coat colors. Rare, non-standard, or heavily faded colors push a dog toward a look-alike breed that usually wears that color. Grooming has the same effect: a pet clip removes the coat texture that separates several small companion breeds.
• Multiple dogs and occlusion. The benchmark rejects photos with more than one dog or a partly hidden subject, so behavior on those uploads is not measured. When the subject is ambiguous the detector can describe the wrong animal.
• Look-alike pairs. Closely related breeds can share structure, coat, and expression. The fifteen confusion packs exist to measure this, and only the upright-ear shepherd family currently has a production safeguard.
• Mixed dogs. Visible resemblance cannot establish genetic ancestry or predict individual behavior. The mixed-dog gate checks detection, the three-match response, and the absence of ancestry claims; it does not score parent breeds against DNA.
• Provider drift. Provider behavior, models, and pricing can change after a release is measured, which is why the suite is rerun before any change to the production model.

Dataset composition, version 2026.1

The table lists every breed in benchmark v2 with its benchmark id, how many of its five reviewed photos sit in each split, the confusion pack it belongs to, and the Creative Commons licenses under which its photos were published. It is generated from the benchmark manifests by an open script, so it changes only when the dataset changes.

Totals: 500 photos across 100 breeds, split 300 development, 100 regression, and 100 holdout. Licenses in use: CC BY 2.0 (166), CC BY-SA 4.0 (135), CC BY-SA 3.0 (108), CC0 (35), CC BY-SA 2.0 (28), CC BY 3.0 (12), CC BY 4.0 (11), CC BY-SA 2.5 (4), CC BY-SA 3.0 de (1). Every photo also records its author and source URL in the benchmark manifest.

How to read it: the three count columns always total five, because every breed has exactly three development photos, one regression photo, and one holdout photo. A breed with no confusion pack is scored on its own; a breed with a pack is also scored inside that family, and the pack name links to its entry on the research page. The licenses column shows which Creative Commons terms cover that breed's photos, so anyone reusing the composition can see at a glance whether attribution or share-alike terms apply. The CSV has the same columns with one header row and no comments, so it opens cleanly in a spreadsheet or a script.

To reproduce it, run the composition script in the repository against the benchmark manifests. The output is deterministic and carries no timestamp, so a diff between two versions shows exactly which breeds, splits, packs, or licenses changed and nothing else.

• Download the dataset composition (CSV)
• Read the confused-breeds research page for what each pack contains and why.

Release and correction policy

Production candidates are compared on the same reviewed fixtures. Prompt changes use development and regression cases; the holdout is reserved for final decisions. A release must pass accuracy, response, stability, and latency gates before replacement.

The gate floors are written into the evaluator so they cannot drift between runs: at least 90% top-1, at least 29 of 30 top-3, 100% response-contract compliance, at least 80% repeated-run stability, a p95 latency no higher than 9 seconds, and an estimated cost no higher than $1.50 per 1,000 scans for the vision stage. A candidate that misses any floor is not released, however well it does on the others. Candidates that clear every floor are compared on a Pareto frontier of quality against cost, and the least expensive one that clears the agreed product floors is chosen.

A winner is never picked from a small smoke run. The comparison uses three full repetitions, every error is read by confusion pair and tag, and provider prices are rechecked on the day of the decision because the costs embedded in the evaluator are dated.

Report a reproducible problem through support. Methodology and results are updated when the model, prompt, benchmark, or release decision changes.

Change log

The version number and the dates on this page move together. A date changes only when the content changes, and a measured number changes only when a new run has been completed on the reviewed fixtures.

• 2026-08-19. First publication. Benchmark v2 dataset description, the production release results measured that day, the confidence-versus-ancestry explanation, and the release policy.
• 2026-09-02. Version 2026.1. Added the per-breed composition table, the downloadable composition CSV, an expanded known-limitations list, and the link to the confused-breeds research page. No measured number changed.