Are AI image detectors reliable, and what do the tests show?

What the academic tests keep finding

A detector learns the fingerprint of the generators in its training set, and that fingerprint does not transfer.

Wang and colleagues (CVPR 2020) trained a classifier on one GAN (ProGAN) and found it generalised "surprisingly well" to ten other CNN-based generators. The caveat arrived with diffusion models. Ojha, Li and Lee (2023) showed that the same kind of classifier, trained on GAN images, fails on newer diffusion and autoregressive models. Their explanation: the classifier tunes itself to the patterns that made the training fakes fake, and the "real" class becomes a sink for everything else, including unseen fakes. Their fix, nearest-neighbour matching in a large vision-language model's feature space, beat the prior state of the art by 15.07 mAP and 25.90 percentage points of accuracy on unseen generators.

GenImage (2023), a benchmark of over one million real and generated image pairs, formalised the two tests that matter: train on one generator and test on the others, and test on degraded images (low resolution, blur, compression).

The largest independent test of commercial tools

Ha and colleagues (2024) tested five automated detectors and three groups of humans on human art and matching images from five generators. On unperturbed images Hive scored 98.03% accuracy with a 0.00% false positive rate; Optic 90.67% with a 24.47% false positive rate; Illuminarty 72.65% with a 67.40% false positive rate. Two open research detectors, DIRE and DE-FAKE, landed near 50%.

The generator mattered more than the art style: Hive caught 100% of SDXL images but marked 6 of 70 Firefly images as human art, which the authors attribute to a lack of training data for the newer model.

JPEG compression barely moved any tool. Gaussian noise cut Optic to 52.63% and Illuminarty to 61.43% detection, near coin-flip, while Hive held at 88.73%. Glaze, an artist protection tool, pulled Hive down to 69.73% at medium intensity and 67.56% at high, from 96.83% unperturbed. Hive and expert artists both did well but failed differently: the tool was weaker against adversarial edits, the experts produced more false positives.

Vendor claims, labelled as claims

Hive's product page calls its performance industry-leading and cites the Ha study. AI or Not states 98.9% accuracy on its homepage without a published method. Resemble AI advertises "up to 99.5%" accuracy for its multimodal detector, with per-generator figures such as 98% for Midjourney and 94% for Stable Diffusion, from its own testing. None are independent, and none say what happens on generators released after training. The same structure of claim versus test appears in the text detector accuracy thread.

What each test or vendor actually reported

Study or tool	Reported result	Condition	Caveat
Ojha et al., 2023	GAN-trained detectors fail on diffusion and autoregressive fakes; CLIP nearest-neighbour adds +15.07 mAP	Unseen generators	Research method, not a product
Ha et al., 2024 (Hive)	98.03% accuracy, 0.00% false positives	Unperturbed art from 5 generators	Falls to 67.56% detection under high Glaze
Ha et al., 2024 (Optic)	90.67% accuracy, 24.47% false positives	Same set	52.63% detection under Gaussian noise
Ha et al., 2024 (Illuminarty)	72.65% accuracy, 67.40% false positives	Same set	Flags most human art as AI
AI or Not (vendor claim)	98.9% accuracy	Undisclosed	No independent evaluation cited
Resemble AI (vendor claim)	Up to 99.5% accuracy	Vendor's own tests	Multimodal figure, generator-dependent

What that means in practice

• Use the best tool and still doubt it. A 98% result on clean images from known generators does not carry over to a new model or an edited file.
• Weight false positives by tool. A 67% false positive rate cannot support an accusation against an artist.
• Pair the score with a human check. The Ha study found the tool and the expert failed in different ways; see the face thread for the human side.