Do AI detectors work on code (GitHub Copilot, programming assignments)?

Short version: the detectors you tried were built for prose and you are feeding them something that is not prose. The signal those tools look for is roughly "how predictable is the next token", and well-written code is extremely predictable by design. Conventions, idioms, standard library names, the same loop structure everyone learns in week three. A clean human solution to a small exercise looks exactly like a model's solution because both are converging on the one obvious answer. So you get noise, which is what you observed.

From a threat model view the more interesting question is what you are actually trying to detect. Copilot completing a line is different from pasting a whole assignment from a chatbot. Pick the behaviour you care about first, then ask whether any tool can see it. For most intro assignments the answer is no, and the department has to fall back on process.

There is research on AI-generated code detection, but I would be careful about reading it as "it works". The published approaches mostly retrain a classifier on pairs of human and model-written solutions to the same problems, and they report decent numbers inside their own benchmark. The caveats are the ones Sam raised: short solutions carry very little signal, conventional code looks model-like, and results drop sharply once code is reformatted, renamed or lightly edited. None of that is surprising. Code has far fewer degrees of freedom than an essay.

It is also worth knowing that Turnitin, which is what most institutions already pay for, says plainly in its AI writing FAQ that its model "does not reliably detect AI-generated text in the form of non-prose, or code," and in the same document that it is "not pursuing ChatGPT code detection at this time." If the biggest vendor in the space is not claiming to do it, a free prose checker is not quietly doing it either.

@Hana Sato thank you, the Turnitin line is exactly what I needed, because the instructor trusts that vendor more than she trusts me. A follow-up: several people have mentioned "process". For a Python course, does that mean asking students to submit a git history? We do not currently require it, and I am not sure first-years could manage git on top of the course.

not a programmer but i help my nephew with his homework sometimes. just paste the code into gptzero? it says it works on any text. we did that with one of his assignments and it said 90% human so he was fine. seems simpler than all this

I will address the process question from the policy side, since that is where it belongs. The departments I work with have largely stopped trying to detect assistant use in code and have instead defined it. The syllabus states which tools are permitted for which assignments, what must be disclosed, and that any student may be asked to explain any submitted line in a short oral check. That last clause does most of the work. It is not an accusation; it is announced in advance and applied to a sample of students each week.

Requiring a git history is defensible for upper-level courses and awkward for first-years. A lighter alternative is to have the course environment save snapshots automatically, which several teaching platforms do without the student touching version control. Whatever you adopt, write it down before the term starts. Retrofitting a rule after a suspicion is where most appeals succeed.

@Jules I would not trust that number in either direction, and I say that gently because it is a very common move. Those tools are trained on natural language. Code is not natural language, so the model is scoring something outside what it learned, and the result is close to arbitrary. Your nephew's 90% human reading is as meaningless as the "likely AI" reading Priya got on a hand-written file. If he wrote it, the thing that protects him is being able to explain it, not a percentage from a tool that was never built for the job.