AI-Generated Comment Detector Revamped to 77% Balanced Accuracy

On September 9, 2026, an improved version of a classifier for detecting AI-generated code comments was released. According to reporting by kqr for entropicthoughts.com on Lobsters, the new version resolves issues in the previous version and was rebuilt on a foundation of public data. It is distinguished by an architecture that combines logistic regression with large language model outputs. Against the backdrop of expanding generative AI use in development work, it is attracting attention as a practical means of verification.

The previous classifier had partially used personal, non-public data for training. Its underlying design also had unstable elements, leaving it in a state where code and data could not be shared. The improved version removes this constraint and was rebuilt using only public data. By moving to a shareable foundation, third-party verification and reuse have become possible. Ensuring reproducibility is an important factor in strengthening trust in detection technology.

AI-assisted coding is spreading rapidly. As shown in OpenAI to Discontinue Codex Skills Collection, Shift to Extensions, the operational form of assistance features continues to change. As shown in Kimi API Now Natively Supports Codex and Claude Code, generative assistance is becoming standardized across multiple development environments. As the boundary between human-written and machine-generated text grows blurred, demand is rising for technology that can determine provenance.

Shift From Private to Public Data

The biggest change from the previous version to the improved version is transparency of the training foundation. Because the previous version included private data, outside researchers could not replicate it. The improved version resolves this problem by using public data. It is also improved in that methodological details and evaluation conditions are easier to disclose. The design is now easier for academic readers to verify.

The reporting by kqr for entropicthoughts.com on Lobsters also emphasizes the hardening of the foundation. The previous version had a strongly experimental character. The improved version has been replaced with a more stable design. Evaluation procedures have also been put in order, with analysis using calibration curves and a confusion matrix added. The approach goes beyond simply reporting accuracy and shows an emphasis on understanding failure modes.

This shift can be assessed as laying the groundwork for practical use of the detector. A detector dependent on non-public data may function within an organization, but its validity externally remains unclear. The move to public data clarifies the basis for comparison. It is significant in making future improvements and comparisons with other methods easier.

In-Browser Trial and Interpretability Features

The improved version can be tried out on a web page. Because it is designed so that entered text is not sent outside the browser, code or text at hand can be pasted in for verification. Even when handling highly confidential snippets, concerns about transmission paths can be limited. Early testers are said to have reacted mostly positively.

Its interpretability design is also distinctive. Selecting any part of the target string shows which features were activated. It also shows how each feature contributed to the overall judgment. While the previous article broke down and explained phrasing characteristic of Claude, the improved version has changed to a method of checking directly on screen. The improvement is that users themselves can trace the evidence.

This approach could change how detection results are received. With only a single judgment label, users cannot tell whether the result should be trusted. Visualizing feature contributions makes it easier to identify potentially misjudged passages. In code review and education, presenting the rationale can become a starting point for discussion.

77% Balanced Accuracy and Calibrated

Probability Design

The central performance figure is 77% balanced accuracy. It indicates the proportion correctly judged under the assumption that human-written and machine-generated inputs appear with equal probability. The figure of 77% shows it is far from perfect discrimination. At the same time, it shows discriminative ability above random guessing.

The classifier outputs a predicted probability for each judgment. This probability is calibrated and can be read as the probability that the individual judgment is correct. Verification with a calibration curve showed predicted probabilities and actual accuracy distributed close to the reference diagonal. This confirms that the predicted probabilities align with reality. The same trend held for comments of different lengths.

A temperature parameter was used to adjust for changes in confidence with increasing data volume. As input grows longer, more evidence becomes available and judgment confidence changes. Correcting for this relationship aims to align the meaning of probabilities for short and long texts. Length-independent interpretation of probabilities is effective for aligning operating conditions in practice.

Since all dots lie very close to the reference diagonal, we know they are approximately correct.

The above is the original author’s explanation of the calibration curve. It visually supports the reliability of the predicted probabilities. Ensuring calibration has more practical significance than the accuracy figure alone. If the certainty of a judgment can be quantified, operators can make selective decisions.

Error Trends and Countermeasures Seen in the

Confusion Matrix

In the confusion matrix, machine-generated text, the detection target, is organized as positive. For human inputs, the true negative rate is 0.73 and the false positive rate is 0.27. For machine inputs, the false negative rate is 0.20 and the true positive rate is 0.80. Human-origin inputs are correctly judged as human 73% of the time, and machine-origin inputs are correctly judged as machine 80% of the time. In both cases, error rates are distributed around 25%.

An aggregate error rate of 25% feels high. However, this figure aggregates all inputs treated uniformly. Because each individual judgment comes with a calibrated predicted probability, results need not be taken uniformly. The risk of error differs between high-confidence and low-confidence cases. It is important to understand aggregate values and individual risks separately.

According to the original author’s summary, when confidence is 80% or higher, the risk of a false positive falls to 5%. When confidence is around 50%, the chance the judgment is wrong is about half. Because calibration holds, this reading is statistically consistent. In operation, it is effective to automate only high-confidence judgments and route uncertain judgments to human review.

When presented with a known-human input, the classifier correctly judges it as human 73 % of the time.

The above is the original author’s description of how to read the confusion matrix. With academic readers in mind, it explicitly defines true and false positive and negative rates. Because confusion matrices are frequently used in classifier discussions, the aim is to ensure comparability. The full picture can be grasped when understood together with calibrated probabilities.

Significance of the Public Implementation and

Remaining Limitations

This rebuild is significant in increasing the openness of detection technology. If sharing of code and data is assumed, independent replication experiments become possible. The validity of feature design and temperature adjustment can also be examined externally. It opens paths for improvement not available from a closed implementation.

At the same time, limitations are clear. 77% balanced accuracy does not mean infallible judgment. An aggregate error rate around 25% is high for adoption as standalone evidence. Operation using calibrated probabilities is essential, and misuse will occur if users do not understand how to interpret probabilities. Operation that treats detector output as conclusive evidence should be avoided.

Changes in the underlying generative models are also a constraint. The style and characteristics of generated text change with model updates. A detector trained on public data from a specific point in time may lose effectiveness over time. Continuous retraining and updated evaluation will be required. As inference infrastructure continues to expand, as shown in NextEra-Brookfield 10-Trillion-Yen Data Center Plan, the volume and diversity of generated output will increase. How to institutionalize maintenance of the detection side will be a challenge.

Editorial Opinion

Over the next 3 to 6 months, management of AI-generated code contamination is likely to emerge as a practical issue in development work. Attempts to incorporate calibrated probabilities into quality gates are likely to increase. In particular, operations that automate only high-confidence judgments and have humans review low-confidence cases are likely to spread.

On a 1-to-3-year view, provenance management for generated output is expected to become a standard function of development infrastructure. A culture of attaching generation probabilities to diffs and comments may take hold. In education, criteria for evaluating human writing ability are also likely to be reconsidered.

The remaining question is who guarantees the legitimacy of detection. Doubts remain over whether a level of 77% balanced accuracy can be treated as evidence in practice. The question is whether adoption should proceed without predefining responsibility for misjudgments.

References

• “Better AI code comment detector”, by entropicthoughts.com by kqr — Lobsters, 2026-09-09T18:27:24.000Z (ARR)
• Source URL: https://entropicthoughts.com/better-ai-comment-classifier