Stability Analysis of OWASP LLM Top 10: Verifying the Discrepancy Between Large-Scale Incident Data and Expert Rankings

AFFILIATE_PRODUCTS:

According to a paper reported by Kyriakos “Rock” Lambros, Steve Wilson, and others on arXiv, there exists a significant discrepancy between the expert-led OWASP ranking of security risks for Large Language Models (LLMs) and rankings based on actual incident data. This research highlights the need for data-driven risk assessment in the field of LLM security.

Research Background and Objectives

The analysis in this paper focuses on the “OWASP Top 10 for LLM Applications,” which ranks the top 10 security risks in LLM applications. This list is created through voting by the security practitioner community. The research by Lambros et al. quantitatively examines how well this expert-based assessment aligns with data from actually recorded incidents.

Construction of a Large-Scale Incident Corpus

Researchers collected LLM-related security incidents from CVE, GitHub Security Advisories (GHSA), Open Source Vulnerabilities (OSV), and the AI Incident Database and Archive (AIAAIC). They obtained a snapshot of 7,714 incidents, of which 6,639 were labeled based on a 20-category taxonomy. Based on this, they corrected the incident counts for each category using a Bayesian measurement error model to derive an incident-based risk ranking.

Alignment Between Expert Rankings and Data

The 2026 candidate list is created by blending expert voting (weight 0.75) and incident data (weight 0.25) in a fixed ratio. Thus, the data serves to correct the consensus but does not override expert judgment. However, the alignment between the two rankings is weak. Cohen’s κ is approximately 0.20, and the 90% confidence interval crosses zero. This suggests that the categories deemed most important by experts do not necessarily correspond to the categories that actually occur frequently in incidents.

Classifier Performance and Ranking Robustness

The study conducted a bake-off performance comparison test with four state-of-the-art language models, but no winner emerged. None of the models surpassed the balanced accuracy (0.863) of a baseline classifier based on the lower bound of incident frequency. Furthermore, verification using held-out ground-truth data confirmed that the rank order based on incident frequency is robust (Spearman ρ = 0.918).

Limitations and Positioning of the Analysis

This analysis is an exploratory study conducted by two members of the OWASP working group and does not replace official OWASP releases or processes. This point is clearly stated in the paper. While the results show a discrepancy between expert judgment and actual incident records, they do not conclude which is “more correct.”

Editorial Opinion

In the short term, these research findings could influence the formulation process of OWASP’s LLM Top 10. Specifically, future list creation might see a shift towards placing greater emphasis on quantitative analysis results of large-scale incident data, like the methodology used in this study, alongside expert voting. A more active discussion on data-driven evaluation methodologies is also expected within the broader LLM security community. In the long term, the establishment of such analyses could force a transformation in the process of setting security standards and best practices itself. Risk rankings will face a more fundamental question of how to construct them by reconciling the intuition of experienced experts with vast, context-lacking incident records. As LLMs evolve, the nature of incidents will continue to change; therefore, rankings are unlikely to be a static answer but rather a dynamic process requiring the periodic re-evaluation of both data and expertise. However, the weak correlation between expert rankings and incident data shown in the results raises several important questions.

References

• “Incident-Data Robustness Analysis of the OWASP Top 10 for LLM Applications (2026): How a Community-Expert Ranking Holds Up Against a Large-Scale LLM Incident Corpus”, by Kyriakos “Rock” Lambros, Steve Wilson — arXiv cs.CR (Cryptography and Security), 2026-08-21T04:00:00.000Z (ARXIV)
• Source URL: https://arxiv.org/abs/2608.19266