Benchmarking Large Language Models for Java Code Smell Detection
2026 2nd International Symposium on AI-Driven Engineering Systems (ISADES), Mbale, Uganda, 19 - 20 Haziran 2026, ss.1-10, (Tam Metin Bildiri)
- Yayın Türü: Bildiri / Tam Metin Bildiri
- Doi Numarası: 10.1109/isades69945.2026.11608239
- Basıldığı Şehir: Mbale
- Basıldığı Ülke: Uganda
- Sayfa Sayıları: ss.1-10
- Ondokuz Mayıs Üniversitesi Adresli: Evet
Özet
Code smells are structural anti-patterns in source code that degrade maintainability and inflate long-term development costs. Existing detection approaches rely on hand-tuned metric thresholds or supervised classifiers that demand labeled training data, both of which limit generalizability across projects and languages. Large Language Models (LLMs) present an alternative: zero-shot classification through natural language prompting, requiring neither labeled datasets nor engineered features. We benchmark seven state-of-the-art LLMs—three commercial (Claude Sonnet 4, GPT-4o, Gemini 2.5 Flash) and four openweight (GPT-OSS 120B, DeepSeek-V4-Flash, Gemma 4-31B-IT, LLaMA 3.3-70B Instruct)—on 1,000 Java code snippets from the MLCQ dataset, a multi-annotator benchmark built from real production code in 20 open-source projects. Each model receives an identical zero-shot prompt and must output exactly one of five labels: clean, god_class, long_method, feature_envy, or data_class. We evaluate binary detection accuracy, multiclass type accuracy, and per-class precision, recall, and F1 score, and introduce a prompt-compliance check to quantify off-label (hallucinated) response rates. LLaMA 3.3-70B Instruct achieves the highest binary accuracy (70.7%) and smelly-class F1(76.4%), though these figures are driven primarily by a highrecall operating point rather than balanced type discrimination; GPT-OSS 120B leads on five-class type accuracy at 58.0%. Persmell analysis reveals that long_method is the most detectable smell with up to 92.6% per-class accuracy, while feature_envy (6.1−42.6%) and god_class (2.0−38.9%) remain largely undetectable at snippet level. Open-weight models achieve competitive type accuracy with their commercial counterparts, clustering within 5 percentage points when LLaMA is excluded. Statistical testing (McNemar’s test) reveals that small accuracy gaps—such as the 1.8% difference between GPT-OSS 120B and GPT-4o (p=0.1443)—lack significance, positioning open-weight models as viable alternatives for cost- or privacy-constrained deployments despite the absence of meaningful empirical advantage. The benchmark establishes a zero-shot performance baseline on a balanced, large-scale industrial dataset and identifies which smell types require context-augmented approaches beyond snippet-level prompting.