Can LLMs Extract Architectural Design Decisions from Source Code Commits? - A Preliminary Exploratory Study

Context: Architectural Design Decisions (ADDs) capture the rationale behind the structure and evolution of software systems but are rarely documented explicitly, and are often hidden inside source code commits. Recovering them is important for Architectural Knowledge Management (AKM). Problem: Extracting ADDs from commits is challenging due to their implicit and unstructured nature. Large Language Models (LLMs) have shown strong capabilities in understanding code and text, yet their effectiveness for this task remains underexplored. Study: We present a preliminary study using four LLMs (Gemini 3 Pro, DeepSeek R1, Kimi K2, Qwen3) with zeroshot and fewshot prompting on 30 developer-written ADDs from open-source projects. We score outputs with ROUGE-L, BLEU, METEOR, and BERTScore, and one author manually reviews the Gemini outputs. Results: All models reach a BERT-F1 above 0.81, and fewshot prompting improves alignment (Gemini BERT-F1: 0.828 to 0.847). However, the generated ADDs are often too long, implementation-focused, and miss the rationale behind the decision. This highlights opportunities for architecture-aware LLM systems and automated AKM.

From Textual Requirements to Microservice Architectures - A Comprehensive Evaluation of LLM-Based Design Synthesis

Microservice architectures have become dominant for modernizing monolithic systems, yet identifying appropriate services remains challenging and largely manual. Existing decomposition approaches are predominantly code-centric, limiting applicability in early design stages where only textual requirements are available. Despite advances in Large Language Models (LLMs), limited empirical evidence exists on their ability to synthesize complete microservice architectures from natural-language requirements, including service definitions and inter-service interactions. This study investigates whether an LLM can bridge requirements engineering and architectural design, generating architectures solely from textual requirements and evaluating structural agreement and perceived quality of results. We conduct a mixed-method study using OpenAI o3 under zero-shot (ZS) and few-shot (FS) prompting across two systems (Bookstore, PetClinic), one execution per system/condition. Architectures are evaluated through (i) comparison with reference architectures using precision, recall, and F1-score for service identification and communication recovery, and (ii) a blinded expert assessment of correctness, completeness, modularity, and plausibility, plus open feedback synthesis. OpenAI o3 identifies services with higher agreement under FS prompting (F1 = 0.79 for ZS versus = 0.97 for FS). Communication recovery is more challenging: ZS produces dense architectures with high recall but low precision (F1 = 0.61), while FS improves agreement, reaching F1 = 0.82 and reducing unsupported dependencies. Expert evaluation corroborates these results, with FS architectures perceived as more modular, coherent, and plausible than ZS outputs. OpenAI o3 shows potential for requirements-driven synthesis when guided by exemplar prompting. Results are model- and context-specific from two small systems, not model-independent proof.

Metaphor-Induced Algorithmic Steering: Cross-Domain Procedural Transfer in LLM Code Generation

Large language models benefit from elements in natural language, such as metaphors and analogies in training data and inference input to achieve generalisability across different domains. However, these language elements may also lead to unwanted behaviors when metaphorical expressions implicitly transfer inappropriate procedural patterns into new tasks. In this paper, we show that metaphorical instructions can induce analogical transfer of procedural mechanisms, thus steering code-generation models towards less efficient algorithms. We refer to this metaphor-induced effect as metaphorical algorithmic steering: a skill that is benign and plausible within its source domain transfers an abstract procedural schema into a programming task, causing the model to favor exhaustive search, full scans, or repeated reconstruction without explicitly mentioning the target algorithm. More broadly, this suggests that code-generation models can carry procedures that are appropriate in a task's background domain into the task's programming problem, where they can lead to unwanted outcomes. To study this phenomenon, we develop MASC (Metaphorical Algorithmic Steering for Code Generation), a framework that iteratively metaphorizes and refines benign skills to elicit low-efficiency code while remaining benign and task-relevant. Beyond behavioral evaluation, we study whether this phenomenon is detectable and mechanistically reflected in model representations. Our method achieves high detection rates for metaphorical skills and less-efficient implementations. We also find that metaphorical skills induce a hidden-state shift towards lower-efficiency procedural behavior prototypes. These results suggest that metaphorical algorithmic steering operates through the transfer of procedural patterns associated with metaphorical source scenarios rather than surface level metaphorical language alone.