Diego Cerda-Mardini, Sarath Chandar, Sreenath Madathilcs.CL cs.AI
LLMs are increasingly deployed as post-hoc explainers of AI-generated outputs, yet it remains unclear whether they can reliably communicate probabilistic information in natural language. For this role to be viable, models must produce identical verbal descriptions for identical inputs, and select descriptions that accurately reflect the magnitude of the underlying numerical quantities. We evaluate whether nine LLMs meet these requirements within a two-stage prediction pipeline, in which an upstream model has produced probabilistic outputs characterized by their likelihood and uncertainty, and LLMs are tasked with selecting an appropriate verbal descriptor for each. We simulate predictions from an upstream model by taking samples from a Beta distribution parameterized by its mode and prior sample size. We then prompt LLMs to explain these predictions under six domain contexts and with ten temperature settings, and repeating each experiment ten times. We find that LLMs are generally consistent but miscalibrated, with substantially weaker performance on uncertainty than on likelihood tasks. Providing models with precomputed summary statistics (mode and prior sample size) reduced sensitivity to contextual framing but did not resolve the underlying miscalibration, suggesting that the bottleneck resides in the verbalization step itself. These findings indicate that current LLMs do not yet constitute reliable zero-shot standalone risk communication tools for probabilistic predictions.
Current AI disclaimers often fail to function as intended due to warning habituation and a transparency paradox. As AI-generated information becomes pervasive in everyday decision-making, effective risk communication is increasingly critical for responsible design. This exploratory study examines how disclaimer placement and persuasive cues shape trust, perceived accuracy, and disclaimer engagement across three high-stakes domains: finance, medicine, and AI-generated content. Using a mixed within-between experimental design with 378 stimulus-level responses from 52 participants, we find that advisory content was generally trusted across conditions, even when disclaimers were present. A significant domain effect showed that medical content received the highest trust ratings. In the AI domain, the findings reveal a transparency paradox: some participants interpreted disclaimers not as warnings, but as signs of system self-awareness and honesty, paradoxically increasing perceived trustworthiness. Evidence of banner blindness further suggests that standardized AI disclaimers are insufficient to prevent over-reliance. Finance and medicine provide useful comparison domains by showing how users interpret warnings differently depending on context and perceived risk. These findings have vital implications for responsible AI design, algorithmic fairness, and consumer protection when users act on potentially misleading information in high-stakes settings.