Sentiment classifiers are increasingly applied to social media content that is either sarcastic or AI-generated --- two distributional regimes where standard evaluations offer little guidance. We present a three-part empirical study of sentiment classifier behaviour under these conditions. First, we find that confidence scores on sarcastic text are significantly lower than on non-sarcastic text (Mann--Whitney $p = 2 \times 10^{-6}$), confirming that classifiers sense their own uncertainty on ironic content even without explicit uncertainty modelling. Second, and counterintuitively, we show that sentiment classifiers achieve higher accuracy on AI-paraphrased reviews than on the original human-authored text (RoBERTa: $+5.8$ pp for Qwen3.5-4B paraphrases, $+3.7$ pp for Gemma4-E4B), revealing a cross-domain stylistic alignment effect: AI paraphrases remove distributional noise that confounds Twitter-trained classifiers, producing cleaner, more prototypical sentiment text. Third, we demonstrate that a lightweight abstention wrapper --- flagging the $14\%$ of inputs with confidence below $0.6$ --- improves accuracy from 82.2\% to 88.9\% ($+6.7$ pp) on the retained set. We further compare Semantic Entropy and MC-Dropout-style disagreement as uncertainty signals and find near-identical AUROC ($0.650$ vs.\ $0.646$) on sarcastic text, suggesting that for short social media inputs, both methods are interchangeable. Our results motivate a shift from confident single-label prediction to uncertainty-aware abstention in high-stakes sentiment applications such as mental health flagging and content moderation.
Cross-dataset generalisation is a fundamental requirement for deploying text classifiers in real-world settings, yet systematic evaluation across corpora from different sources remains uncommon in fake news detection and virtually absent in sarcasm detection research. This paper presents a unified empirical study of zero-shot cross-dataset transfer in three domains: Urdu fake news detection (FND), English FND, and sarcasm detection. For each domain, we fine-tune xlm-roberta-base on one corpus and evaluate it on a second corpus from a different source, comparing against TF-IDF baselines with Logistic Regression (LR) and Support Vector Machines (SVM). In Urdu FND (Ax-to-Grind vs. Notri-Fact), we identify a severe length confound in the Ax-to-Grind dataset, fake articles average 3.4 times more words than real articles, causing catastrophic A to B transfer collapse (macro F1 = 0.005) while B to A achieves F1 = 0.771. Extension to English FND (WELFake vs. ISOT) and sarcasm detection (TweetEval Irony vs. Sarcasm Corpus V2) reveals that such failure modes extend beyond Urdu, confirming that shortcut learning from distributional artefacts is a cross-lingual, cross-domain challenge in binary text classification. We provide a reusable diagnostic methodology, combining class-conditional length analysis, bidirectional transfer asymmetry, and predicted label collapse inspection, applicable across any binary text classification setting.
Nigerian Pidgin is one of Africa's most widely spoken languages, yet remains severely underrepresented in language model evaluation. Existing benchmarks primarily focus on translation, transcription, or generic sentiment analysis, leaving critical aspects of culturally grounded language understanding unmeasured. We introduce Wazobia Eval, a benchmark for evaluating Nigerian Pidgin emotion understanding, sarcasm detection, and cultural reasoning. The benchmark is built on a manually annotated dataset containing over 550 examples and a 16-category emotion taxonomy designed to capture culturally specific emotional registers that are not represented in conventional sentiment frameworks. Wazobia Eval provides standardized evaluation protocols and benchmark tasks for assessing model performance on nuanced Nigerian language understanding. We present the benchmark design, annotation methodology, taxonomy development process, and preliminary pilot evaluation results. Our goal is to provide foundational evaluation infrastructure for Nigerian language AI and establish a reproducible benchmark for future research. The dataset is publicly available at https://huggingface.co/WAZOBIALABS.