Syed Mohammed Arshad Zaidi, Eric Rincon, Shayan Hassantabarcs.LG cs.IR
Vacation rental marketplaces face a structural imbalance on the supply side: a small fraction of properties receive most user interactions, while the long tail of new, niche, and seasonal listings generates too little behavioral signal for collaborative filtering to serve effectively. At Vrbo, item-based k-nearest neighbors (IBKNN) is a core candidate generation channel, but leaves tens of thousands of properties with no candidates and produces weak neighborhoods for sparsely interacted ones. We present a training-free, LLM-based candidate generation pipeline that complements IBKNN using static property metadata alone. An off-the-shelf LLM synthesizes diverse semantic queries per property, a pre-trained text encoder embeds them, and an approximate nearest-neighbor index retrieves candidates from an 11.7M-property catalog. A Union fusion strategy merges these with IBKNN while preserving the behavioral channel's ordering, guaranteeing no degradation on well-served properties, and a downstream learning-to-rank model re-scores the fused pool. Evaluated on 1.6M focal properties, the system extends candidate coverage to tens of thousands of properties IBKNN cannot reach, delivers its largest gains on the long-tail segment where behavioral methods are weakest, and matches or beats IBKNN at every K on shared properties. A downstream learning-to-rank stage further lifts the fused pool, yielding a complete candidate generation and re-ranking stack that serves the long tail without regressing well-served properties. We additionally show that Union fusion collapses the recall gap between a 3B open-weights LLM and frontier API-based models from 27-46% to under 1%, supporting self-hosted small-model deployment at marketplace catalog scale.
Traditional ads recommendation systems have primarily focused on optimizing for prediction accuracy of click or conversion events using canonical metrics such as recall or normalized discounted cumulative gain (NDCG). With the hyper-growth of ads inventory and liquidity with generative AI technologies, the prediction stability and predictability is becoming increasingly critical. Intuitively, prediction stability and predictability can be defined to quantify system robustness with respect to minor/noisy input (ads, creatives) perturbations, the lack of which could lead to advertiser perceivable problems such as repeatability, cold start and under-exploration. In this paper, we introduce a new evaluation framework for quantifying stability and predictability of an ads recommender system, and present an online validated semantic candidate generation framework powered by fine-tuned Large Language Models (LLMs) that showed significant improvement along these metrics by fundamentally improving the semantic-awareness of the system. The approach extracts hierarchical semantic attributes from ad creatives to obtain LLM representations, which serve as the foundation for graph-based expansion, ensuring the retrieved candidates encapsulate semantic variants of an ad, guaranteeing that small creative variants from the advertiser yield consistent and explainable delivery results to the user. We tested this LLM ads retrieval framework in a large-scale industrial ads recommendation system, demonstrating significant improvements across offline and online A/B experiments, showcasing gains in both predictability and traditional performance metrics. Although evaluated in the ads stack, this is a general framework that can be applied broadly to any large-scale recommendation and retrieval systems facing similar scaling and predictability challenges.