Robust Zero-Shot Generalization for Open-Vocabulary Action Recognition via Task Arithmetic
Francesca Morandi, Omayma Moussadek, Federico Venturini, Mauro Suardi, Alessandro Banzatti, Francesco Cannarile, Angelo Porrello, Simone Calderara
Abstract
Open Vocabulary Action Recognition (OVAR) enables the recognition of novel actions by leveraging vision-language representations, overcoming the limitations of traditional closed-set approaches. However, achieving robust performance in real-world scenarios typically requires domain-specific fine-tuning, which is often costly and raises privacy and regulatory concerns. In this work, we propose an alternative paradigm that bypasses target-domain training and recombines knowledge from existing datasets and models. Leveraging model merging and task arithmetic, we extract and combine task vectors from models fine-tuned on diverse public OVAR datasets. We show that, in out-of-distribution settings, the resulting merged model achieves superior zero-shot generalization to the pre-trained base model. Code is available at https://github.com/omaymaMoussadek/robust-ovar
Topics
Classified with taxonomy v2 on Sat, 5 Sept 2026.