Initialization Is Critical: Advancing Federated Short-Term Load Forecasting under Load Heterogeneity via Model Initialization

Short-term load forecasting (STLF) provides essential information for numerous applications in modern power systems. However, accurate STLF often relies on fine-grained smart-meter data from distributed users, raising increasing concerns about data privacy. Federated learning (FL) has therefore emerged as a promising privacy-preserving paradigm for STLF. Nevertheless, this paper reveals structured heterogeneity in clients' load data. Specifically, clients exhibit different responses to exogenous factors and distinct temporal load profiles, which can degrade forecasting performance in FL. To mitigate these issues, this paper studies the role of model initialization in federated STLF, and proposes two initialization strategies from global and local perspectives. For global model initialization, when auxiliary public load data are available, a pretrained initialization strategy is developed to initialize the global model before federated training, thereby reducing client drift during the training process. For local model initialization, we propose SLIAvg, a sequential local initialization strategy that promotes a more consistent training process by allowing participating clients to start from progressively adapted models within each communication round. Since the proposed strategies only modify the initialization process, they are compatible with most existing FL frameworks and privacy-enhancing techniques. Experiments on real smart-meter data with two representative forecasting architectures demonstrate that the proposed strategies effectively improve forecasting performance, as evidenced by reduced client drift, improved convergence behavior, and lower forecasting errors.

SPRI: SVD-Partitioned Residual Initialization for Data-Constrained MoE Upcycling

Mixture-of-Experts (MoE) models enable efficient scaling, but training them from scratch remains prohibitively expensive. MoE upcycling mitigates this cost by converting pretrained dense models into sparse MoE models. However, existing upcycling methods typically rely on large-scale continued training and often perform poorly under data-constrained supervised adaptation, due to either homogeneous experts or overly disruptive perturbations to pretrained parameters. In this setting, effective upcycling must leverage pretrained weight structure while introducing sufficient diversity among routed experts. To this end, we propose SVD-Partitioned Residual Initialization (SPRI), which distributes SVD-partitioned residuals derived from pretrained feed-forward network (FFN) weights across routed experts, introducing controlled expert diversity grounded in pretrained spectral structure. We further introduce a two-stage training strategy to improve adaptation stability. We evaluate SPRI on multilingual speech-to-text translation, where limited supervised data challenges MoE upcycling and multiple target languages provide natural routing heterogeneity. On CoVoST2 across 15 En-to-XX directions, SPRI improves average BLEU and COMET over fully fine-tuned dense models by 2.58 and 3.32 points, respectively, and outperforms the prior best MoE upcycling baseline by 3.39 BLEU and 4.34 COMET points.