SelfDistillCore: a federated learning framework with sparse updates and historical knowledge integration

Abstract

Federated learning (FL) offers a robust approach for privacy-preserving model training across distributed clients, but its performance often declines significantly under non-independent and identically distributed data, a critical challenge in real-world scenarios. Existing methods frequently face issues such as slow convergence, reliance on heuristic regularization, or substantial communication overhead. To address these challenges, we propose SelfDistillCore, a federated learning framework that combines a sparsification technique with a momentum-based aggregation rule using an exponentially weighted moving average. By transmitting only the top K% of parameter updates and using momentum to balance historical and new updates, SelfDistillCore mitigates client drift and reduces communication costs. Experimental results show that SelfDistillCore significantly enhances convergence stability, improves resilience to data heterogeneity, increases scalability, and reduces communication costs by 19% to 70% compared to standard baselines. The code of SelfDistillCore can be found at https://github.com/dongld-2020/selfdistillcore_pj.