Latent Information Sharing for Accelerating Federated Learning
Federated learning suffers from client drift when local data is heterogeneous, slowing convergence and degrading the global model. This paper proposes Activation Sharing Federated Learning (ASFL), a method in which clients exchange hidden-layer activations and labels with a randomly paired peer each round, then perform a partial tail round updating only the deeper layers using the pooled activations before standard local training. The premise is that aligning latent representations directly across clients mitigates heterogeneity more effectively than parameter-space regularization, while partial backpropagation cuts per-round computation.
ASFL is an intuitively appealing and empirically strong method, but several claims are overstated or misaligned with practice. On four non-IID benchmarks, ASFL consistently outperforms FedAvg, FedProx, SCAFFOLD, and SplitFed in accuracy under a fixed round budget, and it materially reduces local wall-clock time. However, the abstract's claim of preserved privacy is contradicted by the paper's own admission of leakage comparable to SplitFed, and the theory's favorable linear-speedup regime requires $\tau'=O(\tau^{1/3})$, whereas experiments set $\tau'=\tau$.
The representation-alignment motivation is substantiated empirically: a t-SNE visualization shows that ASFL yields tighter, more coherent per-client class centroids than FedAvg or SCAFFOLD, indicating genuine cross-client alignment. The cost-variance heuristic for choosing the split layer is practical and well justified, and the wall-clock measurements credibly demonstrate that partial tail-round training lowers per-round compute.
The protocol used in experiments conflates algorithmic novelty with a compute-schedule advantage. By splitting the fixed local update budget equally between partial and full updates ($\tau'=\tau$), ASFL performs fewer full-model gradient steps per round than every baseline, which consume the entire budget on full-model updates. Because partial backpropagation is cheaper, the accuracy comparisons under a fixed round budget are not controlled for equal full-model compute.
A second issue is communication topology: random client pairing for activation exchange requires direct peer-to-peer inter-client communication, a radical departure from canonical star-topology FL that is only briefly discussed in an appendix. Finally, the privacy narrative is inconsistent: the abstract claims data privacy is preserved, yet the introduction explicitly admits privacy leakage comparable to SplitFed, and the proposed remedy is an add-on PixelDP layer rather than an inherent safety property of activation sharing.
Table 1 provides credible evidence that ASFL improves accuracy within the authors' protocol, and the communication-cost comparison shows that reusing a small activation pool is dramatically cheaper than SplitFed's per-step upload. However, the comparison to SplitFed is apples-to-oranges because SplitFed executes tail layers on a central server, whereas ASFL relies on peer-to-peer communication and local tail computation. The gains over variance-reduction baselines are clear, but the paper never isolates whether the accuracy improvement stems from representation alignment or merely from the fact that ASFL devotes half of its update budget to cheap partial steps, effectively giving the full model more frequent access to averaged parameters under the same wall-clock budget.
Reproduction is hampered by the absence of released code or data splits, despite fairly detailed hyperparameter tables in Appendix A.1. The MPI-based gossip communication used for random client pairing is non-standard and would require substantial systems engineering outside typical FL frameworks. Additionally, the layer-selection heuristic depends on an empirical gradient-variance profiling step that is described as using only one local update step per active client, leaving ambiguity about sampling and reproducibility of the chosen split points.
Federated learning (FL) is a communication-efficient distributed learning paradigm. However, client drift remains one of the most critical challenges, hindering the efficient training of a global model. In this study, we propose a novel latent information sharing scheme that directly mitigates data heterogeneity across clients. Our theoretical and empirical results show that sharing a small amount of hidden-layer activations significantly improves training efficiency while preserving convergence guarantees and data privacy. Furthermore, we compare our method with existing FL approaches designed to address client drift, including FedProx, SCAFFOLD, FedPVR, FedProto, and SplitFed, and demonstrate superior model accuracy under a fixed round budget without incurring excessive communication overhead. Overall, this work presents a promising new knowledge aggregation scheme and provides a comprehensive analysis of the impact of activation sharing on federated optimization.
Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.
No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.