https://github.com/bytedance/feddecorr
[ICLR2023] Official Implementation of Towards Understanding and Mitigating Dimensional Collapse in Heterogeneous Federated Learning (https://arxiv.org/abs/2210.00226)
Science Score: 36.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
○CITATION.cff file
-
✓codemeta.json file
Found codemeta.json file -
✓.zenodo.json file
Found .zenodo.json file -
○DOI references
-
✓Academic publication links
Links to: arxiv.org -
○Academic email domains
-
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (10.8%) to scientific vocabulary
Keywords
Repository
[ICLR2023] Official Implementation of Towards Understanding and Mitigating Dimensional Collapse in Heterogeneous Federated Learning (https://arxiv.org/abs/2210.00226)
Basic Info
Statistics
- Stars: 69
- Watchers: 4
- Forks: 7
- Open Issues: 1
- Releases: 0
Topics
Metadata Files
README.md
Introduction
This Repo contains the official implementation of the following paper:
|Venue|Method|Paper Title| |----|-----|-----| |ICLR'23|FedDecorr|Towards Understanding and Mitigating Dimensional Collapse in Heterogeneous Federated Learning|
and unofficial implementation of the following papers:
|Venue|Method|Paper Title| |----|-----|-----| |AISTATS'17|FedAvg|Communication-Efficient Learning of Deep Networks from Decentralized Data| |ArXiv'19|FedAvgM|Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification| |MLSys'20|FedProx|Federated Optimization in Heterogeneous Networks| |NeurIPS'20|FedNova|Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization| |CVPR'21|MOON|Model-Contrastive Federated Learning| |ICLR'21|FedAdagrad/Yogi/Adam|Adaptive Federated Optimization| |KDD'21|FedRS|FedRS: Federated Learning with Restricted Softmax for Label Distribution Non-IID Data| |ICML'22|FedLogitCal|Federated Learning with Label Distribution Skew via Logits Calibration| |ICML'22/ECCV'22|FedSAM|Generalized Federated Learning via Sharpness Aware Minimization/Improving Generalization in Federated Learning by Seeking Flat Minima| |ICLR'23|FedExp|FedExP: Speeding up Federated Averaging via Extrapolation|
Dataset preprocessing
TinyImageNet:
1) Download the dataset to "data" directory from this link: http://cs231n.stanford.edu/tiny-imagenet-200.zip
2) Unzip the downloaded file under "data" directory.
3) Lastly, to reformat the validation set, under the folder "data/tiny-imagenet-200", run:
python3 preprocess_tiny_imagenet.py
Running Instructions
Shell scripts to reproduce experimental results in our paper are under "run_scripts" folder. Simply changing the "ALPHA" variable to run under different degree of heterogeneity.
Here are commands that replicate our results:
FedAvg on CIFAR10:
bash run_scripts/cifar10_fedavg.sh
FedAvg + FedDecorr on CIFAR10:
bash run_scripts/cifar10_fedavg_feddecorr.sh
Experiments on other methods (FedAvgM, FedProx, MOON) and other datasets (CIFAR100, TinyImageNet) follow the similar manner.
Citation
If you find our repo/paper helpful, please consider citing our work :)
@article{shi2022towards,
title={Towards Understanding and Mitigating Dimensional Collapse in Heterogeneous Federated Learning},
author={Shi, Yujun and Liang, Jian and Zhang, Wenqing and Tan, Vincent YF and Bai, Song},
journal={arXiv preprint arXiv:2210.00226},
year={2022}
}
Contact
Yujun Shi (shi.yujun@u.nus.edu)
Acknowledgement
Some of our code is borrowed following projects: MOON, NIID-Bench, SAM(Pytorch)
Owner
- Name: Bytedance Inc.
- Login: bytedance
- Kind: organization
- Location: Singapore
- Website: https://opensource.bytedance.com
- Twitter: ByteDanceOSS
- Repositories: 255
- Profile: https://github.com/bytedance
GitHub Events
Total
- Issues event: 1
- Watch event: 7
- Fork event: 1
Last Year
- Issues event: 1
- Watch event: 7
- Fork event: 1
Issues and Pull Requests
Last synced: 12 months ago
All Time
- Total issues: 1
- Total pull requests: 0
- Average time to close issues: N/A
- Average time to close pull requests: N/A
- Total issue authors: 1
- Total pull request authors: 0
- Average comments per issue: 0.0
- Average comments per pull request: 0
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0
Past Year
- Issues: 1
- Pull requests: 0
- Average time to close issues: N/A
- Average time to close pull requests: N/A
- Issue authors: 1
- Pull request authors: 0
- Average comments per issue: 0.0
- Average comments per pull request: 0
- Merged pull requests: 0
- Bot issues: 0
- Bot pull requests: 0
Top Authors
Issue Authors
- sa96376843 (1)