Is normalization indispensable for training deep neural networks

NeurIPS 2020, Oral.

This is the official implementation for our paper

Is normalization indispensable for training deep neural networks?

I am on vacation now. Now only codes for ImageNet classification experiments are available. See training.md for details. Codes for more experiments are coming soon.

Edit (Nov 28 2020): We think that the theories and experiments in this paper worth further discussion, and we are working on an extended version for some journal. The complete code will be released when the journal version is ready.