{"acknowledgement":"The authors would like to thank Jan Maas, Mahdi Soltanolkotabi, and Daniel Soudry for the helpful discussions, Marius Kloft, Matthias Hein and Quoc Dinh Tran for proofreading portions of a prior version of this paper, and James Martens for a clarification concerning LeCun’s initialization. M. Mondelli was partially supported by the 2019 Lopez-Loreta Prize. Q. Nguyen was partially supported by the German Research Foundation (DFG) award KL 2698/2-1.","project":[{"_id":"059876FA-7A3F-11EA-A408-12923DDC885E","name":"Prix Lopez-Loretta 2019 - Marco Mondelli"}],"intvolume":"        33","volume":33,"year":"2020","citation":{"ieee":"Q. Nguyen and M. Mondelli, “Global convergence of deep networks with one wide layer followed by pyramidal topology,” in <i>34th Conference on Neural Information Processing Systems</i>, Vancouver, Canada, 2020, vol. 33, pp. 11961–11972.","mla":"Nguyen, Quynh, and Marco Mondelli. “Global Convergence of Deep Networks with One Wide Layer Followed by Pyramidal Topology.” <i>34th Conference on Neural Information Processing Systems</i>, vol. 33, Curran Associates, 2020, pp. 11961–11972.","short":"Q. Nguyen, M. Mondelli, in:, 34th Conference on Neural Information Processing Systems, Curran Associates, 2020, pp. 11961–11972.","ama":"Nguyen Q, Mondelli M. Global convergence of deep networks with one wide layer followed by pyramidal topology. In: <i>34th Conference on Neural Information Processing Systems</i>. Vol 33. Curran Associates; 2020:11961–11972.","ista":"Nguyen Q, Mondelli M. 2020. Global convergence of deep networks with one wide layer followed by pyramidal topology. 34th Conference on Neural Information Processing Systems. NeurIPS: Neural Information Processing Systems vol. 33, 11961–11972.","chicago":"Nguyen, Quynh, and Marco Mondelli. “Global Convergence of Deep Networks with One Wide Layer Followed by Pyramidal Topology.” In <i>34th Conference on Neural Information Processing Systems</i>, 33:11961–11972. Curran Associates, 2020.","apa":"Nguyen, Q., &#38; Mondelli, M. (2020). Global convergence of deep networks with one wide layer followed by pyramidal topology. In <i>34th Conference on Neural Information Processing Systems</i> (Vol. 33, pp. 11961–11972). Vancouver, Canada: Curran Associates."},"conference":{"location":"Vancouver, Canada","end_date":"2020-12-12","start_date":"2020-12-06","name":"NeurIPS: Neural Information Processing Systems"},"language":[{"iso":"eng"}],"_id":"9221","page":"11961–11972","author":[{"full_name":"Nguyen, Quynh","last_name":"Nguyen","first_name":"Quynh"},{"full_name":"Mondelli, Marco","id":"27EB676C-8706-11E9-9510-7717E6697425","last_name":"Mondelli","first_name":"Marco","orcid":"0000-0002-3242-7020"}],"publication":"34th Conference on Neural Information Processing Systems","external_id":{"arxiv":["2002.07867"]},"quality_controlled":"1","title":"Global convergence of deep networks with one wide layer followed by pyramidal topology","status":"public","oa":1,"oa_version":"Preprint","user_id":"8b945eb4-e2f2-11eb-945a-df72226e66a9","date_created":"2021-03-03T12:06:02Z","publisher":"Curran Associates","publication_status":"published","date_published":"2020-07-07T00:00:00Z","department":[{"_id":"MaMo"}],"abstract":[{"lang":"eng","text":"Recent works have shown that gradient descent can find a global minimum for over-parameterized neural networks where the widths of all the hidden layers scale polynomially with N (N being the number of training samples). In this paper, we prove that, for deep networks, a single layer of width N following the input layer suffices to ensure a similar guarantee. In particular, all the remaining layers are allowed to have constant widths, and form a pyramidal topology. We show an application of our result to the widely used LeCun’s initialization and obtain an over-parameterization requirement for the single wide layer of order N2.\r\n"}],"main_file_link":[{"open_access":"1","url":"https://arxiv.org/abs/2002.07867"}],"type":"conference","day":"07","article_processing_charge":"No","month":"07","date_updated":"2024-09-10T13:03:17Z"}