{"date_updated":"2024-01-17T09:14:13Z","tmp":{"image":"/images/cc_by.png","name":"Creative Commons Attribution 4.0 International Public License (CC-BY 4.0)","short":"CC BY (4.0)","legal_code_url":"https://creativecommons.org/licenses/by/4.0/legalcode"},"month":"10","article_processing_charge":"Yes (in subscription journal)","day":"01","type":"journal_article","file":[{"success":1,"file_id":"14816","file_size":1510993,"date_created":"2024-01-16T12:13:27Z","date_updated":"2024-01-16T12:13:27Z","content_type":"application/pdf","access_level":"open_access","creator":"dernst","file_name":"2023_JMLR_Beznosikov.pdf","checksum":"c50f2b9db53938b755e30a085f464059","relation":"main_file"}],"abstract":[{"text":"In the last few years, various communication compression techniques have emerged as an indispensable tool helping to alleviate the communication bottleneck in distributed learning. However, despite the fact biased compressors often show superior performance in practice when compared to the much more studied and understood unbiased compressors, very little is known about them. In this work we study three classes of biased compression operators, two of which are new, and their performance when applied to (stochastic) gradient descent and distributed (stochastic) gradient descent. We show for the first time that biased compressors can lead to linear convergence rates both in the single node and distributed settings. We prove that distributed compressed SGD method, employed with error feedback mechanism, enjoys the ergodic rate O(δLexp[−μKδL]+(C+δD)Kμ), where δ≥1 is a compression parameter which grows when more compression is applied, L and μ are the smoothness and strong convexity constants, C captures stochastic gradient noise (C=0 if full gradients are computed on each node) and D captures the variance of the gradients at the optimum (D=0 for over-parameterized models). Further, via a theoretical study of several synthetic and empirical distributions of communicated gradients, we shed light on why and by how much biased compressors outperform their unbiased variants. Finally, we propose several new biased compressors with promising theoretical guarantees and practical performance.","lang":"eng"}],"department":[{"_id":"DaAl"}],"publication_status":"published","date_published":"2023-10-01T00:00:00Z","publisher":"Journal of Machine Learning Research","user_id":"2DF688A6-F248-11E8-B48F-1D18A9856A87","date_created":"2024-01-16T12:13:36Z","oa_version":"Published Version","oa":1,"status":"public","title":"On biased compression for distributed learning","quality_controlled":"1","publication_identifier":{"eissn":["1533-7928"]},"external_id":{"arxiv":["2002.12410"],"isi":["001111578500001"]},"has_accepted_license":"1","author":[{"first_name":"Aleksandr","last_name":"Beznosikov","full_name":"Beznosikov, Aleksandr"},{"last_name":"Horvath","first_name":"Samuel","full_name":"Horvath, Samuel"},{"full_name":"Richtarik, Peter","last_name":"Richtarik","first_name":"Peter"},{"full_name":"Safaryan, Mher","id":"dd546b39-0804-11ed-9c55-ef075c39778d","first_name":"Mher","last_name":"Safaryan"}],"publication":"Journal of Machine Learning Research","page":"1-50","_id":"14815","language":[{"iso":"eng"}],"article_type":"original","citation":{"short":"A. Beznosikov, S. Horvath, P. Richtarik, M. Safaryan, Journal of Machine Learning Research 24 (2023) 1–50.","ama":"Beznosikov A, Horvath S, Richtarik P, Safaryan M. On biased compression for distributed learning. <i>Journal of Machine Learning Research</i>. 2023;24:1-50.","chicago":"Beznosikov, Aleksandr, Samuel Horvath, Peter Richtarik, and Mher Safaryan. “On Biased Compression for Distributed Learning.” <i>Journal of Machine Learning Research</i>. Journal of Machine Learning Research, 2023.","ista":"Beznosikov A, Horvath S, Richtarik P, Safaryan M. 2023. On biased compression for distributed learning. Journal of Machine Learning Research. 24, 1–50.","apa":"Beznosikov, A., Horvath, S., Richtarik, P., &#38; Safaryan, M. (2023). On biased compression for distributed learning. <i>Journal of Machine Learning Research</i>. Journal of Machine Learning Research.","ieee":"A. Beznosikov, S. Horvath, P. Richtarik, and M. Safaryan, “On biased compression for distributed learning,” <i>Journal of Machine Learning Research</i>, vol. 24. Journal of Machine Learning Research, pp. 1–50, 2023.","mla":"Beznosikov, Aleksandr, et al. “On Biased Compression for Distributed Learning.” <i>Journal of Machine Learning Research</i>, vol. 24, Journal of Machine Learning Research, 2023, pp. 1–50."},"year":"2023","volume":24,"intvolume":"        24","ddc":["000"],"acknowledgement":"The work in Sections 1-5 was conducted while A. Beznosikov was a research intern in the Optimizationand Machine Learning Lab of Peter Richtárik at KAUST; this visit was funded by the KAUST Baseline Research Funding Scheme. The work of A. Beznosikov in Section 6 was conducted in Skoltech and was supported by Ministry of Science and Higher Education grant No. 075-10-2021-068. ","file_date_updated":"2024-01-16T12:13:27Z","isi":1}