---
_id: '9571'
abstract:
- lang: eng
  text: As the size and complexity of models and datasets grow, so does the need for
    communication-efficient variants of stochastic gradient descent that can be deployed
    to perform parallel model training. One popular communication-compression method
    for data-parallel SGD is QSGD (Alistarh et al., 2017), which quantizes and encodes
    gradients to reduce communication costs. The baseline variant of QSGD provides
    strong theoretical guarantees, however, for practical purposes, the authors proposed
    a heuristic variant which we call QSGDinf, which demonstrated impressive empirical
    gains for distributed training of large neural networks. In this paper, we build
    on this work to propose a new gradient quantization scheme, and show that it has
    both stronger theoretical guarantees than QSGD, and matches and exceeds the empirical
    performance of the QSGDinf heuristic and of other compression methods.
article_processing_charge: No
article_type: original
arxiv: 1
author:
- first_name: Ali
  full_name: Ramezani-Kebrya, Ali
  last_name: Ramezani-Kebrya
- first_name: Fartash
  full_name: Faghri, Fartash
  last_name: Faghri
- first_name: Ilya
  full_name: Markov, Ilya
  last_name: Markov
- first_name: Vitalii
  full_name: Aksenov, Vitalii
  id: 2980135A-F248-11E8-B48F-1D18A9856A87
  last_name: Aksenov
- first_name: Dan-Adrian
  full_name: Alistarh, Dan-Adrian
  id: 4A899BFC-F248-11E8-B48F-1D18A9856A87
  last_name: Alistarh
  orcid: 0000-0003-3650-940X
- first_name: Daniel M.
  full_name: Roy, Daniel M.
  last_name: Roy
citation:
  ama: 'Ramezani-Kebrya A, Faghri F, Markov I, Aksenov V, Alistarh D-A, Roy DM. NUQSGD:
    Provably communication-efficient data-parallel SGD via nonuniform quantization.
    <i>Journal of Machine Learning Research</i>. 2021;22(114):1−43.'
  apa: 'Ramezani-Kebrya, A., Faghri, F., Markov, I., Aksenov, V., Alistarh, D.-A.,
    &#38; Roy, D. M. (2021). NUQSGD: Provably communication-efficient data-parallel
    SGD via nonuniform quantization. <i>Journal of Machine Learning Research</i>.
    Journal of Machine Learning Research.'
  chicago: 'Ramezani-Kebrya, Ali, Fartash Faghri, Ilya Markov, Vitalii Aksenov, Dan-Adrian
    Alistarh, and Daniel M. Roy. “NUQSGD: Provably Communication-Efficient Data-Parallel
    SGD via Nonuniform Quantization.” <i>Journal of Machine Learning Research</i>.
    Journal of Machine Learning Research, 2021.'
  ieee: 'A. Ramezani-Kebrya, F. Faghri, I. Markov, V. Aksenov, D.-A. Alistarh, and
    D. M. Roy, “NUQSGD: Provably communication-efficient data-parallel SGD via nonuniform
    quantization,” <i>Journal of Machine Learning Research</i>, vol. 22, no. 114.
    Journal of Machine Learning Research, p. 1−43, 2021.'
  ista: 'Ramezani-Kebrya A, Faghri F, Markov I, Aksenov V, Alistarh D-A, Roy DM. 2021.
    NUQSGD: Provably communication-efficient data-parallel SGD via nonuniform quantization.
    Journal of Machine Learning Research. 22(114), 1−43.'
  mla: 'Ramezani-Kebrya, Ali, et al. “NUQSGD: Provably Communication-Efficient Data-Parallel
    SGD via Nonuniform Quantization.” <i>Journal of Machine Learning Research</i>,
    vol. 22, no. 114, Journal of Machine Learning Research, 2021, p. 1−43.'
  short: A. Ramezani-Kebrya, F. Faghri, I. Markov, V. Aksenov, D.-A. Alistarh, D.M.
    Roy, Journal of Machine Learning Research 22 (2021) 1−43.
date_created: 2021-06-20T22:01:33Z
date_published: 2021-04-01T00:00:00Z
date_updated: 2024-03-06T12:22:07Z
day: '01'
ddc:
- '000'
department:
- _id: DaAl
external_id:
  arxiv:
  - '1908.06077'
file:
- access_level: open_access
  checksum: 6428aa8bcb67768b6949c99b55d5281d
  content_type: application/pdf
  creator: asandaue
  date_created: 2021-06-23T07:09:41Z
  date_updated: 2021-06-23T07:09:41Z
  file_id: '9595'
  file_name: 2021_JournalOfMachineLearningResearch_Ramezani-Kebrya.pdf
  file_size: 11237154
  relation: main_file
  success: 1
file_date_updated: 2021-06-23T07:09:41Z
has_accepted_license: '1'
intvolume: '        22'
issue: '114'
language:
- iso: eng
license: https://creativecommons.org/licenses/by/4.0/
main_file_link:
- open_access: '1'
  url: https://www.jmlr.org/papers/v22/20-255.html
month: '04'
oa: 1
oa_version: Published Version
page: 1−43
publication: Journal of Machine Learning Research
publication_identifier:
  eissn:
  - '15337928'
  issn:
  - '15324435'
publication_status: published
publisher: Journal of Machine Learning Research
quality_controlled: '1'
scopus_import: '1'
status: public
title: 'NUQSGD: Provably communication-efficient data-parallel SGD via nonuniform
  quantization'
tmp:
  image: /images/cc_by.png
  legal_code_url: https://creativecommons.org/licenses/by/4.0/legalcode
  name: Creative Commons Attribution 4.0 International Public License (CC-BY 4.0)
  short: CC BY (4.0)
type: journal_article
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
volume: 22
year: '2021'
...