---
_id: '1000'
abstract:
- lang: eng
  text: 'We study probabilistic models of natural images and extend the autoregressive
    family of PixelCNN models by incorporating latent variables. Subsequently, we
    describe two new generative image models that exploit different image transformations
    as latent variables: a quantized grayscale view of the image or a multi-resolution
    image pyramid. The proposed models tackle two known shortcomings of existing PixelCNN
    models: 1) their tendency to focus on low-level image details, while largely ignoring
    high-level image information, such as object shapes, and 2) their computationally
    costly procedure for image sampling. We experimentally demonstrate benefits of
    our LatentPixelCNN models, in particular showing that they produce much more realistically
    looking image samples than previous state-of-the-art probabilistic models. '
acknowledgement: We thank Tim Salimans for spotting a mistake in our preliminary arXiv
  manuscript. This work was funded by the European Research Council under the European
  Unions Seventh Framework Programme (FP7/2007-2013)/ERC grant agreement no 308036.
article_processing_charge: No
arxiv: 1
author:
- first_name: Alexander
  full_name: Kolesnikov, Alexander
  id: 2D157DB6-F248-11E8-B48F-1D18A9856A87
  last_name: Kolesnikov
- first_name: Christoph
  full_name: Lampert, Christoph
  id: 40C20FD2-F248-11E8-B48F-1D18A9856A87
  last_name: Lampert
  orcid: 0000-0001-8622-7887
citation:
  ama: 'Kolesnikov A, Lampert C. PixelCNN models with auxiliary variables for natural
    image modeling. In: <i>34th International Conference on Machine Learning</i>.
    Vol 70. JMLR; 2017:1905-1914.'
  apa: 'Kolesnikov, A., &#38; Lampert, C. (2017). PixelCNN models with auxiliary variables
    for natural image modeling. In <i>34th International Conference on Machine Learning</i>
    (Vol. 70, pp. 1905–1914). Sydney, Australia: JMLR.'
  chicago: Kolesnikov, Alexander, and Christoph Lampert. “PixelCNN Models with Auxiliary
    Variables for Natural Image Modeling.” In <i>34th International Conference on
    Machine Learning</i>, 70:1905–14. JMLR, 2017.
  ieee: A. Kolesnikov and C. Lampert, “PixelCNN models with auxiliary variables for
    natural image modeling,” in <i>34th International Conference on Machine Learning</i>,
    Sydney, Australia, 2017, vol. 70, pp. 1905–1914.
  ista: 'Kolesnikov A, Lampert C. 2017. PixelCNN models with auxiliary variables for
    natural image modeling. 34th International Conference on Machine Learning. ICML:
    International Conference on Machine Learning vol. 70, 1905–1914.'
  mla: Kolesnikov, Alexander, and Christoph Lampert. “PixelCNN Models with Auxiliary
    Variables for Natural Image Modeling.” <i>34th International Conference on Machine
    Learning</i>, vol. 70, JMLR, 2017, pp. 1905–14.
  short: A. Kolesnikov, C. Lampert, in:, 34th International Conference on Machine
    Learning, JMLR, 2017, pp. 1905–1914.
conference:
  end_date: 2017-08-11
  location: Sydney, Australia
  name: 'ICML: International Conference on Machine Learning'
  start_date: 2017-08-06
date_created: 2018-12-11T11:49:37Z
date_published: 2017-08-01T00:00:00Z
date_updated: 2023-09-22T09:50:41Z
day: '01'
department:
- _id: ChLa
ec_funded: 1
external_id:
  arxiv:
  - '1612.08185'
  isi:
  - '000683309501102'
has_accepted_license: '1'
intvolume: '        70'
isi: 1
language:
- iso: eng
main_file_link:
- open_access: '1'
  url: https://arxiv.org/abs/1612.08185
month: '08'
oa: 1
oa_version: Submitted Version
page: 1905 - 1914
project:
- _id: 2532554C-B435-11E9-9278-68D0E5697425
  call_identifier: FP7
  grant_number: '308036'
  name: Lifelong Learning of Visual Scene Understanding
publication: 34th International Conference on Machine Learning
publication_identifier:
  isbn:
  - 978-151085514-4
publication_status: published
publisher: JMLR
publist_id: '6398'
quality_controlled: '1'
scopus_import: '1'
status: public
title: PixelCNN models with auxiliary variables for natural image modeling
type: conference
user_id: c635000d-4b10-11ee-a964-aac5a93f6ac1
volume: 70
year: '2017'
...
---
_id: '432'
abstract:
- lang: eng
  text: 'Recently there has been significant interest in training machine-learning
    models at low precision: by reducing precision, one can reduce computation and
    communication by one order of magnitude. We examine training at reduced precision,
    both from a theoretical and practical perspective, and ask: is it possible to
    train models at end-to-end low precision with provable guarantees? Can this lead
    to consistent order-of-magnitude speedups? We mainly focus on linear models, and
    the answer is yes for linear models. We develop a simple framework called ZipML
    based on one simple but novel strategy called double sampling. Our ZipML framework
    is able to execute training at low precision with no bias, guaranteeing convergence,
    whereas naive quanti- zation would introduce significant bias. We val- idate our
    framework across a range of applica- tions, and show that it enables an FPGA proto-
    type that is up to 6.5 × faster than an implemen- tation using full 32-bit precision.
    We further de- velop a variance-optimal stochastic quantization strategy and show
    that it can make a significant difference in a variety of settings. When applied
    to linear models together with double sampling, we save up to another 1.7 × in
    data movement compared with uniform quantization. When training deep networks
    with quantized models, we achieve higher accuracy than the state-of-the- art XNOR-Net. '
alternative_title:
- PMLR Press
article_processing_charge: No
author:
- first_name: Hantian
  full_name: Zhang, Hantian
  last_name: Zhang
- first_name: Jerry
  full_name: Li, Jerry
  last_name: Li
- first_name: Kaan
  full_name: Kara, Kaan
  last_name: Kara
- first_name: Dan-Adrian
  full_name: Alistarh, Dan-Adrian
  id: 4A899BFC-F248-11E8-B48F-1D18A9856A87
  last_name: Alistarh
  orcid: 0000-0003-3650-940X
- first_name: Ji
  full_name: Liu, Ji
  last_name: Liu
- first_name: Ce
  full_name: Zhang, Ce
  last_name: Zhang
citation:
  ama: 'Zhang H, Li J, Kara K, Alistarh D-A, Liu J, Zhang C. ZipML: Training linear
    models with end-to-end low precision, and a little bit of deep learning. In: <i>Proceedings
    of Machine Learning Research</i>. Vol 70. ML Research Press; 2017:4035-4043.'
  apa: 'Zhang, H., Li, J., Kara, K., Alistarh, D.-A., Liu, J., &#38; Zhang, C. (2017).
    ZipML: Training linear models with end-to-end low precision, and a little bit
    of deep learning. In <i>Proceedings of Machine Learning Research</i> (Vol. 70,
    pp. 4035–4043). Sydney, Australia: ML Research Press.'
  chicago: 'Zhang, Hantian, Jerry Li, Kaan Kara, Dan-Adrian Alistarh, Ji Liu, and
    Ce Zhang. “ZipML: Training Linear Models with End-to-End Low Precision, and a
    Little Bit of Deep Learning.” In <i>Proceedings of Machine Learning Research</i>,
    70:4035–43. ML Research Press, 2017.'
  ieee: 'H. Zhang, J. Li, K. Kara, D.-A. Alistarh, J. Liu, and C. Zhang, “ZipML: Training
    linear models with end-to-end low precision, and a little bit of deep learning,”
    in <i>Proceedings of Machine Learning Research</i>, Sydney, Australia, 2017, vol.
    70, pp. 4035–4043.'
  ista: 'Zhang H, Li J, Kara K, Alistarh D-A, Liu J, Zhang C. 2017. ZipML: Training
    linear models with end-to-end low precision, and a little bit of deep learning.
    Proceedings of Machine Learning Research. ICML: International  Conference  on 
    Machine Learning, PMLR Press, vol. 70, 4035–4043.'
  mla: 'Zhang, Hantian, et al. “ZipML: Training Linear Models with End-to-End Low
    Precision, and a Little Bit of Deep Learning.” <i>Proceedings of Machine Learning
    Research</i>, vol. 70, ML Research Press, 2017, pp. 4035–43.'
  short: H. Zhang, J. Li, K. Kara, D.-A. Alistarh, J. Liu, C. Zhang, in:, Proceedings
    of Machine Learning Research, ML Research Press, 2017, pp. 4035–4043.
conference:
  end_date: 2017-08-11
  location: Sydney, Australia
  name: 'ICML: International  Conference  on  Machine Learning'
  start_date: 2017-08-06
date_created: 2018-12-11T11:46:26Z
date_published: 2017-01-01T00:00:00Z
date_updated: 2023-10-17T12:31:15Z
day: '01'
ddc:
- '000'
department:
- _id: DaAl
file:
- access_level: open_access
  checksum: 86156ba7f4318e47cef3eb9092593c10
  content_type: application/pdf
  creator: dernst
  date_created: 2019-01-22T08:23:58Z
  date_updated: 2020-07-14T12:46:26Z
  file_id: '5869'
  file_name: 2017_ICML_Zhang.pdf
  file_size: 849345
  relation: main_file
file_date_updated: 2020-07-14T12:46:26Z
has_accepted_license: '1'
language:
- iso: eng
month: '01'
oa: 1
oa_version: Submitted Version
page: 4035 - 4043
publication: Proceedings of Machine Learning Research
publication_identifier:
  isbn:
  - 978-151085514-4
publication_status: published
publisher: ML Research Press
publist_id: '7391'
quality_controlled: '1'
scopus_import: '1'
status: public
title: 'ZipML: Training linear models with end-to-end low precision, and a little
  bit of deep learning'
type: conference
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
volume: ' 70'
year: '2017'
...