---
_id: '7479'
abstract:
- lang: eng
  text: "Multi-exit architectures, in which a stack of processing layers is interleaved
    with early output layers, allow the processing of a test example to stop early
    and thus save computation time and/or energy.  In this work, we propose a new
    training procedure for multi-exit architectures based on the principle of knowledge
    distillation. The method encourage searly exits to mimic later, more accurate
    exits, by matching their output probabilities.\r\nExperiments  on  CIFAR100  and
    \ ImageNet  show  that distillation-based training significantly improves the
    accuracy of early exits while maintaining state-of-the-art accuracy  for  late
    \ ones.   The  method  is  particularly  beneficial when  training  data  is  limited
    \ and  it  allows  a  straightforward extension to semi-supervised learning,i.e.
    making use of unlabeled data at training time. Moreover, it takes only afew lines
    to implement and incurs almost no computational overhead at training time, and
    none at all at test time."
article_processing_charge: No
author:
- first_name: Phuong
  full_name: Bui Thi Mai, Phuong
  id: 3EC6EE64-F248-11E8-B48F-1D18A9856A87
  last_name: Bui Thi Mai
- first_name: Christoph
  full_name: Lampert, Christoph
  id: 40C20FD2-F248-11E8-B48F-1D18A9856A87
  last_name: Lampert
  orcid: 0000-0001-8622-7887
citation:
  ama: 'Phuong M, Lampert C. Distillation-based training for multi-exit architectures.
    In: <i>IEEE International Conference on Computer Vision</i>. Vol 2019-October.
    IEEE; 2019:1355-1364. doi:<a href="https://doi.org/10.1109/ICCV.2019.00144">10.1109/ICCV.2019.00144</a>'
  apa: 'Phuong, M., &#38; Lampert, C. (2019). Distillation-based training for multi-exit
    architectures. In <i>IEEE International Conference on Computer Vision</i> (Vol.
    2019–October, pp. 1355–1364). Seoul, Korea: IEEE. <a href="https://doi.org/10.1109/ICCV.2019.00144">https://doi.org/10.1109/ICCV.2019.00144</a>'
  chicago: Phuong, Mary, and Christoph Lampert. “Distillation-Based Training for Multi-Exit
    Architectures.” In <i>IEEE International Conference on Computer Vision</i>, 2019–October:1355–64.
    IEEE, 2019. <a href="https://doi.org/10.1109/ICCV.2019.00144">https://doi.org/10.1109/ICCV.2019.00144</a>.
  ieee: M. Phuong and C. Lampert, “Distillation-based training for multi-exit architectures,”
    in <i>IEEE International Conference on Computer Vision</i>, Seoul, Korea, 2019,
    vol. 2019–October, pp. 1355–1364.
  ista: 'Phuong M, Lampert C. 2019. Distillation-based training for multi-exit architectures.
    IEEE International Conference on Computer Vision. ICCV: International Conference
    on Computer Vision vol. 2019–October, 1355–1364.'
  mla: Phuong, Mary, and Christoph Lampert. “Distillation-Based Training for Multi-Exit
    Architectures.” <i>IEEE International Conference on Computer Vision</i>, vol.
    2019–October, IEEE, 2019, pp. 1355–64, doi:<a href="https://doi.org/10.1109/ICCV.2019.00144">10.1109/ICCV.2019.00144</a>.
  short: M. Phuong, C. Lampert, in:, IEEE International Conference on Computer Vision,
    IEEE, 2019, pp. 1355–1364.
conference:
  end_date: 2019-11-02
  location: Seoul, Korea
  name: 'ICCV: International Conference on Computer Vision'
  start_date: 2019-10-27
date_created: 2020-02-11T09:06:57Z
date_published: 2019-10-01T00:00:00Z
date_updated: 2023-09-08T11:11:12Z
day: '01'
ddc:
- '000'
department:
- _id: ChLa
doi: 10.1109/ICCV.2019.00144
ec_funded: 1
external_id:
  isi:
  - '000531438101047'
file:
- access_level: open_access
  checksum: 7b77fb5c2d27c4c37a7612ba46a66117
  content_type: application/pdf
  creator: bphuong
  date_created: 2020-02-11T09:06:39Z
  date_updated: 2020-07-14T12:47:59Z
  file_id: '7480'
  file_name: main.pdf
  file_size: 735768
  relation: main_file
file_date_updated: 2020-07-14T12:47:59Z
has_accepted_license: '1'
isi: 1
language:
- iso: eng
month: '10'
oa: 1
oa_version: Submitted Version
page: 1355-1364
project:
- _id: 2532554C-B435-11E9-9278-68D0E5697425
  call_identifier: FP7
  grant_number: '308036'
  name: Lifelong Learning of Visual Scene Understanding
publication: IEEE International Conference on Computer Vision
publication_identifier:
  isbn:
  - '9781728148038'
  issn:
  - '15505499'
publication_status: published
publisher: IEEE
quality_controlled: '1'
related_material:
  record:
  - id: '9418'
    relation: dissertation_contains
    status: public
scopus_import: '1'
status: public
title: Distillation-based training for multi-exit architectures
type: conference
user_id: c635000d-4b10-11ee-a964-aac5a93f6ac1
volume: 2019-October
year: '2019'
...