---
_id: '9210'
abstract:
- lang: eng
  text: "Modern neural networks can easily fit their training set perfectly. Surprisingly,
    despite being “overfit” in this way, they tend to generalize well to future data,
    thereby defying the classic bias–variance trade-off of machine learning theory.
    Of the many possible explanations, a prevalent one is that training by stochastic
    gradient descent (SGD) imposes an implicit bias that leads it to learn simple
    functions, and these simple functions generalize well. However, the specifics
    of this implicit bias are not well understood.\r\nIn this work, we explore the
    smoothness conjecture which states that SGD is implicitly biased towards learning
    functions that are smooth. We propose several measures to formalize the intuitive
    notion of smoothness, and we conduct experiments to determine whether SGD indeed
    implicitly optimizes for these measures. Our findings rule out the possibility
    that smoothness measures based on first-order derivatives are being implicitly
    enforced. They are supportive, though, of the smoothness conjecture for measures
    based on second-order derivatives."
article_processing_charge: No
author:
- first_name: Vaclav
  full_name: Volhejn, Vaclav
  id: d5235fb4-7a6d-11eb-b254-f25d12d631a8
  last_name: Volhejn
- first_name: Christoph
  full_name: Lampert, Christoph
  id: 40C20FD2-F248-11E8-B48F-1D18A9856A87
  last_name: Lampert
  orcid: 0000-0001-8622-7887
citation:
  ama: 'Volhejn V, Lampert C. Does SGD implicitly optimize for smoothness? In: <i>42nd
    German Conference on Pattern Recognition</i>. Vol 12544. LNCS. Springer; 2021:246-259.
    doi:<a href="https://doi.org/10.1007/978-3-030-71278-5_18">10.1007/978-3-030-71278-5_18</a>'
  apa: 'Volhejn, V., &#38; Lampert, C. (2021). Does SGD implicitly optimize for smoothness?
    In <i>42nd German Conference on Pattern Recognition</i> (Vol. 12544, pp. 246–259).
    Tübingen, Germany: Springer. <a href="https://doi.org/10.1007/978-3-030-71278-5_18">https://doi.org/10.1007/978-3-030-71278-5_18</a>'
  chicago: Volhejn, Vaclav, and Christoph Lampert. “Does SGD Implicitly Optimize for
    Smoothness?” In <i>42nd German Conference on Pattern Recognition</i>, 12544:246–59.
    LNCS. Springer, 2021. <a href="https://doi.org/10.1007/978-3-030-71278-5_18">https://doi.org/10.1007/978-3-030-71278-5_18</a>.
  ieee: V. Volhejn and C. Lampert, “Does SGD implicitly optimize for smoothness?,”
    in <i>42nd German Conference on Pattern Recognition</i>, Tübingen, Germany, 2021,
    vol. 12544, pp. 246–259.
  ista: 'Volhejn V, Lampert C. 2021. Does SGD implicitly optimize for smoothness?
    42nd German Conference on Pattern Recognition. DAGM GCPR: German Conference on
    Pattern Recognition LNCS vol. 12544, 246–259.'
  mla: Volhejn, Vaclav, and Christoph Lampert. “Does SGD Implicitly Optimize for Smoothness?”
    <i>42nd German Conference on Pattern Recognition</i>, vol. 12544, Springer, 2021,
    pp. 246–59, doi:<a href="https://doi.org/10.1007/978-3-030-71278-5_18">10.1007/978-3-030-71278-5_18</a>.
  short: V. Volhejn, C. Lampert, in:, 42nd German Conference on Pattern Recognition,
    Springer, 2021, pp. 246–259.
conference:
  end_date: 2020-10-01
  location: Tübingen, Germany
  name: 'DAGM GCPR: German Conference on Pattern Recognition '
  start_date: 2020-09-28
date_created: 2021-03-01T09:01:16Z
date_published: 2021-03-17T00:00:00Z
date_updated: 2022-08-12T07:28:47Z
day: '17'
ddc:
- '510'
department:
- _id: ChLa
doi: 10.1007/978-3-030-71278-5_18
file:
- access_level: open_access
  checksum: 3e3628ab1cf658d82524963f808004ea
  content_type: application/pdf
  creator: dernst
  date_created: 2022-08-12T07:27:58Z
  date_updated: 2022-08-12T07:27:58Z
  file_id: '11820'
  file_name: 2020_GCPR_submitted_Volhejn.pdf
  file_size: 420234
  relation: main_file
  success: 1
file_date_updated: 2022-08-12T07:27:58Z
has_accepted_license: '1'
intvolume: '     12544'
language:
- iso: eng
month: '03'
oa: 1
oa_version: Submitted Version
page: 246-259
publication: 42nd German Conference on Pattern Recognition
publication_identifier:
  eissn:
  - 1611-3349
  isbn:
  - '9783030712778'
  issn:
  - 0302-9743
publication_status: published
publisher: Springer
quality_controlled: '1'
scopus_import: '1'
series_title: LNCS
status: public
title: Does SGD implicitly optimize for smoothness?
type: conference
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
volume: 12544
year: '2021'
...