---
_id: '15039'
abstract:
- lang: eng
  text: 'A crucial property for achieving secure, trustworthy and interpretable deep
    learning systems is their robustness: small changes to a system''s inputs should
    not result in large changes to its outputs. Mathematically, this means one strives
    for networks with a small Lipschitz constant. Several recent works have focused
    on how to construct such Lipschitz networks, typically by imposing constraints
    on the weight matrices. In this work, we study an orthogonal aspect, namely the
    role of the activation function. We show that commonly used activation functions,
    such as MaxMin, as well as all piece-wise linear ones with two segments unnecessarily
    restrict the class of representable functions, even in the simplest one-dimensional
    setting. We furthermore introduce the new N-activation function that is provably
    more expressive than currently popular activation functions. We provide code at
    this https URL.'
article_number: '2311.06103'
article_processing_charge: No
arxiv: 1
author:
- first_name: Bernd
  full_name: Prach, Bernd
  id: 2D561D42-C427-11E9-89B4-9C1AE6697425
  last_name: Prach
- first_name: Christoph
  full_name: Lampert, Christoph
  id: 40C20FD2-F248-11E8-B48F-1D18A9856A87
  last_name: Lampert
  orcid: 0000-0001-8622-7887
citation:
  ama: Prach B, Lampert C. 1-Lipschitz neural networks are more expressive with N-activations.
    <i>arXiv</i>. doi:<a href="https://doi.org/10.48550/ARXIV.2311.06103">10.48550/ARXIV.2311.06103</a>
  apa: Prach, B., &#38; Lampert, C. (n.d.). 1-Lipschitz neural networks are more expressive
    with N-activations. <i>arXiv</i>. <a href="https://doi.org/10.48550/ARXIV.2311.06103">https://doi.org/10.48550/ARXIV.2311.06103</a>
  chicago: Prach, Bernd, and Christoph Lampert. “1-Lipschitz Neural Networks Are More
    Expressive with N-Activations.” <i>ArXiv</i>, n.d. <a href="https://doi.org/10.48550/ARXIV.2311.06103">https://doi.org/10.48550/ARXIV.2311.06103</a>.
  ieee: B. Prach and C. Lampert, “1-Lipschitz neural networks are more expressive
    with N-activations,” <i>arXiv</i>. .
  ista: Prach B, Lampert C. 1-Lipschitz neural networks are more expressive with N-activations.
    arXiv, 2311.06103.
  mla: Prach, Bernd, and Christoph Lampert. “1-Lipschitz Neural Networks Are More
    Expressive with N-Activations.” <i>ArXiv</i>, 2311.06103, doi:<a href="https://doi.org/10.48550/ARXIV.2311.06103">10.48550/ARXIV.2311.06103</a>.
  short: B. Prach, C. Lampert, ArXiv (n.d.).
date_created: 2024-02-28T17:59:32Z
date_published: 2023-11-10T00:00:00Z
date_updated: 2024-03-04T07:02:39Z
day: '10'
department:
- _id: GradSch
- _id: ChLa
doi: 10.48550/ARXIV.2311.06103
external_id:
  arxiv:
  - '2311.06103'
language:
- iso: eng
main_file_link:
- open_access: '1'
  url: https://doi.org/10.48550/arXiv.2311.06103
month: '11'
oa: 1
oa_version: Preprint
publication: arXiv
publication_status: submitted
status: public
title: 1-Lipschitz neural networks are more expressive with N-activations
type: preprint
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
year: '2023'
...
---
_id: '11839'
abstract:
- lang: eng
  text: "It is a highly desirable property for deep networks to be robust against\r\nsmall
    input changes. One popular way to achieve this property is by designing\r\nnetworks
    with a small Lipschitz constant. In this work, we propose a new\r\ntechnique for
    constructing such Lipschitz networks that has a number of\r\ndesirable properties:
    it can be applied to any linear network layer\r\n(fully-connected or convolutional),
    it provides formal guarantees on the\r\nLipschitz constant, it is easy to implement
    and efficient to run, and it can be\r\ncombined with any training objective and
    optimization method. In fact, our\r\ntechnique is the first one in the literature
    that achieves all of these\r\nproperties simultaneously. Our main contribution
    is a rescaling-based weight\r\nmatrix parametrization that guarantees each network
    layer to have a Lipschitz\r\nconstant of at most 1 and results in the learned
    weight matrices to be close to\r\northogonal. Hence we call such layers almost-orthogonal
    Lipschitz (AOL).\r\nExperiments and ablation studies in the context of image classification
    with\r\ncertified robust accuracy confirm that AOL layers achieve results that
    are on\r\npar with most existing methods. Yet, they are simpler to implement and
    more\r\nbroadly applicable, because they do not require computationally expensive\r\nmatrix
    orthogonalization or inversion steps as part of the network\r\narchitecture. We
    provide code at https://github.com/berndprach/AOL."
alternative_title:
- LNCS
article_processing_charge: No
arxiv: 1
author:
- first_name: Bernd
  full_name: Prach, Bernd
  id: 2D561D42-C427-11E9-89B4-9C1AE6697425
  last_name: Prach
- first_name: Christoph
  full_name: Lampert, Christoph
  id: 40C20FD2-F248-11E8-B48F-1D18A9856A87
  last_name: Lampert
  orcid: 0000-0001-8622-7887
citation:
  ama: 'Prach B, Lampert C. Almost-orthogonal layers for efficient general-purpose
    Lipschitz networks. In: <i>Computer Vision – ECCV 2022</i>. Vol 13681. Springer
    Nature; 2022:350-365. doi:<a href="https://doi.org/10.1007/978-3-031-19803-8_21">10.1007/978-3-031-19803-8_21</a>'
  apa: 'Prach, B., &#38; Lampert, C. (2022). Almost-orthogonal layers for efficient
    general-purpose Lipschitz networks. In <i>Computer Vision – ECCV 2022</i> (Vol.
    13681, pp. 350–365). Tel Aviv, Israel: Springer Nature. <a href="https://doi.org/10.1007/978-3-031-19803-8_21">https://doi.org/10.1007/978-3-031-19803-8_21</a>'
  chicago: Prach, Bernd, and Christoph Lampert. “Almost-Orthogonal Layers for Efficient
    General-Purpose Lipschitz Networks.” In <i>Computer Vision – ECCV 2022</i>, 13681:350–65.
    Springer Nature, 2022. <a href="https://doi.org/10.1007/978-3-031-19803-8_21">https://doi.org/10.1007/978-3-031-19803-8_21</a>.
  ieee: B. Prach and C. Lampert, “Almost-orthogonal layers for efficient general-purpose
    Lipschitz networks,” in <i>Computer Vision – ECCV 2022</i>, Tel Aviv, Israel,
    2022, vol. 13681, pp. 350–365.
  ista: 'Prach B, Lampert C. 2022. Almost-orthogonal layers for efficient general-purpose
    Lipschitz networks. Computer Vision – ECCV 2022. ECCV: European Conference on
    Computer Vision, LNCS, vol. 13681, 350–365.'
  mla: Prach, Bernd, and Christoph Lampert. “Almost-Orthogonal Layers for Efficient
    General-Purpose Lipschitz Networks.” <i>Computer Vision – ECCV 2022</i>, vol.
    13681, Springer Nature, 2022, pp. 350–65, doi:<a href="https://doi.org/10.1007/978-3-031-19803-8_21">10.1007/978-3-031-19803-8_21</a>.
  short: B. Prach, C. Lampert, in:, Computer Vision – ECCV 2022, Springer Nature,
    2022, pp. 350–365.
conference:
  end_date: 2022-10-27
  location: Tel Aviv, Israel
  name: 'ECCV: European Conference on Computer Vision'
  start_date: 2022-10-23
date_created: 2022-08-12T15:09:47Z
date_published: 2022-10-23T00:00:00Z
date_updated: 2023-05-03T08:00:46Z
day: '23'
department:
- _id: GradSch
- _id: ChLa
doi: 10.1007/978-3-031-19803-8_21
external_id:
  arxiv:
  - '2208.03160'
intvolume: '     13681'
language:
- iso: eng
main_file_link:
- open_access: '1'
  url: ' https://doi.org/10.48550/arXiv.2208.03160'
month: '10'
oa: 1
oa_version: Preprint
page: 350-365
publication: Computer Vision – ECCV 2022
publication_identifier:
  eisbn:
  - '9783031198038'
  isbn:
  - '9783031198021'
publication_status: published
publisher: Springer Nature
quality_controlled: '1'
scopus_import: '1'
status: public
title: Almost-orthogonal layers for efficient general-purpose Lipschitz networks
type: conference
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
volume: 13681
year: '2022'
...