---
_id: '12538'
abstract:
- lang: eng
  text: In this paper, we study the compression of a target two-layer neural network
    with N nodes into a compressed network with M<N nodes. More precisely, we consider
    the setting in which the weights of the target network are i.i.d. sub-Gaussian,
    and we minimize the population L_2 loss between the outputs of the target and
    of the compressed network, under the assumption of Gaussian inputs. By using tools
    from high-dimensional probability, we show that this non-convex problem can be
    simplified when the target network is sufficiently over-parameterized, and provide
    the error rate of this approximation as a function of the input dimension and
    N. In this mean-field limit, the simplified objective, as well as the optimal
    weights of the compressed network, does not depend on the realization of the target
    network, but only on expected scaling factors. Furthermore, for networks with
    ReLU activation, we conjecture that the optimum of the simplified optimization
    problem is achieved by taking weights on the Equiangular Tight Frame (ETF), while
    the scaling of the weights and the orientation of the ETF depend on the parameters
    of the target network. Numerical evidence is provided to support this conjecture.
article_processing_charge: No
article_type: original
arxiv: 1
author:
- first_name: Mohammad Hossein
  full_name: Amani, Mohammad Hossein
  last_name: Amani
- first_name: Simone
  full_name: Bombari, Simone
  id: ca726dda-de17-11ea-bc14-f9da834f63aa
  last_name: Bombari
- first_name: Marco
  full_name: Mondelli, Marco
  id: 27EB676C-8706-11E9-9510-7717E6697425
  last_name: Mondelli
  orcid: 0000-0002-3242-7020
- first_name: Rattana
  full_name: Pukdee, Rattana
  last_name: Pukdee
- first_name: Stefano
  full_name: Rini, Stefano
  last_name: Rini
citation:
  ama: Amani MH, Bombari S, Mondelli M, Pukdee R, Rini S. Sharp asymptotics on the
    compression of two-layer neural networks. <i>IEEE Information Theory Workshop</i>.
    2022:588-593. doi:<a href="https://doi.org/10.1109/ITW54588.2022.9965870">10.1109/ITW54588.2022.9965870</a>
  apa: 'Amani, M. H., Bombari, S., Mondelli, M., Pukdee, R., &#38; Rini, S. (2022).
    Sharp asymptotics on the compression of two-layer neural networks. <i>IEEE Information
    Theory Workshop</i>. Mumbai, India: IEEE. <a href="https://doi.org/10.1109/ITW54588.2022.9965870">https://doi.org/10.1109/ITW54588.2022.9965870</a>'
  chicago: Amani, Mohammad Hossein, Simone Bombari, Marco Mondelli, Rattana Pukdee,
    and Stefano Rini. “Sharp Asymptotics on the Compression of Two-Layer Neural Networks.”
    <i>IEEE Information Theory Workshop</i>. IEEE, 2022. <a href="https://doi.org/10.1109/ITW54588.2022.9965870">https://doi.org/10.1109/ITW54588.2022.9965870</a>.
  ieee: M. H. Amani, S. Bombari, M. Mondelli, R. Pukdee, and S. Rini, “Sharp asymptotics
    on the compression of two-layer neural networks,” <i>IEEE Information Theory Workshop</i>.
    IEEE, pp. 588–593, 2022.
  ista: Amani MH, Bombari S, Mondelli M, Pukdee R, Rini S. 2022. Sharp asymptotics
    on the compression of two-layer neural networks. IEEE Information Theory Workshop.,
    588–593.
  mla: Amani, Mohammad Hossein, et al. “Sharp Asymptotics on the Compression of Two-Layer
    Neural Networks.” <i>IEEE Information Theory Workshop</i>, IEEE, 2022, pp. 588–93,
    doi:<a href="https://doi.org/10.1109/ITW54588.2022.9965870">10.1109/ITW54588.2022.9965870</a>.
  short: M.H. Amani, S. Bombari, M. Mondelli, R. Pukdee, S. Rini, IEEE Information
    Theory Workshop (2022) 588–593.
conference:
  end_date: 2022-11-09
  location: Mumbai, India
  name: 'ITW: Information Theory Workshop'
  start_date: 2022-11-01
date_created: 2023-02-10T13:47:56Z
date_published: 2022-11-16T00:00:00Z
date_updated: 2023-12-18T11:31:47Z
day: '16'
department:
- _id: MaMo
doi: 10.1109/ITW54588.2022.9965870
external_id:
  arxiv:
  - '2205.08199'
language:
- iso: eng
main_file_link:
- open_access: '1'
  url: ' https://doi.org/10.48550/arXiv.2205.08199'
month: '11'
oa: 1
oa_version: Preprint
page: 588-593
publication: IEEE Information Theory Workshop
publication_identifier:
  isbn:
  - '9781665483414'
publication_status: published
publisher: IEEE
quality_controlled: '1'
scopus_import: '1'
status: public
title: Sharp asymptotics on the compression of two-layer neural networks
type: journal_article
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
year: '2022'
...