Outside the box: Abstraction-based monitoring of neural networks
Henzinger TA, Lukina A, Schilling C. 2020. Outside the box: Abstraction-based monitoring of neural networks. 24th European Conference on Artificial Intelligence. ECAI: European Conference on Artificial Intelligence, Frontiers in Artificial Intelligence and Applications, vol. 325, 2433–2440.
Download
Conference Paper
| Published
| English
Department
Series Title
Frontiers in Artificial Intelligence and Applications
Abstract
Neural networks have demonstrated unmatched performance in a range of classification tasks. Despite numerous efforts of the research community, novelty detection remains one of the significant limitations of neural networks. The ability to identify previously unseen inputs as novel is crucial for our understanding of the decisions made by neural networks. At runtime, inputs not falling into any of the categories learned during training cannot be classified correctly by the neural network. Existing approaches treat the neural network as a black box and try to detect novel inputs based on the confidence of the output predictions. However, neural networks are not trained to reduce their confidence for novel inputs, which limits the effectiveness of these approaches. We propose a framework to monitor a neural network by observing the hidden layers. We employ a common abstraction from program analysis - boxes - to identify novel behaviors in the monitored layers, i.e., inputs that cause behaviors outside the box. For each neuron, the boxes range over the values seen in training. The framework is efficient and flexible to achieve a desired trade-off between raising false warnings and detecting novel inputs. We illustrate the performance and the robustness to variability in the unknown classes on popular image-classification benchmarks.
Publishing Year
Date Published
2020-02-24
Proceedings Title
24th European Conference on Artificial Intelligence
Publisher
IOS Press
Acknowledgement
We thank Christoph Lampert and Nikolaus Mayer for fruitful discussions. This research was supported in part by the Austrian Science Fund (FWF) under grants S11402-N23 (RiSE/SHiNE) and Z211-N23 (Wittgenstein Award) and the European Union’s Horizon 2020 research and innovation programme under the Marie SkłodowskaCurie grant agreement No. 754411.
Volume
325
Page
2433-2440
Conference
ECAI: European Conference on Artificial Intelligence
Conference Location
Santiago de Compostela, Spain
Conference Date
2020-08-29 – 2020-09-08
IST-REx-ID
Cite this
Henzinger TA, Lukina A, Schilling C. Outside the box: Abstraction-based monitoring of neural networks. In: 24th European Conference on Artificial Intelligence. Vol 325. IOS Press; 2020:2433-2440. doi:10.3233/FAIA200375
Henzinger, T. A., Lukina, A., & Schilling, C. (2020). Outside the box: Abstraction-based monitoring of neural networks. In 24th European Conference on Artificial Intelligence (Vol. 325, pp. 2433–2440). Santiago de Compostela, Spain: IOS Press. https://doi.org/10.3233/FAIA200375
Henzinger, Thomas A, Anna Lukina, and Christian Schilling. “Outside the Box: Abstraction-Based Monitoring of Neural Networks.” In 24th European Conference on Artificial Intelligence, 325:2433–40. IOS Press, 2020. https://doi.org/10.3233/FAIA200375.
T. A. Henzinger, A. Lukina, and C. Schilling, “Outside the box: Abstraction-based monitoring of neural networks,” in 24th European Conference on Artificial Intelligence, Santiago de Compostela, Spain, 2020, vol. 325, pp. 2433–2440.
Henzinger TA, Lukina A, Schilling C. 2020. Outside the box: Abstraction-based monitoring of neural networks. 24th European Conference on Artificial Intelligence. ECAI: European Conference on Artificial Intelligence, Frontiers in Artificial Intelligence and Applications, vol. 325, 2433–2440.
Henzinger, Thomas A., et al. “Outside the Box: Abstraction-Based Monitoring of Neural Networks.” 24th European Conference on Artificial Intelligence, vol. 325, IOS Press, 2020, pp. 2433–40, doi:10.3233/FAIA200375.
All files available under the following license(s):
Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0):
Main File(s)
File Name
2020_ECAI_Henzinger.pdf
1.69 MB
Access Level
Open Access
Date Uploaded
2020-09-21
MD5 Checksum
80642fa0b6cd7da95dcd87d63789ad5e
Export
Marked PublicationsOpen Data ISTA Research Explorer
Web of Science
View record in Web of Science®Sources
arXiv 1911.09032