---
_id: '13234'
abstract:
- lang: eng
text: Neural-network classifiers achieve high accuracy when predicting the class
of an input that they were trained to identify. Maintaining this accuracy in dynamic
environments, where inputs frequently fall outside the fixed set of initially
known classes, remains a challenge. We consider the problem of monitoring the
classification decisions of neural networks in the presence of novel classes.
For this purpose, we generalize our recently proposed abstraction-based monitor
from binary output to real-valued quantitative output. This quantitative output
enables new applications, two of which we investigate in the paper. As our first
application, we introduce an algorithmic framework for active monitoring of a
neural network, which allows us to learn new classes dynamically and yet maintain
high monitoring performance. As our second application, we present an offline
procedure to retrain the neural network to improve the monitor’s detection performance
without deteriorating the network’s classification accuracy. Our experimental
evaluation demonstrates both the benefits of our active monitoring framework in
dynamic scenarios and the effectiveness of the retraining procedure.
acknowledgement: This work was supported in part by the ERC-2020-AdG 101020093, by
DIREC - Digital Research Centre Denmark, and by the Villum Investigator Grant S4OS.
article_processing_charge: Yes (in subscription journal)
article_type: original
arxiv: 1
author:
- first_name: Konstantin
full_name: Kueffner, Konstantin
id: 8121a2d0-dc85-11ea-9058-af578f3b4515
last_name: Kueffner
orcid: 0000-0001-8974-2542
- first_name: Anna
full_name: Lukina, Anna
id: CBA4D1A8-0FE8-11E9-BDE6-07BFE5697425
last_name: Lukina
- first_name: Christian
full_name: Schilling, Christian
id: 3A2F4DCE-F248-11E8-B48F-1D18A9856A87
last_name: Schilling
orcid: 0000-0003-3658-1065
- first_name: Thomas A
full_name: Henzinger, Thomas A
id: 40876CD8-F248-11E8-B48F-1D18A9856A87
last_name: Henzinger
orcid: 0000-0002-2985-7724
citation:
ama: 'Kueffner K, Lukina A, Schilling C, Henzinger TA. Into the unknown: Active
monitoring of neural networks (extended version). International Journal on
Software Tools for Technology Transfer. 2023;25:575-592. doi:10.1007/s10009-023-00711-4'
apa: 'Kueffner, K., Lukina, A., Schilling, C., & Henzinger, T. A. (2023). Into
the unknown: Active monitoring of neural networks (extended version). International
Journal on Software Tools for Technology Transfer. Springer Nature. https://doi.org/10.1007/s10009-023-00711-4'
chicago: 'Kueffner, Konstantin, Anna Lukina, Christian Schilling, and Thomas A Henzinger.
“Into the Unknown: Active Monitoring of Neural Networks (Extended Version).” International
Journal on Software Tools for Technology Transfer. Springer Nature, 2023.
https://doi.org/10.1007/s10009-023-00711-4.'
ieee: 'K. Kueffner, A. Lukina, C. Schilling, and T. A. Henzinger, “Into the unknown:
Active monitoring of neural networks (extended version),” International Journal
on Software Tools for Technology Transfer, vol. 25. Springer Nature, pp. 575–592,
2023.'
ista: 'Kueffner K, Lukina A, Schilling C, Henzinger TA. 2023. Into the unknown:
Active monitoring of neural networks (extended version). International Journal
on Software Tools for Technology Transfer. 25, 575–592.'
mla: 'Kueffner, Konstantin, et al. “Into the Unknown: Active Monitoring of Neural
Networks (Extended Version).” International Journal on Software Tools for Technology
Transfer, vol. 25, Springer Nature, 2023, pp. 575–92, doi:10.1007/s10009-023-00711-4.'
short: K. Kueffner, A. Lukina, C. Schilling, T.A. Henzinger, International Journal
on Software Tools for Technology Transfer 25 (2023) 575–592.
date_created: 2023-07-16T22:01:11Z
date_published: 2023-08-01T00:00:00Z
date_updated: 2024-01-30T12:06:57Z
day: '01'
ddc:
- '000'
department:
- _id: ToHe
doi: 10.1007/s10009-023-00711-4
ec_funded: 1
external_id:
arxiv:
- '2009.06429'
isi:
- '001020160000001'
file:
- access_level: open_access
checksum: 3c4b347f39412a76872f9a6f30101f94
content_type: application/pdf
creator: dernst
date_created: 2024-01-30T12:06:07Z
date_updated: 2024-01-30T12:06:07Z
file_id: '14903'
file_name: 2023_JourSoftwareTools_Kueffner.pdf
file_size: 13387667
relation: main_file
success: 1
file_date_updated: 2024-01-30T12:06:07Z
has_accepted_license: '1'
intvolume: ' 25'
isi: 1
language:
- iso: eng
month: '08'
oa: 1
oa_version: Published Version
page: 575-592
project:
- _id: 62781420-2b32-11ec-9570-8d9b63373d4d
call_identifier: H2020
grant_number: '101020093'
name: Vigilant Algorithmic Monitoring of Software
publication: International Journal on Software Tools for Technology Transfer
publication_identifier:
eissn:
- 1433-2787
issn:
- 1433-2779
publication_status: published
publisher: Springer Nature
quality_controlled: '1'
related_material:
record:
- id: '10206'
relation: shorter_version
status: public
scopus_import: '1'
status: public
title: 'Into the unknown: Active monitoring of neural networks (extended version)'
tmp:
image: /images/cc_by.png
legal_code_url: https://creativecommons.org/licenses/by/4.0/legalcode
name: Creative Commons Attribution 4.0 International Public License (CC-BY 4.0)
short: CC BY (4.0)
type: journal_article
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
volume: 25
year: '2023'
...
---
_id: '10206'
abstract:
- lang: eng
text: Neural-network classifiers achieve high accuracy when predicting the class
of an input that they were trained to identify. Maintaining this accuracy in dynamic
environments, where inputs frequently fall outside the fixed set of initially
known classes, remains a challenge. The typical approach is to detect inputs from
novel classes and retrain the classifier on an augmented dataset. However, not
only the classifier but also the detection mechanism needs to adapt in order to
distinguish between newly learned and yet unknown input classes. To address this
challenge, we introduce an algorithmic framework for active monitoring of a neural
network. A monitor wrapped in our framework operates in parallel with the neural
network and interacts with a human user via a series of interpretable labeling
queries for incremental adaptation. In addition, we propose an adaptive quantitative
monitor to improve precision. An experimental evaluation on a diverse set of benchmarks
with varying numbers of classes confirms the benefits of our active monitoring
framework in dynamic scenarios.
acknowledgement: We thank Christoph Lampert and Alex Greengold for fruitful discussions.
This research was supported in part by the Simons Institute for the Theory of Computing,
the Austrian Science Fund (FWF) under grant Z211-N23 (Wittgenstein Award), and the
European Union’s Horizon 2020 research and innovation programme under the Marie
Skłodowska-Curie grant agreement No. 754411.
alternative_title:
- LNCS
article_processing_charge: No
arxiv: 1
author:
- first_name: Anna
full_name: Lukina, Anna
id: CBA4D1A8-0FE8-11E9-BDE6-07BFE5697425
last_name: Lukina
- first_name: Christian
full_name: Schilling, Christian
id: 3A2F4DCE-F248-11E8-B48F-1D18A9856A87
last_name: Schilling
orcid: 0000-0003-3658-1065
- first_name: Thomas A
full_name: Henzinger, Thomas A
id: 40876CD8-F248-11E8-B48F-1D18A9856A87
last_name: Henzinger
orcid: 0000-0002-2985-7724
citation:
ama: 'Lukina A, Schilling C, Henzinger TA. Into the unknown: active monitoring of neural
networks. In: 21st International Conference on Runtime Verification. Vol
12974. Cham: Springer Nature; 2021:42-61. doi:10.1007/978-3-030-88494-9_3'
apa: 'Lukina, A., Schilling, C., & Henzinger, T. A. (2021). Into the unknown:
active monitoring of neural networks. In 21st International Conference on Runtime
Verification (Vol. 12974, pp. 42–61). Cham: Springer Nature. https://doi.org/10.1007/978-3-030-88494-9_3'
chicago: 'Lukina, Anna, Christian Schilling, and Thomas A Henzinger. “Into the Unknown:
Active Monitoring of Neural Networks.” In 21st International Conference on
Runtime Verification, 12974:42–61. Cham: Springer Nature, 2021. https://doi.org/10.1007/978-3-030-88494-9_3.'
ieee: 'A. Lukina, C. Schilling, and T. A. Henzinger, “Into the unknown: active monitoring
of neural networks,” in 21st International Conference on Runtime Verification,
Virtual, 2021, vol. 12974, pp. 42–61.'
ista: 'Lukina A, Schilling C, Henzinger TA. 2021. Into the unknown: active monitoring
of neural networks. 21st International Conference on Runtime Verification. RV:
Runtime Verification, LNCS, vol. 12974, 42–61.'
mla: 'Lukina, Anna, et al. “Into the Unknown: Active Monitoring of Neural Networks.”
21st International Conference on Runtime Verification, vol. 12974, Springer
Nature, 2021, pp. 42–61, doi:10.1007/978-3-030-88494-9_3.'
short: A. Lukina, C. Schilling, T.A. Henzinger, in:, 21st International Conference
on Runtime Verification, Springer Nature, Cham, 2021, pp. 42–61.
conference:
end_date: 2021-10-14
location: Virtual
name: 'RV: Runtime Verification'
start_date: 2021-10-11
date_created: 2021-10-31T23:01:31Z
date_published: 2021-10-06T00:00:00Z
date_updated: 2024-01-30T12:06:56Z
day: '06'
department:
- _id: ToHe
doi: 10.1007/978-3-030-88494-9_3
ec_funded: 1
external_id:
arxiv:
- '2009.06429'
isi:
- '000719383800003'
isi: 1
keyword:
- monitoring
- neural networks
- novelty detection
language:
- iso: eng
main_file_link:
- open_access: '1'
url: https://arxiv.org/abs/2009.06429
month: '10'
oa: 1
oa_version: Preprint
page: 42-61
place: Cham
project:
- _id: 260C2330-B435-11E9-9278-68D0E5697425
call_identifier: H2020
grant_number: '754411'
name: ISTplus - Postdoctoral Fellowships
- _id: 25F42A32-B435-11E9-9278-68D0E5697425
call_identifier: FWF
grant_number: Z211
name: The Wittgenstein Prize
publication: 21st International Conference on Runtime Verification
publication_identifier:
eisbn:
- 978-3-030-88494-9
eissn:
- 1611-3349
isbn:
- 9-783-0308-8493-2
issn:
- 0302-9743
publication_status: published
publisher: Springer Nature
quality_controlled: '1'
related_material:
record:
- id: '13234'
relation: extended_version
status: public
scopus_import: '1'
status: public
title: 'Into the unknown: active monitoring of neural networks'
type: conference
user_id: 4359f0d1-fa6c-11eb-b949-802e58b17ae8
volume: '12974 '
year: '2021'
...
---
_id: '7505'
abstract:
- lang: eng
text: Neural networks have demonstrated unmatched performance in a range of classification
tasks. Despite numerous efforts of the research community, novelty detection remains
one of the significant limitations of neural networks. The ability to identify
previously unseen inputs as novel is crucial for our understanding of the decisions
made by neural networks. At runtime, inputs not falling into any of the categories
learned during training cannot be classified correctly by the neural network.
Existing approaches treat the neural network as a black box and try to detect
novel inputs based on the confidence of the output predictions. However, neural
networks are not trained to reduce their confidence for novel inputs, which limits
the effectiveness of these approaches. We propose a framework to monitor a neural
network by observing the hidden layers. We employ a common abstraction from program
analysis - boxes - to identify novel behaviors in the monitored layers, i.e.,
inputs that cause behaviors outside the box. For each neuron, the boxes range
over the values seen in training. The framework is efficient and flexible to achieve
a desired trade-off between raising false warnings and detecting novel inputs.
We illustrate the performance and the robustness to variability in the unknown
classes on popular image-classification benchmarks.
acknowledgement: We thank Christoph Lampert and Nikolaus Mayer for fruitful discussions.
This research was supported in part by the Austrian Science Fund (FWF) under grants
S11402-N23 (RiSE/SHiNE) and Z211-N23 (Wittgenstein Award) and the European Union’s
Horizon 2020 research and innovation programme under the Marie SkłodowskaCurie grant
agreement No. 754411.
alternative_title:
- Frontiers in Artificial Intelligence and Applications
article_processing_charge: No
arxiv: 1
author:
- first_name: Thomas A
full_name: Henzinger, Thomas A
id: 40876CD8-F248-11E8-B48F-1D18A9856A87
last_name: Henzinger
orcid: 0000-0002-2985-7724
- first_name: Anna
full_name: Lukina, Anna
id: CBA4D1A8-0FE8-11E9-BDE6-07BFE5697425
last_name: Lukina
- first_name: Christian
full_name: Schilling, Christian
id: 3A2F4DCE-F248-11E8-B48F-1D18A9856A87
last_name: Schilling
orcid: 0000-0003-3658-1065
citation:
ama: 'Henzinger TA, Lukina A, Schilling C. Outside the box: Abstraction-based monitoring
of neural networks. In: 24th European Conference on Artificial Intelligence.
Vol 325. IOS Press; 2020:2433-2440. doi:10.3233/FAIA200375'
apa: 'Henzinger, T. A., Lukina, A., & Schilling, C. (2020). Outside the box:
Abstraction-based monitoring of neural networks. In 24th European Conference
on Artificial Intelligence (Vol. 325, pp. 2433–2440). Santiago de Compostela,
Spain: IOS Press. https://doi.org/10.3233/FAIA200375'
chicago: 'Henzinger, Thomas A, Anna Lukina, and Christian Schilling. “Outside the
Box: Abstraction-Based Monitoring of Neural Networks.” In 24th European Conference
on Artificial Intelligence, 325:2433–40. IOS Press, 2020. https://doi.org/10.3233/FAIA200375.'
ieee: 'T. A. Henzinger, A. Lukina, and C. Schilling, “Outside the box: Abstraction-based
monitoring of neural networks,” in 24th European Conference on Artificial Intelligence,
Santiago de Compostela, Spain, 2020, vol. 325, pp. 2433–2440.'
ista: 'Henzinger TA, Lukina A, Schilling C. 2020. Outside the box: Abstraction-based
monitoring of neural networks. 24th European Conference on Artificial Intelligence.
ECAI: European Conference on Artificial Intelligence, Frontiers in Artificial
Intelligence and Applications, vol. 325, 2433–2440.'
mla: 'Henzinger, Thomas A., et al. “Outside the Box: Abstraction-Based Monitoring
of Neural Networks.” 24th European Conference on Artificial Intelligence,
vol. 325, IOS Press, 2020, pp. 2433–40, doi:10.3233/FAIA200375.'
short: T.A. Henzinger, A. Lukina, C. Schilling, in:, 24th European Conference on
Artificial Intelligence, IOS Press, 2020, pp. 2433–2440.
conference:
end_date: 2020-09-08
location: Santiago de Compostela, Spain
name: 'ECAI: European Conference on Artificial Intelligence'
start_date: 2020-08-29
date_created: 2020-02-21T16:44:03Z
date_published: 2020-02-24T00:00:00Z
date_updated: 2023-08-18T06:38:16Z
day: '24'
ddc:
- '000'
department:
- _id: ToHe
doi: 10.3233/FAIA200375
ec_funded: 1
external_id:
arxiv:
- '1911.09032'
isi:
- '000650971303002'
file:
- access_level: open_access
checksum: 80642fa0b6cd7da95dcd87d63789ad5e
content_type: application/pdf
creator: dernst
date_created: 2020-09-21T07:12:32Z
date_updated: 2020-09-21T07:12:32Z
file_id: '8540'
file_name: 2020_ECAI_Henzinger.pdf
file_size: 1692214
relation: main_file
success: 1
file_date_updated: 2020-09-21T07:12:32Z
has_accepted_license: '1'
intvolume: ' 325'
isi: 1
language:
- iso: eng
license: https://creativecommons.org/licenses/by-nc/4.0/
month: '02'
oa: 1
oa_version: Published Version
page: 2433-2440
project:
- _id: 260C2330-B435-11E9-9278-68D0E5697425
call_identifier: H2020
grant_number: '754411'
name: ISTplus - Postdoctoral Fellowships
- _id: 25832EC2-B435-11E9-9278-68D0E5697425
call_identifier: FWF
grant_number: S 11407_N23
name: Rigorous Systems Engineering
- _id: 25F42A32-B435-11E9-9278-68D0E5697425
call_identifier: FWF
grant_number: Z211
name: The Wittgenstein Prize
publication: 24th European Conference on Artificial Intelligence
publication_status: published
publisher: IOS Press
quality_controlled: '1'
status: public
title: 'Outside the box: Abstraction-based monitoring of neural networks'
tmp:
image: /images/cc_by_nc.png
legal_code_url: https://creativecommons.org/licenses/by-nc/4.0/legalcode
name: Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0)
short: CC BY-NC (4.0)
type: conference
user_id: 4359f0d1-fa6c-11eb-b949-802e58b17ae8
volume: 325
year: '2020'
...
---
_id: '9040'
abstract:
- lang: eng
text: Machine learning and formal methods have complimentary benefits and drawbacks.
In this work, we address the controller-design problem with a combination of techniques
from both fields. The use of black-box neural networks in deep reinforcement learning
(deep RL) poses a challenge for such a combination. Instead of reasoning formally
about the output of deep RL, which we call the wizard, we extract from it a decision-tree
based model, which we refer to as the magic book. Using the extracted model as
an intermediary, we are able to handle problems that are infeasible for either
deep RL or formal methods by themselves. First, we suggest, for the first time,
a synthesis procedure that is based on a magic book. We synthesize a stand-alone
correct-by-design controller that enjoys the favorable performance of RL. Second,
we incorporate a magic book in a bounded model checking (BMC) procedure. BMC allows
us to find numerous traces of the plant under the control of the wizard, which
a user can use to increase the trustworthiness of the wizard and direct further
training.
acknowledgement: This research was supported in part by the Austrian Science Fund
(FWF) under grant Z211-N23 (Wittgenstein Award).
article_processing_charge: No
author:
- first_name: Par Alizadeh
full_name: Alamdari, Par Alizadeh
last_name: Alamdari
- first_name: Guy
full_name: Avni, Guy
id: 463C8BC2-F248-11E8-B48F-1D18A9856A87
last_name: Avni
orcid: 0000-0001-5588-8287
- first_name: Thomas A
full_name: Henzinger, Thomas A
id: 40876CD8-F248-11E8-B48F-1D18A9856A87
last_name: Henzinger
orcid: 0000-0002-2985-7724
- first_name: Anna
full_name: Lukina, Anna
id: CBA4D1A8-0FE8-11E9-BDE6-07BFE5697425
last_name: Lukina
citation:
ama: 'Alamdari PA, Avni G, Henzinger TA, Lukina A. Formal methods with a touch of
magic. In: Proceedings of the 20th Conference on Formal Methods in Computer-Aided
Design. TU Wien Academic Press; 2020:138-147. doi:10.34727/2020/isbn.978-3-85448-042-6_21'
apa: 'Alamdari, P. A., Avni, G., Henzinger, T. A., & Lukina, A. (2020). Formal
methods with a touch of magic. In Proceedings of the 20th Conference on Formal
Methods in Computer-Aided Design (pp. 138–147). Online Conference: TU Wien
Academic Press. https://doi.org/10.34727/2020/isbn.978-3-85448-042-6_21'
chicago: Alamdari, Par Alizadeh, Guy Avni, Thomas A Henzinger, and Anna Lukina.
“Formal Methods with a Touch of Magic.” In Proceedings of the 20th Conference
on Formal Methods in Computer-Aided Design, 138–47. TU Wien Academic Press,
2020. https://doi.org/10.34727/2020/isbn.978-3-85448-042-6_21.
ieee: P. A. Alamdari, G. Avni, T. A. Henzinger, and A. Lukina, “Formal methods with
a touch of magic,” in Proceedings of the 20th Conference on Formal Methods
in Computer-Aided Design, Online Conference, 2020, pp. 138–147.
ista: 'Alamdari PA, Avni G, Henzinger TA, Lukina A. 2020. Formal methods with a
touch of magic. Proceedings of the 20th Conference on Formal Methods in Computer-Aided
Design. FMCAD: Formal Methods in Computer-Aided Design, 138–147.'
mla: Alamdari, Par Alizadeh, et al. “Formal Methods with a Touch of Magic.” Proceedings
of the 20th Conference on Formal Methods in Computer-Aided Design, TU Wien
Academic Press, 2020, pp. 138–47, doi:10.34727/2020/isbn.978-3-85448-042-6_21.
short: P.A. Alamdari, G. Avni, T.A. Henzinger, A. Lukina, in:, Proceedings of the
20th Conference on Formal Methods in Computer-Aided Design, TU Wien Academic Press,
2020, pp. 138–147.
conference:
end_date: 2020-09-24
location: Online Conference
name: ' FMCAD: Formal Methods in Computer-Aided Design'
start_date: 2020-09-21
date_created: 2021-01-24T23:01:10Z
date_published: 2020-09-21T00:00:00Z
date_updated: 2021-02-09T09:39:59Z
day: '21'
ddc:
- '000'
department:
- _id: ToHe
doi: 10.34727/2020/isbn.978-3-85448-042-6_21
file:
- access_level: open_access
checksum: d616d549a0ade78606b16f8a9540820f
content_type: application/pdf
creator: dernst
date_created: 2021-02-09T09:39:02Z
date_updated: 2021-02-09T09:39:02Z
file_id: '9109'
file_name: 2020_FMCAD_Alamdari.pdf
file_size: 990999
relation: main_file
success: 1
file_date_updated: 2021-02-09T09:39:02Z
has_accepted_license: '1'
language:
- iso: eng
month: '09'
oa: 1
oa_version: Published Version
page: 138-147
project:
- _id: 25F42A32-B435-11E9-9278-68D0E5697425
call_identifier: FWF
grant_number: Z211
name: The Wittgenstein Prize
publication: Proceedings of the 20th Conference on Formal Methods in Computer-Aided
Design
publication_identifier:
eissn:
- 2708-7824
isbn:
- '9783854480426'
publication_status: published
publisher: TU Wien Academic Press
quality_controlled: '1'
scopus_import: '1'
status: public
title: Formal methods with a touch of magic
tmp:
image: /images/cc_by.png
legal_code_url: https://creativecommons.org/licenses/by/4.0/legalcode
name: Creative Commons Attribution 4.0 International Public License (CC-BY 4.0)
short: CC BY (4.0)
type: conference
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
year: '2020'
...