Anytime guarantees for reachability in uncountable Markov decision processes

Grover K, Kretinsky J, Meggendorfer T, Weininger M. 2022. Anytime guarantees for reachability in uncountable Markov decision processes. 33rd International Conference on Concurrency Theory . CONCUR: Conference on Concurrency Theory, LIPIcs, vol. 243, 11.

Download
OA 2022_LIPIcS_Grover.pdf 960.04 KB [Published Version]

Conference Paper | Published | English

Scopus indexed
Author
Grover, Kush; Kretinsky, JanISTA ; Meggendorfer, TobiasISTA ; Weininger, Maimilian
Department
Series Title
LIPIcs
Abstract
We consider the problem of approximating the reachability probabilities in Markov decision processes (MDP) with uncountable (continuous) state and action spaces. While there are algorithms that, for special classes of such MDP, provide a sequence of approximations converging to the true value in the limit, our aim is to obtain an algorithm with guarantees on the precision of the approximation. As this problem is undecidable in general, assumptions on the MDP are necessary. Our main contribution is to identify sufficient assumptions that are as weak as possible, thus approaching the "boundary" of which systems can be correctly and reliably analyzed. To this end, we also argue why each of our assumptions is necessary for algorithms based on processing finitely many observations. We present two solution variants. The first one provides converging lower bounds under weaker assumptions than typical ones from previous works concerned with guarantees. The second one then utilizes stronger assumptions to additionally provide converging upper bounds. Altogether, we obtain an anytime algorithm, i.e. yielding a sequence of approximants with known and iteratively improving precision, converging to the true value in the limit. Besides, due to the generality of our assumptions, our algorithms are very general templates, readily allowing for various heuristics from literature in contrast to, e.g., a specific discretization algorithm. Our theoretical contribution thus paves the way for future practical improvements without sacrificing correctness guarantees.
Publishing Year
Date Published
2022-09-15
Proceedings Title
33rd International Conference on Concurrency Theory
Publisher
Schloss Dagstuhl - Leibniz-Zentrum für Informatik
Acknowledgement
Kush Grover: The author has been supported by the DFG research training group GRK 2428 ConVeY. Maximilian Weininger: The author has been partially supported by DFG projects 383882557 Statistical Unbounded Verification (SUV) and 427755713 Group-By Objectives in Probabilistic Verification (GOPro)
Volume
243
Article Number
11
Conference
CONCUR: Conference on Concurrency Theory
Conference Location
Warsaw, Poland
Conference Date
2022-09-13 – 2022-09-16
ISSN
IST-REx-ID

Cite this

Grover K, Kretinsky J, Meggendorfer T, Weininger M. Anytime guarantees for reachability in uncountable Markov decision processes. In: 33rd International Conference on Concurrency Theory . Vol 243. Schloss Dagstuhl - Leibniz-Zentrum für Informatik; 2022. doi:10.4230/LIPIcs.CONCUR.2022.11
Grover, K., Kretinsky, J., Meggendorfer, T., & Weininger, M. (2022). Anytime guarantees for reachability in uncountable Markov decision processes. In 33rd International Conference on Concurrency Theory (Vol. 243). Warsaw, Poland: Schloss Dagstuhl - Leibniz-Zentrum für Informatik. https://doi.org/10.4230/LIPIcs.CONCUR.2022.11
Grover, Kush, Jan Kretinsky, Tobias Meggendorfer, and Maimilian Weininger. “Anytime Guarantees for Reachability in Uncountable Markov Decision Processes.” In 33rd International Conference on Concurrency Theory , Vol. 243. Schloss Dagstuhl - Leibniz-Zentrum für Informatik, 2022. https://doi.org/10.4230/LIPIcs.CONCUR.2022.11.
K. Grover, J. Kretinsky, T. Meggendorfer, and M. Weininger, “Anytime guarantees for reachability in uncountable Markov decision processes,” in 33rd International Conference on Concurrency Theory , Warsaw, Poland, 2022, vol. 243.
Grover K, Kretinsky J, Meggendorfer T, Weininger M. 2022. Anytime guarantees for reachability in uncountable Markov decision processes. 33rd International Conference on Concurrency Theory . CONCUR: Conference on Concurrency Theory, LIPIcs, vol. 243, 11.
Grover, Kush, et al. “Anytime Guarantees for Reachability in Uncountable Markov Decision Processes.” 33rd International Conference on Concurrency Theory , vol. 243, 11, Schloss Dagstuhl - Leibniz-Zentrum für Informatik, 2022, doi:10.4230/LIPIcs.CONCUR.2022.11.
All files available under the following license(s):
Creative Commons Attribution 4.0 International Public License (CC-BY 4.0):
Main File(s)
File Name
Access Level
OA Open Access
Date Uploaded
2023-09-26
MD5 Checksum
e282e43d3ae0ba6e067b72f4583e13c0


Export

Marked Publications

Open Data ISTA Research Explorer

Sources

arXiv 2008.04824

Search this title in

Google Scholar