{"quality_controlled":"1","title":"Dynamic inference with neural interpreters","status":"public","year":"2021","intvolume":"        34","oa_version":"Preprint","oa":1,"date_created":"2023-08-22T14:04:55Z","user_id":"2DF688A6-F248-11E8-B48F-1D18A9856A87","volume":34,"_id":"14180","language":[{"iso":"eng"}],"department":[{"_id":"FrLo"}],"abstract":[{"lang":"eng","text":"Modern neural network architectures can leverage large amounts of data to generalize well within the training distribution. However, they are less capable of systematic generalization to data drawn from unseen but related distributions, a feat that is hypothesized to require compositional reasoning and reuse of knowledge. In this work, we present Neural Interpreters, an architecture that factorizes inference in a self-attention network as a system of modules, which we call \\emph{functions}. Inputs to the model are routed through a sequence of functions in a way that is end-to-end learned. The proposed architecture can flexibly compose computation along width and depth, and lends itself well to capacity extension after training. To demonstrate the versatility of Neural Interpreters, we evaluate it in two distinct settings: image classification and visual abstract reasoning on Raven Progressive Matrices. In the former, we show that Neural Interpreters perform on par with the vision transformer using fewer parameters, while being transferrable to a new task in a sample efficient manner. In the latter, we find that Neural Interpreters are competitive with respect to the state-of-the-art in terms of systematic generalization. "}],"main_file_link":[{"open_access":"1","url":"https://doi.org/10.48550/arXiv.2110.06399"}],"date_published":"2021-10-12T00:00:00Z","citation":{"short":"N. Rahaman, M.W. Gondal, S. Joshi, P. Gehler, Y. Bengio, F. Locatello, B. Schölkopf, in:, Advances in Neural Information Processing Systems, 2021, pp. 10985–10998.","ama":"Rahaman N, Gondal MW, Joshi S, et al. Dynamic inference with neural interpreters. In: <i>Advances in Neural Information Processing Systems</i>. Vol 34. ; 2021:10985-10998.","apa":"Rahaman, N., Gondal, M. W., Joshi, S., Gehler, P., Bengio, Y., Locatello, F., &#38; Schölkopf, B. (2021). Dynamic inference with neural interpreters. In <i>Advances in Neural Information Processing Systems</i> (Vol. 34, pp. 10985–10998). Virtual.","chicago":"Rahaman, Nasim, Muhammad Waleed Gondal, Shruti Joshi, Peter Gehler, Yoshua Bengio, Francesco Locatello, and Bernhard Schölkopf. “Dynamic Inference with Neural Interpreters.” In <i>Advances in Neural Information Processing Systems</i>, 34:10985–98, 2021.","ista":"Rahaman N, Gondal MW, Joshi S, Gehler P, Bengio Y, Locatello F, Schölkopf B. 2021. Dynamic inference with neural interpreters. Advances in Neural Information Processing Systems. NeurIPS: Neural Information Processing Systems vol. 34, 10985–10998.","ieee":"N. Rahaman <i>et al.</i>, “Dynamic inference with neural interpreters,” in <i>Advances in Neural Information Processing Systems</i>, Virtual, 2021, vol. 34, pp. 10985–10998.","mla":"Rahaman, Nasim, et al. “Dynamic Inference with Neural Interpreters.” <i>Advances in Neural Information Processing Systems</i>, vol. 34, 2021, pp. 10985–98."},"conference":{"end_date":"2021-12-10","location":"Virtual","start_date":"2021-12-07","name":"NeurIPS: Neural Information Processing Systems"},"publication_status":"published","extern":"1","article_processing_charge":"No","month":"10","external_id":{"arxiv":["2110.06399"]},"date_updated":"2023-09-11T11:33:46Z","publication_identifier":{"isbn":["9781713845393"]},"type":"conference","author":[{"first_name":"Nasim","last_name":"Rahaman","full_name":"Rahaman, Nasim"},{"first_name":"Muhammad Waleed","last_name":"Gondal","full_name":"Gondal, Muhammad Waleed"},{"last_name":"Joshi","first_name":"Shruti","full_name":"Joshi, Shruti"},{"full_name":"Gehler, Peter","last_name":"Gehler","first_name":"Peter"},{"first_name":"Yoshua","last_name":"Bengio","full_name":"Bengio, Yoshua"},{"last_name":"Locatello","first_name":"Francesco","full_name":"Locatello, Francesco","id":"26cfd52f-2483-11ee-8040-88983bcc06d4","orcid":"0000-0002-4850-0683"},{"last_name":"Schölkopf","first_name":"Bernhard","full_name":"Schölkopf, Bernhard"}],"publication":"Advances in Neural Information Processing Systems","day":"12","page":"10985-10998"}