{"publication_identifier":{"isbn":["9781713871088"]},"type":"conference","extern":"1","volume":35,"date_updated":"2023-09-06T10:34:43Z","abstract":[{"text":"Since out-of-distribution generalization is a generally ill-posed problem, various proxy targets (e.g., calibration, adversarial robustness, algorithmic corruptions, invariance across shifts) were studied across different research programs resulting in different recommendations. While sharing the same aspirational goal, these approaches have never been tested under the same\r\nexperimental conditions on real data. In this paper, we take a unified view of previous work, highlighting message discrepancies that we address empirically, and providing recommendations on how to measure the robustness of a model and how to improve it. To this end, we collect 172 publicly available dataset pairs for training and out-of-distribution evaluation of accuracy, calibration error, adversarial attacks, environment invariance, and synthetic corruptions. We fine-tune over 31k networks, from nine different architectures in the many- and\r\nfew-shot setting. Our findings confirm that in- and out-of-distribution accuracies tend to increase jointly, but show that their relation is largely dataset-dependent, and in general more nuanced and more complex than posited by previous, smaller scale studies.","lang":"eng"}],"publication_status":"published","publication":"36th Conference on Neural Information Processing Systems","alternative_title":["Advances in Neural Information Processing Systems"],"conference":{"end_date":"2022-12-09","start_date":"2022-11-28","location":"New Orleans, LA, United States","name":"NeurIPS: Neural Information Processing Systems"},"author":[{"first_name":"Florian","full_name":"Wenzel, Florian","last_name":"Wenzel"},{"first_name":"Andrea","full_name":"Dittadi, Andrea","last_name":"Dittadi"},{"full_name":"Gehler, Peter Vincent","first_name":"Peter Vincent","last_name":"Gehler"},{"full_name":"Carl-Johann Simon-Gabriel, Carl-Johann Simon-Gabriel","first_name":"Carl-Johann Simon-Gabriel","last_name":"Carl-Johann Simon-Gabriel"},{"first_name":"Max","full_name":"Horn, Max","last_name":"Horn"},{"last_name":"Zietlow","full_name":"Zietlow, Dominik","first_name":"Dominik"},{"last_name":"Kernert","first_name":"David","full_name":"Kernert, David"},{"full_name":"Russell, Chris","first_name":"Chris","last_name":"Russell"},{"first_name":"Thomas","full_name":"Brox, Thomas","last_name":"Brox"},{"last_name":"Schiele","full_name":"Schiele, Bernt","first_name":"Bernt"},{"first_name":"Bernhard","full_name":"Schölkopf, Bernhard","last_name":"Schölkopf"},{"first_name":"Francesco","full_name":"Locatello, Francesco","orcid":"0000-0002-4850-0683","last_name":"Locatello","id":"26cfd52f-2483-11ee-8040-88983bcc06d4"}],"_id":"14173","date_created":"2023-08-22T14:01:13Z","month":"12","citation":{"ista":"Wenzel F, Dittadi A, Gehler PV, Carl-Johann Simon-Gabriel C-JS-G, Horn M, Zietlow D, Kernert D, Russell C, Brox T, Schiele B, Schölkopf B, Locatello F. 2022. Assaying out-of-distribution generalization in transfer learning. 36th Conference on Neural Information Processing Systems. NeurIPS: Neural Information Processing Systems, Advances in Neural Information Processing Systems, vol. 35, 7181–7198.","chicago":"Wenzel, Florian, Andrea Dittadi, Peter Vincent Gehler, Carl-Johann Simon-Gabriel Carl-Johann Simon-Gabriel, Max Horn, Dominik Zietlow, David Kernert, et al. “Assaying Out-of-Distribution Generalization in Transfer Learning.” In <i>36th Conference on Neural Information Processing Systems</i>, 35:7181–98. Neural Information Processing Systems Foundation, 2022.","apa":"Wenzel, F., Dittadi, A., Gehler, P. V., Carl-Johann Simon-Gabriel, C.-J. S.-G., Horn, M., Zietlow, D., … Locatello, F. (2022). Assaying out-of-distribution generalization in transfer learning. In <i>36th Conference on Neural Information Processing Systems</i> (Vol. 35, pp. 7181–7198). New Orleans, LA, United States: Neural Information Processing Systems Foundation.","short":"F. Wenzel, A. Dittadi, P.V. Gehler, C.-J.S.-G. Carl-Johann Simon-Gabriel, M. Horn, D. Zietlow, D. Kernert, C. Russell, T. Brox, B. Schiele, B. Schölkopf, F. Locatello, in:, 36th Conference on Neural Information Processing Systems, Neural Information Processing Systems Foundation, 2022, pp. 7181–7198.","mla":"Wenzel, Florian, et al. “Assaying Out-of-Distribution Generalization in Transfer Learning.” <i>36th Conference on Neural Information Processing Systems</i>, vol. 35, Neural Information Processing Systems Foundation, 2022, pp. 7181–98.","ama":"Wenzel F, Dittadi A, Gehler PV, et al. Assaying out-of-distribution generalization in transfer learning. In: <i>36th Conference on Neural Information Processing Systems</i>. Vol 35. Neural Information Processing Systems Foundation; 2022:7181-7198.","ieee":"F. Wenzel <i>et al.</i>, “Assaying out-of-distribution generalization in transfer learning,” in <i>36th Conference on Neural Information Processing Systems</i>, New Orleans, LA, United States, 2022, vol. 35, pp. 7181–7198."},"article_processing_charge":"No","department":[{"_id":"FrLo"}],"oa":1,"user_id":"2DF688A6-F248-11E8-B48F-1D18A9856A87","external_id":{"arxiv":["2207.09239"]},"page":"7181-7198","status":"public","language":[{"iso":"eng"}],"intvolume":"        35","oa_version":"Preprint","title":"Assaying out-of-distribution generalization in transfer learning","quality_controlled":"1","scopus_import":"1","main_file_link":[{"open_access":"1","url":"https://arxiv.org/abs/2207.09239"}],"date_published":"2022-12-15T00:00:00Z","day":"15","publisher":"Neural Information Processing Systems Foundation","year":"2022"}