{"department":[{"_id":"GradSch"},{"_id":"MaMo"}],"main_file_link":[{"open_access":"1","url":"https://arxiv.org/abs/2302.01629"}],"abstract":[{"text":"Machine learning models are vulnerable to adversarial perturbations, and a thought-provoking paper by Bubeck and Sellke has analyzed this phenomenon through the lens of over-parameterization: interpolating smoothly the data requires significantly more parameters than simply memorizing it. However, this \"universal\" law provides only a necessary condition for robustness, and it is unable to discriminate between models. In this paper, we address these gaps by focusing on empirical risk minimization in two prototypical settings, namely, random features and the neural tangent kernel (NTK). We prove that, for random features, the model is not robust for any degree of over-parameterization, even when the necessary condition coming from the universal law of robustness is satisfied. In contrast, for even activations, the NTK model meets the universal lower bound, and it is robust as soon as the necessary condition on over-parameterization is fulfilled. This also addresses a conjecture in prior work by Bubeck, Li and Nagaraj. Our analysis decouples the effect of the kernel of the model from an \"interaction matrix\", which describes the interaction with the test data and captures the effect of the activation. Our theoretical results are corroborated by numerical evidence on both synthetic and standard datasets (MNIST, CIFAR-10).","lang":"eng"}],"date_published":"2023-10-27T00:00:00Z","publication_status":"published","article_processing_charge":"No","month":"10","date_updated":"2024-09-10T13:03:19Z","type":"conference","day":"27","alternative_title":["PMLR"],"quality_controlled":"1","title":"Beyond the universal law of robustness: Sharper laws for random features and neural tangent kernels","status":"public","related_material":{"link":[{"url":"https://github.com/simone-bombari/beyond-universal-robustness","relation":"software"}]},"publisher":"ML Research Press","oa_version":"Preprint","oa":1,"date_created":"2023-04-23T16:11:03Z","user_id":"2DF688A6-F248-11E8-B48F-1D18A9856A87","_id":"12859","language":[{"iso":"eng"}],"conference":{"end_date":"2023-07-29","location":"Honolulu, HI, United States","start_date":"2023-07-23","name":"ICML: International Conference on Machine Learning"},"citation":{"ama":"Bombari S, Kiyani S, Mondelli M. Beyond the universal law of robustness: Sharper laws for random features and neural tangent kernels. In: <i>Proceedings of the 40th International Conference on Machine Learning</i>. Vol 202. ML Research Press; 2023:2738-2776.","ista":"Bombari S, Kiyani S, Mondelli M. 2023. Beyond the universal law of robustness: Sharper laws for random features and neural tangent kernels. Proceedings of the 40th International Conference on Machine Learning. ICML: International Conference on Machine Learning, PMLR, vol. 202, 2738–2776.","apa":"Bombari, S., Kiyani, S., &#38; Mondelli, M. (2023). Beyond the universal law of robustness: Sharper laws for random features and neural tangent kernels. In <i>Proceedings of the 40th International Conference on Machine Learning</i> (Vol. 202, pp. 2738–2776). Honolulu, HI, United States: ML Research Press.","chicago":"Bombari, Simone, Shayan Kiyani, and Marco Mondelli. “Beyond the Universal Law of Robustness: Sharper Laws for Random Features and Neural Tangent Kernels.” In <i>Proceedings of the 40th International Conference on Machine Learning</i>, 202:2738–76. ML Research Press, 2023.","short":"S. Bombari, S. Kiyani, M. Mondelli, in:, Proceedings of the 40th International Conference on Machine Learning, ML Research Press, 2023, pp. 2738–2776.","mla":"Bombari, Simone, et al. “Beyond the Universal Law of Robustness: Sharper Laws for Random Features and Neural Tangent Kernels.” <i>Proceedings of the 40th International Conference on Machine Learning</i>, vol. 202, ML Research Press, 2023, pp. 2738–76.","ieee":"S. Bombari, S. Kiyani, and M. Mondelli, “Beyond the universal law of robustness: Sharper laws for random features and neural tangent kernels,” in <i>Proceedings of the 40th International Conference on Machine Learning</i>, Honolulu, HI, United States, 2023, vol. 202, pp. 2738–2776."},"external_id":{"arxiv":["2302.01629"]},"publication":"Proceedings of the 40th International Conference on Machine Learning","author":[{"first_name":"Simone","last_name":"Bombari","id":"ca726dda-de17-11ea-bc14-f9da834f63aa","full_name":"Bombari, Simone"},{"full_name":"Kiyani, Shayan","id":"f5a2b424-e339-11ed-8435-ff3b4fe70cf8","first_name":"Shayan","last_name":"Kiyani"},{"full_name":"Mondelli, Marco","id":"27EB676C-8706-11E9-9510-7717E6697425","first_name":"Marco","last_name":"Mondelli","orcid":"0000-0002-3242-7020"}],"page":"2738-2776","project":[{"name":"Prix Lopez-Loretta 2019 - Marco Mondelli","_id":"059876FA-7A3F-11EA-A408-12923DDC885E"}],"acknowledgement":"Simone Bombari and Marco Mondelli were partially supported by the 2019 Lopez-Loreta prize, and\r\nthe authors would like to thank Hamed Hassani for helpful discussions.\r\n","year":"2023","intvolume":"       202","volume":202}