{"title":"Self-supervised amodal video object segmentation","conference":{"end_date":"2022-12-01","start_date":"2022-11-28","name":"NeurIPS: Neural Information Processing Systems","location":"New Orleans, LA, United States"},"doi":"10.48550/arXiv.2210.12733","author":[{"last_name":"Yao","first_name":"Jian","full_name":"Yao, Jian"},{"first_name":"Yuxin","full_name":"Hong, Yuxin","last_name":"Hong"},{"last_name":"Wang","full_name":"Wang, Chiyu","first_name":"Chiyu"},{"last_name":"Xiao","first_name":"Tianjun","full_name":"Xiao, Tianjun"},{"first_name":"Tong","full_name":"He, Tong","last_name":"He"},{"first_name":"Francesco","full_name":"Locatello, Francesco","orcid":"0000-0002-4850-0683","last_name":"Locatello","id":"26cfd52f-2483-11ee-8040-88983bcc06d4"},{"first_name":"David","full_name":"Wipf, David","last_name":"Wipf"},{"first_name":"Yanwei","full_name":"Fu, Yanwei","last_name":"Fu"},{"full_name":"Zhang, Zheng","first_name":"Zheng","last_name":"Zhang"}],"main_file_link":[{"open_access":"1","url":"https://doi.org/10.48550/arXiv.2210.12733"}],"_id":"14107","oa_version":"Preprint","abstract":[{"text":"Amodal perception requires inferring the full shape of an object that is partially occluded. This task is particularly challenging on two levels: (1) it requires more information than what is contained in the instant retina or imaging sensor, (2) it is difficult to obtain enough well-annotated amodal labels for supervision. To this end, this paper develops a new framework of\r\nSelf-supervised amodal Video object segmentation (SaVos). Our method efficiently leverages the visual information of video temporal sequences to infer the amodal mask of objects. The key intuition is that the occluded part of an object can be explained away if that part is visible in other frames, possibly deformed as long as the deformation can be reasonably learned.\r\nAccordingly, we derive a novel self-supervised learning paradigm that efficiently utilizes the visible object parts as the supervision to guide the training on videos. In addition to learning type prior to complete masks for known types, SaVos also learns the spatiotemporal prior, which is also useful for the amodal task and could generalize to unseen types. The proposed\r\nframework achieves the state-of-the-art performance on the synthetic amodal segmentation benchmark FISHBOWL and the real world benchmark KINS-Video-Car. Further, it lends itself well to being transferred to novel distributions using test-time adaptation, outperforming existing models even after the transfer to a new distribution.","lang":"eng"}],"publication_status":"published","publication":"36th Conference on Neural Information Processing Systems","type":"conference","extern":"1","date_updated":"2023-09-11T09:34:17Z","language":[{"iso":"eng"}],"external_id":{"arxiv":["2210.12733"]},"day":"23","status":"public","year":"2022","oa":1,"date_published":"2022-10-23T00:00:00Z","user_id":"2DF688A6-F248-11E8-B48F-1D18A9856A87","article_processing_charge":"No","department":[{"_id":"FrLo"}],"date_created":"2023-08-21T12:13:25Z","month":"10","citation":{"ieee":"J. Yao <i>et al.</i>, “Self-supervised amodal video object segmentation,” in <i>36th Conference on Neural Information Processing Systems</i>, New Orleans, LA, United States, 2022.","short":"J. Yao, Y. Hong, C. Wang, T. Xiao, T. He, F. Locatello, D. Wipf, Y. Fu, Z. Zhang, in:, 36th Conference on Neural Information Processing Systems, 2022.","mla":"Yao, Jian, et al. “Self-Supervised Amodal Video Object Segmentation.” <i>36th Conference on Neural Information Processing Systems</i>, 2022, doi:<a href=\"https://doi.org/10.48550/arXiv.2210.12733\">10.48550/arXiv.2210.12733</a>.","ama":"Yao J, Hong Y, Wang C, et al. Self-supervised amodal video object segmentation. In: <i>36th Conference on Neural Information Processing Systems</i>. ; 2022. doi:<a href=\"https://doi.org/10.48550/arXiv.2210.12733\">10.48550/arXiv.2210.12733</a>","apa":"Yao, J., Hong, Y., Wang, C., Xiao, T., He, T., Locatello, F., … Zhang, Z. (2022). Self-supervised amodal video object segmentation. In <i>36th Conference on Neural Information Processing Systems</i>. New Orleans, LA, United States. <a href=\"https://doi.org/10.48550/arXiv.2210.12733\">https://doi.org/10.48550/arXiv.2210.12733</a>","ista":"Yao J, Hong Y, Wang C, Xiao T, He T, Locatello F, Wipf D, Fu Y, Zhang Z. 2022. Self-supervised amodal video object segmentation. 36th Conference on Neural Information Processing Systems. NeurIPS: Neural Information Processing Systems.","chicago":"Yao, Jian, Yuxin Hong, Chiyu Wang, Tianjun Xiao, Tong He, Francesco Locatello, David Wipf, Yanwei Fu, and Zheng Zhang. “Self-Supervised Amodal Video Object Segmentation.” In <i>36th Conference on Neural Information Processing Systems</i>, 2022. <a href=\"https://doi.org/10.48550/arXiv.2210.12733\">https://doi.org/10.48550/arXiv.2210.12733</a>."}}