{"abstract":[{"lang":"eng","text":"Photorealistic editing of head portraits is a challenging task as humans are very sensitive to inconsistencies in faces. We present an approach for high-quality intuitive editing of the camera viewpoint and scene illumination (parameterised with an environment map) in a portrait image. This requires our method to capture and control the full reflectance field of the person in the image. Most editing approaches rely on supervised learning using training data captured with setups such as light and camera stages. Such datasets are expensive to acquire, not readily available and do not capture all the rich variations of in-the-wild portrait images. In addition, most supervised approaches only focus on relighting, and do not allow camera viewpoint editing. Thus, they only capture and control a subset of the reflectance field. Recently, portrait editing has been demonstrated by operating in the generative model space of StyleGAN. While such approaches do not require direct supervision, there is a significant loss of quality when compared to the supervised approaches. In this paper, we present a method which learns from limited supervised training data. The training images only include people in a fixed neutral expression with eyes closed, without much hair or background variations. Each person is captured under 150 one-light-at-a-time conditions and under 8 camera poses. Instead of training directly in the image space, we design a supervised problem which learns transformations in the latent space of StyleGAN. This combines the best of supervised learning and generative adversarial modeling. We show that the StyleGAN prior allows for generalisation to different expressions, hairstyles and backgrounds. This produces high-quality photorealistic results for in-the-wild images and significantly outperforms existing methods. Our approach can edit the illumination and pose simultaneously, and runs at interactive rates."}],"department":[{"_id":"BeBi"}],"issue":"4","date_published":"2021-08-01T00:00:00Z","publication_status":"published","tmp":{"image":"/images/cc_by.png","name":"Creative Commons Attribution 4.0 International Public License (CC-BY 4.0)","short":"CC BY (4.0)","legal_code_url":"https://creativecommons.org/licenses/by/4.0/legalcode"},"date_updated":"2023-08-10T14:25:08Z","month":"08","article_processing_charge":"Yes (in subscription journal)","day":"01","file":[{"success":1,"file_id":"9834","date_created":"2021-08-09T11:41:50Z","file_size":49840741,"date_updated":"2021-08-09T11:41:50Z","content_type":"application/pdf","access_level":"open_access","creator":"asandaue","file_name":"2021_ACMTransactionsOnGraphics_Mallikarjun.pdf","checksum":"51b61b7e5c175e2d7ed8fa3b35f7525a","relation":"main_file"}],"type":"journal_article","status":"public","quality_controlled":"1","title":"PhotoApp: Photorealistic appearance editing of head portraits","scopus_import":"1","publisher":"Association for Computing Machinery","user_id":"4359f0d1-fa6c-11eb-b949-802e58b17ae8","date_created":"2021-08-08T22:01:27Z","oa":1,"oa_version":"Published Version","language":[{"iso":"eng"}],"_id":"9819","article_number":"44","article_type":"original","citation":{"mla":"Mallikarjun, B. R., et al. “PhotoApp: Photorealistic Appearance Editing of Head Portraits.” <i>ACM Transactions on Graphics</i>, vol. 40, no. 4, 44, Association for Computing Machinery, 2021, doi:<a href=\"https://doi.org/10.1145/3450626.3459765\">10.1145/3450626.3459765</a>.","ieee":"B. R. Mallikarjun <i>et al.</i>, “PhotoApp: Photorealistic appearance editing of head portraits,” <i>ACM Transactions on Graphics</i>, vol. 40, no. 4. Association for Computing Machinery, 2021.","ama":"Mallikarjun BR, Tewari A, Dib A, et al. PhotoApp: Photorealistic appearance editing of head portraits. <i>ACM Transactions on Graphics</i>. 2021;40(4). doi:<a href=\"https://doi.org/10.1145/3450626.3459765\">10.1145/3450626.3459765</a>","apa":"Mallikarjun, B. R., Tewari, A., Dib, A., Weyrich, T., Bickel, B., Seidel, H. P., … Theobalt, C. (2021). PhotoApp: Photorealistic appearance editing of head portraits. <i>ACM Transactions on Graphics</i>. Association for Computing Machinery. <a href=\"https://doi.org/10.1145/3450626.3459765\">https://doi.org/10.1145/3450626.3459765</a>","ista":"Mallikarjun BR, Tewari A, Dib A, Weyrich T, Bickel B, Seidel HP, Pfister H, Matusik W, Chevallier L, Elgharib MA, Theobalt C. 2021. PhotoApp: Photorealistic appearance editing of head portraits. ACM Transactions on Graphics. 40(4), 44.","chicago":"Mallikarjun, B. R., Ayush Tewari, Abdallah Dib, Tim Weyrich, Bernd Bickel, Hans Peter Seidel, Hanspeter Pfister, et al. “PhotoApp: Photorealistic Appearance Editing of Head Portraits.” <i>ACM Transactions on Graphics</i>. Association for Computing Machinery, 2021. <a href=\"https://doi.org/10.1145/3450626.3459765\">https://doi.org/10.1145/3450626.3459765</a>.","short":"B.R. Mallikarjun, A. Tewari, A. Dib, T. Weyrich, B. Bickel, H.P. Seidel, H. Pfister, W. Matusik, L. Chevallier, M.A. Elgharib, C. Theobalt, ACM Transactions on Graphics 40 (2021)."},"publication_identifier":{"issn":["07300301"],"eissn":["15577368"]},"external_id":{"arxiv":["2103.07658"],"isi":["000674930900011"]},"doi":"10.1145/3450626.3459765","has_accepted_license":"1","author":[{"full_name":"Mallikarjun, B. R.","first_name":"B. R.","last_name":"Mallikarjun"},{"full_name":"Tewari, Ayush","first_name":"Ayush","last_name":"Tewari"},{"full_name":"Dib, Abdallah","first_name":"Abdallah","last_name":"Dib"},{"last_name":"Weyrich","first_name":"Tim","full_name":"Weyrich, Tim"},{"orcid":"0000-0001-6511-9385","id":"49876194-F248-11E8-B48F-1D18A9856A87","full_name":"Bickel, Bernd","first_name":"Bernd","last_name":"Bickel"},{"full_name":"Seidel, Hans Peter","first_name":"Hans Peter","last_name":"Seidel"},{"first_name":"Hanspeter","last_name":"Pfister","full_name":"Pfister, Hanspeter"},{"last_name":"Matusik","first_name":"Wojciech","full_name":"Matusik, Wojciech"},{"full_name":"Chevallier, Louis","last_name":"Chevallier","first_name":"Louis"},{"full_name":"Elgharib, Mohamed A.","last_name":"Elgharib","first_name":"Mohamed A."},{"last_name":"Theobalt","first_name":"Christian","full_name":"Theobalt, Christian"}],"publication":"ACM Transactions on Graphics","ddc":["000"],"file_date_updated":"2021-08-09T11:41:50Z","isi":1,"acknowledgement":"This work was supported by the ERC Consolidator Grant 4DReply (770784). We also acknowledge support from Technicolor and InterDigital. We thank Tiancheng Sun for kindly helping us with the comparisons with Sun et al. [2019].","year":"2021","volume":40,"intvolume":"        40"}