---
_id: '14488'
abstract:
- lang: eng
text: 'Portrait viewpoint and illumination editing is an important problem with
several applications in VR/AR, movies, and photography. Comprehensive knowledge
of geometry and illumination is critical for obtaining photorealistic results.
Current methods are unable to explicitly model in 3D while handling both viewpoint
and illumination editing from a single image. In this paper, we propose VoRF,
a novel approach that can take even a single portrait image as input and relight
human heads under novel illuminations that can be viewed from arbitrary viewpoints.
VoRF represents a human head as a continuous volumetric field and learns a prior
model of human heads using a coordinate-based MLP with individual latent spaces
for identity and illumination. The prior model is learned in an auto-decoder manner
over a diverse class of head shapes and appearances, allowing VoRF to generalize
to novel test identities from a single input image. Additionally, VoRF has a reflectance
MLP that uses the intermediate features of the prior model for rendering One-Light-at-A-Time
(OLAT) images under novel views. We synthesize novel illuminations by combining
these OLAT images with target environment maps. Qualitative and quantitative evaluations
demonstrate the effectiveness of VoRF for relighting and novel view synthesis,
even when applied to unseen subjects under uncontrolled illumination. This work
is an extension of Rao et al. (VoRF: Volumetric Relightable Faces 2022). We provide
extensive evaluation and ablative studies of our model and also provide an application,
where any face can be relighted using textual input.'
acknowledgement: Open Access funding enabled and organized by Projekt DEAL.
article_processing_charge: Yes (via OA deal)
article_type: original
author:
- first_name: Pramod
full_name: Rao, Pramod
last_name: Rao
- first_name: B. R.
full_name: Mallikarjun, B. R.
last_name: Mallikarjun
- first_name: Gereon
full_name: Fox, Gereon
last_name: Fox
- first_name: Tim
full_name: Weyrich, Tim
last_name: Weyrich
- first_name: Bernd
full_name: Bickel, Bernd
id: 49876194-F248-11E8-B48F-1D18A9856A87
last_name: Bickel
orcid: 0000-0001-6511-9385
- first_name: Hanspeter
full_name: Pfister, Hanspeter
last_name: Pfister
- first_name: Wojciech
full_name: Matusik, Wojciech
last_name: Matusik
- first_name: Fangneng
full_name: Zhan, Fangneng
last_name: Zhan
- first_name: Ayush
full_name: Tewari, Ayush
last_name: Tewari
- first_name: Christian
full_name: Theobalt, Christian
last_name: Theobalt
- first_name: Mohamed
full_name: Elgharib, Mohamed
last_name: Elgharib
citation:
ama: Rao P, Mallikarjun BR, Fox G, et al. A deeper analysis of volumetric relightiable
faces. International Journal of Computer Vision. 2023. doi:10.1007/s11263-023-01899-3
apa: Rao, P., Mallikarjun, B. R., Fox, G., Weyrich, T., Bickel, B., Pfister, H.,
… Elgharib, M. (2023). A deeper analysis of volumetric relightiable faces. International
Journal of Computer Vision. Springer Nature. https://doi.org/10.1007/s11263-023-01899-3
chicago: Rao, Pramod, B. R. Mallikarjun, Gereon Fox, Tim Weyrich, Bernd Bickel,
Hanspeter Pfister, Wojciech Matusik, et al. “A Deeper Analysis of Volumetric Relightiable
Faces.” International Journal of Computer Vision. Springer Nature, 2023.
https://doi.org/10.1007/s11263-023-01899-3.
ieee: P. Rao et al., “A deeper analysis of volumetric relightiable faces,”
International Journal of Computer Vision. Springer Nature, 2023.
ista: Rao P, Mallikarjun BR, Fox G, Weyrich T, Bickel B, Pfister H, Matusik W, Zhan
F, Tewari A, Theobalt C, Elgharib M. 2023. A deeper analysis of volumetric relightiable
faces. International Journal of Computer Vision.
mla: Rao, Pramod, et al. “A Deeper Analysis of Volumetric Relightiable Faces.” International
Journal of Computer Vision, Springer Nature, 2023, doi:10.1007/s11263-023-01899-3.
short: P. Rao, B.R. Mallikarjun, G. Fox, T. Weyrich, B. Bickel, H. Pfister, W. Matusik,
F. Zhan, A. Tewari, C. Theobalt, M. Elgharib, International Journal of Computer
Vision (2023).
date_created: 2023-11-05T23:00:54Z
date_published: 2023-10-31T00:00:00Z
date_updated: 2023-11-06T08:52:30Z
day: '31'
department:
- _id: BeBi
doi: 10.1007/s11263-023-01899-3
language:
- iso: eng
main_file_link:
- open_access: '1'
url: https://doi.org/10.1007/s11263-023-01899-3
month: '10'
oa: 1
oa_version: Published Version
publication: International Journal of Computer Vision
publication_identifier:
eissn:
- 1573-1405
issn:
- 0920-5691
publication_status: epub_ahead
publisher: Springer Nature
quality_controlled: '1'
scopus_import: '1'
status: public
title: A deeper analysis of volumetric relightiable faces
type: journal_article
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
year: '2023'
...
---
_id: '6944'
abstract:
- lang: eng
text: 'We study the problem of automatically detecting if a given multi-class classifier
operates outside of its specifications (out-of-specs), i.e. on input data from
a different distribution than what it was trained for. This is an important problem
to solve on the road towards creating reliable computer vision systems for real-world
applications, because the quality of a classifier’s predictions cannot be guaranteed
if it operates out-of-specs. Previously proposed methods for out-of-specs detection
make decisions on the level of single inputs. This, however, is insufficient to
achieve low false positive rate and high false negative rates at the same time.
In this work, we describe a new procedure named KS(conf), based on statistical
reasoning. Its main component is a classical Kolmogorov–Smirnov test that is applied
to the set of predicted confidence values for batches of samples. Working with
batches instead of single samples allows increasing the true positive rate without
negatively affecting the false positive rate, thereby overcoming a crucial limitation
of single sample tests. We show by extensive experiments using a variety of convolutional
network architectures and datasets that KS(conf) reliably detects out-of-specs
situations even under conditions where other tests fail. It furthermore has a
number of properties that make it an excellent candidate for practical deployment:
it is easy to implement, adds almost no overhead to the system, works with any
classifier that outputs confidence scores, and requires no a priori knowledge
about how the data distribution could change.'
article_processing_charge: Yes (via OA deal)
article_type: original
author:
- first_name: Rémy
full_name: Sun, Rémy
last_name: Sun
- first_name: Christoph
full_name: Lampert, Christoph
id: 40C20FD2-F248-11E8-B48F-1D18A9856A87
last_name: Lampert
orcid: 0000-0001-8622-7887
citation:
ama: 'Sun R, Lampert C. KS(conf): A light-weight test if a multiclass classifier
operates outside of its specifications. International Journal of Computer Vision.
2020;128(4):970-995. doi:10.1007/s11263-019-01232-x'
apa: 'Sun, R., & Lampert, C. (2020). KS(conf): A light-weight test if a multiclass
classifier operates outside of its specifications. International Journal of
Computer Vision. Springer Nature. https://doi.org/10.1007/s11263-019-01232-x'
chicago: 'Sun, Rémy, and Christoph Lampert. “KS(Conf): A Light-Weight Test If a
Multiclass Classifier Operates Outside of Its Specifications.” International
Journal of Computer Vision. Springer Nature, 2020. https://doi.org/10.1007/s11263-019-01232-x.'
ieee: 'R. Sun and C. Lampert, “KS(conf): A light-weight test if a multiclass classifier
operates outside of its specifications,” International Journal of Computer
Vision, vol. 128, no. 4. Springer Nature, pp. 970–995, 2020.'
ista: 'Sun R, Lampert C. 2020. KS(conf): A light-weight test if a multiclass classifier
operates outside of its specifications. International Journal of Computer Vision.
128(4), 970–995.'
mla: 'Sun, Rémy, and Christoph Lampert. “KS(Conf): A Light-Weight Test If a Multiclass
Classifier Operates Outside of Its Specifications.” International Journal of
Computer Vision, vol. 128, no. 4, Springer Nature, 2020, pp. 970–95, doi:10.1007/s11263-019-01232-x.'
short: R. Sun, C. Lampert, International Journal of Computer Vision 128 (2020) 970–995.
date_created: 2019-10-14T09:14:28Z
date_published: 2020-04-01T00:00:00Z
date_updated: 2024-02-22T14:57:30Z
day: '01'
ddc:
- '004'
department:
- _id: ChLa
doi: 10.1007/s11263-019-01232-x
ec_funded: 1
external_id:
isi:
- '000494406800001'
file:
- access_level: open_access
checksum: 155e63edf664dcacb3bdc1c2223e606f
content_type: application/pdf
creator: dernst
date_created: 2019-11-26T10:30:02Z
date_updated: 2020-07-14T12:47:45Z
file_id: '7110'
file_name: 2019_IJCV_Sun.pdf
file_size: 1715072
relation: main_file
file_date_updated: 2020-07-14T12:47:45Z
has_accepted_license: '1'
intvolume: ' 128'
isi: 1
issue: '4'
language:
- iso: eng
license: https://creativecommons.org/licenses/by/4.0/
month: '04'
oa: 1
oa_version: Published Version
page: 970-995
project:
- _id: 2532554C-B435-11E9-9278-68D0E5697425
call_identifier: FP7
grant_number: '308036'
name: Lifelong Learning of Visual Scene Understanding
- _id: B67AFEDC-15C9-11EA-A837-991A96BB2854
name: IST Austria Open Access Fund
publication: International Journal of Computer Vision
publication_identifier:
eissn:
- 1573-1405
issn:
- 0920-5691
publication_status: published
publisher: Springer Nature
quality_controlled: '1'
related_material:
link:
- relation: erratum
url: https://doi.org/10.1007/s11263-019-01262-5
record:
- id: '6482'
relation: earlier_version
status: public
scopus_import: '1'
status: public
title: 'KS(conf): A light-weight test if a multiclass classifier operates outside
of its specifications'
tmp:
image: /images/cc_by.png
legal_code_url: https://creativecommons.org/licenses/by/4.0/legalcode
name: Creative Commons Attribution 4.0 International Public License (CC-BY 4.0)
short: CC BY (4.0)
type: journal_article
user_id: 3E5EF7F0-F248-11E8-B48F-1D18A9856A87
volume: 128
year: '2020'
...
---
_id: '6952'
abstract:
- lang: eng
text: 'We present a unified framework tackling two problems: class-specific 3D reconstruction
from a single image, and generation of new 3D shape samples. These tasks have
received considerable attention recently; however, most existing approaches rely
on 3D supervision, annotation of 2D images with keypoints or poses, and/or training
with multiple views of each object instance. Our framework is very general: it
can be trained in similar settings to existing approaches, while also supporting
weaker supervision. Importantly, it can be trained purely from 2D images, without
pose annotations, and with only a single view per instance. We employ meshes as
an output representation, instead of voxels used in most prior work. This allows
us to reason over lighting parameters and exploit shading information during training,
which previous 2D-supervised methods cannot. Thus, our method can learn to generate
and reconstruct concave object classes. We evaluate our approach in various settings,
showing that: (i) it learns to disentangle shape from pose and lighting; (ii)
using shading in the loss improves performance compared to just silhouettes; (iii)
when using a standard single white light, our model outperforms state-of-the-art
2D-supervised methods, both with and without pose supervision, thanks to exploiting
shading cues; (iv) performance improves further when using multiple coloured lights,
even approaching that of state-of-the-art 3D-supervised methods; (v) shapes produced
by our model capture smooth surfaces and fine details better than voxel-based
approaches; and (vi) our approach supports concave classes such as bathtubs and
sofas, which methods based on silhouettes cannot learn.'
acknowledgement: Open access funding provided by Institute of Science and Technology
(IST Austria).
article_processing_charge: Yes (via OA deal)
article_type: original
arxiv: 1
author:
- first_name: Paul M
full_name: Henderson, Paul M
id: 13C09E74-18D9-11E9-8878-32CFE5697425
last_name: Henderson
orcid: 0000-0002-5198-7445
- first_name: Vittorio
full_name: Ferrari, Vittorio
last_name: Ferrari
citation:
ama: Henderson PM, Ferrari V. Learning single-image 3D reconstruction by generative
modelling of shape, pose and shading. International Journal of Computer Vision.
2020;128:835-854. doi:10.1007/s11263-019-01219-8
apa: Henderson, P. M., & Ferrari, V. (2020). Learning single-image 3D reconstruction
by generative modelling of shape, pose and shading. International Journal of
Computer Vision. Springer Nature. https://doi.org/10.1007/s11263-019-01219-8
chicago: Henderson, Paul M, and Vittorio Ferrari. “Learning Single-Image 3D Reconstruction
by Generative Modelling of Shape, Pose and Shading.” International Journal
of Computer Vision. Springer Nature, 2020. https://doi.org/10.1007/s11263-019-01219-8.
ieee: P. M. Henderson and V. Ferrari, “Learning single-image 3D reconstruction by
generative modelling of shape, pose and shading,” International Journal of
Computer Vision, vol. 128. Springer Nature, pp. 835–854, 2020.
ista: Henderson PM, Ferrari V. 2020. Learning single-image 3D reconstruction by
generative modelling of shape, pose and shading. International Journal of Computer
Vision. 128, 835–854.
mla: Henderson, Paul M., and Vittorio Ferrari. “Learning Single-Image 3D Reconstruction
by Generative Modelling of Shape, Pose and Shading.” International Journal
of Computer Vision, vol. 128, Springer Nature, 2020, pp. 835–54, doi:10.1007/s11263-019-01219-8.
short: P.M. Henderson, V. Ferrari, International Journal of Computer Vision 128
(2020) 835–854.
date_created: 2019-10-17T13:38:20Z
date_published: 2020-04-01T00:00:00Z
date_updated: 2023-08-17T14:01:16Z
day: '01'
ddc:
- '004'
department:
- _id: ChLa
doi: 10.1007/s11263-019-01219-8
external_id:
arxiv:
- '1901.06447'
isi:
- '000491042100002'
file:
- access_level: open_access
checksum: a0f05dd4f5f64e4f713d8d9d4b5b1e3f
content_type: application/pdf
creator: dernst
date_created: 2019-10-25T10:28:29Z
date_updated: 2020-07-14T12:47:46Z
file_id: '6973'
file_name: 2019_CompVision_Henderson.pdf
file_size: 2243134
relation: main_file
file_date_updated: 2020-07-14T12:47:46Z
has_accepted_license: '1'
intvolume: ' 128'
isi: 1
language:
- iso: eng
month: '04'
oa: 1
oa_version: Published Version
page: 835-854
project:
- _id: B67AFEDC-15C9-11EA-A837-991A96BB2854
name: IST Austria Open Access Fund
publication: International Journal of Computer Vision
publication_identifier:
eissn:
- 1573-1405
issn:
- 0920-5691
publication_status: published
publisher: Springer Nature
quality_controlled: '1'
scopus_import: '1'
status: public
title: Learning single-image 3D reconstruction by generative modelling of shape, pose
and shading
tmp:
image: /images/cc_by.png
legal_code_url: https://creativecommons.org/licenses/by/4.0/legalcode
name: Creative Commons Attribution 4.0 International Public License (CC-BY 4.0)
short: CC BY (4.0)
type: journal_article
user_id: 4359f0d1-fa6c-11eb-b949-802e58b17ae8
volume: 128
year: '2020'
...