Real-Time Relighting of Human Faces with a Low-Cost Setup

Video-streaming services usually feature post-processing effects to replace the background. However, these often yield inconsistent lighting. Machine-learning-based relighting methods can address this problem, but, at real-time rates, are restricted to a low resolution and can result in an unrealistic skin appearance. Physically-based rendering techniques require complex skin models that can only be acquired using specialised equipment. Our method is lightweight and uses only a standard smartphone. By correcting imperfections during capture, we extract a convincing physically-based skin model. In combination with suitable acceleration techniques, we achieve real-time rates on commodity hardware.

@InProceedings { MUEM22,
  author       = "Macek, Nejc and Usta, Baran and Eisemann, Elmar and Marroquim, Ricardo",
  title        = "Real-Time Relighting of Human Faces with a Low-Cost Setup",
  booktitle    = "Proceedings of the ACM in Computer Graphics and Interactive Techniques (PACM CGIT)",
  volume       = "5",
  month        = "may",
  year         = "2022",
  keywords     = "Relighting, reflectance function, face capture",
  url          = "http://graphics.tudelft.nl/Publications-new/2022/MUEM22"
}