Computational lighting in video

Project

This project will explore how advances in Generative networks and deep-learning approaches to beautification and stylistic transfer, can be combined to artificially recreate the appearance of professional lighting. The system will use recordings from one or more viewpoints of a scene with an unknown lighting setup, and will generate a video of the same scene but which a viewer would judge to have been recorded under “three point lighting”. The developed technology is of interest as a potential extension to the capabilities of the Ed system currently being developed by BBC R&D.

The Ed system is an automated system for creating edited coverage of live events. Its inputs are high-resolution locked-off cameras that are deployed at venues with live audiences. The system then creates “virtual camera” views by cropping these raw camera feeds and cuts between them to produce output. In real-world deployments, the lighting configuration is often constrained to a suboptimal setup. The ability to synthesise more appealing data than what can practically be achieved, would improve the output of the Ed system

Program of research

The first 15 months will be based at the University of Surrey, and will focus on integrating deep style transfer with computational lighting and monocular scene understanding. The initial research will focus primarily on static environments captured at Surrey’s Audio-Visual lab. This period will also include extensive personal development and training opportunities covering technical and professional skills.

The following 18 months will be undertaken at the BBC North site in Manchester, and will explore the temporal aspects of the problem including temporal lighting consistency and correct shadowing for dynamic objects. This will use dynamic footage captured at the BBC site and consequently the majority of the data collection will also happen during this period, with radio theatre as a likely use-case. Difficulties in capturing multiple lighting conditions for a dynamic scene and may necessitate an exploration of unsupervised/semi-supervised learning or domain-transfer from simulation or between frequency spectrums.

The remainder of the PhD is expected to be based primarily at BBC North, but some flexibility may be possible, subject to the requirements of the project and the student. The research during this period will focus on preliminary feasibility studies for potential follow-on research, such as conditional generation and structure/lighting disentanglement to allow the simulated lighting setup to be varied dynamically by Ed or a human user.

Key highlights of this Studentship

This studentship should appeal particularly to scientists and engineers who are interested in working with the creative industries or have a particular interest in cutting edge and next generation vision technologies.  The project will be very interdisciplinary, with exposure to both academic and industrial research centres. Furthermore, this studentship could potentially benefit from the University of Surrey’s Centre for Doctoral Training in Audio-Visual Machine Perception. This will constitute a rigorous fully-funded programme of personal development, including professional research and software skills, interdisciplinary hackathon events, and a co-located peer-support network of approximately 100 other PhD students in related areas. Student’s receiving their doctorate through this scheme are expected to be exceptionally well-qualified for a subsequent career in research.

Candidates are expected to have (by October 2019) either a 2:1 or First Class Honours undergraduate degree or a masters in one of the areas highlighted below.

The ideal candidate for this studentship should have a strong academic background in software development from studying computer science, electronic engineering or a related subject. Furthermore, the candidate should have a keen interest in AI/machine learning. Prior experience (including formal study, project work or MOOCs) is advantageous but not required.

Non-native speakers of English will normally be required to have IELTS 6.5 or above (or equivalent) with no sub-test of less than 6.

Due to the nature of funding the studentship is available to UK. Overseas applications will not be considered.

• Centre for Vision, Speech and Signal Processing
• AI in Production