Your next Teams meeting could have an extra dimension to it, with 3D video conferencing looming large on the tech horizon.
A Cornell University doctoral student has developed a remote conferencing tool which allows a remote user to manipulate the view of a scene in 3D.
Mose Sakashita said the system, called SharedNeRF, combined two graphics rendering techniques – one that is slow and photorealistic, and another that is instantaneous but less precise – to help the remote user experience the physical space of the collaborator.
“It would enable people to work on tasks that have never been possible and that are very difficult to convey through video-based systems with only one angle,” he said.
Mr Sakashita designed the remote conferencing tool as an intern at Microsoft in 2023, working with Andrew Wilson, formerly a computer science major at Cornell.
SharedNeRF takes a novel approach to remote collaboration by employing a graphics rendering method called a neural radiance field (NeRF).
NeRF uses artificial intelligence to construct a 3D representation of a scene using 2D images.
In the SharedNeRF system, the local collaborator wears a head-mounted camera to record the scene, with the resulting images fed into a NeRF deep learning model, which renders the scene in 3D for the remote collaborator.
Mr Sakashita said when the scene changed it triggered the NeRF model to update the view, but this update could take about 15 seconds, so his team merged the detailed visuals created by NeRF with point cloud rendering, a faster technology.
“By merging the two rendering techniques, a remote user can view the scene from various angles in high quality through NeRF while also seeing real-time movements in the scene through point clouds,” Mr Sakashita said.
“SharedNeRF also showed an avatar of the local collaborator’s head, so the remote user can see where they are looking.”
