Volumetric Video | Point Clouds | Lightfields
I have been captivated by the realm of volumetric reconstruction since the early days of photogrammetry. Over the years, I’ve crafted tools and workflows for handling vast datasets from lidar, photogrammetry, and real-time solutions such as the Zed Camera or Kinect. My pioneering work includes collaboration with Lytro’s light-field camera and Google’s exploration of depth-based volumetric reconstruction for virtual reality, showcased in a Siggraph Publication and a masterclass back in 2017 titled “Video for Virtual Reality.”
Point Clouds as Volumetric Capture
Using techniques that I developed, and some amazing footage that Ian Forester captured in Africa we partnered with Nuralize and Atom View, a real-time point-cloud player. Point clouds were initially processed through Nuke and then filtered in Houdini. Below I demonstrate how we can use the point clouds as collision surfaces for simulations.
The Emergence of Volumetric Video
Volumetric video, an emerging media format, transcends the boundaries of traditional 3D movies or 360-degree videos. It grants users the ability to experience recorded events in three dimensions with six degrees of freedom (6DoF), encompassing X, Y, and Z axes, as well as pitch, yaw, and roll. This newfound freedom allows users to immerse themselves in the event, rather than passively watching it, courtesy of VR (virtual reality) or AR (augmented reality) headsets.
Girl | Virtual Reality | VR Playhouse
Girl tells the story of a 16-year-old girl who finds herself living with her grandmother in Joshua Tree, California, following the breakdown of her parent’s marriage. Audiences experience the story from this young girl’s perspective through the use of Foundry’s Cara VR for Nuke software – the virtual reality plug-in toolset that dramatically speeds up the challenging process of stitching and compositing 360° video footage. As the story unfolds, she discovers her grandmother’s house, which at first appears haunting, but later becomes more inviting and cheerful.
Girl was done as a collaborative project between VR Playhouse and the Foundry. Read the case study on the Foundry website here…
Moon | Lytro | Virtual Reality | VR Playhouse
VR Playhouse conceived of the concept of the idea for “Moon’, the first Lightfield and CGI integrated short ever produced by the camera company Lytro. I was the Technical Director and lead artist on the production and post-production side.
I was the Technical Director on the Production and Post-production side and the Lead Artist. The production had to be meticulous in the amount of information that we had as well as exacting in its technical accuracy. To cover the post-production, we captured Lidar using a Velodyne. We also shot extensive photogrammetry. The photogrammetry software Reality Capture had just been released, making this a much easier process. We also captured extensive HDRs under the highly controlled different lighting scenarios.
Real-Time Volumetric Video with Lidar
The advent of smaller and faster lidar, driven by advancements like Ouster and Touchdesigner, is reshaping the landscape of real-time volumetric video. My recent experimentation with a compact real-time lidar unit underscores the transformative potential of this technology. Lidar, running in real-time, has the power to redefine how we represent large, realistic datasets in both virtual and augmented reality.