Real-Time Point Clouds with the Zed Mini and Touchdesigner

Lately, I have been experimenting with real-time point clouds with the Zed Mini from Stereolab. Firstly, I am a big believer in real-time point clouds being a viable solution for co-located, virtual reality experiences. Concurrently, I am also interested in examining the development of this technology, and how we use artificial intelligence and machine learning to examine the world we live in. The Zed Mini functions much in the same way that the Kinect Azure does. The big difference is that the Kinect is more plug and play, while the Zed needs external libraries. Lastly, another big difference is the image and point-cloud quality, with the Zed being far superior. For some people, the fact that some of the tools need to be built using CMake and Visual Studio will be a deal-breaker. Stereolab provides resources in their Git Repository.

The Zed Tools

After I built the tools from the source, I now have access to some tools in samples/bin. As an example, the most immediately useful is ZED_SVO_Recording.exe. Firstly, this allows me to write an SVO file to disk. SVO is a proprietary format from Stereolab. Most importantly, this format allows for the recording of all the data from the Zed camera into a compressed file format. This will allow me to construct the point-clouds after shooting. In order to run these tools, I need to call them from the command-line. For ZED_SVO_Recording the only argument I need is a path to where I want the SVO file saved. So assuming I am in the shell in the correct directory, you would just write:

ZED_SVO_Recording.exe c:SVO_save_folder/mySVOFile.svo

The prompt should start scrolling frame numbers, and a ctrl-c will break the operation. One the SVO is saved, one can use some of the other tools that Stereolab provides.

Zed in Touchdesigner

I am working with a friend, Shaoyu Su, to get tensorflow and YOLO working through Zed and Touchdesigner for object recognition and tracking.You can download a free version of Touchdesigner here.