Jetbot + VR180 : Part 1 – Introduction
2019-07-30 2019-10-15
Jetbot + VR180
In March 2019, NVIDIA introduced Jetson Nano, a compact and inexpensive computer that can take advantage of state-of-the-art AI. In addition, they released the open source hardware robot Jetbot. Jetbot consists of 3D printed parts, and the software runs on Ubuntu Linux. The hardware design and the source code are available on Github, and an SD card image with software installed is also distributed. Jetbot is a good platform for learning AI computing and robot development.
In this article, we will introduce a series of steps to connect two Picam360 (4K type) to Jetbot and live stream 3D-180 video. With Picam360’s real-time 3D video, you can experience the feeling as if you were riding into a Jetbot and driving.
Is stereo 3D available on regular Jetson Nano instead of Jetbot?
You may want to install Picam360 software on regular Jetson Nano instead of Jetbot. That is possible. The procedure is basically the same as installing Picam360 software on Jetbot. Please refer to the steps in this series.
3D-180 Video
The left and right images are slightly different because they are taken with separate left and right cameras. If you look at this separately with the right eye and the left eye and focus it, it looks three-dimensional.
Immersive 3D vision of 180 degrees in both vertical and horizontal directions can be realized by capturing the front with two fisheye cameras and arranging images from two viewpoints.
What is Picam360?
Picam360 is the panoramic camera system is being developed by the Picam360 developer community. Picam36o can stream 360 degree video lightly using proprietary image compression technology. Panoramic cameras generally use high resolution cameras to capture a wide area. Therefore, one image data is heavy and not suitable for live streaming. By own unique compression processes such as dynamically compresses images in the viewing direction, Picam36o reduces the data size to 1/50 compared to a typical system equipped with two 4K cameras and using VR180 format (*1). This enables live streaming even on narrow bandwidth networks such as mobile networks. Details will be explained below.
Technology for streaming panoramic video in real time
A panoramic camera generally uses a high-resolution camera in order to capture a wide area clearly. Therefore, the image data becomes heavier than a normal camera.
Foveated Compression
When you watch a 360 degree video uploaded to YouTube, you may feel that the image quality is rough. This is due to the fact that the image quality of 360 videos with a large amount of data is reduced according to the communication speed of the connected Internet.
There is an image compression method called Foveated Compression as a way to reduce the amount of data without reducing image quality as much as possible. The human eye can clearly see the center of the field of view and blur toward the periphery. In other words, even if the clarity around the field of view is reduced, it is difficult for the video experience to be affected. Using this property, Foveated Compression is a compression method that reduces the amount of image information as the distance from the point of interest increases.
Picam360 has a unique Foveated Compression technology optimized for 360 degree images.
H.265/HEVC
H.265 is one of the video compression standards. It is the successor to H.264 / MPEG-4 AVC, which is the current mainstream video compression technology used for Blu-ray and general video distribution, and about half that of H.264 Equal image quality can be achieved with data size. Picam360 software (Jetson version) supports H.265.
Data size is reduced to 1/50
The Picam360 uses a unique Foveated Compression technology optimized for 360-degree images and the H.265 codec, Picam360 uses the proprietary Foveated Compression technology optimized for 360-degree images and the H.265 codec, reducing the data size by a factor of 50 compared to a typical 5.7K panoramic camera(*1) while maintaining the image quality sufficient for the video experience.
By significantly reducing the video data size, 360-degree video can be streamed real-time at mobile network-level speeds.
The following Youtube video is the actual video. Connected to iPhone with Jetbot + Picam360 and captured the screen.
The above video is accessed from the iPhone Chrome browser. Picam360 viewer is a browser app. There is no need to install a dedicated app, and you can access it with a smartphone, PC, VR headset, etc. (* 2). The viewing direction is linked to the posture of the device (PC is operated with the mouse pointer).
That’s it for part one of the series. Look forward to the second.
(* 1) Specifications to be compared: {Camera: 3840 (4 k) pixels camera x 2, {Image format: Equidistant cylindrical projection map, VR180}, Video codec: H264}
(* 2) Availability depends on OS, OS version, and device hardware.
Series of content:
