2022-09-08 2022-09-08

audio image

Hello, this is GAKU from the Picam360 developer community.

We are pleased to announce a new feature. The Picam360 software actually did not support audio streaming. Since many users of Picam360’s real-time streaming were using it to remotely control work machines such as heavy machinery and drones, and often did not need audio as a usage scenario, it was not a priority for development. Recently, however, we have been asked if it would be possible to use VR to connect large products that cannot be transported to an exhibition site far from where they are installed, so that customers can view them in real time. The remote area where the product was located had a person in charge of explaining the product, and it was decided that voice was needed, so the system was finally voice-enabled.

So the new feature will be voice.

What you need to prepare

The audio function supports USB or Bluetooth microphones. In addition, the PICAM360-4KHDR is actually equipped with a microphone. If you have a model with an onboard microphone, you can use the onboard microphone.

microphones

Software Version Upgrades

Upgrade the PServer and PViewer applications. Please check <Confirmed Environment> for the version of each software.

Upgrade node-pserver @ Jetson Nano

$ npm -g uninstall node-pserver # Uninstall the old version
$ npm -g install node-pserverVERIFIED_VERSION # Install new one. VERIFIED_VERSION part specifies the version that has been confirmed to work, such as @v0.1.1.

Upgrade Pviewer @ viewer device. PViewer has its own apps for iOS, Android, Mac, etc. Please follow the app update procedure for each platform. Here we describe the procedure on Mac for reference.

$ npm -g uninstall node-pviewer
$ npm -g install node-pviewer@VERIFIED_VERSION

Setup

Connect the microphone to the Jetson Nano.

Start Pserver as usual. Then the device name of the recognized microphone is displayed in the console log of Pserver, so copy it.

audio device name screen

Open PServer’s config.json.

 "stream_params" : {
　　"default" : {

Add the following elements to the end of the stream_params > default element. Don’t forget to put a comma at the end of the element above “amux.ain0”!

　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　,
　　　　"amux.ain0" : "dup name=audio def='oal_capture name=capture' ! opus_encoder",
　　　　"amux.ain0.capture.devicename" : "HD USB Camera Analog Stereo"

Next.

"stream_defs" : {
　　"default" : "mux name...

Add the following to the end of stream_defs > default

 　　　... ! mux name=amux

The entire file is included for reference. Keep your settings except where noted.

{
　　"wrtc_enabled" : false,
　　"wrtc_key" : "your_key",
　　"stream_def" : "default",
　　"stream_defs" : {
　　　　"default" : "mux name=mux ! mjpeg_tegra_decoder name=decoder cudamem=1 ! pcuda_remapper name=remapper dup=1 edge_r=\"0.1\" s=@APP.SIZE@ ! pipe name=encoder t=H265 ! mux name=amux"
　　},
　　"plugin_paths" : [
　　　　# "plugins/jetbot/jetbot_service.js"
　　],
　　"stream_params" : {
　　　　"default" : {
　　　　　　"renderer.realtime" : "1",
　　　　　　"mux.vin0" : "v4l2_capture name=capture s=@APP.CAM0@",
　　　　　　"mux.vin0.capture.meta" : "<meta maptype=\"FISH\" deg_offset=\"-90,0,0\" lens_params=\"file:///home/picam360/pserver/lens_params0.json\" />",
　　　　　　"mux.vin0.capture.set_sharpness" : "7",
　　　　　　"mux.vin0.capture.set_backlight_compensation" : "30",
　　　　　　"mux.vin1" : "v4l2_capture name=capture s=@APP.CAM1@",
　　　　　　"mux.vin1.capture.meta" : "<meta maptype=\"FISH\" deg_offset=\"-90,0,0\" lens_params=\"file:///home/picam360/pserver/lens_params1.json\" />",
　　　　　　"mux.vin1.capture.set_sharpness" : "7",
　　　　　　"mux.vin1.capture.set_backlight_compensation" : "30",
　　　　　　"encoder.def" : "gst-launch-1.0 -q fdsrc ! rawvideoparse width=@IWIDTH@ height=@IHEIGHT@ framerate=30/1 format=i420 ! omxh265enc iframeinterval=300 bitrate=2000000 control-rate=2 ! h265parse ! video/x-h265,stream-format=byte-stream,alignment=nal ! fdsink",
　　　　　　"amux.ain0" : "dup name=audio def='oal_capture name=capture' ! opus_encoder",
　　　　　　"amux.ain0.capture.devicename" : "Wireless Microphone RX, USB Audio (CARD=RX,DEV=0)"
　　　　}
　　},
　　"params" : {
　　　　"APP.SIZE" : "1440x720",
　　　　"APP.CAM0" : "3840x2880@15:/dev/video0",
　　　　"APP.CAM1" : "3840x2880@15:/dev/video1"
　　},
　　# "pviewer_config_ext" : "www/jetbot-config.json",
　　"license" : {
　　　　"app_key" : "your_app_key",
　　　　"sku" : "P360SWLIC-STEREO-V1",
　　　　"iface" : "wlan0"
　　},
　　"debug" : 0
}

This completes the setup.

Let’s connect

There are no settings in the viewer. If you try to connect it and audio is played, it is a success.

 

This is an introduction to the voice function.

Picam360 Links：