3D FPV with Google Cardboard, Oculus Rift and more

user-interface

#1

Hi everyone,

I am playing with this unexpensive 3D kit made by Google https://cardboard.withgoogle.com/ and I was wondering if we could pair it with OpenROV.


I am a huge fan of FPV for UAVs and I believe that such immersive experience with a platform like OpenROV would be a game changer.
The democratization of VR systems is a great opportunity.

Now, how can we make it happen?

The main challenge is to turn the video stream from the ROV into a 3D split screen video that can be played on an Android smartphone.


There are apps that can do that but it is not streaming ready yet: https://play.google.com/store/apps/details?id=co.mobius.vrcinema

Building an Android app with this built-in feature would be great but then it generates new challenges. We would have interface the TCP IP connection to the node.js application. We could plug the ethernet jack into a wireless router... Not sure what would be the best road but having a tablet/smartphone compatible app would open many doors including the one of virtual reality.

Benoit


#2

Hey Benoit,

Getting 2 video streams topside:

In the short term, the system should be able to handle 2 cameras, though we may have to bring the resolution down. That would require a small USB bus adapter to be placed in the electronics tube.

From the software side, we can spin up another instance of mjpeg streamer that is reads from the second video source and emits it on another port such as 8091.

This gets 2 video streams running up the tether.

-Brian


#3

Hey Brian,

Thanks for looking into this!


Actually, we might not even need 2 cameras. We would end up adding some unecessary payload and it would be difficult to synch both cameras and to find the right angles.

As long as we get a streamed video signal, it should be possible to convert it into some fake VR video.

Take a look at this app: https://play.google.com/store/apps/details?id=com.loxai.trinus.test

Apparently, this handles most of the actual VR work of streaming video in one direction and sensor data in the other, and combined with a client-side Windows program it makes VR streaming a snap.

Check this out: https://www.youtube.com/watch?v=AJ1x3u79qBg

If this works with Skyrim, perhaps it will work with Cockpit.

Benoit


#4

Hey Benoit,

We are in the same boat !(or OpenROV...) :P I'm also very interested in being able to view the cockpit with Google Cardboard.

Virtual Desktop might be another option to try. I'm running on a Mac so unfortunately I couldn't test it. https://developer.oculusvr.com/forums/viewtopic.php?f=29&t=8182

Google Chrome also has an App/Chrome remote desktop that you can use to stream your desktop screen to your phone/tablet, but it doesn't create the VR images for you to be able to view it in cardboard.

Something that I'm going to try out later today is a Oculus Rift plugin that Brian had forwarded to me. https://github.com/covedesign/openrov-oculus-rift-plugin.

Please let me know if you come across anything that works! I'll post my progress here as well.

Thanks,

Mike


#5

Hey Mike,

Thanks for your help. It is only a matter of time before we find a viable solution. We are getting close.

To connect a controller (Ps3 paddle for example) to and Android phone, we use an Otg device paired with an app called "USB BT Joystick Center".

To Stream a desktop to a smartphone, we should experiment with the following apps:

Kainy
https://play.google.com/store/apps/details?id=com.kainy.clientads

Kimoni Remote Desktop:

https://play.google.com/store/apps/details?id=com.kinoni.remotedesktop&hl=en


Splashtop
https://play.google.com/store/apps/details?id=com.splashtop.remote.pad.v2&hl=en

Teamviewer
https://play.google.com/store/apps/details?id=com.teamviewer.teamviewer.market.mobile&hl=en

From there, we just need to find a way to project side by side video.

Regarding Oculus Rift, we might also use this app: http://vire.io/


#6

I am getting some decent results with Trinus Gyre and Google Cardboard.

Did anyone else tried that?


#7

HI, I just did a simple implementation of Google Cardboard without doing any modifications of the OpenROV software. To see if this would work, I just duplicated my Google Chrome sessions into OpenROV and manually sized them to create a left and right image. I then did a Remote Desktop(RD) into my PC with my Android and sized the screen to fit both images. Once in the Google Cardboard, it worked out great and the delay in image update was mild. Though I had only guessed how to size and setup the stereo images on the laptop, it worked out perfect. I tried it again many times and each time I was successful. I think my eyes were able to compensate for any minor error in size/alignment.

I think creating a dashboard for OpenROV that would be a stereo image of the one camera is a great idea, but I think going in the direction of RD to the laptop would be fine for delivering it to the Android. Though I'm currently using a web base RD I think setting up a local network and using the MS-RDP would eliminate that issue.

I just posted an image of my testing of the idea. https://openexplorer.com/expedition/newenglandexplorers

~Michael


#8

Hi, as i mentioned, I had success with remotely accessing my desktop and manually splitting the video image, but will give Trinus Gyre a try too. You idea sounds like it would better streamline the process.


#9

Tonight I tried Trinus Gyre as Benoit suggested, along with Vireio, and got great results! I attached my phone directly to the PC for the best connection and I was impressed with the quality. The only issue I saw was that the google cardboard has the phone's screen too close and the lens are too small. This make it difficult to see the entire image. I have a better VR glasses for my phone on the way and will report on any improvement.

~Michael


#10

I got the new mount for my Galaxy S4 and its quite an improvement. It has bigger lenses and places the phone a little further from the optics, which helps to give a nice full view.

I also have changed some of the settings in the Trinus Gyre server on the PC and it seems to also improve quality and performance. The last thing I'm going to try to so is map the camera tilt for the OpenROV to the head tracking. If I'm successful with that I might try to map the steering of OpenROV to the head tracking too.

~Michael


#11

Hey Michael,

I tried your dual-window idea using remote desktop and google cardboard and was amazed! I'll try Trinus Gyre later this weekend to get that going.

I had the same idea as you, trying to map the camera to the OpenROV tilt and steering. When I was in university, our underwater robotics team had implemented this using their onboard pan-tilt camera and projected the feed to a pair of head mounted displays they had on a hardhat. Control of the ROV was via a 3D mouse. It was super easy and intuitive to drive.

Please keep us updated on any progress that you make!

Mike


#12

Hi Mike,

With Trinus Gyre you won't be able to do a dual window into the OpenROV. It only will show the currently active window. I've only tried it with software that Trinua Gyre seemed to have no issue splitting it into two screens on its own, and the results are fantastic. There are plugins that are supposed to be able to manage the split for most software, and there is one posted in this forum, but I have not had the change to try yet. http://community.openrov.com/forum/topics/plugin-wip-oculus-rift?id=6365107%3ATopic%3A103246&page=2#comments

With the weather prediction of blizzard for tonight and tomorrow I should have a chance to see what needs to be done to get the split activate with OpenROV cockpit up in Google Chrome.

I think mapping the controls for looking up and down should be easy. The code is already there it will just be changing where they're already mapped to. Though I think it will be fun to remap the steering, I think using the game pad will be best. The left and right movement detection is a little sloppy, and I found it is the phone not the software. It causes the view/movement you have to constantly drift to the left or right on it own, even when you're perfectly still. A fix for that is being worked on.

~Michael


#13

Any updates! Did you get the dual window software working?


#14

Hi all,
I think @badevguru has the right idea, because really? who want a fake 3D experience? really thats like putting a Ferrari badge on a VW beatle so you can experience the speed of a race car lol.
But seriously, in the 1st quarter of 2016 the rift will be released giving the world a new tool to experience something truly amazing ! why are we not trying to set this as the minimum bench mark? Having a OpenROV Rift ready will increase its profile enormously and then further help in development & R&D. This should also be fully ready for the Trident released version !!!