Oculus Rift for Tele-Presence?



I think that the new Oculus Rift VR headset display, which seems to be taking the gaming world by storm, could also be a useful way to pilot Remotely Operated Vehicles. It would be great to see this type of consumer technology making its way into OpenROV, because it could pave the way for greater things. Today an ROV submersible, but tomorrow aerial drones, or robot spacecraft, and even lunar rovers. There seems to be such a frenzy of creative activity going on with Oculus Rift in regards to developing new gaming experiences, that it would be a shame to neglect the possible uses for tele-presence, which has many important practical applications.

What kind of camera setup would be best for capturing a more fully encompassing visual field necessary for immersive tele-presence?


YES! You're exactly right. We talk about this a lot around the office. Colin has an Oculus dev kit.

The camera setup will be key. Eric had some ideas on that. What do you think?


Well, I'm assuming that it should be a twin-camera setup for stereoscopic vision. Since the lenses inside the Oculus provide a fish-eye effect, this would have to be accounted for thru the camera lenses or else via some post-processing of the video signal.

Another thing is that the Rift provides additional immersion by having the head-tracking sensors whose feedback are used to influence the rendered video output. In a remote camera telepresence scenario, it might also be nice to have some way for your natural head motion to similarly be reflected in what you're seeing from the camera feed. I guess this might mean either having camera motors pivoting them in reaction to the head-tracking feedback, or else somehow using a digital camera panning method based on sub-selecting from a wider camera field of view.

One thing I've read about the Rift is the potential for motion sickness or dizziness based on latency in the response of the rendered video output. I guess that things have to be made to feel as natural as possible, in order to keep the mind at ease.

I've read that the people at OculusVR are very friendly and engaging with the developer community, and I really think that you ought to get in touch with them to solicit their thoughts on the subject. Getting support from the OEM might provide you with the best insight, as well as reminding them of the wider potential applications beyond just computer-generated Virtual Reality, which they should be attentive to.


MEMS might also potentially be useful for camera panning/tilting with minimal latency, in response to the head-tracking sensor feedback:


The technology may not be there yet, but this is the kind of thing where if you could convince people there was a future market for it, they might work on developing the tech. Sort of a chicken-and-egg situation.


The camera setup is the problem. You need 2 wide angle Kameras with high res. I thing it is not Possible to place Place both (or even one) inside of the actual Tube.

You need a beeter optical port for such an wide angle, to get a good visual feedback. Normaly dom-port are used by underwater photgraphers for wide angle, but action cams like intova Sports Pro or Hero HD has a wide angle with a Flat port.

Next: you have to Streem 2 HD signals to the Surface :)

Like Sanjay tells it would be a nice option to use the "headtraking".

So you can rotate the ROV or the Cameras if the user rotates the had ... :)


Hmm, wide separation between cameras - what about something like the hammerhead shark?


Would it be possible to extend the tube beyond the width of the main frame, on either side? Then have each camera "eye" mounted on each end of the tube. But even then you still can't swivel in all dimensions...


Actually, I was looking around on various FPV flying forums, and dome cameras seem to be a popular solution.


So why not put a dome on each end of the tube, like fish eyes?

They can then swivel around inside their domes, along the tube's axis of rotation. Panning side-to-side might be more of a problem.


Another popular solution among RC flyers are the Fatshark glasses:



I like the Idea. But it I think will not work for stereo.

By stereo you have to turn fix the cameras to each other and turn than around the middle between the both cameras.

And yo always has to keep the refraction in mind. Underwater it change the optical behavior. A Doom port is good solution to minimize it. One of my co-worker doing research in the area of underwater 3D-Reconstruction. And the first is the Camera Calibration under Water what is a big research theme for itself.


I'm currently trying to make this work as part of my senior year project at school/for my school's ROV team.

I'm still in the planning phase but would like to start testing and building software soon. I was thinking of picking up some cheap webcams just as a proof of concept, maybe these cameras. or these ones. I've read that in order to reduce sickness 720p video at 60 fps is ideal, so this seems like a close match and I can return them easily if it doesn't work.

I'm thinking that this might be the camera I put on the ROV but I'm not 100% positive it will work. I have access to lots of cat5/5e/6 cable or could convert it to fiberop cable with stuff at the school. the tether will be 50-100 feet. so USB 3.0 might be out of the question, but I have considered this camera with an extender. The USB 3.0 camera is cheaper, but the cables could make it more expensive/ more of a hassle.

As for the distortion, the bubble dome from our old ROV seems to cause some barrel distortion. I'm hoping this will counter act the pin cushion distortion from the Rift. If not, I should be able to get some fish eye lenses or maybe implement a software fix.

The cameras will be mounted to a 2-axis gimbal so we can take advantage of the Rift's head tracking. I'm hoping this will allow the pilot to be able to look around in a natural way, while still piloting the ROV in a straight line. It should be interesting to see how the distortion of the dome affects the panning cameras.

I'm really not sure how to go from video output to something the OR can handle so it should be interesting to say the least. A friend of mine has suggested I use Linux for the OS to have the most control over the video processing as possible. Which I have no experience with. Also, our current system is windows based, mainly just running labview, which can be used on Linux, but the members of years past have said they use Windows XP because they had problems with labview and newer versions of windows. I don't want to bring in any new problems if I don't have to.

To recap: I'm going to start with webcams and work on a proof of concept then work up from there. Any insight/tips/ things I've just completely missed, would be a great help.




maybe if added sth. like this to the ROV it can use with the Oculus ...



I've accomplished this using the arduino to control the three servos on my 3 axis gimbal that's servos positions are updated by the IMU in the Rift, two webcams with custom M12 150 degree fish eye lenses. Res/stream is not the best, but that will come with time. With that said....for now....its AWESOME! Have not have a chance to put it in the water yet since I don't have an enclosure...that's in the works.... but looking around the office from a different corner of the office is a weird but awesome feeling. I dream of the day for it's maiden voyage! :)


Awesome! :)


As soon as I'm confident in it's performance (if that ever happens lol) I'll post all the steps and code. Right now we're working through quite a few issues we knew we would experience.

I can't wait for the day of integrated pupil tracking. This was we can have the cameras converge with our eyes....much like the camera James Cameron built for Avatar. This would avoid stereoscopic discomfort from close objects. (Servos spinning cameras inwards towards each other as you naturally focus on something close)


How are you guys dealing with the head origin movement? Does that become an issue? In my mind we would need to replicate not just the head rotation but also the slight head translation movements (up/down/back/forward/left/right). Or is that small enough to not be noticeable?