GoPro underwater Structure from Motion (SfM) / Photogrammetry

payloads
camera

#1

Probably adding a GoPro to an OpenROV is one of the simplest payloads but I just thought I would share how this simple addition can significantly add to the capability of the unit to document wreck or reef sites

Structure from motion (SfM) is the process of estimating three-dimensional structures from image sequences whilst photogrammetry is just making measurements from photographs

OpenROV first utilised some of these techniques back in 2012

Firstly a few images to show what I have been able to do with a simple GoPro Hero 3 Black (note some images were done whist scuba diving)

The Wreck of the Annie M Miller lies in 44m of water off Sydney Harbour and was a collier of 706 Gross tonnage and 49 meters long and sank in 1929 (for more history on the vessel see here )

The image bellow is an automated 3D model of the wreck built up from 475 images off a GoPro Hero Black shooting at 12 MegaPixel (larger image here) you can sore of see the 3D nature of the photo in this isometric style projection


The Wreck of the Undola lies in 44m of water about 40km south of Sydney and was another collier of 429 Gross tonnage and 41 meters long and sank in 1918 (for more history on the vessel see here)

The image bellow is built up from 1050 images (larger image here)


The wreck of the 1861 passenger schooner Colonist lies in Sydney harbour under the main ferry channel and has very little left but the Muntz metal hull outline and is made up of 550 images

This image shows how well the system works as the visibility was at the most 2 meters such that from the centre line tape you could not see the edges of the vessel


Therefore, how can this be done

Basically the images were just shot at a frame every second with no lighting pointing the camera straight down

I then used the Agisoft PhotoScan Professional edition software although other open source pieces of software should also be capable of producing similar images and dumped them in there in an "unordered" state and the software was then able to spit out the images above

The image bellow shows the 3D nature of the data as well as the camera pose (location) of the shot (maybe useful in a "towards autonomy" future SLAM implementation??)


I have a modified GoPro Structure from Motion payload on my ROV where the camera sits under and just forward of the E-tube (E tube removed for clarity and as the GoPro is negatively buoyant I wanted the weight down load and just forward of the centre of buoyancy)


The white plastic backplate is broadly made as shown in the attached dxf file (the file has a couple of alternatives so watch what layers you turn on and off)

GoPro%20ROV%20mount.dxf

The GoPro mount is just a simple Aluminium mount on the standard (slightly longer of the two) arm

Future Steps

There is plenty to play around with here, there are several videos showing GoPro WiFi working underwater (to about 150mm) streaming images such that a USB WiFi module mounted in the E-tube and then a bridging IP address should allow what the GoPro see's be streamed back to the surface

A more complex payload could include stereo vision (although monocular vision seems to be working fine currently) lighting and even a couple of lasers (see Laser Pointers and YouTube ) to provide "altitude" off the bottom which would be useful for both bottom tracking (as opposed to depth) and scale given the cameras FOV all of this would allow adaption "towards autonomy" and future SLAM implementations (EXIF tag data from ROV IMU of heading/pitch/roll/yawl??)

Finally of course find and document more wrecks have a look at this for what else we have been up to and yes David when I pull my finger out I will put some of the wreck hunting we are doing up on Openexplorer


Structure from Motion for documenting for Citizen / Community Science and Marine Archaeology
Underwater Archaeological Camera
#2

Scott,

First, great write-up! I was completely unaware of the Structure from Motion technique until now. I also spent some time reviewing your newsletter and you guys are having one heck of an adventure down there.

You guys have the "gold standard" of shipwreck site recording right here and I am very impressed. I would like to do the same for documenting the various wrecks in Southern California.

Although I think the SfM technique is great, is there a way to simply do 2D photomosaics? $4000 for the PhotoScan software is more than I can justify for software right now.

What settings were you using for the GoPro? I am interested because taking out the fisheye is a big issue with these cameras.

If you are using natural light, vice organic lighting from the ROV/diver, would a red filter be better at bringing out the natural color of the site?

What altitude above the site were the pictures taken? Or does it matter as long as it's even?


#3

Kevin, thanks for the kind comments

Yes SfM is a great technique for documenting a wreck or any other underwater site

Whilst I have been using the "profession edition" everything you see in the images here can be done in the "standard edition" (comparison)

I'm almost 99% sure that the software is just a nice front end GUI to a lot of the open source stuff out there, where you use it really to just "kick off" a number of discrete processing steps and keep out of Linux command line stuff

Have a look at the Wikipedia article on Structure from Motion that has a list of the open source software solutions in particular Bundler and Python Photogrammetry Toolbox

For the GoPro it is a GoPro Hero Black on the 12 megapixel wide setting taking a shot every second. The fish eye effect is significantly less underwater than it is on the surface in part due to the refractive index of water and the software I think does do some camera calibration during the process. For the height above the wreck its typically a 2-3 meters due to the FOV of 120 degrees by 90 degrees of the GoPro and the software liking about 80% over coverage in the direction of travel and about 60% over coverage between the lanes I generated a cheat sheet (below height above bottom) and the software handles changes in height well as long as it can still make out features on the bottom


I have had it work in visibility of about 1 meter (so images shot about 800mm above the bottom) although the images are not as "pretty". In an Ideal SfM / Photogrammetry payload I would include lighting to help with the colours although for the mosaic style image the loss of red isn't as much of an issue compared with the pretty macros diver shots you see


#4

I have been asked a couple of questions about how much the GoPro camera may feature in the OpenROV web camera and in general I would say not very much at all

Basically when the OpenROV webcam is in the neutral looking straight forward location (A button on the keyboard in Cockpit) you don't see the GoPro


As you pan the webcam down the tip of the GoPro only starts to come into view of the 6th and last "step" down of the webcam (see image full pan down)


You can see how far it sits in front of the E-Tuber in this image




#5

Thanks for sharing.
This is helpful!


#6

This is really great to read about! For a long time I've been dreaming about using SfM to do stuff like this.

I'm eager to hear about your progress!

Eric


#7

Thanks Eric

I had been looking around for several years and had tested out quite a few different (and very labour intensive) techniques to try and get some "mosaics" of wreck sites until I came across the SfM, although I have been using agisoft photoscan I know of a few people who have been doing similar with the opensource VisualSFM : A Visual Structure from Motion System

It is also well worth having a look at some 3D sections of the Annie Miller (one of the wreck images above) that can be manipulated on line (and even embedded) done by another guy over here in Australia

You can also follow what I have been up to looking for and documenting new wrecks over at open explorer

Scott


#8

Hello Scott:

As you may have read at one of my posts, Im into a project for the location, exploration and re-construction of Spanish Historical Vessels.

The Atlantic exploration and high seas ships building, started around 3500 years ago, when Phoenicians set a naval factory at Cadiz for the exploration of Africa west coast, and the tin metal trade with the British islands.

Wrecks around that area cover the whole period from that beginning to nowadays.

We have two real ships already sailing. A Carvel (C-XV) and a Phillipines Galeon (C-XVI) .

Now almost finished with the reseach of another two vessels.

I've got impressed with your post and I am sure my Dpt director will be as well.

Kind regards and thanks a lot.


#9

Ion

Yes I saw your post and was quite interested, as you can most likely tell I'm also into looking for wrecks and documenting them on a budget, and it's always great to help and be helped by others doing sort of similar stuff (and hence the post). The more minds on any problem the better the answer and the more info and techniques are passed around the higher quality work we can all produce

From this, you can most likely see why I'm interested in even simplistic planned navigation (eg an expending square search pattern even if just based on time but not distance, to generate these mosaics).

The old wrecks you people have in that part of the world are always interesting compared to the just over 200 years of maritime history we have over here


#10

Hi again:

Conditions now at the Atlantic european coast starts to be too hard for any exploration. Sea state starts to be rough and visibility too short. From now on, conditions will go worse. Hence our underwater program is planned to start around next June.

Our safe working depth (SWD) has to be around 250m, I see as a minor problem to improve the OpenROV SWD.

The size of the ship wreck (belongs to the biggest wooden ship of the line ever built) , will require an exploration area of around 100 x 100 meters, that, underwater means quite a big place, hence location accuracy is a must.

Im now working on some propellers, but as soon as I finish, I'll try to start with the mathematical procedure for accurate inertial navigation.

It must be said, that those inertial systems + dynamic possitioning, are daily used in the Oil industry. BLS procedures would be impossible without such systems.

Hence we know, for sure, that it can be done. :-) :-)

Kind regards


#11

Hi,


I've been working on doing SfM on land or underwater with an ROV, and have had great success. fro those who are interested in SfM, I'm working on a step by step How-To guide. Once I get it all together I'll share it with everyone here. Meanwhile, if you have any questions please feel free to contact me via my OpenExplorers page https://openexplorer.com/expedition/newenglandexplorers or directly at photogrammetry@neexplorers.org

~Michael


#12

For photo mosaics, have a look at the AutoPanoPro software from Kolor : http://www.kolor.com/

They do a few variations of Panorama software for photographers and I've used their AutoPanoPro software to create 2d panoramas that are over a Gigapixel in size. They make the software that's used for those huge 360 degree city shots.


#13

This is great!
Looking forward to it!

I am having issues using a GoPro for Photogrammetry because of the distortion caused by the camera lens.

I use PT Lens to fix this issue : http://epaperpress.com/ptlens/


#14

Looking forward to it Michael

Love the work that you have done integrating the 3D SfM model into Blender for the fly around and also work you have done in Sketchfab

PS David L has sent through a message about some other 3D SfM software from Autodesk Project Memento that looks interesting

Scott

Benoit I've always intended to calibrate the lens but have never got around to it good to hear you have had success with it


#15

Forget about calibrating the lens. PT Lens is a software. Everything has to be done in "post-prod".


#16

Benoit, I would consider you a more proficient GoPro user than I, would you consider, or see any merit in switching out the lens vice post-correcting the images/video?

I've been looking at the Peau Production 2.97mm replacement lens and I might just buy a Hero 4 silver with it already installed.

I really hate messing with software if I don't have to and I just haven't been happy with some of the videos around the net using post corrections. The settings seem very finicky.

Here's some video with the 2.97mm lens: Peau Productions 2.97mm lens on a quad copter


#17

Hi Kevin

PT Lens is a sort of software type of glasses to clear any distortions made by the lens (so well worth doing to improve the final image) most of the SfM software allows for a calibrated lens to be used (the output of PT Lens or others) and some pieces of software do a lens correction as part of the stitch.

However, as you indicate the better the original image the better the result. The advantage of the GoPro is its small robust and simple but better lens and or cameras and higher resolutions or narrower field of view (and hence more lanes) will all produce better results

The SfM images above are just de rated and downscaled to make them more usable on the net, basically at full resolution they are giga pixel resolution images (each of the 500 odd images that went into them where a 12MP image to start with)

Scott


#18

Scott,

Thanks again for the kind words.

David mentioned Memento on my page too. I signed up and have been checking out the software. The results are a much higher quality. Lots of helpful features too. It could be useful, but I won't let myself get to distracted by it. I'd like to try to use only open source software for the How-To. We'll never know what the access to Memento will be after Jan 31st.

~Michael


#19

Hey Scott,

Did you manage to get the GoPro rig + OpenROV to produce any Photogrammetry?


#20

Hi Brian

Short answer is No, but’s that’s only because my OpenROV is currently in multiple pieces with a new wiring harness and configuration in it’s near future

I have had the SfM work as a snorkeler, as a free swimming diver, on a diver scooter and @Michael_Girard has also had it working from a Kayak and it has worked off other ROV platforms. I don’t see an issue with running it off the OpenROV as the frame I built (dxf files and images above) holds well on the OpenROV in the water (I did some basic sea trials with the configeration and the GoPro on the ROV but din’t try to capture anything with the camera as I was jsut playing and checking)

The only picture I have off it in that configuration in the water was taken from my phone

As a side issue we have been putting together a guide with a bit more details similar to the Multibeam Guide

@Kevin_K is just playing with the finishing touches of the Guide to make it look more professional in addition to his we have also roped in Michael_Girard to add some info on how to take it into blender to get some pretty rotational and swooping videos of the output. We are just trying to give some assistance to the community on stuff they can do with the ROV and take the outputs and uses to the next step

Scott