IMU Debugging


#1

One the largest challenges we've faced at OpenROV has been trying to figure out how to make tools with advanced functionality using common, low cost resources and a very minimal internal engineering team. The IMU/ Depth sensor we've developed is one of my favorite examples of this because it uses parts that are not very expensive (the depth sensor is commonly use in SCUBA diving computers, and the IMU can be found in many cell phones) yet it makes people's ROVs much more capable.

In a few cases, we've started to notice that the compass and orientation data in Cockpit sometimes freezes up during a dive, and we're not sure why. The behavior we're seeing with the data bus on the IMU chip right now is one of these types of challenges where getting outside help from our community is something we could really use. This is one of the reasons why we've made our project open source. We've done quite a bit of experimentation and head pounding, but we still can't get to the bottom of exactly what’s causing the problem or how to fix it.

Here is everything we know about the issue so far:

NAK - On some ROVs, data from the IMU sensor "locks up" (the number of Not Acknowledge or "NAK" increases dramatically and does not recover) causing the the compass, roll and pitch data to freeze in cockpit.

IN WATER - The problem seems to almost always occur only when the ROV is placed in water (both fresh and salt water).

WET TETHER - The problem seems to only occur when the tether and the ROV chassis are in the water at the same time.

DRY IMU - The problem can occur when the IMU module itself is outside the water but the I2C lines and the chassis are in the water.

DEPTH OKAY - The problem does not happen with the depth sensor data which uses the same I2C bus to communicate- the depth sensor data continues to stream in just fine. In other words, The problem only affects the MPU9150, the Arduino to depth sensor communication remains unaffected during these events.

FOLLOWS ROV - The problem seems to follow the the ROV and not the IMU sensor- in other words, if we have a working ROV and one with this problem, and we switch the IMU sensors, the problem stays with the ROV even though it has a new IMU sensor.

ESCs OFF - The problem has been reproduced when the ESC primary power is turned off, but we acknowledge that the servo line is still providing a bit of power to the ESCs.

WAITING FOR REPLY - When the problem occurs, the Arduino default wire library gets stuck in a state where it is waiting for a reply on the I2C bus from the MPU9150. The modified Arduino library adds a timeout that resets? the MPU9150.

DEPTH DISABLED - The problem occurs even when the depth sensor code has been disabled.

RESET ARDUINO - After the problem occurs, resetting the Arduino (using the software reset script) will eventually restore connectivity to the MPU9150, but not right away. We often see delays of up to a minute where the reset of the Arduino has no effect. This only works if the ROV has been taken out of the water. If the ROV stays in the water, resetting the system will not recover the IMU.

V2.7 - Although similar behavior had been noticed once or twice previously, instances of the problem occurring seemed to increase dramatically with the release of V2.7. V2.7 has a different wiring harness as well as other changes.

TWISTED HARNESS - Previous designs had the (twisted) tether routed all the way through the endcap to the DB-25 connector, but v2.7 calls for the tethered to be soldered to untwisted leads in the wiring harness that go to the DB25. When we’ve manually twisted the tether wires (twisting helps reduce electrical noise transmission) in the harness as well as the I2C wires going to the IMU/Depth Sensor, the problem seems to occur less frequently.



Theories on cause:

NOISY TETHER/ SENSITIVE IMU - Electrical noise generated by the tether capacitively couples with the I2C bus wires in the wiring harness (capacitive coupling is enhanced with water as a dielectric between the wires) which causes the IMU (which is overly prone to failure from noise) to drop out.



Background information:

IMU chip used: MPU9150

SW used to communicate with IMU: https://github.com/OpenROV/openrov-software-arduino/blob/master/OpenROV/MPU9150.cpp

Depth Sensor chip used: MS5803





IMU/Depth Sensor is not working
New IMU / Depth Sensor
Acoustic Location System
IMU error with 2.5.1-101 version
I can't get the IMU working
#2

Eric

Thanks for the update and having seen this issue with my unit and seen it continue with all elements changed out bar the wiring harness (eg different E tube and different IMU). The one comment I would add is that as I added a ferrite core to the IMU wiring it decreased the issue (less outages in a given period) and as I passed the wiring through the ferrite core multiple times it was even further diminished (eg more EMI suppression = more resistance to the issue) till it was working fine on land. When the unit next hit salt water it was again present till even more ferrite suppression was added which seems to have further diminished it.

I am reworked my wiring harness (eg complete wiring rebuild) and aside from using a couple of ferrite cores I have also purchased some EMI Shielding Wire Mesh (a tinned copper version so will be OK in salt water) and intend using it on each of the motor cable groupings and the IMU- given your comments on the tether most likely I will add a few hundred mm to that as well

Thanks again to Dom (for letting me physically cut apart his unit to assist in troubleshooting) and all the other people who assisted in overcoming this to an extent on my unit

I hope someone other there in the community can add a few more thoughts and assist in a good workable answer

Scott


#3

Thanks for the reply Scott. As a matter of fact, hearing about your ferrite experiments was a big part of us hunting down the suspect area. It really does seem likely that this phenomenon is the result of two things working against each other- an electrically noisy tether, and an overly-sensitive-to-noise IMU. We're currently evaluating some alternative IMU chips (since the depth sensor works fine on the same bus, so maybe another IMU would be less picky), but of course we'll see what we can do to get the current system working as well as possible.

So far, we've gotten what seems to be improved behavior when we twist all the I2C wires in a bundle along the length of the harness, and also make sure the wires connecting the tether are twisted all the way up to the endcap.

Please let us know if you make any other discoveries, and thanks again for the advice here!

Eric


#4

Hi Eric

As indicted, I am upping my RFI protection by as well as adding the Ferrite cores in the next rebuild of my unit, I'm adding some RFI shielded mesh (I've purchased it already and its waiting for me to get around and add it into the unit) where I am using the 3.4mm version of this (I think for a small quantities it was $9/meter). As you indicated the "twisting" and "plaiting" of the IMU wires are in part a form of this style of shielding

When you think about it in the bigger picture in the ROV it is somewhat easy to have RFI inducted into its cables as you are basically sending a modulating frequency signal to the motors, the motors are also just big rotating magnets right next to conductive wires all great ways to induct RFI into parallel data cables

If I could suggest a trial, it could also be worth trying using some micro coax to connect to the IMU rather than standard wire (eg its own in build shielding). An additional trial could also be mounting the IMU as far away from the motors and their cable as possible (maybe mount it near the front of the unit??) so there is little (aside from inside the E-Tube) as possible parallel wire routing of data (I2C) and modulating power

Unfortunately, it is a bit outside my world as I do not have an oscilloscope capable of looking at the magnitude of the noise inducted (if that is the case) and compare each of the different mitigation strategies (and / or others)

Scott


#5

#6

Hello,
for what it is worth… (and please use this with caution considering that 6 month ago I thought Arduino is something you would order after lunch)…

Beginning of the year, I bought on ebay a GY-88 Module (cant remember the price but roughly 7 EUR including shipping to Singapore). This module contains an MPU 6050 Motion Sensor, an HMC 5883L compass and a BMP 085 Pressure sensor (air pressure not water…used in drones as altimeter). A standard breakout board used in most Quadcopters. My original motive was that I needed something that would allow better compass functionality as I did not want to manually calibrate the OpenROV IMU. Plus, given my depth requirements I wanted something that sits inside the ROV. (still have to work on a pressure transmitter though but that is a different story)

Having now changed the arduino code and added this module as an additional equipment (using interesting libraries from the Internet… one that uses a nice calibration function for the compass (for this I need to shortly put the watchdog on hold else it jumps into reboot during calibration) and uses a library that uses the DMP inside the MPU 6050 for motion (pitch, roll, yaw) it gives me extremely stable readings over all 3 axis… and I can put the whole module inside the ROV hence no tether / water / thruster interference. My current assumption is that the interference inside the ROV cant be more than the ones in the boards of most drones…or inside my phone. (caution with this assumption)

Currently I am comparing both readings OpenROV IMU vs. the ebay GY-88 module (with the different software / hardware approach) and various outside interferences over a longer period of time (ill do a datalog over the weekend). Lets see where this leads to…

@OpenROV: What was the original train of thought that made you put the motion sensor outside of the ROV?

Greetings Carsten


#7

I look forward to your results!!
The reason why we put the motion sensor outside the ROV is two fold:

  1. We were concerned about EMI issues with the IMU being so close to the ESCs in the e-tube; the gyro and accelerometer would be fine, but the magnetometer might throw erroneous readings when so close to strong currents flowing intermittently. MEMs magnetometers are quite sensitive and are intrinsically noisy (at least for the MPU-9150’s magnetometer, we noticed it jumping around alot around EMI).
  • We wanted to have a single integrated IMU+Depth module for ease of integration; since both the depth and IMU modules run over I2C, we would be able to have a single sensor package that would only require 2 data lines, and also be able to mount the IMU much further away from the motors. However it seems the conductivity of the water changes the properties of the EMI, and we didn’t anticipate the effects of the tether…

#8

Hi

At work we use same MPU but for a quadcopter. we did some work for an American plane manufacturer where we used beaglebone black. It turned out that the beaglebone made a lot of noise in the system - anyone checked that???

We ended up with a cardboardbox with only the beaglebone inside and ethernet + powecable going in. The box was covered with this type of tape to isolate the noise and ferrites on powerlines. http://export.rsdelivers.com/product/hi-bond-tapes-limited/hb-720a-12mm-x-33m/hi-bond-0025mm-thick-copper-tape-12mm-x-33m-700-g-in/8326316.aspx?query=832-6316


#9

Colin, I still owe you the results of my datalogging comparison 9150 vs. HMC5883L. I have now successfully installed both in the software. (had to add a datalogger on SD card as well). As of now, I still see inconsitend readings I can not explain. the 9150 has periodic drop outs. (I might be oversampling and reaching a limit here…) I am still debugging. Hope over Chinese New Year here in Singapore I will find the time to look into this issue in more depth and with more time. Its been a bit of a mental time for me here in the last couple of weeks. I am also looking into shielding options of the BBB and the motor controllers (I have 1400 Watt motor controllers). Once I have something that is only mearly proper to post here, Ill get back to you on that. sorry for the delay.

Greetings Carsten


#10

Eric,

Here is my experience with my v2.7 and IMU to date:

Week of 02-06 MAR:
Freshwater pool testing in 6ft of water. Results satisfactory with both depth and heading working properly.

Saturday 07 MAR:
Saltwater testing in ocean. IMU locked up on contact with water.

Week of 09-10 MAR:
Freshwater pool testing in 6ft of water. Results satisfactory with both depth and heading working properly.

Wednesday 11 MAR:
Saltwater testing in San Diego Bay with @Stretch. IMU locked up on contact with water. Depth sensor working satisfactory and holding appropriate depth with the auto depth feature.

My conclusion is that mine is fine in freshwater but reacts badly in saltwater, probably due to the higher conductivity.

I built a payload bay tray with a spare crossbar and a piece of plastic for my GoPro. I’ll try mounting the IMU as far forward and away from the motors and wiring harness as I can.


#11

Thats a great data point. @Walt_Holm, next time we do the tank tests we will have to see if testing in salt water will give us consistent failures.


#12

I went wandering thought the MPU9150 posts again today. Complete speculation on my part right now, but I wonder if something might be triggering the chip to drop in to sleep mode. I’ll try loading the simpler sketch which clearly removes sleep mode and see if I can get basic output from it. If so, I will post the alternate driver code here and perhaps we can get some folks to test and see if it “clear” the lockups they have been seeing.


#13

Fingers crossed! Of it worked, would this be a pretty easy change for people to make to existing software?


#14

“easy” is awful subjective :smiling_imp: For those motivated to test the change, it will require some ssh command-line work that I can document step by step.

If the behavior can be fixed in software than ultimately we issue a software update with a new image that people can install.


#15

hey all
I’m almost done with my OpenROV 2.7 build, the only thing missing is the IMU, I have it assembled but not yet soldered on and positioned. I’m reading a lot about problems with the IMU in the forums, so I’m kinda worried a little.
I would like to know if you suggest putting the IMU somewhere else, not so close to the motors, as I understand that there might be some electromagnetic interference from the motors giving trouble?
Some other points I need to pay special attention?

Thanks for your help,
Patrick


Navigation, Location and Dead Reckoning
#16

Hi all,

I just ordered a kit and IMU, haven’t received it yet. I started reading through this and had some thoughts. Without seeing an ROV and IMU in front of me, these are general suggestions.

First, take a look at power integrity at the IMU as motors are switched on and off. Anything electro-mechanical for that matter. Large current draws at switching moments may impact power at the IMU. The pressure sensor may be less susceptible to this as a virtue of its design (it is a separate chip on the IMU if I understand correctly). This isn’t a fix, but may get to a root cause.

To aid in troubleshooting, getting a scope on power and triggering off of control of the electro-mechanics might show you if this couples into the system as noise.

Second, if ferrite and additional twists are improving performance for some folks, try some caps as well to suppress noise from the system. I realize this is a hardware mod, but may be necessary. This doesn’t identify a root cause either, but it does treat the symptom like the ferrite and may get people going with less trouble until a root cause is identified.

Again, general suggestions, I hope this helps.

Best regards,
Lenny


#17

So I was playing with using the MPU9150 in pass through mode which means it no longer tries to manage the magnetometer. What I have found is that the magnetometer is super sketchy and stops responding on the I2c bus. I suspect that the MPU9150 was not robust enough to handle the magnetometer lock ups and created a cascading failure. In the pass through mode, the gyro and accelerometer seem to continue to run fine even though the mag chokes. Still testing.

Right now I am working on ways to recover the magnetometer when it gets in this state. I’ll need to do some more testing on other units to confirm these initial findings, but I am hopeful.

-Brian


#18

-Update-
So in my testing, driving the magnetometer directly seems to have worked around the issue. I’m finishing up some code with the right orientation (it also re-enabled true compass) and will post the patch for early testers soon.


#19

Can you throw out some details when you get a chance? I’d like to familiarize myself with the issues. I haven’t seen it yet, but I’ve been limited in depth of the IMU, focusing on the math for position estimation, and not into operational characteristics. I assume you are benchtop testing, any strong EMI around that may throw the mag into disarray?


#20

@badevguru has been hard at work and has a software patch that we would like to have users test. If you have an OpenROV with a non-working IMU that matches the description in this post and would like to help us out by testing this patch, please PM me your email address so we can get the new software sent to you.

-Brian Grau