One the largest challenges we've faced at OpenROV has been trying to figure out how to make tools with advanced functionality using common, low cost resources and a very minimal internal engineering team. The IMU/ Depth sensor we've developed is one of my favorite examples of this because it uses parts that are not very expensive (the depth sensor is commonly use in SCUBA diving computers, and the IMU can be found in many cell phones) yet it makes people's ROVs much more capable.
In a few cases, we've started to notice that the compass and orientation data in Cockpit sometimes freezes up during a dive, and we're not sure why. The behavior we're seeing with the data bus on the IMU chip right now is one of these types of challenges where getting outside help from our community is something we could really use. This is one of the reasons why we've made our project open source. We've done quite a bit of experimentation and head pounding, but we still can't get to the bottom of exactly what’s causing the problem or how to fix it.
Here is everything we know about the issue so far:
NAK - On some ROVs, data from the IMU sensor "locks up" (the number of Not Acknowledge or "NAK" increases dramatically and does not recover) causing the the compass, roll and pitch data to freeze in cockpit.
IN WATER - The problem seems to almost always occur only when the ROV is placed in water (both fresh and salt water).
WET TETHER - The problem seems to only occur when the tether and the ROV chassis are in the water at the same time.
DRY IMU - The problem can occur when the IMU module itself is outside the water but the I2C lines and the chassis are in the water.
DEPTH OKAY - The problem does not happen with the depth sensor data which uses the same I2C bus to communicate- the depth sensor data continues to stream in just fine. In other words, The problem only affects the MPU9150, the Arduino to depth sensor communication remains unaffected during these events.
FOLLOWS ROV - The problem seems to follow the the ROV and not the IMU sensor- in other words, if we have a working ROV and one with this problem, and we switch the IMU sensors, the problem stays with the ROV even though it has a new IMU sensor.
ESCs OFF - The problem has been reproduced when the ESC primary power is turned off, but we acknowledge that the servo line is still providing a bit of power to the ESCs.
WAITING FOR REPLY - When the problem occurs, the Arduino default wire library gets stuck in a state where it is waiting for a reply on the I2C bus from the MPU9150. The modified Arduino library adds a timeout that resets? the MPU9150.
DEPTH DISABLED - The problem occurs even when the depth sensor code has been disabled.
RESET ARDUINO - After the problem occurs, resetting the Arduino (using the software reset script) will eventually restore connectivity to the MPU9150, but not right away. We often see delays of up to a minute where the reset of the Arduino has no effect. This only works if the ROV has been taken out of the water. If the ROV stays in the water, resetting the system will not recover the IMU.
V2.7 - Although similar behavior had been noticed once or twice previously, instances of the problem occurring seemed to increase dramatically with the release of V2.7. V2.7 has a different wiring harness as well as other changes.
TWISTED HARNESS - Previous designs had the (twisted) tether routed all the way through the endcap to the DB-25 connector, but v2.7 calls for the tethered to be soldered to untwisted leads in the wiring harness that go to the DB25. When we’ve manually twisted the tether wires (twisting helps reduce electrical noise transmission) in the harness as well as the I2C wires going to the IMU/Depth Sensor, the problem seems to occur less frequently.
Theories on cause:
NOISY TETHER/ SENSITIVE IMU - Electrical noise generated by the tether capacitively couples with the I2C bus wires in the wiring harness (capacitive coupling is enhanced with water as a dielectric between the wires) which causes the IMU (which is overly prone to failure from noise) to drop out.
IMU chip used: MPU9150
SW used to communicate with IMU: https://github.com/OpenROV/openrov-software-arduino/blob/master/OpenROV/MPU9150.cpp
Depth Sensor chip used: MS5803