Troubleshooting IMU2 orientation data freezing shortly after boot


#1

I recently installed the new IMU, and upgraded to software 30.0.2, but can’t get the new IMU to work. Although the new IMU is wired up, I have left it unpotted at this stage, to test it before committing to epoxy.

The cockpit displays normally, but I’m receiving no heading or attitude information. The HUD shows a level attitude, and does not move when I move the ROV. The heading tape shows North, and does not move at all.

A possible cause (although I’m guessing here) is that my software version is reported differently in two locations. Hovering the cursor over the COCKPIT logo inside the ROV cockpit (192.168.254.1:8080), it says that I’m running 30.0.0. On the DASHBOARD (192.168.254.1) under the SOFTWARE tab however, it says I’m running 30.0.2 and that my ROV is up to date.

Any ideas, anyone?

My ROV is operational with the exception of the IMU data. One foible is that the lights occasionally start flashing, and hitting the I,O and P keys to change the illumination level doesn’t seem to stop the flashing.


New IMU / Depth Sensor
#2

Hello, I have the exact same problem, not sure how to solve it. We have upgraded to 30.0.2 and on the dashboard it is the old one 30.0.0 as on the picture above. Everything else works except the IMU. It is the BNO005 version. Please advice how to debug and address this issue. Still haven’t put the ROV in action because of this problem.

The upgrade was done through a mindsdk without any problems, but the IMU was not working even with the stock firmware which version I haven’t written down.
I am pretty sure it is well wired and connection is good.
On the Dashboard there is a message BNO055.status failed. Not sure what this means!


#3

The BNO055.status failed means the Arduino was unable to communicate to the IMU chip.

If there are other BNO055 messages such as BNO055.test2. something and you have the status=failed message, it would indicate the IMU was working and then stopped. If the only message is the BNO055.staus=failed, then it indicates the IMU was not able to communicate at all.

If you are getting depth information (other than 0) then the I2C bus in general is working and the issue is limited to the IMU.

I belive support is also troubleshooting this issue with James so I’ll make sure we circle back an update.


#4

Hi, Sorry, but I am new and can’t post images. Here is what I see - https://www.dropbox.com/s/4vgpj9h4f0a3qco/photo.JPG?dl=0

I guess the -0.76 value is the depth that you are talking about. It changes from time to time.

p.s. just installed the software from master through the dashboard. No change. Just fixed the motors issue that were not running properly when left/right.

Thank you for the support, we need to run this ROV for a project pretty soon and this is essential.


#5

Hello,

After two days of trying different approaches, we still haven’t reached much of a success. The best story we have so far is

  • The sensor seems to works after reboot for less than 1 second then the ESCs beep again and then stops
  • Uploaded on hand the arduino code only from master - the sensor is reported as working, but nothing changes on the UI. We have seen this happening once.

We will try the development image today, but can you give us some clues on how to debug this issue. Do you have any progress with James?


#6

Thanks for the detailed report. We are waiting on one of the units that exhibit this problem to make it back to our lab so that we can take a look. It’s on the way.

Which version rov controller board do you have? 2.7, 2.6, 2.5…


#7

It is 2.7. In the meantime is there anything I can do to trace the problem? Any code that just tests the sensor only?


#8

The primary issue is to isolate the problem. Basically, if the IMU runs on a dedicated Arduino without fail that is a good data point. If then the IMU runs with our existing software and controller board with its own dedicated power to the IMU, that would be a good data point. Knowing if the IMU continues to respond to an I2C bus scan even when not responding correctly to the device ID check would also be a good data point.

Some ideas in troubleshooting it:

  1. If you have an Arduino handy, you can easily adapt the code we have on board for the BNO055 to work alone. You basically cut and paste the cpp file code in to a new sketch and rename the device_setup and device_loop to setup and loop. Just a few other lines to cleanup and it should work. There are also some libraries on github for the BNO055 you can grab.

2a. Assume that there is something regarding the power to the device. We know the 2.7 board has limited headroom on the 5V regulator it is using. Could the board be drawing more power than the 2.7 board can handle? If you have a bench supply, go ahead and power the IMU with dedicate 5V power, link the ground between the power supply the the ground off of the ROV, and see if the issue goes away.

2b. Alternately, the BNO055 has low power mode options that are currently disabled, review the spec sheet and try initializing it in low power mode to see if that helps.

I’ll keep thinking about it, but I don’t have much else to go on without a failing device in hand.


#9

The ROV completes the boot-up (beeps when powered up, beeps again about a minute later to indicate that it’s ready) but beeps again about 15 seconds later for some reason. If I open the Chrome browser within this 15 second period, I see the cockpit with a heading other than “N” and an attitude other than level. After that third beep however, the compass displays “N” and the attitude is fixed at level. No amount of rotating the ROV causes the attitude or heading information to be displayed.

If I shut everything down (including the laptop), restart the laptop, and reboot the ROV, and allow all three “beeps” to complete before opening the Chrome browser, I get a level attitude and a compass fixed on North. No modification of boot sequence has any effect.

I’ve done all this on batteries so far, always ensuring that I’m using well charged batteries. I’ll now try again on external power, although this would obviously only be of benefit in troubleshooting, as I wouldn’t be able to boot with external power in the field.


#10

This is the current data being returned.
This is an unmodified build. ROV 2.7 + IMU2.
Any ideas, anyone?

· hdgd 0.00
· deap -0.14
· pitc 0.00
· roll 0.00
· yaw 0.00
· fthr 0.00
· motorAttached 1
· servo 1500
· starg 1500
· fmem 3542.00
· vout 9.03
· iout 0.54
· atmp 0.00
· verb1ccff3fc29c1f3676bb773083295d449f526985 -
· cmpd Jul 29 2015, 10
· cpuUsage 0.36986301369863017
· time 69276.00
· pres 1001.30
· temp 26.33
· dlms0|69091|48412|18683|12574|181005|40526|13167|16268|23819|416210|336811|2265
· alps 2725
· BRDT 29.57
· SC1I 0.02
· SC2I 0.03
· SC3I 0.04
· BRDI 0.53
· BT1I 0.37
· BT2I 0.34
· BRDV 9.01
· AVCC 5023
· mtarg 1500,1500,1500
· mtrmod-1.00,1.00,-1.00,-2.00,2.00,-2.00
· BNO055.CALIB_MAG 0
· BNO055.CALIB_ACC 1
· BNO055.CALIB_GYR 3
· BNO055.CALIB_SYS 0
· BNO055.MODE 12
· log done
· MPU9150.status Not found
· MS5803.status reset
· Depth Calibration constants are
· Depth.C0 45057
· Depth.C1 34226
· Depth.C2 33853
· Depth.C3 20402
· Depth.C4 21146
· Depth.C5 27271
· Depth.C6 26426
· Depth.C7 26426
· BNO055.Address 40
· BNO055.WHO_AM_I 0
· BNO055.IAM_160 255
· BNO055.status failed
· BNO055.SW_Revision_ID FF.FF
· BNO055.bootloader 255
· BNO055.selftest.acc passed
· BNO055.selftest.mag passed
· BNO055.selftest.gyro passed
· BNO055.selftest.mcu passed
· BNO055.selftest2.acc passed
· BNO055.selftest2.mag passed
· BNO055.selftest2.gyro passed
· BNO055.Initial_Mode 255
· boot 1
· rawcmd9E,70,69,6E,67,28,30,29,3B,
· crc pass
· cmd ping(0)
· CAPA 206
· LIGT 0
· LIGP 0.00
· pong 1,68932


#11

I confirm that everything works after the first dive. We have debugged this for three days on a dry dock. I have attached external power supply, the arduino was restarting after the initialisation of phase 2 and we had big surprise to see everything functional when we put the rov into water. It had to be restarted 1-2 times as it was not responsive at the beginning and we had to wait some extra time on the reboot, but then suddenly the sensor was fully functional for the first time in the water, video stream, compass hold and depth hold work very good. All this with the stock image from the stable release. Really not sure what was causing this issue!


#12

Are you able to boot it successfully without external power now?


#13

Hi all,

Just a suggestion that following the first dive the ROV has restarted itself few times then all did start to work properly including the IMU2 sensor which we have been struggling with Nmanchovski.
So, I guess you should definitely get it dive then will find out whether the IMU will be working or not. It seems that nothing else fixes the problem so far. We have also connected the BB to external power supply and updated several times the software but did not work out.
Currently have spent half an hour in the tube having 30.0.2 OpenROV firmware from the stable branch.
Everything is seems to be working properly so far. Will update in a few days as well.
Enjoy your build and get it dive right away.
Stefan


openROV V 2.6 software
#14

Hi James,

yes, it works on its own, but it need one-two reboots at the beginning and a little patience.


#15

Hey guys,

For anyone that is still experiencing the problem we have a couple things to explain and play with.

The code for the Bosch sensor tries to talk to the Bosch, if the sensor is not responding then the system tries again after 30 seconds. When the Bosch does respond, the code first runs the sensor through it’s self test, which typically takes about 4 seconds.

For the system’s that are getting a third beep, I suspect that the Bosch sensor is not immediately available after the 2nd beep (which is an intentional arduino reset). So the system waits 30 seconds and then tries to initialize it. As it does so… for some reason I cannot yet explain, it is possible the self test code takes more than 8 seconds which triggers a hardware reset of the Arduino… the 3rd beep.

One thing you can try, and I would like to know how it goes, it disabling the “2nd beep”, which is the intentionally Arduino reset. That line of code can be found here: https://github.com/OpenROV/openrov-cockpit/blob/master/etc/rc.local#L36

To disable it you have to ssh in to the rov. The you can use
sudo pico /opt/openrov/cockpit/etc/rc.local and then you can put a Hash tag in front of the line that says /opt/openrov/cockpit/linux/reset.sh to disable that line. Then press control-x to exit and save.

Reboot, and the system should only have the single beep on initial power up. My hope is that the Bosch sensor is stable on initial power and just works… and that something about how we are reseting the system might be causing an issue. Just speculation at this point.


#16

Interesting update. Not sure the change I recommended above actually does what I wanted. Tried it locally and the “2nd” beep was still happening. :frowning:

I have a pending code update to the Arduino firmware for the BNO055 file that will remove the “third” beep from the system and hopefully make a more graceful auto retry of the sensor when it has initialization issues. Will have that posted in the next 24 hours or so.


#17

Thank you. I’m looking forward to getting it up and running. Any progress?


#18

Update:

So @spiderkeys has been kicking butt characterizing the issue with the IMU. We have been able to successfully recreate the issues that folks have seen. If you have a 2.8 controller board, you should be fine. For earlier controller boards, we have updated Arduino firmware code that we are putting out in a new beta release sometime this week.

Going deeper…

Ultimately, when we were cycling the Arduino, we had situations where the I2C conversation with the sensor can be left incomplete. The sensor in this case is very intolerant of that and does not recover gracefully. Those 2nd and 3rd beeps of the ROV startup sequence are restarts of the Arduino. Given the frequency of the communication with the IMU sensor, the Arduino reset almost guarantees hanging the sensor. Those restarts have been isolated and removed from the system.

Totally geeky temporary work around to prove to yourself that your IMU works: If you have a 2.7 controller board or earlier model with the IMU2 there is a hack to see it working while you wait for our software fix. This hack is only good for those with the ROV running on your desk. You can wait for the entire system to boot up, remove the power from the IMU, wait a second, and then reapply the power to the IMU and it will respond.

Thanks for your patients. Happy to answer questions.

-Brian


#19

Hi, guys!

This is my first post here! Finished building OpenROV 2.7 last week. Upgraded software to 30.0.2., but I have the same issues as jamesyoung. Cockpit shows I’m running 30.0.0., dashboard that I’m running 30.0.2. Controls are a bit twitchy, but after some time it all starts to work fine. Except IMU, which worked at some point, but now it constantly sais BNO055.status failed. And it peeps and (I guess) resets constantly.

So, my question and request goes mainly to badevguru - when can we try the new BETA? I’m supposed to test the ROV in the sea next week, so I hope it will be available soon!

Best regards,

Ozi


#20

Hi Brian
Is there any progress? I’m getting a bit time critical to get this up and running.
James