Feature: New UI theme based on webcomponents

software

#1

Software cleanup and adding multiple UI’s

Over the last few weeks I was busy working on the implementation of a new UI based on Eric’s ideas.

Around that, there was a lot of cleanup and streamlining done.

Google Ploymer/Webcomponents

With the rework we descided to move a lot of the current and new UI over to Google Polymer based on Webcomponents.
To learn more about Polymer check out the Polymer Project: https://www.polymer-project.org/

Extending the UI with additional elements

So far, plugins would use jQuery to find elements on the UI that they want to extend, for example adding something to the menu bar.
As this gets a bit tricky with webcomponents (because a so called Shadow DOM is used, basically independent sub documents that are loaded similar to iframe’s) we introduced extensionPoints where you can register your elements.

You can find the extensionPoints on the cockpit object when you define a plugin.
Current extensionPoints are:

cockpit.extensionPoints = {
    rovSettings: // * <-- The settings panel
    rovDiagnostics: // * <-- The diagnostics panel
    videoContainer: // * <-- The video container
    keyboardInstructions: // * <-- Keyboard Instructions
    buttonPanel: // * <-- the buttons panel
    menu: // * <-- the menu bar 
    inputController: // <-- the input controller
    headsUpMenu: // <-- the heads up menu (can be undefined if the menu is not loadded)
}

The extension points marked with a * are jQuery objects and you can use functions like .append().

InputController

So far, to register something with the inputController, you would send a message. With the extension points you register things like this:

self.cockpit.extensionPoints.inputController.register(
  [{
    name: "example.keyBoardMapping",
    description: "Example for keymapping.",
    defaults: { keyboard: 'alt+0', gamepad: 'X' },
    down: function() { console.log('0 down'); },
    up: function() { console.log('0 up'); }
  }]);

HeadsUpMenu

To register something in the heads-up menu, use this:

var item = {
  label: ko.observable("Example menu"),
  callback: function () {
    alert('example menu item from heads up menu');
    item.label(this.label() + " Foo Bar");
  }
};
if (this.cockpit.extensionPoints.headsUpMenu) {
  this.cockpit.extensionPoints.headsUpMenu.register(item);
}

Messaging

One of the biggest changes beside the UI is certainly the way the UI and the NodeJS communicate.
So far, every UI had to listen to the ‘connection’ event on the io.sockets object.
Now, if you want to listen to something sent by the NodeJS backend:

cockpit.rov.on('plugin.example.message', function(message) {
  alert(message);
});

This registers for the message plugin.example.message send by the NodeJS part of the plugin (or another plugin).
You want to send something to the ROV:

cockpit.rov.emit('plugin.example.foo');

On the NodeJS end:

deps.cockpit.on('plugin.example.foo', function() {
  deps.cockpit.emit('plugin.example.message', 'bar');
});

If you want to send something from the NodeJS to arduino:

deps.rov.send('arduino_cmd(0)');

To read something from the status response on arduino:

deps.rov.on('status', function (status) {
  if ('someArduinoValue' in status) { 
    // do something with status.someArduinoValue
  }

Simplified communication between the ROV/Arduino and the Client/Cockpit

Especially when you’re going development for new plugins with code running on Arduino, you had so far to write quite a bit of boilerplate code to get the messages across the NodeJS backend even if you don’t want to do anything with the values there.
To minimize this boilerplate code use the following code:

deps.rov.registerPassthrough({
  messagePrefix: 'plugin.example',
  fromROV: [
    'example_foo',
    'example_bar'
  ],
  toROV: [
    'example_to_foo',
    'example_to_bar'
  ]
});

This copies all the messages that come from the ROV (in this case example_foo and example_bar) over to the cockpit, prefixing it with plugin.example..
On the other hand, messages from the UI plugin.example.example_to_foo and plugin.example.example_to_bar are stripped of their prefix and forwarded to Arduino.

Cleanup

Beside the changes mentioned above there was a ton of cleanup done.
Where the rovpilot and capestatus plugins where pretty intermixed so far, they are now split and have actually no direct interactions with the UI itself beside the settings and diagnostics. All the UI action is now done via messages and webcomponents.

Beside that, a lot of processing was done on the client and only ‘status’ messages where sent from the backend to the client.
This was changed so that the actual handling is done on the NodeJS side.
This means that we have now the option to interact with the NodeJS end from other frontends, for example from other languages.

Messages that can be sent to the ROV:

videoStatus
ping
update_settings
disconnect
arduinofirmware-startupload
arduinofirmware-uploadfromsource
arduinofirmware-upload
plugin-finder.search
plugin-finder.list
plugin-finder.install
plugin-finder.uninstall
plugin.cameraTilt.set
plugin.cameraTilt.adjust
plugin.diveprofile.watertype.toggle
plugin.laser.toggle
plugin.lights.toggle
plugin.lights.adjust
callibrate_escs
motor-diag.motorTest
plugin.navigationData.zeroDepth
plugin.navigationData.calibrateCompass
plugin.photoCapture.snapshot
rovpilot.allStop
rovpilot.setThrottle
rovpilot.setYaw
rovpilot.setLift
rovpilot.setPitchControl
rovpilot.setRollControl
rovpilot.setPowerLevel
rovpilot.adjustVerticalTrim
rovpilot.adjustThrottleTrim
rovpilot.incrementPowerLevel
rovpilot.powerOnESCs
rovpilot.powerOffESCs
rovpilot.headingHold.toggle
rovpilot.headingHold.set
rovpilot.depthHold.toggle
rovpilot.depthHold.set
rovpilot.manualMotorThrottle
rovpilot.disable
rovpilot.enable
rovpilot.powerLevel.request
plugin.serial-monitor.toggle

Messages from the ROV:

arduinoFirmware-output
arduinofirmware-requestmoredata
arduinoFirmware-status
arduinofirmware-uploaddone
capestatus.battery.config
capestatus.battery.current.out
capestatus.battery.voltage
capestatus.connection.connected
capestatus.connection.disconnected
capestatus.cpu
capestatus.request.battery.config
capestatus.time.runtime
capestatus.time.time
cockpit.pluginsLoaded
plugin.cameraTilt.angle
plugin.input.gamepad.connected
plugin.input.gamepad.disconnected
plugin.laser.disabled
plugin.laser.enabled
plugin.lights.level
plugin.navigationData.data
plugin.photoCapture.photos.added
plugin.photoCapture.photos.updated
plugin.serial-monitor.serial-received
plugin.telemetry.cycleTextColor
plugin.telemetry.logData
pong
rovpilot.control_update
rovpilot.depthHold.disabled
rovpilot.depthHold.enabled
rovpilot.headingHold.disabled
rovpilot.headingHold.enabled
rovpilot.powerLevel
rovsys
settings
status

Roadmap

Currently, the development is done on https://github.com/codewithpassion/openrov-software/tree/feature/new-ui.
In the next few days there will be a lot of testing done and then it should end up in master pretty soon.

What do you think?
What would you like to see?


#2

All the code is pretty beyond me, but I am looking forward to the new UI!

My only request is that it still remains “open” and we can still try out the software builds before it becomes an official release candidate. I really liked beta testing the last round.


#3

When it comes to codes and stuff, I am a big noob :confused:

I therefore hope the new cockpit will be more plug n play…and maybe a option for just drag n drop or just click on plugins and different themes we like?


#4

@Kevin_K: Yes, the code is still open, it’s currently in my development branch (see the Roadmap heading) and after some testing in the next days will be moved to the main OpenROV development branch and then be available as beta versions.

@Tom_Vidar_Salangli: There is an option to change between the different UI’s, yes. When it comes Plugins: We work on make it easier to install them too.


#5

@codewithpassion, any screen shots on what the new UI is shaping in to?


#6

Great work Dominic! I look forward to see how this shapes up!

I’m currently taking an interface aesthetics course for my graduate studies, and I’m working on designing a OpenROV cockpit UI for the final project. I’m focusing on the design of the aesthetics and usability of the interface, and I’m wondering if I could build it upon the webcomponents theme you are making? I will have a UI design by mid-May. I think it would be cool and good to build this as a theme we could use for the OpenROV.

I’ve attached two UI design process documents that I’ve made so far:
Process2: details the UX background research; What are the objectives, who are the users, what are their interactions, and a preliminary user flow diagram.
Process3: Is an aesthetic analysis of two UI’s I want to draw inspiration from; Metroid Prime and Star Citizen. I plan on going for a diegetic interface, since my design objective is to create a compelling and immersive UI.

Process2- User and Interaction Research (587.0 KB)

Process3 - Comparative Aesthetic Analysis (2.0 MB)

I’m currently designing different UI wireframes to experiment with different interface layouts. I will post my work after I finish it today. I’m leaning towards a HUD-focused interface that maximizes the video area through overlayed UI elements.

My questions:

  • Will that be possible/easy to implement a complete UI (menus and all) with your webcomponents theme framework?

  • Are overlayed translucent UI elements possible?
    @codewithpassion
    @Eric_Stackpole


#7

@badevguru here you go


#8

@Colin_Ho The thing with UI will be that we won’t get a ‘one fits all’. There’s always going to be different opinions. But great work on the UI patterns.

For you questions:

  1. Yes, the ‘New-UI’ is implemented as a ui-plugin and the existing one was moved to one as well. Common elements that should be useful for others, are added as webcomponents. things that are fairly specific to the old ui are not yet exposed as a component. Check out the 2 current UIs.
    On that note, I already converted the simple UI from the cummunity into a new plugin. This will be a separate PR to the owner.

  2. Yes. the artificial horizon is already such a UI element.

  • Dominik

#9

@Eric_Stackpole, @codewithpassion what do you think about making the switches clickable


#10

There is code in place to allow them to change on click bit it needs to be set up to the appropriate messages.


#11

Ok, The change was merged to master!

There are a issues that are setup to address a few questions or tasks:
https://github.com/OpenROV/openrov-software/issues/346
https://github.com/OpenROV/openrov-software/issues/347
https://github.com/OpenROV/openrov-software/issues/348
https://github.com/OpenROV/openrov-software/issues/349


#12

Hi

How can I download and use this UI?
I really love it. It is alot more intuitive for me :smile:

Regards
Tom-Vidar Salangli


#13

@codewithpassion
On the new UI. Got some more feedback from @Eric_Stackpole / @Walt_Holm trying it on some Tahoe dives this weekend:

  • The HUD does not really add value with this theme. It would be nice to set it to not be shown by default.
  • Really miss not having the keyboard icon to hover over to remind what the key mappings are.

#14

This looks great Dom. I’m lost with the code, but I’m certainly looking forward to testing the new UI.


#15

@codewithpassion, I would like to test this UI. If I download the following folder:

https://github.com/codewithpassion/openrov-software/tree/feature/new-ui/src/ui-plugins/new-ui

and paste it into the plugins folder on the ROV at /opt/openrov/cockpit/src/plugins

will I be able to run it as a plugin in the same way as @Joakim_Karlsson simpletheme?


#16

I’ve posted the todo list for the next refactor for the themes infrastructure. As I have been working through the themes, we always end up with a list of things we did not get too or later discover fall short of the experience we would like. This is that list. It will probably be the last significant refactor before a final release.

https://github.com/OpenROV/openrov-software/issues/506