With a voice interface, there is no longer a need for screens – right?
Wrong! Natural language voice interfaces are great for issuing commands without having to navigate menus and dialogs to “get to the right screen”. At times a quick verbal response contains all the info you need. But humans are visual creatures and the old saying is true, a picture is worth a thousand words.
The classic example is a basic weather skill. “What’s the weather like?” is easily answered with “63 degrees and sunny”. But nobody wants the response to “Tell me the weather forecast for this weekend” to drone on with the high, low and expected condition for three days. And the 7-day forecast? Ain’t nobody got time for that. But a single graphic display can convey all of that information in a second to your highly efficient visual cortex.
Visual interfaces can also answer questions without even being asked. Start a timer and see the remaining time on the screen with a silent glance. Get the name of the current song without breaking the mood. Visual interfaces are excellent mechanisms for conveying information.
For anyone who has extended Mycroft for their own purposes, the concept of a Skill is well-known. Vocabulary, Intents, Intent Handlers, and Dialog are the basic pieces needed to connect your cool API or custom code to a verbal interface. The Mycroft GUI extends this elegantly, providing access to the simple-but-powerful visual interface technology built by KDE and Qt.
For simple interactions, Mycroft provides basic functions for displaying text, images or web pages. WAV and MP3 files can also be played and controlled with an attractive playback interface.
The hello world of the voice world is “What time is it?” Mycroft easily handles this with code like this:
@intent_file_handler("what.time.is.it") def handle_time_query(self, message): current = now_local() self.speak(nice_time(current))
This will catch phrases similar to the examples in the “what.time.is.it” file, answering with something like “six oh two”. Nice and easy.
Adding a visualization to this is almost as easy:
@intent_file_handler("what.time.is.it") def handle_time_query(self, message): current = now_local() hh_mm = nice_time(current, speech=False, use_24hour=False) self.gui["time_string"] = hh_mm # e.g. "7:00" self.gui.show_page("clock_face.qml") self.speak(nice_time(current))
These extra three lines tell Mycroft to look for an available GUI, load up the page called
"clock_face.qml", and pass it the value
"hh_mm" with the current time as a string like
"time_string" variable into Hour and Minute at the colon, adjusting the QML elements to show that time. With that, you have your clock!
You don’t want to have to constantly speak to your clock just to check the time. With the Mycroft GUI we can easily update the
"hhmm" variable in a simple timer callback, something like this:
def initialize(self): self.schedule_repeating_event(self.update_display, None, 1) def callback_time(self): current = now_local() hh_mm = nice_time(current, speech=False, use_24hour=False) self.gui["hhmm"] = hh_mm # e.g. "7:00"
This calls your function once a second, updating the
"hhmm" variable with the current time. That’s all you need to do. As long as the page is showing, the QML will receive a notification only when the
"hhmm" value actually changes, then the position of the hands will update accordingly. Easy peasy!
QML can also change values, sending a notification back to the skill when some screen interaction has occurred. A skill can capture this by defining a notification handler using code like:
def initialize(self): self.gui.set_on_gui_changed(self.handle_gui_notification) def handle_gui_notification(): # Look at your skill variables (one of them is different) and # react accordingly!
These notifications could occur in response to the user moving something on-screen: things like dragging the clock hands, clicking a button, or filling a form with text and pressing the Save button.
With just those basic hooks, you have what you need to display nearly anything you can imagine AND react to the user interactions on a screen. Verbal and graphical interfaces can live in harmony, each doing what they do best.
One of the most powerful things about this GUI design is that you don’t even need to have a device with a screen. Other devices can register on the Mycroft Messagebus, allowing for future devices to show a GUI on things like a TV connected to a Raspberry Pi or a streaming media player which implements the Mycroft GUI protocols. Mycroft will be able to take advantage of whatever screens are visible to the user!
You can learn more about the Mycroft GUI by reading the tutorial on Github, and by examining the code of our examples. There is excellent documentation for the QML markup language and the Kirigami visual framework.
Use your creativity to push the bounds of QML and Mycroft. When you get something cool going, post a link to it on the forum!
Steve has been building cutting edge yet still highly usable technology for over 25 years, previously leading teams at Autodesk and the Rhythm Engineering. He now leads the development team at Mycroft as a partner and the CTO.