Sr. UX Designer, Microsoft, 2011-2013
I designed the voice interaction on Xbox One from initial concepts through launch. Working with the dedicated Xbox speech designer, who was new to voice recognition, we used the strengths and shortcomings of Xbox 360 to build a whole new interaction from the ground up.
The Problem
The speech team at Microsoft, housed inside the Bing division, was working toward an intelligent system called “conversational understanding” (CU), which allowed for robust speech recognition, complex intent detection, and routing to provide an appropriate response. That response could be an answer to the question, a follow-up prompt, or a collection of results.
My job was to integrate CU into the most appropriate systems. The first system was Xbox One and the best candidate was the Search experience.
What I Did
For search, we needed to explain to users how this new robust interaction was different from existing keyword search. We also needed to explain the scope of the CU interaction, namely that it would only work within the Bing search area.
The Speech Interaction Framework proved invaluable as I worked with Xbox One. We were able to use it to articulate some of the interaction shortcomings of the two-word model (“Xbox” for global commands, “Xbox select” for on-screen commands). It also helped show areas where we could explicitly teach the new interaction of flexible querying in CU for search.
What Shipped
Xbox One shows how the intelligent system of CU allows for complex queries.
It begins by giving users an open-ended prompt (“Say what you’re searching for”). There are also tips of what you can say at the bottom of the screen.
The user says what they want in a natural way (e.g., “show me some new dance music”), which is easier than trying to figure out how to ask for the same thing using onscreen menus. The CU system translates that into a query (music – electronic / dance – new) that returns rich results.