"
The true sign of intelligence is not knowledge but imagination.
"
- Albert Einstein
User experience for virtual assistant, defining instructions and workflow
2016 - 2018
Galaxy S8, Galaxy S9
Bixby is a virtual assistant developed from 2016 to 2017 and applied to Galaxy S8. At that time, Samsung tried to renew a virtual assistant system to enhance the performance of voice recognition and personalized experience by utilizing AI. The messages team is selected as one of the three apps (Calendar, Gallery, Messages) to design the initial instructions for the demonstration. I collaborated with AI team, developers, and pear designers to build initial instructions to enable users to operate every function by voice in the messaging app, built over 40,000 instructions and workflows. Bixby is now applied to many of the Samsung devices, including smart TV, and home appliances such as robotic vacuum, refrigerator, washing machine, etc.
The assistant should understand when a user is natural or ambiguous in their spoken utterance. It led us to design one standard instruction and varies into 4 to 5 different variations in each of every function.
We see many cases that when making an error, an assistant will answer, “Sorry, I cannot do that.” But many of the cases, they can still cover the mistakes by analyzing previous behavior or asking questions to complete tasks.
One of the pain points users feels when completing tasks with voice is time inefficiency. It mostly takes more time than swiping and tapping to complete tasks. I compared the usage data and listed down all the functions so that when there is an overlap between high usage features, contacts, or messages and low usage counterparts, It should run main functions based on usage data and previous behaviors.
I listed all the features of the messaging application and started to create map instructions in an excel chart. First, I listed down all the screens and functions on the screen. And then, I created the basic utterance related to the features. After the job is finished, I started to work on combinations. It was 40,000 instructions only for messages.
In each of the instructions, I created a flow chart to see how the assistant work. What users see when interacting with a virtual assistant might seems simple and nice. But the hardest part for designers to create flow is the combination of different functions and apps. For example, when a user utters “Hey Bixby, send dad photos from Tokyo and say I love you.”, I had to define all the side cases, including errors and questions.
Bixby was successfully launched and gained positive feedbacks from multiple media and the public. The system is integrated not only to mobile devices but smart home appliances such as Smart TV, Robot vacuum, refrigerator, etc. to control by voice by phone or Samsung hub.
