Assistant

Find someone unfamiliar with your project to try out your dialog. Getting feedback during the design process exposes usability issues and gives you the opportunity to self-correct early. Before you write a single line of code, it’s important to run a usability test on your conversational experience. We recommend conducting a quick and dirty Wizard of Oz (WOZ) experiment to help you figure out if you’re on the right path.

Use what you have. All you need is your sample dialogs (which you should already have at this point). Simply find someone unfamiliar with your project (e.g., family, friends, colleagues) and ask them to role-play your dialog with you—you’ll read your persona’s lines and observe how they react as the user. If the user goes “off script”, feel free to improvise what your persona would say.

For the most realistic experience, simulate the persona’s role by playing the persona’s prompts using the TTS Simulator in the Actions on Google Developer Console. Download the audio to have it ready to play on demand.

This version requires four things:

• A conversation script that provides directions on what the persona should say after each user response. The high-level flow (or a simplified version of it) is ideal for this.
• Downloaded audio of all the persona’s spoken prompts. Use file names that will help you quickly identify the correct file to play.
• Someone to play the "user." This should be someone who’s unfamiliar with your Action.
• Someone to play the “wizard." This should be someone highly familiar with your Action.

Have the wizard start the conversation by playing the audio for your Action’s greeting, for example, “Welcome to your launchpad for all things Google I/O. The festival's underway right now. Are you one of the lucky attendees?” The wizard will then wait for the user to respond, hopefully with a synonym of “yes” or “no”. Once the user has responded, the wizard will have to quickly consult the high level-flow to determine what prompt to play next, then find and play the correct audio file.

Of course, once you’ve started building your Action, you should test it often using the the Actions Simulator in the Actions on Google Developer Console. Have your friends, family, or colleagues test it too!

No matter what experiment you use, be sure to do the following:
Talk it out	Since your goal is to update your design to reflect what works best for real users, you want your WOZ prototype to be as close to reality as possible. What looks good on paper doesn’t necessarily sound or feel natural in real conversation, so make sure users are hearing your prompts and speaking their response.
Record your sessions	Get permission to record your sessions so you can go back and listen to them. Take note of any issues that arose during the session.
Ask for feedback	Ask the user to describe their experience in their own words. How did it meet or fail to meet their expectations? Did anything surprise them? Were they satisfied? Remember that the focus is on their behavior, not their opinion.

Running a WOZ experiment allows you to understand how people will engage with your design. You may find that users were doing something very different than what you had expected, requiring you to alter the design to better align with their needs and expectations.

Bottom line: Focus on the usability of your design (and not on users’ opinions). Iterate based on user behavior, and test again if time permits.

Things to look for (and how you might improve your dialog):
Natural conversation	Pay attention to the way users naturally ask for things. Do they feel like they can only speak in short keyword-like phrases, or do they sound more conversational? Do they sound hesitant or confident when speaking to your persona? Does the flow make users feel like they can only provide one piece of information at a time, or does it encourage them to provide multiple details in one sentence?
User confusion	Look for places where users look confused or are unsure of what to say or do. Examine the previous prompts to see where you could make some clarifications. Was the call to action clear?
Unexpected  utterances	Users might say something you didn’t expect. Take note of it and add handling for it in your design.
Signs of frustration or impatience	This is typically a sign that the interaction is too long-winded. Review your prompts to see if you can be more concise. Are there details that can be omitted?
Observe who’s speaking the most	Do users seem to be in control of the conversation? If not, how can you change that?

Robust testing is essential for developing high-quality software and creating user satisfaction.

This video is a deep dive into developing end-to-end tests for your Actions, and it covers the tools that are available to make the process easier. It will also share best practices on a variety of topics, like how to handle unexpected user queries.