Insights · guide

Prototype a voice-first AR tool on your phone

Drafted with Insight · edited by The WE ARE VR team

design

Voice-first AR should begin with a useful command, not expensive glasses.

I wrote this because hardware delays can stall a good idea for months. A phone lets us test the interaction, language and value in one afternoon.

What's the problem?

A voice-first AR concept often starts with a long feature list. Someone wants navigation, object recognition, live translation, notes and an assistant with a personality.

That is five projects wearing one headset.

The team then waits for smart glasses, access to a device programme or a working software development kit. When the hardware arrives, nobody knows whether people will say the expected command. They also do not know what should appear afterwards.

Voice adds another difficulty. People speak differently, especially around names, places and South African accents. A command that works in a quiet office may struggle during a busy activation in Rosebank or Braamfontein.

The useful question is smaller: can one spoken command complete one task?

What's the solution?

Use your phone as a stand-in for the glasses.

Choose one command with a visible result. For example: “Save a note about this chair.” The workflow has four parts:

  1. The user looks through the phone camera.
  2. They speak the command.
  3. Speech becomes text in an ordinary input field.
  4. A small note card appears over the camera view.

You do not need full spatial tracking yet. Place the card near the centre of the screen and keep it there. The prototype is testing whether the command makes sense, not whether the virtual card remains attached to a chair from every angle.

For the quickest version, build a simple mobile web page. Use the phone camera for context and the phone keyboard's microphone for voice typing. That avoids adding a speech service, account or new dependency. The user taps a text field, speaks, checks the transcription and presses “Place note”.

This is deliberately plain. The keyboard microphone may require an extra tap, but it still exposes the important problems: unclear commands, incorrect words and confusing results.

Test the flow with five people. Give them the goal, but do not give them the exact sentence. Write down what each person naturally says. If everyone uses different wording, your tool needs flexible command handling. If they hesitate after speaking, your visual response is probably unclear.

The recent meta-oakley-hstn-sandbox work used phone testing while developing five voice-first AR tools. That is a sensible order: prove the small workflow first, then connect it to glasses.

[The headset will still find new ways to be difficult later.]

What's next?

  1. Pick one task that takes under ten seconds, such as placing a note, identifying an item or starting directions.
  2. Make a phone prototype with a camera view, one voice-typed field and one visible result. Skip accounts, databases and polished graphics.
  3. Test it somewhere with real background noise. Record the words people use, where they pause and what they expect to happen next.

Share what you prototype with the Jozi XR community, or send us the command that keeps failing.

The WE ARE VR team