Building an AI Voice Assistant: Speech Recognition to Response

Voice assistants are becoming ubiquitous. This guide shows how to build your own AI voice assistant from scratch.

Architecture Overview

A voice assistant has four components:

1. Speech-to-Text (STT): Convert audio to text
2. Natural Language Understanding (NLU): Understand intent
3. Response Generation: Generate appropriate response
4. Text-to-Speech (TTS): Convert response to speech

Step 1: Speech Recognition

import speech_recognition as sr

def listen():
    recognizer = sr.Recognizer()
    with sr.Microphone() as source:
        print("Listening...")
        audio = recognizer.listen(source)
    
    try:
        text = recognizer.recognize_whisper(audio, language="english")
        return text
    except sr.UnknownValueError:
        return "Could not understand audio"
    except sr.RequestError:
        return "Could not request results"

Step 2: Intent Recognition

def understand_intent(text):
    import openai
    
    response = openai.chat.completions.create(
        model="gpt-4",
        messages=[
            {"role": "system", "content": "Classify the intent of this user message: weather, reminder, search, general"},
            {"role": "user", "content": text}
        ]
    )
    return response.choices[0].message.content.strip()

Step 3: Response Generation

def generate_response(intent, text):
    if intent == "weather":
        # Call weather API
        return "The weather today is sunny with a high of 25 degrees."
    elif intent == "reminder":
        # Set reminder logic
        return "I've set a reminder for you."
    else:
        # General conversation
        response = openai.chat.completions.create(
            model="gpt-4",
            messages=[{"role": "user", "content": text}]
        )
        return response.choices[0].message.content

Step 4: Text-to-Speech

from gtts import gTTS
import pygame

def speak(text, lang='en'):
    tts = gTTS(text=text, lang=lang)
    tts.save('response.mp3')
    
    pygame.mixer.init()
    pygame.mixer.music.load('response.mp3')
    pygame.mixer.music.play()
    while pygame.mixer.music.get_busy():
        continue

Multilingual Support

For Arabic or French voice assistants:

• STT: Whisper supports 100+ languages
• NLU: GPT-4 works in Arabic and French
• TTS: Google TTS supports Arabic and French

Real-World Use Case: Medical Assistant

A health tech startup in Casablanca built a Darija-speaking voice assistant for:

• Appointment scheduling
• Medication reminders
• Symptom triage
• Health information

The assistant handles 1,000+ calls daily in Moroccan Arabic.

Deployment

• Use WebSocket for real-time communication
• Deploy STT on GPU instances for low latency
• Cache common responses for speed
• Monitor accuracy and user satisfaction

Next Steps

• Add wake word detection ("Hey Assistant")
• Implement multi-turn conversations
• Add custom actions (send email, control smart home)
• Support code-switching between languages