Skip to content

Speech AI

Voice & Speech AI Toolkit

Live Hindi-to-English translation in your own voice, video dubbing with voice cloning, and a lip-syncing 3D avatar you can talk to.

Discuss a similar project
My role
End-to-end engineering
Stack
PythonWhisperXTTS-v2GPT-4oNode.jsWeb Audio

The problem

Speech models are easy to demo and hard to chain into something that feels live. These tools stitch them into pipelines that run in real time or produce finished media.

What I built

  • Live translator: speak Hindi, be heard in English in your own cloned voice, routed through a virtual audio device so it works as the microphone in Zoom or Meet.
  • Video dubbing: ffmpeg, voice and background separation (Demucs), faster-whisper transcription, translation, XTTS-v2 voice cloning or stock voices, per-line time fitting, then mixing and muxing back to MP4.
  • Avatar chat: microphone, voice activity detection, speech-to-text, GPT-4o, neural text-to-speech, and a Web Audio analyser that drives mouth, face and head motion on a 3D VRM avatar.
  • A live voice changer with presets for calls, built in Python.

How it fits together

  1. Mic / video
  2. VAD + speech-to-text
  3. Translate / LLM
  4. TTS or voice clone
  5. Audio out / dubbed MP4

Highlights

  • Real-time and offline pipelines
  • Voice cloning with time-fit dubbing
  • Runs locally on a Mac