ShowcaseRealtime AI
Voice Avatar Platform
VisionsCraft's Voice Avatar Platform is a GPU-backed conversational avatar stack: LiveKit WebRTC in the browser, MuseTalk lip-sync inference, ElevenLabs (or TTS) speech, GFPGAN face restore, and RAG so the avatar speaks from a private knowledge base—built for tutoring and interactive demos.
The problem
Text chatbots feel flat. Pre-rendered talking-head videos cannot listen. Most avatar demos break the illusion—laggy lips, invented facts, or GPU pipelines that never leave a research notebook.
The solution
A production realtime path: browser joins a LiveKit room, a MuseTalk agent on RunPod/GPU drives lip-synced frames from speech, TTS cleans text for pause-free audio, and RAG grounds answers in private docs before the avatar speaks.
The CIC Academy / MuseTalk LiveKit deployment shows the full stack. A Flash serverless endpoint issues LiveKit JWTs, lists avatars from network volume, and reports CUDA health. A persistent MuseTalk LiveKit agent runs inference states—loading, greeting, idle, listening, thinking, speaking—with cosine-eased onset/offset blending between frames. Speech text is sanitized for TTS (no markdown bullets that create acoustic hangs). GFPGAN can restore face quality. The browser sees a live avatar feed, transcript, WebRTC connection status, and knowledge-retrieval confidence. RAG supplies the words; MuseTalk supplies the mouth; LiveKit supplies the room. Together they deliver presence that chat alone cannot.
Implementation
LiveKit WebRTC rooms with JWT minting via RunPod Flash handlers. MuseTalk realtime inference on GPU with avatar assets on network volume. Speech cleaning pipeline for TTS. Optional GFPGAN restore. RAG over a private corpus before spoken responses. Operator UI for session, avatar selection, knowledge hits, and connection health—deployable as tutoring or product-demo experiences.
Tech stack
- LiveKit
- MuseTalk
- ElevenLabs
- RAG
- PyTorch
- RunPod Flash
- GFPGAN
Results
- Live avatar sessions with lip-sync tied to spoken audio
- Answers grounded in a private knowledge base before speech
- GPU-backed inference path suitable for tutoring demos
- Browser WebRTC join without shipping models to the client
- Operational hooks for avatar inventory, CUDA health, and room tokens
FAQ
How does lip-sync stay realtime?
MuseTalk runs on GPU (e.g. RTX-class via RunPod) and streams lip-synced frames into the LiveKit room while TTS audio plays—onset/offset blending keeps transitions smooth.
Can the avatar use private documents?
Yes. RAG retrieves from a private knowledge base first; the avatar only speaks grounded content suitable for tutoring and demos.
What does the browser connect to?
LiveKit WebRTC. A serverless handler can mint room tokens and report volume/avatar/CUDA health so clients never talk to the GPU pod directly.
Is this only a research demo?
The stack is production-shaped: persistent agent states, Flash deploy hooks, network volumes for avatars, and an operator-facing session UI—not a one-off Gradio clip.