Multi-model AI pipeline for video localization
Four AI services in one coherent pipeline. A source video goes in, a localized derivative video comes out, for under $1.
The business problem
A media localization agency wanted to adapt source videos into new variations for specific markets: different languages, audiences, and niches. The requirement was to create truly derivative videos (newly assembled footage, rewritten scripts, new voiceover) rather than 1:1 translations, so that each result is a separate asset for its market.
This was a custom commercial request, not an off-the-shelf product, and nothing like it existed on the market. The client wanted to test a content-scaling hypothesis: can AI adapt source material into new variations with enough quality and volume to make the operation economically viable?
What made it hard
The naive approach of translating, re-voicing, and republishing the same footage doesn't work, for two reasons:
- Originality: reusing the source footage and structure gives a near-duplicate; the result has to be a genuinely new asset
- Distinctness at scale: every result has to differ visually and structurally, not copy the original
The system had to produce videos with:
- New footage (assembled from a library, not from the original)
- Rewritten scripts (meaning kept, wording changed)
- A new voiceover in the target language
- And still be semantically coherent: the new footage has to actually match what the new audio says
Architecture: a two-stage pipeline
Stage 1: building the library
First the system builds a searchable visual library:
- The user sends video URLs in bulk through a Telegram bot (processed through a queue)
- Videos are downloaded on the server, then segmented by scene detection (detecting cuts, not fixed intervals)
- Each segment gets a semantic embedding through Vertex AI
- Embeddings and segments are stored locally in the Qdrant vector database
- Each segment also gets a JSON description of its content (which later improves matching accuracy)
Stage 2: generating the new video
When the client wants to make a new video:
- They send a link to the original plus a configuration (language, voice, music, emoji, subtitles, all chosen through the bot)
- FFmpeg extracts the audio from the source video
- The audio is transcribed with self-hosted Whisper Large (locally, to avoid API costs at scale)
- The transcript is rewritten by Gemini, keeping the meaning while changing the wording
- The rewritten script is translated into the target language
- Multilingual TTS generates the voiceover in the chosen voice and language
- Vertex AI matches library footage to the new audio segments by embedding similarity
- FFmpeg assembles the final video: matched footage + new audio + chosen enhancements (background music, sounds, memes, subtitles)
- The finished video is sent to the client in Telegram
Key technical decisions
Why Vertex AI for embeddings
At the time, OpenAI didn't offer API access to the video embedding model we needed. Local alternatives were expensive to run. Vertex AI gave the best price/quality balance for production.
Why self-hosted Whisper
At scale, transcription API costs become significant. Self-hosting on a local GPU removed recurring transcription costs entirely.
Why multilingual TTS through a reseller
Instead of a direct subscription with hard plan limits, we used a pay-as-you-go reseller. Same quality, no subscription lock-in, and costs are easier to scale.
Why Qdrant locally
A vector database on a local server removed recurring cloud database costs. The whole library lived on a single home server (10th-gen i5 + GTX 1070).
Cost engineering
Cost breakdown for one video, for a 20-minute result:
- Creating embeddings (once per source video): negligible
- Whisper transcription: free (self-hosted)
- Gemini rewrite + translation: ~cents
- Multilingual voiceover (through the reseller): the main cost item
- Storage: free (local)
- Processing: electricity only
This cost structure is what makes localization at scale economically viable. Manually localizing a 20-minute video would take a designer or editor 8–15 hours of work.
Production environment
- Deployed on a home server (i5-10K + GTX 1070, 16 GB RAM)
- A single Telegram bot interface: the client sends a URL and gets finished videos
- FFmpeg with GPU acceleration for video assembly
- Throughput: ~2 videos per hour for 20-minute clips (the bottleneck is assembly)
- Scalable design: the architecture supports parallel deployment across several GPU nodes (top-end cards aren't needed, a 2060/3060 is enough for this load)
- Embedding namespaces by category (for example, separate libraries for cooking and gaming content), which keeps semantic matching relevant within each domain
Problems we solved
1. Embedding quality for visual matching
The first implementation gave poor semantic matches: audio about topic X got paired with footage that had no visual connection. The fix: we supplemented each segment's embedding with a JSON description of its content, which sharply improved match relevance.
2. Pacing and rhythm in assembled videos
Auto-assembled videos first looked unnatural: segments too short (under 1.5s) or too long (over 15s), cuts at awkward moments. We built constraints into the assembly logic: minimum and maximum segment length, avoiding repeated segments close together, and audio level normalization.
3. Migrating from Firebase to local Qdrant
The inherited architecture stored embeddings in Firebase with recurring costs. We moved the whole pipeline to local Qdrant, removing ongoing database costs entirely.
4. Whisper translation quality
Standard Whisper translations sometimes came out clumsy. We added Gemini as a rewrite layer, which improved both meaning preservation and how natural the language sounds in the target language.
Result
End-to-end automation: the client sends a URL in Telegram and gets a publish-ready video with no manual steps in between. Under $1 per 20-minute clip. The scalable architecture is built to expand across several GPU instances and category libraries.
Tech stack
| Language | Python |
| Video processing | FFmpeg (GPU-accelerated) |
| Embeddings | Vertex AI |
| Transcription | Whisper Large (self-hosted) |
| LLM rewriting | Gemini |
| Speech synthesis | Multilingual TTS |
| Vector DB | Qdrant (self-hosted) |
| Interface | Telegram Bot |
What this demonstrates
- Multi-model AI orchestration: 4+ AI services combined into one coherent pipeline
- Semantic understanding of video content: embedding-based matching keeps video and audio coherent
- End-to-end product engineering: from a raw URL to a finished result, all automated
- Cost optimization through architecture: strategic decisions on what to self-host and what to run on APIs, keeping cost per video under $1 even on a premium AI stack
- Custom solution development: we built something that didn't exist as a product, for a specific commercial task