PawSearch, Multimodal RAG on Home Camera Feed
A full multimodal-RAG pipeline that searches home video by natural language, no tags, no keywords. 5+ users have asked for a production release.
- FFmpeg
- Gemini Multimodal Embeddings
- ChromaDB
- Cosine similarity
- clip chunking
- 90s
- users want production
- 5+
- tags required
- 0
PawSearch demonstrates the full multimodal-RAG pipeline applied to video. Describe what you're looking for, 'orange cat sitting on the floor', and the system retrieves the exact clip window. No tags. Pure semantic understanding. Hobby project; 5+ users asked for a production release.
The premise
Search your videos like Google. Describe what's happening in plain English, the system finds the exact moment.
No keyword matching. No tags. Pure semantic understanding powered by multimodal embeddings.
The pipeline
Ingestion: video, FFmpeg chunked into 90-second clips, Gemini multimodal embedding per clip, ChromaDB vector store, persisted locally.
Search: natural-language query, Gemini text embedding, cosine similarity against the clip vectors, ranked results, LLM-generated 'why this matched' explanation surfaced alongside the clip.
The product decisions hiding inside the ML
Chunk size: longer clips embed more context but lose precision on where the moment starts; shorter clips are precise but fragment a continuous scene. 90 seconds was the empirical sweet spot for indoor home footage.
Result explanation: returning a ranked list is not enough; non-technical users do not trust vector results without a sentence of why. The LLM-generated rationale was the difference between 'cool' and 'usable'.
Local-first: the home-video use case is intimate. Running embeddings and storage locally was a product decision, not just a technical one.
Why this matters as PM evidence
Built to demonstrate end-to-end applied AI: product thinking, full-stack ML system design, and the realities of a multimodal-RAG architecture from raw video to natural-language search.
Shipped as a hobby project. 5+ users surfaced themselves asking when it goes to production, which is the only signal that actually counts for a v0.
"The instructive part wasn't the model choice, it was deciding where to chunk, how to balance recall vs. clip length, and how to explain matches back to a non-technical user. The PM work hid inside what looked like an ML problem."