DM Explains · No. 024 · RAG
Multi-modal RAG: making one search find the photo, the recording and the page
Most company knowledge isn't text. Here's how retrieval systems learn to search images, audio and video alongside documents, and where stitching them together still breaks.
· 15 min read
Read me first
Multi-modal retrieval-augmented generation genuinely does let one query find the right page, photo and recording together, but only when every format was converted into the same shared language of numbers first; the search still never understands any of them.
Ask: 'who fixed the wiring last year' → 1 scanned invoice, work photo, voice note all ingested → 2 all embedded into one shared space → 3 FAISS searches 148,854 vectors in ~7.04ms → 4 reranker prunes redundant matches → 5 answer assembled with image and quote → 6 TTS streams the voice reply before the full LLM response finishes
Papa wants to know which electrician fixed the wiring last year. The invoice is a scanned PDF, sitting unopened in a cupboard folder. The proof-of-work photo is three weeks deep in the family chat. Dadi mentioned the man's name in a voice message nobody wrote down. A text search box would find nothing, because none of those three things is text — not really, not yet.
Most organisations are in Papa's position. What they "know" lives in slides, scanned forms, recordings and video, not clean text, so a search tool built only for text looks ignorant when it is really just blind to everything that isn't text.
Everyday picture. Amma keeps recipe cards, photographs of finished dishes, and Dadi's voice notes in different drawers. When Arjun asks how to make the dish in a photo, someone has to check the photo drawer, match it to a recipe card, and maybe play Dadi's voice note too — all without anyone writing a single new note down.
Where the picture breaks. In the kitchen, a person manually cross-references drawers one at a time. Multi-modal retrieval-augmented generation (RAG) instead pre-converts everything — card, photo, voice note — into comparable numerical representations sitting in one shared space, so there is no drawer-by-drawer searching at all.
Why it exists now
Text-only RAG solved a narrower problem: fetch the right paragraph, feed it to a language model, answer with it instead of a guess. But a paragraph is a convenient fiction for most real archives. A product catalogue is photographs. A support history is call recordings. A compliance file is scanned signatures. Recent work has converged to make searching across all of this practical in one pipeline rather than as separate bolted-on tools: cross-modal encoders that put text and images in one shared vector space, and voice-agent architectures fast enough to answer out loud without an awkward pause.
One framework studied combines a cross-modal encoder (BridgeTower), automatic speech recognition (Whisper), and the FAISS vector index, and reports measured results across three enterprise use cases: 94.2% cross-modal retrieval accuracy and a 67.3% reduction in hallucination compared with unimodal baselines (Multi-Modal RAG Framework Integrating Text, Image, Audio and Video — IJSET). That is a single vendor's measured result, not an industry average, but it is the clearest evidence that unifying modalities measurably reduces invented answers rather than just adding engineering complexity for its own sake.
How it works
Unified ingestion
Every format gets a door into the same house
What it is. Documents, images, audio and video enter one pipeline through format-specific extraction: optical character recognition (OCR) pulls text out of scans and images, Whisper transcribes audio, and video is sampled into frames.
Why it exists. Nothing can be searched in its raw form. A scanned invoice is pixels; a voice note is a waveform. Each needs converting into something a search system can compare before anything else happens.
Where it breaks. Extraction quality varies wildly by modality. Poor OCR on a creased scan, or a mistranscribed name in a recording, poisons everything retrieved downstream — the system will search confidently on the wrong text.
How that's handled. The same framework reports a measured multi-modal document throughput of 1,200 pages per hour — a sign that ingestion speed is tracked as its own number, separate from retrieval accuracy, precisely because errors introduced here are invisible later.
In technical terms
In technical terms: OCR, Whisper-based automatic speech recognition (ASR), and automatic frame sampling each run as modality-specific extraction stages feeding one downstream representation layer (Multi-Modal RAG Framework Integrating Text, Image, Audio and Video — IJSET).
Cross-modal embedding
One shared space, not three separate ones
What it is. Extracted content from every modality gets projected into one shared mathematical space, where meaning is stored as a list of numbers — an embedding — and similar meanings sit close together regardless of what format they started in.
Why it exists. Without a shared space, a text query can only ever match other text. Captioning an image into a sentence first is the obvious workaround, but it throws away whatever detail the caption didn't think to mention.
Where it breaks. A shared embedding space is built by training an encoder on paired examples — a photo next to its description, say. If an organisation's actual photos and documents don't resemble whatever pairs the encoder was trained on, cross-modal matches degrade quietly, with no error message to flag it.
How that's handled. The encoder studied, BridgeTower, is trained jointly on text and image pairs, and the surrounding framework supports embeddings of up to 8K dimensions for this cross-modal search (Multi-Modal RAG Framework Integrating Text, Image, Audio and Video — IJSET).
In technical terms
In technical terms: BridgeTower jointly encodes text and images into one representation; the studied implementation supports embeddings up to 8K dimensions, letting a text query land near a semantically related image vector with no intermediate captioning step.
Vector indexing
Searching millions of vectors in milliseconds
What it is. Embeddings get stored in an index, commonly FAISS, and searched approximately rather than exhaustively — trading a small chance of missing the single best match for speed.
Why it exists. Comparing a query against every stored vector one by one does not scale once an archive holds hundreds of thousands of items. Approximate nearest-neighbour search finds items that are close enough, fast enough to feel live.
Where it breaks. Approximate search can settle for several good-enough matches over the one best match — which matters exactly when the correct invoice looks similar to a dozen others filed the same month.
How that's handled. One voice-enabled RAG system measured roughly 7.04 ms retrieval latency (p95: 7.97 ms) against a 50 ms budget, searching a large collection of in-memory vectors — evidence that the speed trade-off, done well, leaves plenty of budget to spare rather than cutting it fine.
In technical terms
In technical terms: a dense vector search over FAISS, often paired with lexical matching, returns an approximate nearest-neighbour result set that is fast by design and inexact by design.
Reranking
More retrieved isn't more relevant
What it is. Candidates pulled back by the first search pass get filtered and reordered before any of them reach the model that writes the answer.
Why it exists. Raw similarity search tends to return redundant or loosely related material — three near-duplicate photos of the same invoice, say — which dilutes rather than helps the eventual answer.
Where it breaks. Reranking adds its own latency and its own failure mode: a blunt relevance scorer can discard something genuinely relevant just because it's phrased unusually.
How that's handled. M3KG-RAG's pruning module, GRASP, grounds retrieved entities back to the query, checks whether each piece of retrieved content actually supports an answer, and discards what doesn't, rather than passing everything retrieved straight through (M3KG-RAG: Multi-hop Multimodal Knowledge Graph-enhanced Retrieval-Augmented Generation — CVPR 2026 / Computer Vision Foundation).
In technical terms
In technical terms: GRASP grounds entities to the query, scores retrieved content for whether it supports an answer, and prunes what fails that check, ahead of generation.
Modality-aware output
The answer wears whatever format fits
What it is. The generation stage decides whether to answer in text, point to a retrieved image, or speak the answer aloud, depending on what the query needs and what evidence was found.
Why it exists. An answer that's technically correct but delivered in the wrong format is still unusable — reading out a long technical table makes no more sense than captioning a photo of damaged wiring instead of showing it.
Where it breaks. Choosing the wrong output modality for a query — speaking a long answer that really needed to be read, or showing an image when a quick spoken confirmation would do — makes a correct retrieval feel like a failure.
How that's handled. The large vision-language model (LVLM) in the measured framework produces the answer directly from whichever modality was retrieved, with text-to-speech (TTS) layered on when needed, and that combination is what drives the reported 94.2% cross-modal retrieval accuracy and 67.3% hallucination reduction across its three enterprise test cases (Multi-Modal RAG Framework Integrating Text, Image, Audio and Video — IJSET).
In technical terms
In technical terms: an LVLM generates from whichever retrieved modality is most relevant, optionally paired with TTS for spoken output, rather than forcing every answer through text alone.
Streaming for voice
Overlap the stages, don't wait for all of them
What it is. When the output needs to be spoken, splitting speech-to-text (STT), the language model, and text-to-speech into separate stages that each stream — rather than waiting for the whole chain to finish before saying anything — is what makes the reply feel conversational.
Why it exists. A single blocking call that processes audio in, waits for the model, then produces audio out, has unpredictable and often high latency. Streaming lets TTS begin speaking the start of an answer before the model has finished generating the end of it.
Where it breaks. A theoretical cascaded stack — STT, then the language model, then TTS, then network — can take roughly 780 ms for a first response if nothing is streamed, which breaches the roughly 300–500 ms window that feels like natural conversation (The Architecture of a Real Voice Agent — GRX10, 2026).
How that's handled. Streaming each stage brings the cascaded figure down toward a theoretical 220 ms for a unified speech-to-speech forward pass, by the same whitepaper's reckoning, and models that skip the cascade altogether can go further: Ichigo, an early-fusion mixed-modal model, reports 111 ms to first token by processing speech and text together rather than handing off between separate stages (Ichigo: Mixed-Modal Early-Fusion Realtime Voice Assistant — arXiv, 2024).
Ask aloud “who fixed the wiring last year” → 1 STT streams the transcript as you speak → 2 LLM drafts a reply, time-to-first-token the slow step → 3 TTS starts speaking before the draft finishes → 4 cascaded stack totals ≈780 ms, over the 300-500ms window → 5 native speech-to-speech model answers directly at ≈220 ms → spoken reply names the electrician, cascaded or end-to-end
In technical terms
In technical terms: a cascaded pipeline streams STT, large language model (LLM) generation and TTS as overlapping stages to approximate the latency of a native speech-to-speech model, which instead takes audio in and produces audio out through one forward pass with no discrete handoffs.
Compared with what came before
| Text-only RAG | Multi-modal RAG | |
|---|---|---|
| What gets indexed | Documents and text chunks | Documents, images, audio, video |
| How images/audio enter | Converted to captions first, if at all | Embedded natively alongside text |
| Shared representation | One embedding space, one modality | One embedding space, several modalities (e.g. BridgeTower) |
| Voice handling | Rare, usually bolted on | Streamed cascade or native speech-to-speech |
| Measured result | Established benchmarks across many vendors | 94.2% cross-modal accuracy, 67.3% less hallucination, one studied framework (IJSET) |
The judgement is plain: multi-modal RAG doesn't make retrieval smarter, it widens what counts as retrievable. What it costs is engineering surface — more encoders, more extraction stages, more places for errors to enter before anyone reads a generated answer.
Where it breaks
If the organisation's actual photos, scans and recordings resemble whatever an encoder was trained on, cross-modal matches hold up; if they don't, matches degrade with no obvious error to point to.
Captioning is not multi-modal RAG
Converting an image into a text caption and then running ordinary text RAG on the caption is a different, older approach. It discards whatever detail the caption didn't think to mention. Native embedding, as with BridgeTower, retrieves on the image itself, not a lossy description of it.
Streaming adds its own complexity
Splitting a voice pipeline into streamed STT, LLM and TTS stages solves a latency problem, but it introduces buffering, partial-result handling, and interruption logic that a single blocking call never needed. That complexity is the price of hitting the roughly 300–500 ms conversational window rather than the 780 ms a naive cascade can produce.
A knowledge graph-based approach like M3KG-RAG exists for a different reason entirely: it targets multi-hop reasoning over structured relationships between entities, not broader modality coverage, and shouldn't be confused with modality unification just because both prune retrieved content before answering.
Numbers to carry
| Figure | What it measures |
|---|---|
| 94.2% | Cross-modal retrieval accuracy, one studied MM-RAG framework across three enterprise use cases (IJSET) |
| 67.3% | Reduction in hallucination versus unimodal baselines, same framework |
| 1,200 pages/hour | Multi-modal document ingestion throughput, same framework |
| 8K dimensions | Maximum embedding size supported for cross-modal search, same framework |
| ~7.04 ms (p95: 7.97 ms) | Vector retrieval latency, one voice-enabled RAG system, against a 50.0 ms budget |
| ~780 ms vs ~220 ms | Theoretical first-response latency: naive cascaded STT→LLM→TTS vs unified speech-to-speech pass (GRX10, 2026) |
| 111 ms | Time to first token, Ichigo early-fusion mixed-modal model (arXiv, 2024) |
| 300–500 ms | Target latency window for a conversation to feel natural |
Myths worth killing
| Commonly repeated | What is actually true |
|---|---|
| "Captioning images is basically multi-modal RAG" | Captioning converts an image to text before retrieval and loses visual detail; multi-modal RAG retrieves native embeddings of the image itself |
| "A single giant model is always faster than a pipeline" | A naive single blocking call can be slower and less predictable than a well-streamed cascade of separate stages |
| "Approximate vector search means unreliable search" | It trades a small chance of missing the single best match for speed; one measured system still ran at ~7.04 ms over a large in-memory index |
| "More retrieved candidates means a better answer" | Redundant or off-topic candidates dilute an answer; reranking exists specifically to prune, not pad, what gets passed to the generator |
| "Voice agents and multi-modal RAG are the same thing" | Voice agents are one output format for a multi-modal system; the retrieval and embedding stages exist independently of whether the final answer is spoken |
Worked example
- Papa asks, out loud or by typing, "who fixed the wiring last year" (The electrician search).
- Unified ingestion has already turned the scanned invoice into searchable text through OCR, turned Dadi's voice note into a transcript through Whisper, and extracted frames from the photo.
- Cross-modal embedding has placed the invoice text, the photo, and the voice transcript all into one shared vector space, so none of the three had to be matched to the others by hand beforehand.
- The query is embedded and compared against that shared space using FAISS; illustratively, three candidates come back close together — the invoice, the photo, and the voice transcript — because all three describe the same event even though they started in three different formats.
- Reranking checks whether each candidate actually supports an answer to "who fixed the wiring," and in this illustrative case keeps all three, since each names or shows the same electrician.
- Modality-aware output decides the answer should be spoken, since Papa asked out loud, so a text-to-speech stage reads out the electrician's name while the photo and invoice stay available to view.
- If Papa had asked by voice and expected an instant reply, the streaming-for-voice stage would have started speaking the first part of the answer before the full reasoning chain had finished, rather than making him wait through the whole retrieval-and-generation process in silence.
Should you use it?
Reach for multi-modal RAG when an organisation's actual knowledge is mixed — scanned forms, product photos, recordings, video — and a single query genuinely needs to search across all of it rather than text alone. Skip it when the archive is text already; adding image and audio encoders to a system that doesn't hold any images or audio worth retrieving is pure overhead. And skip the streamed-voice complexity specifically if the application needs a guaranteed sub-100 ms response that even a well-streamed cascade can't reach — that calls for a native speech-to-speech model, not a cascade dressed up to look fast.
The cheapest first experiment is narrow: take one modality beyond text that already causes real search pain — scanned invoices, say — and measure cross-modal retrieval accuracy on its own, before any voice or generation layer gets added.
Say it in the room
"Use it when the knowledge genuinely lives in photos, scans or recordings and one query has to find all of them; skip it when the archive is already text and the only thing multi-modal would add is latency."
Lines to carry
- "Most knowledge isn't text, it just got treated like it was."
- "A shared space beats a captioned detour."
- "Fast retrieval is approximate on purpose."
- "More candidates retrieved is not the same as more relevance found."
- "Streaming the pipeline beats waiting for the whole answer."
- "Measure each modality's retrieval accuracy on its own before trusting the combined answer."
FAQ
Is multi-modal RAG just text RAG with extra steps bolted on?
Not quite. The extra steps — OCR, Whisper, frame sampling — matter, but the structural change is the shared embedding space underneath. Without that, text, image and audio stay in separate silos no matter how many extraction steps run beforehand.
Does adding voice always make a system slower?
Not if the pipeline is streamed. A naive single call that waits for everything can run close to 780 ms before saying anything, by one whitepaper's reckoning, while streaming the same stages targets the roughly 300–500 ms window that feels conversational.
Why not just caption every image and run normal text search?
Because captioning throws detail away before search even begins. A caption like "a damaged wire" cannot stand in for the actual pattern of damage in the photo; native cross-modal embedding keeps that detail searchable.
Is reranking always worth the extra latency it adds?
It depends on how noisy the first-pass retrieval is. If approximate search already returns clean, relevant candidates, reranking adds cost for little gain. If candidates are redundant or loosely related, pruning them before generation is what keeps the eventual answer faithful.
How is this different from a multi-modal knowledge graph?
A knowledge graph approach, such as M3KG-RAG, is built to trace multi-hop relationships between entities for deeper reasoning. Multi-modal RAG, as covered here, is built to widen what counts as searchable in the first place. They can share pruning techniques without solving the same problem.
Glossary
- RAG (Retrieval-Augmented Generation) — an approach where a language model retrieves relevant content before generating an answer, instead of answering from training alone.
- MM-RAG (Multi-modal RAG) — RAG extended to retrieve and reason across text, images, audio and video.
- Embedding — a numerical representation of content that captures its meaning, so similar items can be found by comparing numbers.
- Vector index / FAISS — a structure and library for quickly searching large collections of embeddings for the closest matches.
- Cross-modal retrieval — finding content in one format relevant to a query in another format.
- BridgeTower — a model architecture that jointly encodes text and images into one shared representation.
- Whisper — an automatic speech recognition model that converts spoken audio into text.
- LVLM (Large Vision-Language Model) — a large model trained to process images and text together and generate text responses.
- STT (Speech-to-Text) — converting spoken audio into written text.
- TTS (Text-to-Speech) — converting written text into spoken audio.
- Cascaded pipeline — separate specialised stages (STT, then LLM, then TTS) chained in sequence.
- Speech-to-speech (S2S) model — a single model taking audio in and producing audio out directly, without separate text stages.
- Latency budget — the maximum delay a response can take before it stops feeling natural.
- Reranking — reordering initially retrieved results to better reflect true relevance.
- Knowledge graph — a structured representation of entities and relationships, used for deeper multi-hop reasoning.
Sources
- Multi-Modal RAG Framework Integrating Text, Image, Audio and Video — IJSET, n.d.
- ELLSA: End-to-end Listen, Look, Speak and Act — arXiv, 2026
- M3KG-RAG: Multi-hop Multimodal Knowledge Graph-enhanced Retrieval-Augmented Generation — CVPR 2026 / Computer Vision Foundation, 2026
- The Architecture of a Real Voice Agent — GRX10, 2026
- Ichigo: Mixed-Modal Early-Fusion Realtime Voice Assistant — arXiv, 2024