DM Explains · No. 013 · RAG
Multi-modal index fusion: why mixing text, image and audio search can backfire
Combining text, image and audio retrieval into one system sounds obvious, but shoving everything into a single embedding space quietly breaks pathways nobody touched. Here is how the working versions actually fuse evidence.
· 14 min read
Read me first
Mixing text, image, video and audio search into one retrieval system genuinely does find evidence that text-only search misses, but only when the fusion is checked per modality; stitched into one careless embedding space, it quietly breaks a pathway nobody touched.
The scanned equipment manual: 1 page images, markdown text and training-video audio each encoded by their own specialist → 2 fused into one backbone initialised from a visual-language model → 3 nothing in audio changed yet its retrieval quality drops — Projector Drift → 4 full-parameter projector fine-tuning plus balanced rehearsal repairs it → 5 a visual index of page images and a text index of markdown are kept side by side, consistency-constrained fusion down-weights disagreement → 6 a router sends a diagram question to the image index and a wiring-step question to markdown instead of one merged space
A support team builds a retrieval-augmented generation (RAG) system over scanned equipment manuals full of diagrams and tables. Text-only search keeps missing the wiring diagram that actually answers the question, so someone adds an image index for the diagrams and an audio index for the training videos, then merges everything into one model for simplicity.
Weeks later, audio search quietly gets worse, even though nobody touched a line of audio code. The cause sits one layer down: fusing the specialist models into a single backbone left the small module that translates audio into that backbone's language miscalibrated. If nobody checks retrieval quality per modality after a fusion step, that regression ships silently, and the training videos stop surfacing when someone asks a spoken question.
Everyday picture. Meera keeps the family photo albums, Amma keeps the written diary, and Dadi keeps her box of cassette recordings. When Arjun asks what happened on Bruno's birthday, nobody dumps all three into one drawer and hopes for the best; they check each person's record and compare notes. Later, a cousin rebuilds the stereo around Dadi's cassette player, and suddenly her tapes sound muffled — nobody touched the tapes, but the speakers around them changed.
Where the picture breaks. Real fusion happens inside a trained model's numerical parameters, not by people literally comparing shelves, and "comparing notes" in these systems is a learned consistency score, not a chat at the dinner table.
Why it exists now
Real documents and real questions are rarely pure text. A manual page mixes a diagram with a caption and a table; a training video mixes speech with slides. A system that indexes only text misses the evidence sitting in the chart, the scanned table, or the spoken aside, which is why teams keep reaching for image and audio indexes once text-only retrieval plateaus.
What changed is that several 2026 papers moved this from folklore to named, measured failure modes with fixes. Conan-embedding-v3 names and repairs "Projector Drift", where fusing modality-specialist models into one backbone leaves an audio pathway miscalibrated despite no audio module being altered (Conan-embedding-v3 — arXiv, 2026). UniversalRAG formalises the "modality gap" in unified embedding spaces, where a shared space favours same-modality matches over the actually relevant one, and proposes routing instead (UniversalRAG — ACL Anthology, 2026). VLD-RAG and MIDR show that keeping two separate indexes and checking them against each other, rather than collapsing everything into one space, measurably improves retrieval over multi-page scanned documents (VLD-RAG — arXiv, 2026; MIDR — arXiv, 2026).
How it works
Encode per modality
Each expert learns its own language first
What it is. Images, text, video and audio are each turned into numbers by their own specialist model, not by one shared encoder trained on everything at once.
Why it exists. Different data types have different statistical patterns and training dynamics; forcing one encoder to learn all of them simultaneously causes the modalities to compete during optimisation, which degrades quality in each one.
Where it breaks. Training separate specialists means separate indexes to build and maintain, and no shared space yet exists — fusion still has to happen somewhere downstream.
How that's handled. Each specialist starts from the same shared vision-language initialisation, so later fusion has a common point of departure, as in Conan-embedding-v3.
In technical terms
In technical terms: Conan-embedding-v3 trains image, video, visual-document and grafted-audio specialists independently from the same visual-language initialisation before fusing them (Conan-embedding-v3 — arXiv, 2026).
Fuse into backbone
One model, stitched from many, is not automatically whole
What it is. The trained specialists are combined into a single deployable model, rather than being kept as separate systems a query has to be sent to one by one.
Why it exists. Serving several specialist models separately multiplies infrastructure and latency; one backbone covering visual, video and document retrieval is cheaper to run and simpler to reason about.
Where it breaks. Merging can leave pathways that depend on an external encoder plus a small translation module — audio, typically — pointing at a backbone that no longer matches what that module was trained against.
How that's handled. The merge is done by combining task vectors or adapters rather than retraining from nothing, which composes capabilities without discarding the specialists' training.
In technical terms
In technical terms: task-vector or LoRA (Low-Rank Adaptation) merging composes capabilities such as visual, video and document retrieval into one backbone, as described for Conan-embedding-v3, which reports 74.9 on the MMEB (Massive Multimodal Embedding Benchmark) as of 2026 (Conan-embedding-v3 — arXiv, 2026).
Projector drift
Nothing in audio changed; audio got worse anyway
What it is. A regression in one modality's retrieval quality that appears purely because the backbone it connects to changed shape, not because that modality's own training changed.
Why it exists. An audio pathway is usually attached through a small module — a projector — trained to translate an audio encoder's output into the old specialist backbone's space. Once that backbone is replaced during fusion, the projector's translation no longer lands correctly.
Where it breaks. This is detectable only if retrieval is benchmarked per modality after fusion; a single blended score across all modalities can hide a large audio-specific regression entirely.
How that's handled. Conan-embedding-v3 names this failure mode "Projector Drift" explicitly and treats it as a fusion-time problem to be measured and fixed, not a bug in the audio data.
In technical terms
In technical terms: Projector Drift is the degradation of a projector-based modality's retrieval quality caused by changing the backbone it was calibrated against, even though the projector and its source encoder were never modified (Conan-embedding-v3 — arXiv, 2026).
Projector recovery
Retrain the plug, not the whole machine
What it is. A targeted repair that retrains only the small translation module, leaving the fused backbone untouched.
Why it exists. Retraining the entire fused model to fix one drifted pathway would be slow and risks undoing the gains from fusing the other modalities; retraining just the projector is far cheaper and more precise.
Where it breaks. Recovery needs training data balanced across modalities; fine-tune the projector on audio data alone and the fix risks overcorrecting the model toward being an audio-only retriever.
How that's handled. Conan-embedding-v3 freezes the new backbone, fully fine-tunes the projector against it, then rehearses on a balanced mix of audio, image, video and document data to restore all pathways together.
In technical terms
In technical terms: full-parameter projector fine-tuning (Projector Recovery) followed by balanced multi-modal rehearsal brought Conan-embedding-v3 to 55.61 on the 30-task MAEB (multi-task audio embedding benchmark) audio suite as of 2026, alongside its 74.9 MMEB score (Conan-embedding-v3 — arXiv, 2026).
The manual indexed as page images, markdown text, and training-video audio → 1 visual and text indexes fused for a query → 2 the two disagree on a diagram's label, triggering the verifier → 3 offline, the fused projector drifts audio performance even though audio training never changed → 4 Projector Recovery fine-tunes and rehearses it back → 5 modality-aware routing sends the next query to the markdown index instead of forcing one shared space → fused, cross-checked answer
Dual-index fusion
Keep two records and listen when they disagree
What it is. A page or document is indexed twice — once as a visual embedding of the page image, once as structured text extracted from it — and the two are fused at query time rather than collapsed into one representation.
Why it exists. A page image preserves structure, tables and figures that get lost when text is extracted; the text index preserves literal wording that a visual embedding may blur. Each catches what the other misses.
Where it breaks. Down-weighting candidates where the two indexes disagree assumes the two are independently informative; if both the visual and the text index encode the same underlying error, disagreement won't surface it.
How that's handled. VLD-RAG blends sparse lexical and dense multimodal scores, down-weights disagreement, and lets a verifier trigger another retrieval pass when the fused evidence still looks thin.
In technical terms
In technical terms: VLD-RAG combines a visual index of page images using dense vision-language embeddings with a text index built from structured markdown serialisation, fusing them via consistency-constrained scoring that penalises cross-modal disagreement (VLD-RAG — arXiv, 2026). A related approach, MIDR, shows enrichment-augmented indexing lifting average nDCG (normalised Discounted Cumulative Gain) across five English domains to 0.6219, a 23.0% relative gain over BM25, and lifting two French-document domains from 0.1532 to 0.5448 nDCG by bridging English queries to French page text (MIDR — arXiv, 2026).
Route, don't merge
A unified drawer is not a shared language
What it is. Instead of embedding every modality into one shared space, a query is sent to whichever modality-specific index actually suits it, and only that index is searched.
Why it exists. Forcing every modality into one aggregated embedding space creates what UniversalRAG calls a "modality gap": retrieval tends to favour results from the same modality as the query, even when the real answer sits in a different modality entirely.
Where it breaks. Routing depends on correctly classifying what the query needs; a wrong routing decision means the right evidence is never even searched, regardless of how good that index is on its own.
How that's handled. UniversalRAG validates its router across 10 benchmarks spanning multiple modalities and granularities, choosing the corpus and level of detail per query rather than per fixed configuration.
In technical terms
In technical terms: UniversalRAG performs retrieval-augmented generation over corpora of diverse modalities and granularities, routing queries to avoid the modality gap measured in unified embedding spaces, and reports validation across 10 benchmarks (UniversalRAG — ACL Anthology, 2026). A related system, OmniRet, extends this to composed queries spanning text, vision and audio together, trained on roughly 6 million query-target pairs across 30 datasets and benchmarked on 13 retrieval tasks plus an MMEBv2 subset (OmniRet — CVPR, 2026).
Compared with what came before
| Text-only RAG | Unified embedding fusion | Dual-index / routed fusion | |
|---|---|---|---|
| What it searches | Text chunks only | Everything in one shared vector space | Separate modality indexes, merged or routed per query |
| Catches diagrams and audio | No | Yes, but biased toward query's own modality | Yes, with disagreement checks or routing |
| Risk of silent regression | Low — one pathway to watch | High — fusing backbones can drift a module nobody touched | Lower — per-modality indexes stay separately measurable |
| Engineering cost | Lowest | Medium to build, high to debug | Higher upfront, more stable to operate |
The judgement is not "fusion is bad". Collapsing everything into one space is the move that creates both the modality gap and the Projector Drift failure; keeping modality-specific records and either checking them against each other or routing between them, rather than merging them unconditionally, is what the measured gains above actually come from.
Where it breaks
Fusion earns its complexity only when the corpus genuinely mixes modalities and the task needs cross-checking between them; applied to a purely text corpus it adds latency and engineering cost for no retrieval gain.
A blended score hides per-modality regressions
If retrieval quality is only ever measured as one averaged number across all modalities, a large drop in one pathway — such as the audio regression Conan-embedding-v3 documents — can sit underneath an unchanged or even improved overall score. Measuring each modality's retrieval separately after any fusion step is the only way to catch it.
Low-latency, high-throughput settings are a poor fit for a projector-recovery-and-rehearsal cycle or an iterative agentic retrieval loop, both of which add real time per query or per deployment. And projector recovery itself depends on balanced training data across modalities; skip the rehearsal step and the fix risks collapsing the fused model toward whichever modality was tuned last, undoing gains elsewhere.
Numbers to carry
| Figure | Value |
|---|---|
| Conan-embedding-v3 on MMEB | 74.9, as of 2026 |
| Conan-embedding-v3 on 30-task MAEB audio suite | 55.61, as of 2026 |
| MIDR average nDCG, five English domains | 0.6219, a 23.0% relative gain over BM25 |
| MIDR nDCG lift, two French-document domains | 0.1532 to 0.5448, via English-to-French enrichment |
| OmniRet training data | ~6 million query-target pairs across 30 datasets |
| OmniRet benchmark coverage | 13 retrieval tasks plus an MMEBv2 subset |
| UniversalRAG validation | 10 benchmarks across modalities and granularities |
Myths worth killing
| Commonly repeated | What is actually true |
|---|---|
| "One shared embedding space is strictly better than separate indexes." | A unified space creates a measured modality gap favouring same-modality retrieval, which is why UniversalRAG routes instead of merging into one space. |
| "If you didn't touch the audio code, audio retrieval can't regress." | Conan-embedding-v3's Projector Drift is exactly this: an untouched audio projector miscalibrates because the backbone it feeds into changed during fusion. |
| "Fusing specialist models into one backbone is a one-off step with no further cost." | Recovering a drifted projector requires full fine-tuning plus balanced rehearsal across all modalities, not a quick patch. |
| "Combining two indexes just means averaging their scores." | VLD-RAG explicitly checks for cross-modal disagreement and down-weights candidates where the two indexes conflict, rather than blending blindly. |
| "More modalities always means better retrieval." | Only when the corpus and query genuinely need that modality; otherwise fusion adds cost without a measured gain. |
Worked example
- Papa scans the equipment manual's wiring diagram pages and markdown-extracts the surrounding text; both go into separate indexes, following the dual-index design VLD-RAG describes.
- Amma asks a question that depends on reading the diagram, not the caption; the visual index surfaces the right page while the text index, searching only the caption, ranks it lower.
- The fusion step notices the two indexes disagree and, instead of discarding the visual hit, down-weights it only partially and keeps it in the result set, as VLD-RAG's consistency-constrained fusion is designed to do.
- Later, the team merges their image, video, document and audio specialist models into one backbone to simplify deployment, the way Conan-embedding-v3 merges task vectors.
- Illustrative only: before the merge, a sample of audio queries against the training video retrieves the right clip for roughly 8 out of 10 queries; after the merge, without anyone touching the audio code, that illustrative figure drops to roughly 5 out of 10, because the audio projector no longer matches the new backbone — a Projector Drift pattern.
- The team retrains only the projector against the frozen fused backbone and rehearses on a balanced mix of audio, image, video and document data, following Conan-embedding-v3's Projector Recovery step, and the audio pathway's measured retrieval returns to its earlier level.
- Going forward, the team benchmarks each modality separately after every fusion or merge step, rather than trusting one blended score.
Should you use it?
Use multi-modal fusion when the corpus genuinely mixes modalities that carry the answer — diagrams, scanned tables, spoken asides — and the questions users ask could plausibly need any one of them. Skip it when the corpus is purely text, or when the team cannot commit to measuring retrieval per modality after every change to the system.
The cheapest first experiment is not fusion at all: index text and images separately, measure each modality's retrieval quality on its own, and only then decide whether a shared backbone or a router is worth the complexity it adds.
Say it in the room
"Fuse modalities when the evidence genuinely lives across them, but measure each modality's retrieval separately after every merge — a blended score will hide the one pathway that just broke."
Lines to carry
- "One drawer is not one language."
- "Nothing in audio changed, and audio broke anyway."
- "Disagreement between records is a signal, not a nuisance."
- "Routing asks where to look before it asks what it means."
- "A blended score hides a regression that per-modality measurement would catch."
- "Fuse the specialists, but keep the specialists' own scoreboard."
FAQ
Why not just train one encoder on text, images and audio together from the start?
Training one encoder jointly on all modalities at once causes them to compete during optimisation, degrading quality in each — this is why Conan-embedding-v3 trains specialists independently from a shared starting point and fuses them afterward rather than training everything together in one pass.
Is Projector Drift specific to audio?
It applies to any modality attached through an external encoder plus a translation module rather than natively trained into the backbone; audio is the documented case in Conan-embedding-v3, but the same pattern — a module calibrated to an old backbone, now facing a new one — could affect any similarly attached pathway.
Does keeping separate indexes always beat one shared embedding space?
Not universally, but UniversalRAG's measured modality gap shows a shared space tends to favour retrieval of the query's own modality even when a different one holds the real answer, which is a concrete reason to prefer routing or dual indexes when the corpus is genuinely mixed.
How do you know fusion has broken something if the overall score looks fine?
A single blended score across all modalities can hide a large regression in just one of them, which is exactly what happened to the audio pathway in Conan-embedding-v3 before Projector Drift was diagnosed; the fix is to benchmark each modality separately after any fusion or merge step.
What's the cheapest way to try this without committing to full fusion?
Build and measure a text index and an image or audio index separately first, following the dual-index pattern VLD-RAG uses, and only consider merging them into one backbone once you can see, modality by modality, that each one is actually pulling its weight.
Glossary
- RAG (Retrieval-Augmented Generation) — a technique where an AI model retrieves relevant external documents or data before generating an answer, so its response is grounded in real evidence.
- Embedding — a list of numbers representing the meaning of a piece of text, image, audio or other data, so similar items end up with similar numbers.
- Multi-modal / omni-modal — involving more than one type of data at once, such as text, images, video and audio together.
- Index fusion — combining results or representations from multiple separate retrieval indexes into a single ranked set of results.
- Modality gap — a measured tendency for a shared embedding space to cluster items by data type, so a query favours results of its own type.
- Task vector — the difference between a model's parameters before and after training on a specific task, used to transfer that capability by combining with other models.
- Projector — a small neural network that converts the output of one encoder into a format compatible with another model.
- Projector Drift — a failure where a projector becomes miscalibrated after the backbone it feeds into is changed, even though the projector itself was not altered.
- Backbone — the main shared neural network that processes combined representations from different modalities.
- LoRA (Low-Rank Adaptation) — a method for fine-tuning a large model efficiently using small trainable adapter layers instead of retraining all parameters.
- nDCG (normalised Discounted Cumulative Gain) — a standard metric for ranking quality in information retrieval.
- BM25 — a classic text-only ranking algorithm based on keyword matching, used as a baseline.
- Composed query — a search query that itself combines more than one modality, such as an image together with text.
- MMEB / MAEB — benchmark suites used to score embedding models across many retrieval tasks, including audio-specific tasks.
- Rehearsal (in training) — continuing to train a model on a balanced mix of old task data while adding new data, to prevent it forgetting earlier capabilities.
Sources
- Conan-embedding-v3: Fusing Modality-Specific Models for Omni-Modal Embedding — arXiv, 2026
- VLD-RAG: Vision–Language Document Retrieval-Augmented Generation — arXiv, 2026
- MIDR: Enrichment-Augmented Indexing for Multimodal Document Retrieval — arXiv, 2026
- Efficient and High-Fidelity Omni Modality Retrieval — CVPR / Computer Vision Foundation, 2026
- UniversalRAG: Retrieval-Augmented Generation over Corpora of Diverse Modalities and Granularities — ACL Anthology, 2026