MMLongEmbed: Benchmarking Multimodal Embedding
Models in Long-Context Scenarios

Overview: the benchmark and what it found

MEMs map heterogeneous modalities — text, document images, video frames — into a shared vector space, making "retrieve the right image/video/document with a text query" possible. The new MLLM-backed generation of MEMs (Qwen3-VL-Embedding, GME, …) has pushed context windows past 32K tokens, powering multimodal RAG and agent workflows. The authors put their finger on a sharp contradiction: the theoretical window length is not the model's real long-context understanding or representation ability — "it fits" is not "it reads".

MMLongEmbed fills that evaluation gap as the first systematic long-context MEM benchmark:

Four evaluation tasks span three input formats (pure text, pure image, interleaved image-text): Visual Grounding (a synthetic NIAH task) plus three real tasks — Multi-Source Reasoning, Logical Summarization, and Temporal Summarization. Dataset statistics:

Length is the benchmark's central controlled variable: the synthetic task slices contexts into 8 length bins (256 to 32K); the real tasks use three pools at 8K / 16K / 32K. The figure below shows the length distribution of all candidates, with the four tasks cleanly separated:

💡 Click any image to view the original 300-DPI version; click again or press Esc to close.

Three headline findings (spoilers)

• Scale and capability are decoupled:bigger models with higher embedding dimensions barely improve long-context performance — Qwen3 8B gains only ~1 point over 2B (53.41 vs 52.31), and on MMDocRAG's Hard subset the 8B (25.42) actually loses to the 2B (32.29).
• Fine-grained retrieval decays systematically with length:most models retrieve reliably only at ≤2K tokens, then degrade steadily; summarization-type tasks stay comparatively stable — real long documents have redundancy and dispersed evidence.
• Deep understanding is the weak point:once high-similarity distractors close off the surface-matching shortcut, scores fall off a cliff, and robustness to cross-modal redundancy varies wildly across models.

Motivation: why MMLongEmbed was needed

Three generations of MEMs, and three gaps in existing benchmarks

MEMs have gone through roughly three generations: contrastive dual-tower architectures (CLIP) building a shared semantic space from massive image-text pretraining; MLLM-driven models (E5-V, VLM2Vec, GME, MM-Embed) learning more general embedding spaces on visual backbones; and reasoning-driven representation learning (Embed-RL, MMEmb-R1) embedding implicit reasoning into representation training. Context windows kept growing — to 32K, to Qwen3-VL-Embedding's document-level understanding, to a proprietary model's 128K.

Evaluation lagged behind. Mainstream embedding benchmarks (MTEB/MMEB/MIEB) focus on short inputs and simple understanding; the few long-input benchmarks (e.g. LongEmbed) carry three structural deficiencies, which MMLongEmbed addresses one by one:

• ① Insufficient context diversity:text-only, no coverage of interleaved web pages, document pages, video frames → MMLongEmbed brings four data sources (documents, web pages, video-frame sequences, long text) and three input formats.
• ② No length stratification:input length isn't a controlled variable, so "how does performance evolve as context grows" can't be analyzed → length-stratified pooling, evaluated bin by bin.
• ③ Insufficient task difficulty:difficulty isn't explicitly controlled; pooling whatever data is available lets many tasks be solved by surface matching → high-quality distractors: high-similarity needles in the synthetic task, "looks-alike" distractors in the real tasks.

Key findings: three counterintuitive conclusions

Before the method, fix the paper's three most important empirical conclusions (all from §5–§6, evidenced later):

Finding 1: scaling up ≠ stronger long-context ability

Within a model family, 2–4× parameters bring marginal gains: Qwen3 8B vs 2B is 53.41 vs 52.31; GME 7B vs 2B is 24.63 vs 23.10 — with the 8B even losing to the 2B on the fine-grained Hard subset. Paired t-tests (Table 8) show the Qwen3 family difference is statistically insignificant (p=0.5375). Conclusion: without long-context-specific optimization, scale and embedding dimension alone cannot solve fine-grained discrimination in long contexts.

Finding 2: degradation varies systematically with context length and key-information position

With length as a controlled variable, clear patterns emerge: fine-grained retrieval (Visual Grounding) slides persistently after ~2K tokens; targets in the middle of the sequence score worst (positional bias, §6); summarization tasks stay relatively stable — redundancy is high, evidence dispersed, the "global semantic label" remains recoverable.

Finding 3: current MEMs lean heavily on surface-feature matching

Faced with carefully built high-similarity distractors, most models' high Easy scores collapse instantly: Logical Summarization drops from 80–90% to 10–20%; Temporal Summarization's Hard set is equally brutal. Cross-modal interference robustness is highly uneven — some models are modality-biased, some get "pierced" by mixed image-text noise, and only a few (e.g. Rzen-v2-7B) stay stable.

Method: four tasks and hard-distractor construction

Length-stratified pooling + unified token counting

All queries are comparatively short while candidate multimodal documents are much longer; query–candidate pairs are pooled by candidate length (8 bins for the synthetic task, 8K/16K/32K for the real ones), each pool evaluated independently, and candidate density (C/Q) is held constant across pools within a task. Because models tokenize images very differently, the paper adopts a unified multimodal token-count baseline (revised from MMLongBench): text via the Qwen3 tokenizer; images as 16×16 patch embeddings + 2×2 spatial merging — each visual token covering an effective 32×32 pixels:

Total context length sums over modalities:

Why unify: different models may differ by several times in tokens for the same image; if each model pooled by its own counting, one "length bin" would mean different difficulty for different models. The unified scheme keeps length a fair, comparable control variable.

The construction pipeline

Task 1 · Visual Grounding (synthetic — visual anchor localization)

The multimodal needle-in-a-haystack: 25 text and 25 image query-anchor pairs selected from NeedleBench and MM-NIAH, with anchors embedded in interleaved image-text haystacks sampled from OBELICS. Contexts span 256–32,768 tokens in eight bins; insertion depth is discretized into 6 levels (0%–100%); each length×depth cell gets its own 25-candidate corpus. The pivotal design: anchors are highly similar to their distractors. Text anchors share nearly identical syntax and lexical overlap with distractors, differing only in the core entity — e.g. the target "The first band to play on the Moon was the Virtual Rocket Band." versus the distractor "…on Venus …"; a query like "which band first performed on the Moon?" forces a choice. Visual anchors are synthetically generated unique objects/scenes that cannot have leaked from training, placed among random backgrounds and visually similar distractor images — forcing zero-shot fine-grained alignment instead of coarse scene recognition. This task orthogonalizes "fine-grained instance-level discrimination" against "position/length" — the paper's sharpest scalpel.

Task 2 · Multi-Source Reasoning (real — evidence chains across sources)

Built on HotpotQA and MMDocRAG, testing the ability to capture structured evidence chains across 2–5 evidence fragments. Queries without "strongly linked anchors" are filtered out first (the auditor prompt below); original contexts ≤8K are expanded to 16K/32K with semantically irrelevant but natural filler while keeping the relative positions of key evidence unchanged. Hard distractors are generated by breaking the reasoning chain: a key evidence span (first / last / middle random segment) is removed from the expanded candidate and replaced with strictly length-matched coherent text — breaking the chain while erasing the length cue. Easy pools are undamaged versions padded to the same length.

Task 3 · Logical Summarization (real — global document semantics)

Built on GovReport (text) and MMLongBench (document images), testing alignment of a summary with a long document's global semantics, keeping only the subset requiring macro-level understanding rather than local excerpting. Hard variants replace large contiguous spans (first 25% / last 25% / middle 50%) with query-irrelevant content — and the replacement source is the key: for text, BM25 retrieves from a global index documents highly similar to the summary but factually different, and paragraphs are dynamically drawn from them; for multimodal, "unseen consecutive image pages" from the same source document's extended version are used. The injected noise has high lexical overlap, fluent prose, uniform visual format — locally coherent, globally broken — so local matching can't steal points.

Task 4 · Temporal Summarization (real — long-range temporal structure)

Built on YoukuDenseCaption (Chinese) and ActivityNet Captions, testing understanding of long-range temporal structure: all videos are downsampled to 1 FPS into standard frame sequences, and the retrieval unit is the whole candidate video (an ordered frame sequence). A text query describes an event sequence ("first… then… finally…") and must retrieve the most relevant video; queries are rewritten from concatenated subtitles by Qwen2.5-32B-Instruct into coherent sequences with explicit transition words (prompt below), and Chinese annotations are zero-shot translated by the same model.

The core innovation of the Hard distractors is event-level temporal perturbation: instead of naive frame shuffling, videos are segmented at semantic boundaries first, then macro-structure transforms are applied (wholesale temporal reversal, block permutation) — preserving event-level semantic units while destroying global narrative and order, with a 0.5-second anti-overlap filter keeping event boundaries clean. Easy pools use other unperturbed videos.

Models and overall results

The 11 evaluated MEMs

Eleven MEMs with ≥32K context windows: 10 open-source (2B–8B, embedding dimensions 1536–4096) + the proprietary Doubao-Seed-1.6 (128K window, parameters undisclosed), spanning three generations of techniques. To prevent OOM, all models use middle truncation (keep head and tail).

Overall performance (Precision@1)

The main table reports per-model P@1 over 14 subsets (4 tasks × Easy/Hard and per-dataset splits). Two coordinates for reading it: HARD columns (shaded) are the "surface matching closed off" scores, EASY the conventional retrieval scores; read model families vertically, difficulty decay horizontally.

Supplementary metrics (Recall@k / MRR@k / nDCG@k) qualify the P@1 story:

The scale effect, statistically

Paired t-tests (10,000 bootstrap resamples for 95% CIs) compare each family's largest vs smallest variant on query-level P@1:

The difficulty ladder on Easy tasks

Compared jointly, the Easy tasks form a clear ladder: Visual Grounding is hardest — instance-level detail discrimination is most models' long-context weakness; Multi-Source Reasoning is middling — HotpotQA clearly beats multimodal MMDocRAG, showing that cross-modal alignment (not the reasoning itself) is the bottleneck; Summarization is easiest — many models reach 80%+ even on videos, because summary-type queries allow global semantic matching without preserving fine-grained evidence.

Analysis: position bias, modality interference, length scaling

Position bias and fine-grained retention

Visual Grounding's length×depth heatmaps (P@5) separate "length degradation" from "position sensitivity". Each heatmap: x axis is 8 context-length bins (0.25K–32K), y axis is 6 insertion depths (0%–100%), brighter is higher.

Two bottlenecks are thereby established: ① mid-position information loss + boundary preference — retrieval drops significantly when evidence isn't at the sequence's start or end: representation across positions is non-uniform, and current models cannot retain information uniformly over long sequences; ② fine-grained retention decays quickly with length — most models hold stable exact retrieval only at ≤2K tokens, and the decay appears across families, not tied to any architecture or scale. A linear mixed-effects model (needle instance as random intercept) confirms the positional bias statistically:

Robustness to heterogeneous cross-modal interference

In the synthetic task, background distractors are split by modality — pure text, pure image, mixed — exposing how models handle heterogeneous background noise. The 8 representative models sort into three groups:

The appendix extends the typing to all 11 models with formal definitions of the mean and sample standard deviation:

Performance dynamics as context scales

The real tasks share one pattern: unlike the synthetic task's "sudden collapse" past some length, they degrade gradually — real corpora have higher density of salient evidence, so success doesn't hinge on perfect fine-grained retention.

Shared failure modes on Hard tasks

• Summarization is most fragile:distractors that "keep partial semantics, break global structure" cause the largest drops — the very evidence that models live on global semantic matching.
• Multi-Source Reasoning is comparatively stable:it still catches evidence chains coarsely under high-similarity distractors, but chain completeness and logical-consistency assessment remain limited.
• Temporal Summarization fails independently:scores dive the moment order is perturbed — models are insensitive to long-range event structure; order-agnostic shallow visual similarity is what they rely on.

The appendix's complete heatmaps add the models omitted from Figure 3:

Conclusion, limitations, and commentary

Conclusions (from the authors)

MMLongEmbed's evaluation exposes a wide gap between "effective information use" and "representation ability": current MEMs lean heavily on surface feature matching and struggle to capture deep semantics and long-range structural dependencies. Fine-grained information retention decays significantly as context grows; summarization performance looks stable until fine-grained discrimination with distractors begins, where it collapses; and simply enlarging the context window, model scale, or embedding dimension solves none of it — fundamental long-context semantic modeling innovation is needed.

Limitations (self-stated)

• The evaluation caps at 32K tokens; the design doesn't directly scale to arbitrarily longer inputs.
• Only text–image modalities — no audio or continuous 3D spatial data.
• All datasets are English; multilingual long-context multimodal understanding is not assessed.

Commentary

Strengths:filling a real gap with rigorous methodology — the first benchmark to systematize "long context" and "multimodal embedding" together, and every design choice (length stratification, constant C/Q, length-matched replacement, unified token counting) closes a known evaluation shortcut. The quantitative diagnostics go beyond a leaderboard: LMM significance tests, bootstrap CIs, Cohen's d turn "8B isn't better than 2B" from an impression into a statistical conclusion; positional-bias and modality-robustness typing are reusable analysis paradigms. The distractor design is imaginative — event-level temporal perturbation (semantic segmentation + block permutation) tests temporal understanding far better than naive frame shuffling, and the BM25 "look-alike" text injection is worth borrowing.

Debatable points:the real tasks sample only three lengths (8K/16K/32K), so the resolution of length dynamics is coarser than the synthetic task's, and the 8K floor already exceeds many practical retrieval scenarios; "Hard" is heterogeneous — each task's corruption mechanism differs, so cross-task Hard-score comparisons must be made carefully (differences may partly come from task definition rather than model ability); the proprietary model competes at a different weight (128K window + unknown parameters, mitigated but not eliminated by unified token counting); and video compressed to 1 FPS frame sequences means long-range temporal evaluation happens over frame-level representations, with audio and motion excluded — consistent with the authors' own stated limitation.

Glossary

Hover any dotted-underlined abbreviation in the text, or come back here any time.