Multimodal fusion & projection

How a language model learns to see: encode the image into vectors, translate those vectors into the language model's own embedding space, and decide where in the network the two streams meet. Add an action output and you have the core of a robot vision-language-action model.

Source: Suraj Sharma's roadmap post, stage 7 · saved in sources/08-suraj-ai-research-engineer.md

Going deeper: everything below is the full version, with some maths. On a first read, skim the headings and the "Why it matters for robotics" box, and come back for the rest later.

1. What this stage covers

• Learn: how contrastive image encoders (CLIP, SigLIP) are trained and why their features line up with language; cross-attention as a fusion mechanism; how audio becomes tokens; and the early-versus-late fusion design choice.
• Practice: freeze a SigLIP encoder and a small LLM, and train only an MLP projector between them so the LLM can caption images. This is the LLaVA recipe in miniature.
• Why (the post's claim): text is only part of the data in the world, and the leading models increasingly reason over images, audio and text together. Mostly true, with caveats in section 4.

2. Concepts, explained

Vision encoders: CLIP and SigLIP

A Vision Transformer (ViT) cuts an image into P × P patches, flattens each, applies a linear layer, adds a position embedding and runs a standard transformer. The output is one vector per patch. What makes CLIP and SigLIP special is how they're trained: on huge sets of image–caption pairs, with a text encoder alongside, so that an image and its caption land near each other in a shared space. Those patch features are already "close to language", which is why a small projector is enough to hand them to an LLM.

CLIP (Radford et al., 2021) scores every image against every caption in a batch and applies a softmax in both directions. Each image must pick out its own caption from all the others.

SigLIP (Zhai, Mustafa, Kolesnikov and Beyer, 2023) replaces the softmax with an independent sigmoid on each image–text pair: "do these two match, yes or no?" No batch-wide normalisation is needed, which makes very large batches cheaper and also works better at small batch sizes. The paper found gains from growing the batch fade quickly, and that around 32k is enough.

CLIP  (softmax, per image i):  −log  exp(t·xᵢ·yᵢ) / Σⱼ exp(t·xᵢ·yⱼ)     (plus the text→image direction)
SigLIP (sigmoid, every pair):  −Σᵢⱼ log σ( zᵢⱼ · (t·xᵢ·yⱼ + b) ),   zᵢⱼ = +1 if i = j else −1

x, y = normalised image / text embeddings,  t = learned temperature,  b = learned bias

Token count is set by geometry: an image of side H with patch size P gives (H/P)² tokens. A 224-pixel image with 14-pixel patches is 16 × 16 = 256 tokens. At 384 pixels it's 27 × 27 = 729. The demo below makes this concrete, because for robots with several cameras it becomes the dominant cost.

Projection: the bridge into the LLM

The encoder's vectors have its own width (say 768 or 1152) and live in its own space. The LLM expects vectors of its hidden size, in its embedding space. A projector learns that map. LLaVA (Liu, Li, Wu and Lee, 2023) used a single linear layer from a CLIP ViT-L/14 into Vicuna. Its follow-up, LLaVA-1.5, switched to a two-layer MLP and found it better. Training is two stages: first train only the projector on image–caption pairs (feature alignment), then unfreeze the LLM and fine-tune on visual instruction data, which LLaVA generated with text-only GPT-4.

v   = Encoder(image)            # [N_patches, d_vision]
h   = W₂ · GELU(W₁ · v)         # [N_patches, d_llm]   the projector
seq = [h ; Embed(prompt_tokens)]   # image tokens first, then text
LLM(seq) → next-token loss on the answer tokens only

Other bridges compress the image into a fixed, smaller set of tokens. A Q-Former (BLIP-2) or Perceiver resampler (Flamingo) uses a few dozen learned queries that cross-attend to the patches. Pixel shuffle (used in SmolVLM-style small models) folds each 2 × 2 block of neighbouring patches into one wider vector, cutting tokens by 4×. Fewer tokens means cheaper inference and a less precise view of small details.

Cross-attention as fusion

In cross-attention, queries come from one stream and keys and values from another. Flamingo (Alayrac et al., 2022) keeps a pretrained LLM frozen and inserts new cross-attention layers between its blocks. Text hidden states form the queries; resampled image tokens form the keys and values. Each new layer's output passes through a tanh gate initialised at zero, so at the start of training the model behaves exactly like the original LLM and the visual pathway fades in.

Q = H_text · W_Q      K = V_img · W_K      V = V_img · W_V
H_text ← H_text + tanh(α) · softmax(Q Kᵀ / √d) · V       # α starts at 0

The trade-off against LLaVA-style input fusion: cross-attention leaves the text context short (image tokens never enter the sequence) and protects the LLM's text skills, but it adds parameters and a more complex architecture. Input-token fusion is simpler and lets every layer of the LLM attend to the image, at the cost of a longer sequence.

Audio tokenisation

Audio is handled in two broad ways. For understanding, an encoder such as Whisper's (Radford et al., 2022) turns a log-mel spectrogram into a sequence of vectors, which are then projected into the LLM exactly as with images. For generation, a neural codec such as EnCodec (Défossez et al., 2022) compresses the waveform into discrete codes using residual vector quantisation: several codebooks, each encoding what the previous ones missed. Those integer codes can then be predicted like text tokens. The same split, continuous features to read and discrete codes to write, reappears with robot actions below.

Early versus late fusion

The terms are used loosely, so be precise about which you mean:

• Late fusion: each modality has its own model, and they meet only at the end, for example CLIP's dot product between an image vector and a text vector, or averaging two classifiers' scores. Cheap and modular, but the modalities can't inform each other's processing.
• Intermediate (deep) fusion: separate encoders, then mixing inside the network. Flamingo's cross-attention layers, and LLaVA's "encoder, then projected tokens into the LLM", both sit here. Most open VLMs and VLAs use this.
• Early fusion: all modalities become tokens in one shared vocabulary and one transformer processes them from the first layer. Chameleon (Chameleon Team, Meta, 2024) quantises images into discrete tokens and trains on interleaved image and text from scratch, so it can also generate images. The paper spends real effort on training stability, a known difficulty of this approach.

The direction of travel at the frontier is towards models trained natively on several modalities, rather than bolting a frozen encoder onto a finished LLM. Module 05 records that Kimi K3 trained its vision encoder from scratch inside the language model and reported that a pretrained SigLIP-style encoder was unnecessary and less stable. That's a real counterpoint to this stage's recipe. It comes from a lab with trillion-token budgets, though. With a small dataset, reusing a pretrained encoder is still the sensible default.

3. The practice project

(a) Laptop/Colab version: a tiny captioning VLM

Freeze a small SigLIP and a small LLM. Train only the projector. This runs on a free Colab GPU in an hour or two.

1. Models: google/siglip-base-patch16-224 (196 patch tokens) and HuggingFaceTB/SmolLM2-135M. Read both hidden sizes from their configs; don't hard-code them.
2. Data: a few thousand image–caption pairs (a slice of COCO Captions or Flickr30k). Keep 500 pairs aside for validation.
3. Cache the vision features once, since the encoder is frozen. Training then only touches the projector and is fast.
4. Train the two-layer MLP with AdamW (learning rate around 1e-3), loss on caption tokens only.
5. Measure validation caption loss under three conditions: the true image, a shuffled image from another example, and no image. The gap between "true" and "shuffled" is the only number that proves the projector passes visual information. Also read 20 generated captions by eye.

import torch, torch.nn as nn
from transformers import AutoModel, AutoModelForCausalLM

vis = AutoModel.from_pretrained("google/siglip-base-patch16-224").vision_model.eval()
lm = AutoModelForCausalLM.from_pretrained("HuggingFaceTB/SmolLM2-135M").eval()
for p in [*vis.parameters(), *lm.parameters()]:
    p.requires_grad = False

dv, dt = vis.config.hidden_size, lm.config.hidden_size
proj = nn.Sequential(nn.Linear(dv, dt), nn.GELU(), nn.Linear(dt, dt))
opt = torch.optim.AdamW(proj.parameters(), lr=1e-3)

def loss_fn(pix, ids, pad_id):
    with torch.no_grad():
        v = vis(pixel_values=pix).last_hidden_state      # [B, 196, dv]
    img = proj(v)                                         # [B, 196, dt]
    txt = lm.get_input_embeddings()(ids)                  # [B, T, dt]
    lab = ids.masked_fill(ids == pad_id, -100)            # no loss on padding
    lab = torch.cat([torch.full(img.shape[:2], -100), lab], 1)  # or on image slots
    return lm(inputs_embeds=torch.cat([img, txt], 1), labels=lab).loss

Don't expect fluent captions from a 135M model. Expect short, roughly topical ones ("a man riding a horse") and a clear loss gap over shuffled images. If the gap is near zero, the LLM is just writing generic captions from its language prior.

(b) The full version

The post's project at full size is LLaVA-1.5: a CLIP or SigLIP encoder at 336–384 px, an MLP projector and a 7B LLM. Stage 1 trains the projector on several hundred thousand caption pairs. Stage 2 unfreezes the LLM for visual instruction tuning. The LLaVA-1.5 authors report about a day of training on a single 8×A100 node. Rented, that's a few hundred dollars per run, before the failed runs. The nanoVLM repo is the middle ground: a complete, readable VLM trainable on one GPU.

For a robotics follow-on, fine-tune OpenVLA on a small task with LoRA, or train a SmolVLA-sized policy with LeRobot on a simulated task. Both are in the course list below.

(c) Pitfalls

• Wrong preprocessing. Use the encoder's own processor (resize, normalisation). SigLIP and CLIP normalise differently, and a mismatch silently degrades features.
• Loss on the wrong tokens. Mask image positions, the prompt and padding. Otherwise the model learns to predict its own inputs.
• Scale mismatch. Projected vectors can have a very different norm from text embeddings. Check both norms at the start; a large mismatch slows or destabilises training.
• Trusting fluent output. A strong LLM writes plausible captions with no image at all. Always run the shuffled-image control.
• CLS versus patch tokens. SigLIP has no CLS token; CLIP does. Know which tokens you're passing and how many.

4. Reality check

The post says text is only half the internet and the current meta is models that natively reason over vision, audio and text together.

• True: the leading models are multimodal, and a large share of useful data (video, images, audio, sensor streams) isn't text. For robotics this isn't a trend, it's the job: a robot has no text input except the instruction.
• Overstated: "natively" is doing a lot of work. Most open VLMs and nearly all VLAs are still the bolt-on recipe in this stage: pretrained encoder, projector, pretrained LLM. Natively trained multimodal models exist (Chameleon, and Kimi K3's from-scratch vision encoder), but they need budgets few teams have.
• Missing: the projector exercise teaches the plumbing, not the hard parts. Data quality, image resolution strategy, and evaluating whether the model actually looks at the image matter more for results than the projector's shape.
• "Half the internet" is a slogan, not a measured figure.

5. In this course

• Module 05 · LLM architecture at scale: Kimi K3 training its vision encoder from scratch, the counterpoint to the SigLIP recipe.
• 30u30 Day 28 · CS231n: conv shapes, strides and why robot vision reuses pretrained backbones.
• 30u30 Day 13 · The Transformer and Day 14 · The Annotated Transformer: the attention maths that cross-attention reuses.
• Module 04 · Big teacher, small student: distilling large VLAs into smaller, faster policies.
• Module 03 · Human–AI help levels: what happens around a VLA when a human shares control.

6. Why it matters for robotics

7. Self-check

8. Courses & content

Start with nanoVLM: it's the practice project already built, small enough to read in an evening, and you can swap its projector for your own. Then move to OpenVLA for the robotics side.

Full course map for all 12 stages →

9. Papers & docs

• CLIP: Learning Transferable Visual Models From Natural Language Supervision (Radford et al., 2021)
• SigLIP: Sigmoid Loss for Language Image Pre-Training (Zhai, Mustafa, Kolesnikov, Beyer, 2023)
• Flamingo: a Visual Language Model for Few-Shot Learning (Alayrac et al., 2022)
• Visual Instruction Tuning (LLaVA) (Liu, Li, Wu, Lee, 2023) and Improved Baselines with Visual Instruction Tuning (LLaVA-1.5) (Liu et al., 2023)
• Chameleon: Mixed-Modal Early-Fusion Foundation Models (Chameleon Team, 2024)
• Whisper: Robust Speech Recognition via Large-Scale Weak Supervision (Radford et al., 2022) and EnCodec: High Fidelity Neural Audio Compression (Défossez et al., 2022)
• RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control (Brohan et al., 2023)
• OpenVLA: An Open-Source Vision-Language-Action Model (Kim et al., 2024)
• π0: A Vision-Language-Action Flow Model for General Robot Control (Black et al., 2024)
• Docs: Transformers · SigLIP