Vision-language model

In short

A vision-language model (VLM) is an AI model that takes in both images and text, so it can describe a photo, read the text in it, or answer questions about what it shows.

Also called: VLM, Visual language model, Large multimodal model
Topic: How AI models work
Level: Some background helps
First described: Alayrac and colleagues introduced Flamingo as "a family of Visual Language Models (VLM)" in April 2022, building on image and text matching work such as OpenAI's CLIP in 2021²¹.

A vision-language model joins two skills: seeing and writing. You give it a picture plus a question, and it replies in ordinary language. OpenAI's documentation calls this capability vision and lists describing images, reading visible text and answering questions about objects as typical uses⁴.

A common design bolts a vision encoder onto a large language model. LLaVA, a 2023 research model, connects a vision encoder to an LLM and trains the pair end to end on instruction-following examples³. The encoder turns the image into a sequence of pieces the language model can read alongside words. Anthropic, for example, says Claude views images in 28 by 28 pixel patches, each counted as a visual token⁵.

Much of the groundwork came from OpenAI's CLIP in 2021, which learned to match images with captions using 400 million image and text pairs from the internet¹. DeepMind's Flamingo in 2022 went further and could handle images and text mixed together in one prompt².

OpenAI's GPT models, Anthropic's Claude and Google's Gemini all accept images⁴⁵⁶. Google says its Gemini models are "built to be multimodal from the ground up"⁶. A VLM understands images; it does not necessarily make them. Claude, for instance, can analyse images but cannot generate or edit them⁵.

They have clear limits. OpenAI says its models are not suitable for specialised medical images such as CT scans, struggle with precise spatial tasks such as chess positions, and may give only approximate counts⁴.

Example

You photograph a confusing set of parking signs on a London street and ask a chatbot whether you can park there at 6pm on a Saturday. Reading the signs and reasoning about them in words is the VLM's job.

Why it matters

It is why chatbots can now read a screenshot, a receipt or a hand-drawn chart. It also means their mistakes can come from misreading a picture, not just from the text.

Questions people ask

Is a vision-language model the same as an image generator?

No. A VLM takes images in and writes text out. Image generators work the other way. Anthropic, for example, says Claude can interpret images but cannot generate or edit them⁵.

Can OpenAI's GPT models read images?

Yes. OpenAI's recent vision-capable models can describe images, read visible text and answer questions about objects. OpenAI says they can struggle with non-Latin text, precise spatial reasoning and exact counts⁴.

Can I use a VLM to read my X-ray or scan?

Not reliably. OpenAI says its models are not suitable for interpreting specialised medical images like CT scans⁴, and Anthropic says Claude is not designed to interpret complex diagnostic scans such as CTs or MRIs⁵.

Related terms

Tools that use it

Sources

Back to the glossary