Multimodal AI

Imagine a friend who learned about the world only from books. You show them a photo of a red apple. They could recite every description they've read. But the vivid red of the apple in front of them, or its sweet smell? That would be hard to picture right away. Early chatbots were just like this. They looked at the world through one narrow window: text.

The channels we use to take in information every day are called modalities. Words and photos we see, voices and noises we hear, and textures we feel by touch are all different modalities. The "multi" in multimodal means using several of these channels together.

So multimodal AI is technology that weaves several senses into one. AI no longer stops at reading a question typed in words. It can look at a photo of your fridge and suggest what to make for dinner. It can even listen to a shaky voice and make a rough guess about how the speaker feels.

When the senses grow, so does the depth of understanding. The AI can pick up subtle changes in facial expression and the mood of a scene, things that are hard to put into words. That makes it a much more useful assistant.

To a computer, a photo and a sentence are completely different kinds of data. A photo is a huge grid of colored dots (pixels). Text is a string of letters and symbols. Sound is a wave that rises and falls over time. These things have nothing in common, so how does a computer tie them together?

The secret is translating each kind of information into "number coordinates" that a computer can work with easily. In AI, this is called embedding (turning information into lists of numbers). After lots of training, the AI places a photo of a red apple and the word "apple" side by side in one shared space of meaning.

That's why you can point your phone camera at something and ask a quick "What's this?" and get a natural reply. The AI instantly links the coordinates of the object in the photo with the coordinates of your question, and answers, "That's a ripe, juicy red apple."

Sound can be placed in the very same space. An audio clip of a dog barking, the word "woof," and a photo of a puppy all land in neighboring spots because they mean the same thing. Together they form one giant web of knowledge.

Tech that reads text in photos or transcribes speech has existed for a long time. Linking a dedicated image-recognition AI to a text-reading AI, like stepping stones across a stream, is also a multimodal system in the broad sense. In fact, early systems often used this kind of pipeline (a step-by-step chain of separate tools).

More recently, some models attach an image-reading component to a language model and then train them further together. Others are unified models trained to learn from the start across everything at once, with text, images, and audio bundled together from early in development. These models handle several senses without a translation step in the middle.

It's like crossing at a traffic light. You hear a car horn while you watch how fast a car is approaching, and you stop in your tracks. You don't turn the sound into words in your head and then compare it with what you see. You combine your senses in a split second and decide right away. Unified models work more like that.

Thanks to this integration, AI can do higher-level reasoning that takes both the context of the text and the details of the image into account. Researchers are even working on helping AI notice things like sarcasm, where the tone of voice and the facial expression don't match.