Embeddings Explained: From a Map to How AI Understands Meaning

When I first started learning about embeddings, one question kept bothering me:

What does a dimension actually mean?

People often explain embeddings by saying:

"A word is converted into a vector and placed somewhere in a high-dimensional space."

That sounds simple enough.

But then I started thinking about a word like cat.

A cat can be:

• a pet
• an animal
• a predator
• something that eats rats
• something that chases rabbits
• something that falls from a mountain

So if cat is just one point on a map, how can the model understand all these different meanings?

The easiest way I found to understand this is to first forget AI for a moment and think about a map.

---

#1. Imagine a map

Think about Google Maps.

Every city has a location:

Delhi  → (28.6, 77.2)
Mumbai → (19.0, 72.8)
London → (51.5, -0.1)

The two numbers are coordinates.

They tell us where something is.

Now imagine a completely different kind of map.

Instead of cities, this map contains concepts.

And instead of only two coordinates, it has hundreds or thousands.

This is roughly the idea behind an embedding space.

             Embedding Space

                  ● dog

        ● wolf

                         ● cat


                             ● car

Things that are related can end up closer together in this space.

In AI terms:

• the map → embedding space
• the coordinates → dimensions
• the location of something → vector
• the distance/relationship between locations → similarity between representations

So when someone says:

"A cat is represented by a 1,536-dimensional vector"

you can imagine:

"The model has given the concept a location on a map that has 1,536 coordinates."

---

#2. But there is one big difference from a normal map

On a geographical map, we know what the coordinates mean.

Latitude tells us north/south.

Longitude tells us east/west.

But with AI embeddings, we usually don't know that dimension 1 means one specific human concept and dimension 2 means another.

It isn't:

Dimension 1 → animal
Dimension 2 → intelligence
Dimension 3 → food
Dimension 4 → happiness

Instead, the model learns the numerical representation itself.

Information can be spread across many dimensions.

So if an embedding looks like:

[0.21, -0.73, 0.14, 0.91, -0.12, ...]

we shouldn't assume:

"0.21 means animal."

The meaning comes from the whole pattern of numbers and how that vector relates to other vectors.

This is called a distributed representation.

---

#3. Now let's bring "cat" onto the map

Suppose the model has some representation for:

cat → [0.12, -0.44, 0.81, ...]

At first, you might think:

"Okay, so cat has one fixed location."

And this is where things get interesting.

Consider these three sentences:

The cat eats rats.

The cat is a pet.

The cat fell from the mountain.

The word cat is identical in all three.

But the context is completely different.

Modern language models don't simply look at cat in isolation.

They process the surrounding words too.

Very roughly:

cat + eats + rats
        ↓
contextual representation


cat + pet
   ↓
contextual representation


cat + fell + mountain
        ↓
contextual representation

In AI terms, this is a contextual representation.

And one of the major mechanisms that allows this is attention.

---

#4. What does attention do?

Think about this sentence:

The cat fell from the mountain because it was chasing a bird.

When the model encounters the word "it", it needs to figure out what "it" refers to.

The surrounding words provide clues.

Conceptually:

The   cat   fell   from   the   mountain   because   it   was   chasing   a   bird
      ↑                                                    ↑
      └──────────────────── relationship ─────────────────┘

The actual mathematics behind attention is much more complicated, but the basic idea is:

The model looks at other parts of the input to understand what is relevant to the current word.

So the model isn't just doing:

cat → fixed meaning

It's doing something more like:

cat
 ↓
look at surrounding context
 ↓
combine relevant information
 ↓
new contextual representation

This is why the same word can behave differently in different sentences.

---

#5. But what about sentences the model has never seen?

This is probably the most interesting part.

Suppose during training the model has seen:

cats eat mice
cats eat rats
cats hunt small animals
rabbits are small animals
rabbits can be prey

But it has never seen:

A cat eats a rabbit.

Can it understand the sentence?

Potentially, yes.

Why?

Because it doesn't need to memorize every possible sentence.

It can learn relationships between concepts.

Think of LEGO.

You don't need a separate LEGO piece called:

CAT-EATS-RABBIT

You already have:

CAT
EAT
RABBIT

The model learns how these pieces can relate to each other.

This ability to apply learned patterns to new combinations is called generalization.

---

#6. This is why infinite combinations aren't a problem

There are practically infinite possible sentences.

For example:

The cat eats a rabbit.

The small cat eats a rabbit.

The small orange cat eats a rabbit.

The small orange cat eats a rabbit near the garden.

The small orange cat suddenly eats a rabbit near the garden while...

We obviously cannot create and store a separate representation for every possible sentence.

Instead, the model learns patterns that can be reused.

Conceptually:

                 Training examples
                        ↓
                learn relationships
                        ↓
        ┌───────────────┼───────────────┐
        ↓               ↓               ↓
      CAT             EAT            RABBIT
        └───────────────┼───────────────┘
                        ↓
              new combination
                        ↓
              "cat eats rabbit"

This is one of the fundamental reasons neural networks are useful.

They can generalize beyond the exact examples they were trained on.

---

#7. So does the model actually "reason" that cats can eat rabbits?

This needs a little caution.

Suppose the model has learned:

Cats are mammals.
Cats are predators.
Cats hunt small animals.
Cats eat mice.
Cats eat rats.
Rabbits are animals.
Rabbits can be prey.

Then you ask:

Can a cat eat a rabbit?

The model may produce a sensible answer.

But it doesn't necessarily execute a clean rule like:

Cat → predator
Rabbit → prey
Predator + prey → can eat
Therefore → cat can eat rabbit

Instead, its neural network has learned a huge number of relationships through training.

Those learned relationships influence the representation and prediction it produces.

This distinction is important because it explains both the power and the limitations of language models.

---

#8. And this brings us back to dimensions

Now we can answer the original question more clearly.

If a model has a 1,536-dimensional embedding, don't imagine:

Dimension 1 → cat
Dimension 2 → rabbit
Dimension 3 → food
Dimension 4 → mountain
...

That's not how we should think about it.

Instead, imagine the information being distributed throughout the coordinate system.

             1,536-dimensional space

        ┌─────────────────────────────┐
        │  0.12  -0.81  0.33  ...     │
        │  0.44   0.19 -0.63  ...     │
        │  0.91  -0.02  0.41  ...     │
        └─────────────────────────────┘
                       ↓
               learned representation
                       ↓
              relationships between
                    concepts

The individual number isn't necessarily meaningful by itself.

The overall pattern is what matters.

---

#9. One more important distinction: word embeddings vs sentence embeddings

There are different kinds of embeddings.

A simple word embedding might represent:

cat → vector

But a sentence embedding represents something like:

"The cat fell from the mountain."
             ↓
           vector

That vector is intended to capture useful information about the meaning of the entire sentence.

Now compare:

The cat fell from the mountain.

with:

The cat fell down the mountain.

Their embeddings should generally be relatively similar because the meanings are similar.

But:

The cat is sleeping on the sofa.

should be different.

This is why embeddings are useful for things like:

• semantic search
• finding similar documents
• recommendation systems
• clustering
• retrieval in RAG systems

---

#10. So what is physically stored?

At the end of the day, all of this really does come down to numbers.

For example:

cat → [0.12, -0.44, 0.81, ...]

Those numbers are actually stored and processed by the computer.

There isn't a tiny object inside the GPU labeled:

CAT

and there isn't a special transistor labeled:

CAT = ANIMAL

Instead, the model contains a huge number of learned numerical parameters.

Those parameters collectively create the behavior we interpret as understanding relationships.

That's why the meaning is said to be distributed.

---

#11. The simplest way to remember all of this

Think of an embedding as an AI-generated map of meaning.

The analogy maps to the real AI terms like this:

Analogy	AI term
Map	Embedding space
Coordinates	Dimensions
Location on the map	Vector
Distance between locations	Similarity
Different context around a place	Context
Looking at nearby/relevant information	Attention
Learning how the map should look	Training
Learning from examples and handling new combinations	Generalization
Numbers that create the learned behavior	Model parameters

The most important thing to remember is:

An embedding is not a dictionary where every dimension represents one concept. It's more like a learned coordinate system where information is distributed across many dimensions and relationships between representations carry meaning.

And when language is involved, the model doesn't simply find the fixed location of a word and stop.

It uses the context around that word to build a richer representation.

That's what allows the same cat to participate in:

cat + pet

cat + rats

cat + rabbit

cat + mountain

without needing a separate pre-trained entry for every possible combination.

Once I started thinking about embeddings as a learned map rather than a list of properties, the whole idea became much easier to understand.